This library provides a Dagster integration with Airbyte.
For more information on getting started, see the Airbyte integration guide.
The Airbyte Connection ID that this op will sync. You can retrieve this value from the “Connections” tab of a given connector in the Airbyte UI.
The time (in seconds) that will be waited between successive polls.
Default Value: 10
The maximum time that will waited before this operation is timed out. By default, this will never time out.
Default Value: None
If True, materializations corresponding to the results of the Airbyte sync will be yielded when the op executes.
Default Value: True
If provided and yield_materializations is True, these components will be used to prefix the generated asset keys.
Default Value: [‘airbyte’]
Executes a Airbyte job sync for a given connection_id, and polls until that sync
completes, raising an error if it is unsuccessful. It outputs a AirbyteOutput which contains
the job details for a given connection_id.
It requires the use of the airbyte_resource, which allows it to
communicate with the Airbyte API.
Examples
from dagster import job
from dagster_airbyte import airbyte_resource, airbyte_sync_op
my_airbyte_resource = airbyte_resource.configured(
{
"host": {"env": "AIRBYTE_HOST"},
"port": {"env": "AIRBYTE_PORT"},
}
)
sync_foobar = airbyte_sync_op.configured({"connection_id": "foobar"}, name="sync_foobar")
@job(resource_defs={"airbyte": my_airbyte_resource})
def my_simple_airbyte_job():
sync_foobar()
@job(resource_defs={"airbyte": my_airbyte_resource})
def my_composed_airbyte_job():
final_foobar_state = sync_foobar(start_after=some_op())
other_op(final_foobar_state)
The Airbyte Server Address.
Port for the Airbyte Server.
Username if using basic auth.
Password if using basic auth.
Use https to connect in Airbyte Server.
Default Value: False
The maximum number of times requests to the Airbyte API should be retried before failing.
Default Value: 3
Time (in seconds) to wait between each request retry.
Default Value: 0.25
Time (in seconds) after which the requests to Airbyte are declared timed out.
Default Value: 15
Any additional kwargs to pass to the requests library when making requests to Airbyte.
{}
Whether to forward Airbyte logs to the compute log, can be expensive for long-running syncs.
Default Value: True
Whether to cancel a sync in Airbyte if the Dagster runner is terminated. This may be useful to disable if using Airbyte sources that cannot be cancelled and resumed easily, or if your Dagster deployment may experience runner interruptions that do not impact your Airbyte deployment.
Default Value: True
This resource allows users to programatically interface with the Airbyte REST API to launch syncs and monitor their progress. This currently implements only a subset of the functionality exposed by the API.
For a complete set of documentation on the Airbyte REST API, including expected response JSON schema, see the Airbyte API Docs.
To configure this resource, we recommend using the configured method.
Examples:
from dagster import job
from dagster_airbyte import airbyte_resource
my_airbyte_resource = airbyte_resource.configured(
{
"host": {"env": "AIRBYTE_HOST"},
"port": {"env": "AIRBYTE_PORT"},
# If using basic auth
"username": {"env": "AIRBYTE_USERNAME"},
"password": {"env": "AIRBYTE_PASSWORD"},
}
)
@job(resource_defs={"airbyte":my_airbyte_resource})
def my_airbyte_job():
...
This class exposes methods on top of the Airbyte REST API.
Loads Airbyte connection assets from a configured AirbyteResource instance. This fetches information about defined connections at initialization time, and will error on workspace load if the Airbyte instance is not reachable.
airbyte (ResourceDefinition) – An AirbyteResource configured with the appropriate connection details.
workspace_id (Optional[str]) – The ID of the Airbyte workspace to load connections from. Only required if multiple workspaces exist in your instance.
key_prefix (Optional[CoercibleToAssetKeyPrefix]) – A prefix for the asset keys created.
create_assets_for_normalization_tables (bool) – If True, assets will be created for tables created by Airbyte’s normalization feature. If False, only the destination tables will be created. Defaults to True.
connection_to_group_fn (Optional[Callable[[str], Optional[str]]]) – Function which returns an asset group name for a given Airbyte connection name. If None, no groups will be created. Defaults to a basic sanitization function.
io_manager_key (Optional[str]) – The I/O manager key to use for all assets. Defaults to “io_manager”. Use this if all assets should be loaded from the same source, otherwise use connection_to_io_manager_key_fn.
connection_to_io_manager_key_fn (Optional[Callable[[str], Optional[str]]]) – Function which returns an I/O manager key for a given Airbyte connection name. When other ops are downstream of the loaded assets, the IOManager specified determines how the inputs to those ops are loaded. Defaults to “io_manager”.
connection_filter (Optional[Callable[[AirbyteConnectionMetadata], bool]]) – Optional function which takes in connection metadata and returns False if the connection should be excluded from the output assets.
connection_to_asset_key_fn (Optional[Callable[[AirbyteConnectionMetadata, str], AssetKey]]) – Optional function which takes in connection metadata and table name and returns an asset key for the table. If None, the default asset key is based on the table name. Any asset key prefix will be applied to the output of this function.
connection_to_freshness_policy_fn (Optional[Callable[[AirbyteConnectionMetadata], Optional[FreshnessPolicy]]]) – Optional function which takes in connection metadata and returns a freshness policy for the connection’s assets. If None, no freshness policies will be applied to the assets.
Examples:
Loading all Airbyte connections as assets:
from dagster_airbyte import airbyte_resource, load_assets_from_airbyte_instance
airbyte_instance = airbyte_resource.configured(
{
"host": "localhost",
"port": "8000",
}
)
airbyte_assets = load_assets_from_airbyte_instance(airbyte_instance)
Filtering the set of loaded connections:
from dagster_airbyte import airbyte_resource, load_assets_from_airbyte_instance
airbyte_instance = airbyte_resource.configured(
{
"host": "localhost",
"port": "8000",
}
)
airbyte_assets = load_assets_from_airbyte_instance(
airbyte_instance,
connection_filter=lambda meta: "snowflake" in meta.name,
)
Loads an Airbyte project into a set of Dagster assets.
Point to the root folder of an Airbyte project synced using the Octavia CLI. For more information, see https://github.com/airbytehq/airbyte/tree/master/octavia-cli#octavia-import-all.
project_dir (str) – The path to the root of your Airbyte project, containing sources, destinations, and connections folders.
workspace_id (Optional[str]) – The ID of the Airbyte workspace to load connections from. Only required if multiple workspace state YAMLfiles exist in the project.
key_prefix (Optional[CoercibleToAssetKeyPrefix]) – A prefix for the asset keys created.
create_assets_for_normalization_tables (bool) – If True, assets will be created for tables created by Airbyte’s normalization feature. If False, only the destination tables will be created. Defaults to True.
connection_to_group_fn (Optional[Callable[[str], Optional[str]]]) – Function which returns an asset group name for a given Airbyte connection name. If None, no groups will be created. Defaults to a basic sanitization function.
io_manager_key (Optional[str]) – The I/O manager key to use for all assets. Defaults to “io_manager”. Use this if all assets should be loaded from the same source, otherwise use connection_to_io_manager_key_fn.
connection_to_io_manager_key_fn (Optional[Callable[[str], Optional[str]]]) – Function which returns an I/O manager key for a given Airbyte connection name. When other ops are downstream of the loaded assets, the IOManager specified determines how the inputs to those ops are loaded. Defaults to “io_manager”.
connection_filter (Optional[Callable[[AirbyteConnectionMetadata], bool]]) – Optional function which takes in connection metadata and returns False if the connection should be excluded from the output assets.
connection_directories (Optional[List[str]]) – Optional list of connection directories to load assets from. If omitted, all connections in the Airbyte project are loaded. May be faster than connection_filter if the project has many connections or if the connection yaml files are large.
connection_to_asset_key_fn (Optional[Callable[[AirbyteConnectionMetadata, str], AssetKey]]) – Optional function which takes in connection metadata and table name and returns an asset key for the table. If None, the default asset key is based on the table name. Any asset key prefix will be applied to the output of this function.
connection_to_freshness_policy_fn (Optional[Callable[[AirbyteConnectionMetadata], Optional[FreshnessPolicy]]]) – Optional function which takes in connection metadata and returns a freshness policy for the connection’s assets. If None, no freshness policies will be applied to the assets.
Examples:
Loading all Airbyte connections as assets:
from dagster_airbyte import load_assets_from_airbyte_project
airbyte_assets = load_assets_from_airbyte_project(
project_dir="path/to/airbyte/project",
)
Filtering the set of loaded connections:
from dagster_airbyte import load_assets_from_airbyte_project
airbyte_assets = load_assets_from_airbyte_project(
project_dir="path/to/airbyte/project",
connection_filter=lambda meta: "snowflake" in meta.name,
)
Builds a set of assets representing the tables created by an Airbyte sync operation.
connection_id (str) – The Airbyte Connection ID that this op will sync. You can retrieve this value from the “Connections” tab of a given connector in the Airbyte UI.
destination_tables (List[str]) – The names of the tables that you want to be represented in the Dagster asset graph for this sync. This will generally map to the name of the stream in Airbyte, unless a stream prefix has been specified in Airbyte.
normalization_tables (Optional[Mapping[str, List[str]]]) – If you are using Airbyte’s normalization feature, you may specify a mapping of destination table to a list of derived tables that will be created by the normalization process.
asset_key_prefix (Optional[List[str]]) – A prefix for the asset keys inside this asset. If left blank, assets will have a key of AssetKey([table_name]).
upstream_assets (Optional[Set[AssetKey]]) – A list of assets to add as sources.
freshness_policy (Optional[FreshnessPolicy]) – A freshness policy to apply to the assets
The following APIs are used as part of the experimental ingestion-as-code functionality. For more information, see the Airbyte ingestion as code guide.
Reconciles Python-specified Airbyte connections with an Airbyte instance.
Passing the module containing an AirbyteManagedElementReconciler to the dagster-airbyte CLI will allow you to check the state of your Python-code-specified Airbyte connections against an Airbyte instance, and reconcile them if necessary.
This functionality is experimental and subject to change.
Loads Airbyte connection assets from a configured AirbyteResource instance, checking against a list of AirbyteConnection objects. This method will raise an error on repo load if the passed AirbyteConnection objects are not in sync with the Airbyte instance.
airbyte (ResourceDefinition) – An AirbyteResource configured with the appropriate connection details.
connections (Iterable[AirbyteConnection]) – A list of AirbyteConnection objects to build assets for.
key_prefix (Optional[CoercibleToAssetKeyPrefix]) – A prefix for the asset keys created.
create_assets_for_normalization_tables (bool) – If True, assets will be created for tables created by Airbyte’s normalization feature. If False, only the destination tables will be created. Defaults to True.
connection_to_group_fn (Optional[Callable[[str], Optional[str]]]) – Function which returns an asset group name for a given Airbyte connection name. If None, no groups will be created. Defaults to a basic sanitization function.
io_manager_key (Optional[str]) – The IO manager key to use for all assets. Defaults to “io_manager”. Use this if all assets should be loaded from the same source, otherwise use connection_to_io_manager_key_fn.
connection_to_io_manager_key_fn (Optional[Callable[[str], Optional[str]]]) – Function which returns an IO manager key for a given Airbyte connection name. When other ops are downstream of the loaded assets, the IOManager specified determines how the inputs to those ops are loaded. Defaults to “io_manager”.
connection_to_asset_key_fn (Optional[Callable[[AirbyteConnectionMetadata, str], AssetKey]]) – Optional function which takes in connection metadata and table name and returns an asset key for the table. If None, the default asset key is based on the table name. Any asset key prefix will be applied to the output of this function.
connection_to_freshness_policy_fn (Optional[Callable[[AirbyteConnectionMetadata], Optional[FreshnessPolicy]]]) – Optional function which takes in connection metadata and returns a freshness policy for the connection. If None, no freshness policy will be applied.
Examples:
from dagster_airbyte import (
AirbyteConnection,
airbyte_resource,
load_assets_from_connections,
)
airbyte_instance = airbyte_resource.configured(
{
"host": "localhost",
"port": "8000",
}
)
airbyte_connections = [
AirbyteConnection(...),
AirbyteConnection(...)
]
airbyte_assets = load_assets_from_connections(airbyte_instance, airbyte_connections)
A user-defined Airbyte connection, pairing an Airbyte source and destination and configuring which streams to sync.
name (str) – The display name of the connection.
source (AirbyteSource) – The source to sync from.
destination (AirbyteDestination) – The destination to sync to.
stream_config (Mapping[str, AirbyteSyncMode]) – A mapping from stream name to the sync mode for that stream, including any additional configuration of primary key or cursor field.
normalize_data (Optional[bool]) – Whether to normalize the data in the destination.
destination_namespace (Optional[Union[AirbyteDestinationNamespace, str]]) – The namespace to sync to in the destination. If set to AirbyteDestinationNamespace.SAME_AS_SOURCE, the namespace will be the same as the source namespace. If set to AirbyteDestinationNamespace.DESTINATION_DEFAULT, the namespace will be the default namespace for the destination. If set to a string, the namespace will be that string.
Example
from dagster_airbyte.managed.generated.sources import FileSource
from dagster_airbyte.managed.generated.destinations import LocalJsonDestination
from dagster_airbyte import AirbyteConnection, AirbyteSyncMode
cereals_csv_source = FileSource(...)
local_json_destination = LocalJsonDestination(...)
cereals_connection = AirbyteConnection(
name="download-cereals",
source=cereals_csv_source,
destination=local_json_destination,
stream_config={"cereals": AirbyteSyncMode.full_refresh_overwrite()},
)
Represents a user-defined Airbyte source.
name (str) – The display name of the source.
source_type (str) – The type of the source, from Airbyte’s list of sources https://airbytehq.github.io/category/sources/.
source_configuration (Mapping[str, Any]) – The configuration for the source, as defined by Airbyte’s API.
Represents a user-defined Airbyte destination.
name (str) – The display name of the destination.
destination_type (str) – The type of the destination, from Airbyte’s list of destinations https://airbytehq.github.io/category/destinations/.
destination_configuration (Mapping[str, Any]) – The configuration for the destination, as defined by Airbyte’s API.
Represents the sync mode for a given Airbyte stream, which governs how Airbyte reads from a source and writes to a destination.
For more information, see https://docs.airbyte.com/understanding-airbyte/connections/.
Syncs the entire data stream from the source, appending rows to the destination.
https://docs.airbyte.com/understanding-airbyte/connections/full-refresh-append/
Syncs the entire data stream from the source, replaces data in the destination by overwriting it.
https://docs.airbyte.com/understanding-airbyte/connections/full-refresh-overwrite
Syncs only new records from the source, appending rows to the destination. May optionally specify the cursor field used to determine which records are new.
https://docs.airbyte.com/understanding-airbyte/connections/incremental-append/
Syncs new records from the source, appending to an append-only history table in the destination. Also generates a deduplicated view mirroring the source table. May optionally specify the cursor field used to determine which records are new, and the primary key used to determine which records are duplicates.
https://docs.airbyte.com/understanding-airbyte/connections/incremental-append-dedup/
Airbyte Source for Strava
Documentation can be found at https://docs.airbyte.com/integrations/sources/strava
name (str) – The name of the destination.
client_id (str) – The Client ID of your Strava developer application.
client_secret (str) – The Client Secret of your Strava developer application.
refresh_token (str) – The Refresh Token with the activity: read_all permissions.
athlete_id (int) – The Athlete ID of your Strava developer application.
start_date (str) – UTC date and time. Any data before this date will not be replicated.
Airbyte Source for Appsflyer
name (str) – The name of the destination.
app_id (str) – App identifier as found in AppsFlyer.
api_token (str) – Pull API token for authentication. If you change the account admin, the token changes, and you must update scripts with the new token. Get the API token in the Dashboard.
start_date (str) – The default value to use if no bookmark exists for an endpoint. Raw Reports historical lookback is limited to 90 days.
timezone (Optional[str]) – Time zone in which date times are stored. The project timezone may be found in the App settings in the AppsFlyer console.
Airbyte Source for Google Workspace Admin Reports
Documentation can be found at https://docs.airbyte.com/integrations/sources/google-workspace-admin-reports
name (str) – The name of the destination.
credentials_json (str) – The contents of the JSON service account key. See the docs for more information on how to generate this key.
email (str) – The email of the user, who has permissions to access the Google Workspace Admin APIs.
lookback (Optional[int]) – Sets the range of time shown in the report. The maximum value allowed by the Google API is 180 days.
Airbyte Source for Cart
Documentation can be found at https://docs.airbyte.com/integrations/sources/cart
name (str) – The name of the destination.
start_date (str) – The date from which you’d like to replicate the data
Airbyte Source for Linkedin Ads
Documentation can be found at https://docs.airbyte.com/integrations/sources/linkedin-ads
name (str) – The name of the destination.
start_date (str) – UTC date in the format 2020-09-17. Any data before this date will not be replicated.
account_ids (Optional[List[int]]) – Specify the account IDs separated by a space, to pull the data from. Leave empty, if you want to pull the data from all associated accounts. See the LinkedIn Ads docs for more info.
Airbyte Source for Mongodb
Documentation can be found at https://docs.airbyte.com/integrations/sources/mongodb
name (str) – The name of the destination.
host (str) – Host of a Mongo database to be replicated.
port (int) – Port of a Mongo database to be replicated.
database (str) – Database to be replicated.
user (str) – User
password (str) – Password
auth_source (str) – Authentication source where user information is stored. See the Mongo docs for more info.
replica_set (Optional[str]) – The name of the set to filter servers by, when connecting to a replica set (Under this condition, the ‘TLS connection’ value automatically becomes ‘true’). See the Mongo docs for more info.
ssl (Optional[bool]) – If this switch is enabled, TLS connections will be used to connect to MongoDB.
Airbyte Source for Stock Ticker Api Tutorial
Documentation can be found at https://polygon.io/docs/stocks/get_v2_aggs_grouped_locale_us_market_stocks__date
name (str) – The name of the destination.
stock_ticker (str) – The stock ticker to track
api_key (str) – The Polygon.io Stocks API key to use to hit the API.
Airbyte Source for Wrike
name (str) – The name of the destination.
access_token (str) – Permanent access token. You can find documentation on how to acquire a permanent access token here
wrike_instance (str) – Wrike’s instance such as app-us2.wrike.com
start_date (Optional[str]) – UTC date and time in the format 2017-01-25T00:00:00Z. Only comments after this date will be replicated.
Airbyte Source for Commercetools
Documentation can be found at https://docs.airbyte.com/integrations/sources/commercetools
name (str) – The name of the destination.
region (str) – The region of the platform.
host (str) – The cloud provider your shop is hosted. See: https://docs.commercetools.com/api/authorization
start_date (str) – The date you would like to replicate data. Format: YYYY-MM-DD.
project_key (str) – The project key
client_id (str) – Id of API Client.
client_secret (str) – The password of secret of API Client.
Airbyte Source for Gutendex
Documentation can be found at https://docs.airbyte.com/integrations/sources/gutendex
name (str) – The name of the destination.
author_year_start (Optional[str]) – (Optional) Defines the minimum birth year of the authors. Books by authors born prior to the start year will not be returned. Supports both positive (CE) or negative (BCE) integer values
author_year_end (Optional[str]) – (Optional) Defines the maximum birth year of the authors. Books by authors born after the end year will not be returned. Supports both positive (CE) or negative (BCE) integer values
copyright (Optional[str]) – (Optional) Use this to find books with a certain copyright status - true for books with existing copyrights, false for books in the public domain in the USA, or null for books with no available copyright information.
languages (Optional[str]) – (Optional) Use this to find books in any of a list of languages. They must be comma-separated, two-character language codes.
search (Optional[str]) – (Optional) Use this to search author names and book titles with given words. They must be separated by a space (i.e. %20 in URL-encoded format) and are case-insensitive.
sort (Optional[str]) – (Optional) Use this to sort books - ascending for Project Gutenberg ID numbers from lowest to highest, descending for IDs highest to lowest, or popular (the default) for most popular to least popular by number of downloads.
topic (Optional[str]) – (Optional) Use this to search for a case-insensitive key-phrase in books’ bookshelves or subjects.
Airbyte Source for Iterable
Documentation can be found at https://docs.airbyte.com/integrations/sources/iterable
name (str) – The name of the destination.
api_key (str) – Iterable API Key. See the docs for more information on how to obtain this key.
start_date (str) – The date from which you’d like to replicate data for Iterable, in the format YYYY-MM-DDT00:00:00Z. All data generated after this date will be replicated.
Airbyte Source for Quickbooks Singer
Documentation can be found at https://docs.airbyte.com/integrations/sources/quickbooks
name (str) – The name of the destination.
client_id (str) – Identifies which app is making the request. Obtain this value from the Keys tab on the app profile via My Apps on the developer site. There are two versions of this key: development and production.
client_secret (str) – Obtain this value from the Keys tab on the app profile via My Apps on the developer site. There are two versions of this key: development and production.
refresh_token (str) – A token used when refreshing the access token.
realm_id (str) – Labeled Company ID. The Make API Calls panel is populated with the realm id and the current access token.
user_agent (str) – Process and email for API logging purposes. Example: tap-quickbooks .
start_date (str) – The default value to use if no bookmark exists for an endpoint (rfc3339 date string). E.g, 2021-03-20T00:00:00Z. Any data before this date will not be replicated.
sandbox (bool) – Determines whether to use the sandbox or production environment.
Airbyte Source for Bigcommerce
Documentation can be found at https://docs.airbyte.com/integrations/sources/bigcommerce
name (str) – The name of the destination.
start_date (str) – The date you would like to replicate data. Format: YYYY-MM-DD.
store_hash (str) – The hash code of the store. For https://api.bigcommerce.com/stores/HASH_CODE/v3/, The store’s hash code is ‘HASH_CODE’.
access_token (str) – Access Token for making authenticated requests.
Airbyte Source for Shopify
Documentation can be found at https://docs.airbyte.com/integrations/sources/shopify
name (str) – The name of the destination.
shop (str) – The name of your Shopify store found in the URL. For example, if your URL was https://NAME.myshopify.com, then the name would be ‘NAME’.
credentials (Union[ShopifySource.APIPassword, ShopifySource.OAuth20]) – The authorization method to use to retrieve data from Shopify
start_date (str) – The date you would like to replicate data from. Format: YYYY-MM-DD. Any data before this date will not be replicated.
Airbyte Source for Appstore Singer
Documentation can be found at https://docs.airbyte.com/integrations/sources/appstore
name (str) – The name of the destination.
key_id (str) – Appstore Key ID. See the docs for more information on how to obtain this key.
private_key (str) – Appstore Private Key. See the docs for more information on how to obtain this key.
issuer_id (str) – Appstore Issuer ID. See the docs for more information on how to obtain this ID.
vendor (str) – Appstore Vendor ID. See the docs for more information on how to obtain this ID.
start_date (str) – UTC date and time in the format 2017-01-25T00:00:00Z. Any data before this date will not be replicated.
Airbyte Source for Greenhouse
Documentation can be found at https://docs.airbyte.com/integrations/sources/greenhouse
name (str) – The name of the destination.
api_key (str) – Greenhouse API Key. See the docs for more information on how to generate this key.
Airbyte Source for Zoom Singer
Documentation can be found at https://docs.airbyte.com/integrations/sources/zoom
name (str) – The name of the destination.
jwt (str) – Zoom JWT Token. See the docs for more information on how to obtain this key.
Airbyte Source for Tiktok Marketing
Documentation can be found at https://docs.airbyte.com/integrations/sources/tiktok-marketing
name (str) – The name of the destination.
credentials (Union[TiktokMarketingSource.OAuth20, TiktokMarketingSource.SandboxAccessToken]) – Authentication method
start_date (Optional[str]) – The Start Date in format: YYYY-MM-DD. Any data before this date will not be replicated. If this parameter is not set, all data will be replicated.
end_date (Optional[str]) – The date until which you’d like to replicate data for all incremental streams, in the format YYYY-MM-DD. All data generated between start_date and this date will be replicated. Not setting this option will result in always syncing the data till the current date.
report_granularity (Optional[str]) – The granularity used for aggregating performance data in reports. See the docs.
Airbyte Source for Zendesk Chat
Documentation can be found at https://docs.airbyte.com/integrations/sources/zendesk-chat
name (str) – The name of the destination.
start_date (str) – The date from which you’d like to replicate data for Zendesk Chat API, in the format YYYY-MM-DDT00:00:00Z.
subdomain (Optional[str]) – Required if you access Zendesk Chat from a Zendesk Support subdomain.
Airbyte Source for Aws Cloudtrail
Documentation can be found at https://docs.airbyte.com/integrations/sources/aws-cloudtrail
name (str) – The name of the destination.
aws_key_id (str) – AWS CloudTrail Access Key ID. See the docs for more information on how to obtain this key.
aws_secret_key (str) – AWS CloudTrail Access Key ID. See the docs for more information on how to obtain this key.
aws_region_name (str) – The default AWS Region to use, for example, us-west-1 or us-west-2. When specifying a Region inline during client initialization, this property is named region_name.
start_date (str) – The date you would like to replicate data. Data in AWS CloudTrail is available for last 90 days only. Format: YYYY-MM-DD.
Airbyte Source for Okta
Documentation can be found at https://docs.airbyte.com/integrations/sources/okta
name (str) – The name of the destination.
domain (Optional[str]) – The Okta domain. See the docs for instructions on how to find it.
start_date (Optional[str]) – UTC date and time in the format YYYY-MM-DDTHH:MM:SSZ. Any data before this date will not be replicated.
Airbyte Source for Insightly
Documentation can be found at https://docs.airbyte.com/integrations/sources/insightly
name (str) – The name of the destination.
token (Optional[str]) – Your Insightly API token.
start_date (Optional[str]) – The date from which you’d like to replicate data for Insightly in the format YYYY-MM-DDT00:00:00Z. All data generated after this date will be replicated. Note that it will be used only for incremental streams.
Airbyte Source for Linkedin Pages
Documentation can be found at https://docs.airbyte.com/integrations/sources/linkedin-pages/
name (str) – The name of the destination.
org_id (int) – Specify the Organization ID
Airbyte Source for Persistiq
Documentation can be found at https://docs.airbyte.com/integrations/sources/persistiq
name (str) – The name of the destination.
api_key (str) – PersistIq API Key. See the docs for more information on where to find that key.
Airbyte Source for Freshcaller
Documentation can be found at https://docs.airbyte.com/integrations/sources/freshcaller
name (str) – The name of the destination.
domain (str) – Used to construct Base URL for the Freshcaller APIs
api_key (str) – Freshcaller API Key. See the docs for more information on how to obtain this key.
requests_per_minute (Optional[int]) – The number of requests per minute that this source allowed to use. There is a rate limit of 50 requests per minute per app per account.
start_date (str) – UTC date and time. Any data created after this date will be replicated.
sync_lag_minutes (Optional[int]) – Lag in minutes for each sync, i.e., at time T, data for the time range [prev_sync_time, T-30] will be fetched
Airbyte Source for Appfollow
Documentation can be found at https://docs.airbyte.com/integrations/sources/appfollow
name (str) – The name of the destination.
ext_id (str) – for App Store — this is 9-10 digits identification number; for Google Play — this is bundle name;
cid (str) – client id provided by Appfollow
api_secret (str) – api secret provided by Appfollow
country (str) – getting data by Country
Airbyte Source for Facebook Pages
Documentation can be found at https://docs.airbyte.com/integrations/sources/facebook-pages
name (str) – The name of the destination.
access_token (str) – Facebook Page Access Token
page_id (str) – Page ID
Airbyte Source for Jira
Documentation can be found at https://docs.airbyte.com/integrations/sources/jira
name (str) – The name of the destination.
api_token (str) – Jira API Token. See the docs for more information on how to generate this key.
domain (str) – The Domain for your Jira account, e.g. airbyteio.atlassian.net
email (str) – The user email for your Jira account.
projects (Optional[List[str]]) – List of Jira project keys to replicate data for.
start_date (Optional[str]) – The date from which you’d like to replicate data for Jira in the format YYYY-MM-DDT00:00:00Z. All data generated after this date will be replicated. Note that it will be used only in the following incremental streams: issues.
additional_fields (Optional[List[str]]) – List of additional fields to include in replicating issues.
expand_issue_changelog (Optional[bool]) – Expand the changelog when replicating issues.
render_fields (Optional[bool]) – Render issue fields in HTML format in addition to Jira JSON-like format.
enable_experimental_streams (Optional[bool]) – Allow the use of experimental streams which rely on undocumented Jira API endpoints. See https://docs.airbyte.com/integrations/sources/jira#experimental-tables for more info.
Airbyte Source for Google Sheets
Documentation can be found at https://docs.airbyte.com/integrations/sources/google-sheets
name (str) – The name of the destination.
spreadsheet_id (str) – Enter the link to the Google spreadsheet you want to sync
row_batch_size (Optional[int]) – Number of rows fetched when making a Google Sheet API call. Defaults to 200.
credentials (Union[GoogleSheetsSource.AuthenticateViaGoogleOAuth, GoogleSheetsSource.ServiceAccountKeyAuthentication]) – Credentials for connecting to the Google Sheets API
Airbyte Source for Dockerhub
Documentation can be found at https://docs.airbyte.com/integrations/sources/dockerhub
name (str) – The name of the destination.
docker_username (str) – Username of DockerHub person or organization (for https://hub.docker.com/v2/repositories/USERNAME/ API call)
Airbyte Source for Us Census
Documentation can be found at https://docs.airbyte.com/integrations/sources/us-census
name (str) – The name of the destination.
query_params (Optional[str]) – The query parameters portion of the GET request, without the api key
query_path (str) – The path portion of the GET request
api_key (str) – Your API Key. Get your key here.
Airbyte Source for Kustomer Singer
Documentation can be found at https://docs.airbyte.com/integrations/sources/kustomer
name (str) – The name of the destination.
api_token (str) – Kustomer API Token. See the docs on how to obtain this
start_date (str) – The date from which you’d like to replicate the data
Airbyte Source for Azure Table
name (str) – The name of the destination.
storage_account_name (str) – The name of your storage account.
storage_access_key (str) – Azure Table Storage Access Key. See the docs for more information on how to obtain this key.
storage_endpoint_suffix (Optional[str]) – Azure Table Storage service account URL suffix. See the docs for more information on how to obtain endpoint suffix
Airbyte Source for Scaffold Java Jdbc
Documentation can be found at https://docs.airbyte.com/integrations/sources/scaffold_java_jdbc
name (str) – The name of the destination.
host (str) – Hostname of the database.
port (int) – Port of the database.
database (str) – Name of the database.
username (str) – Username to use to access the database.
password (Optional[str]) – Password associated with the username.
jdbc_url_params (Optional[str]) – Additional properties to pass to the JDBC URL string when connecting to the database formatted as ‘key=value’ pairs separated by the symbol ‘&’. (example: key1=value1&key2=value2&key3=value3)
replication_method (str) – Replication method to use for extracting data from the database. STANDARD replication requires no setup on the DB side but will not be able to represent deletions incrementally. CDC uses the Binlog to detect inserts, updates, and deletes. This needs to be configured on the source database itself.
Airbyte Source for Tidb
Documentation can be found at https://docs.airbyte.com/integrations/sources/tidb
name (str) – The name of the destination.
host (str) – Hostname of the database.
port (int) – Port of the database.
database (str) – Name of the database.
username (str) – Username to use to access the database.
password (Optional[str]) – Password associated with the username.
jdbc_url_params (Optional[str]) – Additional properties to pass to the JDBC URL string when connecting to the database formatted as ‘key=value’ pairs separated by the symbol ‘&’. (example: key1=value1&key2=value2&key3=value3)
ssl (Optional[bool]) – Encrypt data using SSL.
Airbyte Source for Qualaroo
Documentation can be found at https://docs.airbyte.com/integrations/sources/qualaroo
name (str) – The name of the destination.
token (str) – A Qualaroo token. See the docs for instructions on how to generate it.
key (str) – A Qualaroo token. See the docs for instructions on how to generate it.
start_date (str) – UTC date and time in the format 2017-01-25T00:00:00Z. Any data before this date will not be replicated.
survey_ids (Optional[List[str]]) – IDs of the surveys from which you’d like to replicate data. If left empty, data from all surveys to which you have access will be replicated.
Airbyte Source for Yahoo Finance Price
name (str) – The name of the destination.
tickers (str) – Comma-separated identifiers for the stocks to be queried. Whitespaces are allowed.
interval (Optional[str]) – The interval of between prices queried.
range (Optional[str]) – The range of prices to be queried.
Airbyte Source for Google Analytics V4
Documentation can be found at https://docs.airbyte.com/integrations/sources/google-analytics-universal-analytics
name (str) – The name of the destination.
credentials (Union[GoogleAnalyticsV4Source.AuthenticateViaGoogleOauth, GoogleAnalyticsV4Source.ServiceAccountKeyAuthentication]) – Credentials for the service
start_date (str) – The date in the format YYYY-MM-DD. Any data before this date will not be replicated.
view_id (str) – The ID for the Google Analytics View you want to fetch data from. This can be found from the Google Analytics Account Explorer.
custom_reports (Optional[str]) – A JSON array describing the custom reports you want to sync from Google Analytics. See the docs for more information about the exact format you can use to fill out this field.
window_in_days (Optional[int]) – The time increment used by the connector when requesting data from the Google Analytics API. More information is available in the the docs. The bigger this value is, the faster the sync will be, but the more likely that sampling will be applied to your data, potentially causing inaccuracies in the returned results. We recommend setting this to 1 unless you have a hard requirement to make the sync faster at the expense of accuracy. The minimum allowed value for this field is 1, and the maximum is 364.
Airbyte Source for Jdbc
Documentation can be found at https://docs.airbyte.com/integrations/sources/postgres
name (str) – The name of the destination.
username (str) – The username which is used to access the database.
password (Optional[str]) – The password associated with this username.
jdbc_url (str) – JDBC formatted URL. See the standard here.
jdbc_url_params (Optional[str]) – Additional properties to pass to the JDBC URL string when connecting to the database formatted as ‘key=value’ pairs separated by the symbol ‘&’. (example: key1=value1&key2=value2&key3=value3).
Airbyte Source for Faker
Documentation can be found at https://docs.airbyte.com/integrations/sources/faker
name (str) – The name of the destination.
count (int) – How many users should be generated in total. This setting does not apply to the purchases or products stream.
seed (Optional[int]) – Manually control the faker random seed to return the same values on subsequent runs (leave -1 for random)
records_per_sync (Optional[int]) – How many fake records will be returned for each sync, for each stream? By default, it will take 2 syncs to create the requested 1000 records.
records_per_slice (Optional[int]) – How many fake records will be in each page (stream slice), before a state message is emitted?
Airbyte Source for Tplcentral
Documentation can be found at https://docs.airbyte.com/integrations/sources/tplcentral
name (str) – The name of the destination.
user_login_id (Optional[int]) – User login ID and/or name is required
user_login (Optional[str]) – User login ID and/or name is required
start_date (Optional[str]) – Date and time together in RFC 3339 format, for example, 2018-11-13T20:20:39+00:00.
Airbyte Source for Clickhouse
Documentation can be found at https://docs.airbyte.com/integrations/destinations/clickhouse
name (str) – The name of the destination.
host (str) – The host endpoint of the Clickhouse cluster.
port (int) – The port of the database.
database (str) – The name of the database.
username (str) – The username which is used to access the database.
password (Optional[str]) – The password associated with this username.
jdbc_url_params (Optional[str]) – Additional properties to pass to the JDBC URL string when connecting to the database formatted as ‘key=value’ pairs separated by the symbol ‘&’. (Eg. key1=value1&key2=value2&key3=value3). For more information read about JDBC URL parameters.
ssl (Optional[bool]) – Encrypt data using SSL.
Airbyte Source for Freshservice
Documentation can be found at https://docs.airbyte.com/integrations/sources/freshservice
name (str) – The name of the destination.
domain_name (str) – The name of your Freshservice domain
api_key (str) – Freshservice API Key. See here. The key is case sensitive.
start_date (str) – UTC date and time in the format 2020-10-01T00:00:00Z. Any data before this date will not be replicated.
Airbyte Source for Zenloop
Documentation can be found at https://docs.airbyte.com/integrations/sources/zenloop
name (str) – The name of the destination.
api_token (str) – Zenloop API Token. You can get the API token in settings page here
date_from (Optional[str]) – Zenloop date_from. Format: 2021-10-24T03:30:30Z or 2021-10-24. Leave empty if only data from current data should be synced
survey_id (Optional[str]) – Zenloop Survey ID. Can be found here. Leave empty to pull answers from all surveys
survey_group_id (Optional[str]) – Zenloop Survey Group ID. Can be found by pulling All Survey Groups via SurveyGroups stream. Leave empty to pull answers from all survey groups
Airbyte Source for Oracle
Documentation can be found at https://docs.airbyte.com/integrations/sources/oracle
name (str) – The name of the destination.
host (str) – Hostname of the database.
port (int) – Port of the database. Oracle Corporations recommends the following port numbers: 1521 - Default listening port for client connections to the listener. 2484 - Recommended and officially registered listening port for client connections to the listener using TCP/IP with SSL
connection_data (Union[OracleSource.ServiceName, OracleSource.SystemIDSID]) – Connect data that will be used for DB connection
username (str) – The username which is used to access the database.
password (Optional[str]) – The password associated with the username.
schemas (Optional[List[str]]) – The list of schemas to sync from. Defaults to user. Case sensitive.
jdbc_url_params (Optional[str]) – Additional properties to pass to the JDBC URL string when connecting to the database formatted as ‘key=value’ pairs separated by the symbol ‘&’. (example: key1=value1&key2=value2&key3=value3).
encryption (Union[OracleSource.Unencrypted, OracleSource.NativeNetworkEncryptionNNE, OracleSource.TLSEncryptedVerifyCertificate]) – The encryption method with is used when communicating with the database.
Airbyte Source for Klaviyo
Documentation can be found at https://docs.airbyte.com/integrations/sources/klaviyo
name (str) – The name of the destination.
api_key (str) – Klaviyo API Key. See our docs if you need help finding this key.
start_date (str) – UTC date and time in the format 2017-01-25T00:00:00Z. Any data before this date will not be replicated.
Airbyte Source for Google Directory
Documentation can be found at https://docs.airbyte.com/integrations/sources/google-directory
name (str) – The name of the destination.
credentials (Union[GoogleDirectorySource.SignInViaGoogleOAuth, GoogleDirectorySource.ServiceAccountKey]) – Google APIs use the OAuth 2.0 protocol for authentication and authorization. The Source supports Web server application and Service accounts scenarios.
Airbyte Source for Instagram
Documentation can be found at https://docs.airbyte.com/integrations/sources/instagram
name (str) – The name of the destination.
start_date (str) – The date from which you’d like to replicate data for User Insights, in the format YYYY-MM-DDT00:00:00Z. All data generated after this date will be replicated.
access_token (str) – The value of the access token generated. See the docs for more information
Airbyte Source for Shortio
Documentation can be found at https://developers.short.io/reference
name (str) – The name of the destination.
secret_key (str) – Short.io Secret Key
start_date (str) – UTC date and time in the format 2017-01-25T00:00:00Z. Any data before this date will not be replicated.
Airbyte Source for Square
Documentation can be found at https://docs.airbyte.com/integrations/sources/square
name (str) – The name of the destination.
is_sandbox (bool) – Determines whether to use the sandbox or production environment.
start_date (Optional[str]) – UTC date in the format YYYY-MM-DD. Any data before this date will not be replicated. If not set, all data will be replicated.
include_deleted_objects (Optional[bool]) – In some streams there is an option to include deleted objects (Items, Categories, Discounts, Taxes)
Airbyte Source for Amazon Sqs
Documentation can be found at https://docs.airbyte.com/integrations/sources/amazon-sqs
name (str) – The name of the destination.
queue_url (str) – URL of the SQS Queue
region (str) – AWS Region of the SQS Queue
delete_messages (bool) – If Enabled, messages will be deleted from the SQS Queue after being read. If Disabled, messages are left in the queue and can be read more than once. WARNING: Enabling this option can result in data loss in cases of failure, use with caution, see documentation for more detail.
max_batch_size (Optional[int]) – Max amount of messages to get in one batch (10 max)
max_wait_time (Optional[int]) – Max amount of time in seconds to wait for messages in a single poll (20 max)
attributes_to_return (Optional[str]) – Comma separated list of Mesage Attribute names to return
visibility_timeout (Optional[int]) – Modify the Visibility Timeout of the individual message from the Queue’s default (seconds).
access_key (Optional[str]) – The Access Key ID of the AWS IAM Role to use for pulling messages
secret_key (Optional[str]) – The Secret Key of the AWS IAM Role to use for pulling messages
Airbyte Source for Youtube Analytics
Documentation can be found at https://docs.airbyte.com/integrations/sources/youtube-analytics
name (str) – The name of the destination.
Airbyte Source for Looker
Documentation can be found at https://docs.airbyte.com/integrations/sources/looker
name (str) – The name of the destination.
domain (str) – Domain for your Looker account, e.g. airbyte.cloud.looker.com,looker.[clientname].com,IP address
client_id (str) – The Client ID is first part of an API3 key that is specific to each Looker user. See the docs for more information on how to generate this key.
client_secret (str) – The Client Secret is second part of an API3 key.
run_look_ids (Optional[List[str]]) – The IDs of any Looks to run
Airbyte Source for Gitlab
Documentation can be found at https://docs.airbyte.com/integrations/sources/gitlab
name (str) – The name of the destination.
api_url (str) – Please enter your basic URL from GitLab instance.
private_token (str) – Log into your GitLab account and then generate a personal Access Token.
groups (Optional[str]) – Space-delimited list of groups. e.g. airbyte.io.
projects (Optional[str]) – Space-delimited list of projects. e.g. airbyte.io/documentation meltano/tap-gitlab.
start_date (str) – The date from which you’d like to replicate data for GitLab API, in the format YYYY-MM-DDT00:00:00Z. All data generated after this date will be replicated.
Airbyte Source for Exchange Rates
Documentation can be found at https://docs.airbyte.com/integrations/sources/exchangeratesapi
name (str) – The name of the destination.
start_date (str) – Start getting data from that date.
access_key (str) – Your API Key. See here. The key is case sensitive.
base (Optional[str]) – ISO reference currency. See here. Free plan doesn’t support Source Currency Switching, default base currency is EUR
ignore_weekends (Optional[bool]) – Ignore weekends? (Exchanges don’t run on weekends)
Airbyte Source for Amazon Ads
Documentation can be found at https://docs.airbyte.com/integrations/sources/amazon-ads
name (str) – The name of the destination.
client_id (str) – The client ID of your Amazon Ads developer application. See the docs for more information.
client_secret (str) – The client secret of your Amazon Ads developer application. See the docs for more information.
refresh_token (str) – Amazon Ads refresh token. See the docs for more information on how to obtain this token.
region (Optional[str]) – Region to pull data from (EU/NA/FE). See docs for more details.
report_wait_timeout (Optional[int]) – Timeout duration in minutes for Reports. Default is 60 minutes.
report_generation_max_retries (Optional[int]) – Maximum retries Airbyte will attempt for fetching report data. Default is 5.
start_date (Optional[str]) – The Start date for collecting reports, should not be more than 60 days in the past. In YYYY-MM-DD format
profiles (Optional[List[int]]) – Profile IDs you want to fetch data for. See docs for more details.
state_filter (Optional[List[str]]) – Reflects the state of the Display, Product, and Brand Campaign streams as enabled, paused, or archived. If you do not populate this field, it will be ignored completely.
Airbyte Source for Mixpanel
Documentation can be found at https://docs.airbyte.com/integrations/sources/mixpanel
name (str) – The name of the destination.
credentials (Union[MixpanelSource.ServiceAccount, MixpanelSource.ProjectSecret]) – Choose how to authenticate to Mixpanel
project_id (Optional[int]) – Your project ID number. See the docs for more information on how to obtain this.
attribution_window (Optional[int]) – A period of time for attributing results to ads and the lookback period after those actions occur during which ad results are counted. Default attribution window is 5 days.
project_timezone (Optional[str]) – Time zone in which integer date times are stored. The project timezone may be found in the project settings in the Mixpanel console.
select_properties_by_default (Optional[bool]) – Setting this config parameter to TRUE ensures that new properties on events and engage records are captured. Otherwise new properties will be ignored.
start_date (Optional[str]) – The date in the format YYYY-MM-DD. Any data before this date will not be replicated. If this option is not set, the connector will replicate data from up to one year ago by default.
end_date (Optional[str]) – The date in the format YYYY-MM-DD. Any data after this date will not be replicated. Left empty to always sync to most recent date
region (Optional[str]) – The region of mixpanel domain instance either US or EU.
date_window_size (Optional[int]) – Defines window size in days, that used to slice through data. You can reduce it, if amount of data in each window is too big for your environment.
Airbyte Source for Orbit
Documentation can be found at https://docs.airbyte.com/integrations/sources/orbit
name (str) – The name of the destination.
api_token (str) – Authorizes you to work with Orbit workspaces associated with the token.
workspace (str) – The unique name of the workspace that your API token is associated with.
start_date (Optional[str]) – Date in the format 2022-06-26. Only load members whose last activities are after this date.
Airbyte Source for Amazon Seller Partner
Documentation can be found at https://docs.airbyte.com/integrations/sources/amazon-seller-partner
name (str) – The name of the destination.
app_id (Optional[str]) – Your Amazon App ID
lwa_app_id (str) – Your Login with Amazon Client ID.
lwa_client_secret (str) – Your Login with Amazon Client Secret.
refresh_token (str) – The Refresh Token obtained via OAuth flow authorization.
aws_access_key (str) – Specifies the AWS access key used as part of the credentials to authenticate the user.
aws_secret_key (str) – Specifies the AWS secret key used as part of the credentials to authenticate the user.
role_arn (str) – Specifies the Amazon Resource Name (ARN) of an IAM role that you want to use to perform operations requested using this profile. (Needs permission to ‘Assume Role’ STS).
replication_start_date (str) – UTC date and time in the format 2017-01-25T00:00:00Z. Any data before this date will not be replicated.
replication_end_date (Optional[str]) – UTC date and time in the format 2017-01-25T00:00:00Z. Any data after this date will not be replicated.
period_in_days (Optional[int]) – Will be used for stream slicing for initial full_refresh sync when no updated state is present for reports that support sliced incremental sync.
report_options (Optional[str]) – Additional information passed to reports. This varies by report type. Must be a valid json string.
max_wait_seconds (Optional[int]) – Sometimes report can take up to 30 minutes to generate. This will set the limit for how long to wait for a successful report.
aws_environment (str) – An enumeration.
region (str) – An enumeration.
Airbyte Source for Courier
Documentation can be found at https://docs.airbyte.io/integrations/sources/courier
name (str) – The name of the destination.
api_key (str) – Courier API Key to retrieve your data.
Airbyte Source for Close Com
Documentation can be found at https://docs.airbyte.com/integrations/sources/close-com
name (str) – The name of the destination.
api_key (str) – Close.com API key (usually starts with ‘api\_’; find yours here).
start_date (Optional[str]) – The start date to sync data. Leave blank for full sync. Format: YYYY-MM-DD.
Airbyte Source for Bing Ads
Documentation can be found at https://docs.airbyte.com/integrations/sources/bing-ads
name (str) – The name of the destination.
tenant_id (Optional[str]) – The Tenant ID of your Microsoft Advertising developer application. Set this to “common” unless you know you need a different value.
client_id (str) – The Client ID of your Microsoft Advertising developer application.
client_secret (Optional[str]) – The Client Secret of your Microsoft Advertising developer application.
refresh_token (str) – Refresh Token to renew the expired Access Token.
developer_token (str) – Developer token associated with user. See more info in the docs.
reports_start_date (str) – The start date from which to begin replicating report data. Any data generated before this date will not be replicated in reports. This is a UTC date in YYYY-MM-DD format.
Airbyte Source for Primetric
name (str) – The name of the destination.
client_id (str) – The Client ID of your Primetric developer application. The Client ID is visible here.
client_secret (str) – The Client Secret of your Primetric developer application. You can manage your client’s credentials here.
Airbyte Source for Elasticsearch
Documentation can be found at https://docs.airbyte.com/integrations/source/elasticsearch
name (str) – The name of the destination.
endpoint (str) – The full url of the Elasticsearch server
authenticationMethod (Union[ElasticsearchSource.None\_, ElasticsearchSource.ApiKeySecret, ElasticsearchSource.UsernamePassword]) – The type of authentication to be used
Airbyte Source for Bigquery
Documentation can be found at https://docs.airbyte.com/integrations/sources/bigquery
name (str) – The name of the destination.
project_id (str) – The GCP project ID for the project containing the target BigQuery dataset.
dataset_id (Optional[str]) – The dataset ID to search for tables and views. If you are only loading data from one dataset, setting this option could result in much faster schema discovery.
credentials_json (str) – The contents of your Service Account Key JSON file. See the docs for more information on how to obtain this key.
Airbyte Source for Woocommerce
Documentation can be found at https://docs.airbyte.com/integrations/sources/woocommerce
name (str) – The name of the destination.
shop (str) – The name of the store. For https://EXAMPLE.com, the shop name is ‘EXAMPLE.com’.
start_date (str) – The date you would like to replicate data. Format: YYYY-MM-DD.
api_key (str) – The CUSTOMER KEY for API in WooCommerce shop.
api_secret (str) – The CUSTOMER SECRET for API in WooCommerce shop.
conversion_window_days (Optional[int]) – A conversion window is the period of time after an ad interaction (such as an ad click or video view) during which a conversion, such as a purchase, is recorded in Google Ads.
Airbyte Source for Search Metrics
Documentation can be found at https://docs.airbyte.com/integrations/sources/seacrh-metrics
name (str) – The name of the destination.
country_code (str) – The region of the S3 staging bucket to use if utilising a copy strategy.
start_date (str) – Data generated in SearchMetrics after this date will be replicated. This date must be specified in the format YYYY-MM-DDT00:00:00Z.
Airbyte Source for Typeform
Documentation can be found at https://docs.airbyte.com/integrations/sources/typeform
name (str) – The name of the destination.
start_date (str) – UTC date and time in the format: YYYY-MM-DDTHH:mm:ss[Z]. Any data before this date will not be replicated.
token (str) – The API Token for a Typeform account.
form_ids (Optional[List[str]]) – When this parameter is set, the connector will replicate data only from the input forms. Otherwise, all forms in your Typeform account will be replicated. You can find form IDs in your form URLs. For example, in the URL “https://mysite.typeform.com/to/u6nXL7” the form_id is u6nXL7. You can find form URLs on Share panel
Airbyte Source for Webflow
Documentation can be found at https://docs.airbyte.com/integrations/sources/webflow
name (str) – The name of the destination.
site_id (str) – The id of the Webflow site you are requesting data from. See https://developers.webflow.com/#sites
api_key (str) – The API token for authenticating to Webflow. See https://university.webflow.com/lesson/intro-to-the-webflow-api
Airbyte Source for Firebolt
Documentation can be found at https://docs.airbyte.com/integrations/sources/firebolt
name (str) – The name of the destination.
username (str) – Firebolt email address you use to login.
password (str) – Firebolt password.
account (Optional[str]) – Firebolt account to login.
host (Optional[str]) – The host name of your Firebolt database.
database (str) – The database to connect to.
engine (Optional[str]) – Engine name or url to connect to.
Airbyte Source for Fauna
Documentation can be found at https://github.com/fauna/airbyte/blob/source-fauna/docs/integrations/sources/fauna.md
name (str) – The name of the destination.
domain (str) – Domain of Fauna to query. Defaults db.fauna.com. See the docs.
port (int) – Endpoint port.
scheme (str) – URL scheme.
secret (str) – Fauna secret, used when authenticating with the database.
collection (FaunaSource.Collection) – Settings for the Fauna Collection.
Airbyte Source for Intercom
Documentation can be found at https://docs.airbyte.com/integrations/sources/intercom
name (str) – The name of the destination.
start_date (str) – UTC date and time in the format 2017-01-25T00:00:00Z. Any data before this date will not be replicated.
access_token (str) – Access token for making authenticated requests. See the Intercom docs for more information.
Airbyte Source for Freshsales
Documentation can be found at https://docs.airbyte.com/integrations/sources/freshsales
name (str) – The name of the destination.
domain_name (str) – The Name of your Freshsales domain
api_key (str) – Freshsales API Key. See here. The key is case sensitive.
Airbyte Source for Adjust
Documentation can be found at https://docs.airbyte.com/integrations/sources/adjust
name (str) – The name of the destination.
additional_metrics (Optional[List[str]]) – Metrics names that are not pre-defined, such as cohort metrics or app specific metrics.
api_token (str) – Adjust API key, see https://help.adjust.com/en/article/report-service-api-authentication
dimensions (List[str]) – Dimensions allow a user to break down metrics into groups using one or several parameters. For example, the number of installs by date, country and network. See https://help.adjust.com/en/article/reports-endpoint#dimensions for more information about the dimensions.
ingest_start (str) – Data ingest start date.
metrics (List[str]) – Select at least one metric to query.
until_today (Optional[bool]) – Syncs data up until today. Useful when running daily incremental syncs, and duplicates are not desired.
Airbyte Source for Bamboo Hr
Documentation can be found at https://docs.airbyte.com/integrations/sources/bamboo-hr
name (str) – The name of the destination.
subdomain (str) – Sub Domain of bamboo hr
api_key (str) – Api key of bamboo hr
custom_reports_fields (Optional[str]) – Comma-separated list of fields to include in custom reports.
custom_reports_include_default_fields (Optional[bool]) – If true, the custom reports endpoint will include the default fields defined here: https://documentation.bamboohr.com/docs/list-of-field-names.
Airbyte Source for Google Ads
Documentation can be found at https://docs.airbyte.com/integrations/sources/google-ads
name (str) – The name of the destination.
customer_id (str) – Comma separated list of (client) customer IDs. Each customer ID must be specified as a 10-digit number without dashes. More instruction on how to find this value in our docs. Metrics streams like AdGroupAdReport cannot be requested for a manager account.
start_date (str) – UTC date and time in the format 2017-01-25. Any data before this date will not be replicated.
end_date (Optional[str]) – UTC date and time in the format 2017-01-25. Any data after this date will not be replicated.
login_customer_id (Optional[str]) – If your access to the customer account is through a manager account, this field is required and must be set to the customer ID of the manager account (10-digit number without dashes). More information about this field you can see here
conversion_window_days (Optional[int]) – A conversion window is the period of time after an ad interaction (such as an ad click or video view) during which a conversion, such as a purchase, is recorded in Google Ads. For more information, see Google’s documentation.
Airbyte Source for Sendgrid
Documentation can be found at https://docs.airbyte.com/integrations/sources/sendgrid
name (str) – The name of the destination.
apikey (str) – API Key, use admin to generate this key.
start_time (Union[int, str]) – Start time in ISO8601 format. Any data before this time point will not be replicated.
Airbyte Source for Monday
Documentation can be found at https://docs.airbyte.com/integrations/sources/monday
name (str) – The name of the destination.
Airbyte Source for Dixa
Documentation can be found at https://docs.airbyte.com/integrations/sources/dixa
name (str) – The name of the destination.
api_token (str) – Dixa API token
start_date (str) – The connector pulls records updated from this date onwards.
batch_size (Optional[int]) – Number of days to batch into one request. Max 31.
Airbyte Source for Salesforce
Documentation can be found at https://docs.airbyte.com/integrations/sources/salesforce
name (str) – The name of the destination.
is_sandbox (Optional[bool]) – Toggle if you’re using a Salesforce Sandbox
client_id (str) – Enter your Salesforce developer application’s Client ID
client_secret (str) – Enter your Salesforce developer application’s Client secret
refresh_token (str) – Enter your application’s Salesforce Refresh Token used for Airbyte to access your Salesforce account.
start_date (Optional[str]) – Enter the date in the YYYY-MM-DD format. Airbyte will replicate the data added on and after this date. If this field is blank, Airbyte will replicate all data.
streams_criteria (Optional[List[SalesforceSource.FilterSalesforceObjectsEntry]]) – Filter streams relevant to you
Airbyte Source for Pipedrive
Documentation can be found at https://docs.airbyte.com/integrations/sources/pipedrive
name (str) – The name of the destination.
authorization (Union[PipedriveSource.SignInViaPipedriveOAuth, PipedriveSource.APIKeyAuthentication]) – Choose one of the possible authorization method
replication_start_date (str) – UTC date and time in the format 2017-01-25T00:00:00Z. Any data before this date will not be replicated. When specified and not None, then stream will behave as incremental
Airbyte Source for File
Documentation can be found at https://docs.airbyte.com/integrations/sources/file
name (str) – The name of the destination.
dataset_name (str) – The Name of the final table to replicate this file into (should include letters, numbers dash and underscores only).
format (str) – The Format of the file which should be replicated (Warning: some formats may be experimental, please refer to the docs).
reader_options (Optional[str]) – This should be a string in JSON format. It depends on the chosen file format to provide additional options and tune its behavior.
url (str) – The URL path to access the file which should be replicated.
provider (Union[FileSource.HTTPSPublicWeb, FileSource.GCSGoogleCloudStorage, FileSource.S3AmazonWebServices, FileSource.AzBlobAzureBlobStorage, FileSource.SSHSecureShell, FileSource.SCPSecureCopyProtocol, FileSource.SFTPSecureFileTransferProtocol, FileSource.LocalFilesystemLimited]) – The storage Provider or Location of the file(s) which should be replicated.
Airbyte Source for Glassfrog
Documentation can be found at https://docs.airbyte.com/integrations/sources/glassfrog
name (str) – The name of the destination.
api_key (str) – API key provided by Glassfrog
Airbyte Source for Chartmogul
Documentation can be found at https://docs.airbyte.com/integrations/sources/chartmogul
name (str) – The name of the destination.
api_key (str) – Chartmogul API key
start_date (str) – UTC date and time in the format 2017-01-25T00:00:00Z. When feasible, any data before this date will not be replicated.
interval (str) – Some APIs such as Metrics require intervals to cluster data.
Airbyte Source for Orb
Documentation can be found at https://docs.withorb.com/
name (str) – The name of the destination.
api_key (str) – Orb API Key, issued from the Orb admin console.
start_date (Optional[str]) – UTC date and time in the format 2022-03-01T00:00:00Z. Any data with created_at before this data will not be synced.
lookback_window_days (Optional[int]) – When set to N, the connector will always refresh resources created within the past N days. By default, updated objects that are not newly created are not incrementally synced.
string_event_properties_keys (Optional[List[str]]) – Property key names to extract from all events, in order to enrich ledger entries corresponding to an event deduction.
numeric_event_properties_keys (Optional[List[str]]) – Property key names to extract from all events, in order to enrich ledger entries corresponding to an event deduction.
Airbyte Source for Cockroachdb
Documentation can be found at https://docs.airbyte.com/integrations/sources/cockroachdb
name (str) – The name of the destination.
host (str) – Hostname of the database.
port (int) – Port of the database.
database (str) – Name of the database.
username (str) – Username to use to access the database.
password (Optional[str]) – Password associated with the username.
jdbc_url_params (Optional[str]) – Additional properties to pass to the JDBC URL string when connecting to the database formatted as ‘key=value’ pairs separated by the symbol ‘&’. (Eg. key1=value1&key2=value2&key3=value3). For more information read about JDBC URL parameters.
ssl (Optional[bool]) – Encrypt client/server communications for increased security.
Airbyte Source for Confluence
name (str) – The name of the destination.
api_token (str) – Please follow the Jira confluence for generating an API token: https://support.atlassian.com/atlassian-account/docs/manage-api-tokens-for-your-atlassian-account/
domain_name (str) – Your Confluence domain name
email (str) – Your Confluence login email
Airbyte Source for Plaid
Documentation can be found at https://plaid.com/docs/api/
name (str) – The name of the destination.
access_token (str) – The end-user’s Link access token.
api_key (str) – The Plaid API key to use to hit the API.
client_id (str) – The Plaid client id
plaid_env (str) – The Plaid environment
start_date (Optional[str]) – The date from which you’d like to replicate data for Plaid in the format YYYY-MM-DD. All data generated after this date will be replicated.
Airbyte Source for Snapchat Marketing
Documentation can be found at https://docs.airbyte.com/integrations/sources/snapchat-marketing
name (str) – The name of the destination.
client_id (str) – The Client ID of your Snapchat developer application.
client_secret (str) – The Client Secret of your Snapchat developer application.
refresh_token (str) – Refresh Token to renew the expired Access Token.
start_date (Optional[str]) – Date in the format 2022-01-01. Any data before this date will not be replicated.
end_date (Optional[str]) – Date in the format 2017-01-25. Any data after this date will not be replicated.
Airbyte Source for Microsoft Teams
Documentation can be found at https://docs.airbyte.com/integrations/sources/microsoft-teams
name (str) – The name of the destination.
period (str) – Specifies the length of time over which the Team Device Report stream is aggregated. The supported values are: D7, D30, D90, and D180.
credentials (Union[MicrosoftTeamsSource.AuthenticateViaMicrosoftOAuth20, MicrosoftTeamsSource.AuthenticateViaMicrosoft]) – Choose how to authenticate to Microsoft
Airbyte Source for Lever Hiring
Documentation can be found at https://docs.airbyte.com/integrations/sources/lever-hiring
name (str) – The name of the destination.
credentials (LeverHiringSource.OAuthCredentials) – Choose how to authenticate to Lever Hiring.
start_date (str) – UTC date and time in the format 2017-01-25T00:00:00Z. Any data before this date will not be replicated. Note that it will be used only in the following incremental streams: comments, commits, and issues.
environment (Optional[str]) – The environment in which you’d like to replicate data for Lever. This is used to determine which Lever API endpoint to use.
Airbyte Source for Twilio
Documentation can be found at https://docs.airbyte.com/integrations/sources/twilio
name (str) – The name of the destination.
account_sid (str) – Twilio account SID
auth_token (str) – Twilio Auth Token.
start_date (str) – UTC date and time in the format 2020-10-01T00:00:00Z. Any data before this date will not be replicated.
lookback_window (Optional[int]) – How far into the past to look for records. (in minutes)
Airbyte Source for Stripe
Documentation can be found at https://docs.airbyte.com/integrations/sources/stripe
name (str) – The name of the destination.
account_id (str) – Your Stripe account ID (starts with ‘acct\_’, find yours here).
client_secret (str) – Stripe API key (usually starts with ‘sk_live\_’; find yours here).
start_date (str) – UTC date and time in the format 2017-01-25T00:00:00Z. Only data generated after this date will be replicated.
lookback_window_days (Optional[int]) – When set, the connector will always re-export data from the past N days, where N is the value set here. This is useful if your data is frequently updated after creation. More info here
slice_range (Optional[int]) – The time increment used by the connector when requesting data from the Stripe API. The bigger the value is, the less requests will be made and faster the sync will be. On the other hand, the more seldom the state is persisted.
Airbyte Source for Db2
Documentation can be found at https://docs.airbyte.com/integrations/sources/db2
name (str) – The name of the destination.
host (str) – Host of the Db2.
port (int) – Port of the database.
db (str) – Name of the database.
username (str) – Username to use to access the database.
password (str) – Password associated with the username.
jdbc_url_params (Optional[str]) – Additional properties to pass to the JDBC URL string when connecting to the database formatted as ‘key=value’ pairs separated by the symbol ‘&’. (example: key1=value1&key2=value2&key3=value3).
encryption (Union[Db2Source.Unencrypted, Db2Source.TLSEncryptedVerifyCertificate]) – Encryption method to use when communicating with the database
Airbyte Source for Slack
Documentation can be found at https://docs.airbyte.com/integrations/sources/slack
name (str) – The name of the destination.
start_date (str) – UTC date and time in the format 2017-01-25T00:00:00Z. Any data before this date will not be replicated.
lookback_window (int) – How far into the past to look for messages in threads.
join_channels (bool) – Whether to join all channels or to sync data only from channels the bot is already in. If false, you’ll need to manually add the bot to all the channels from which you’d like to sync messages.
channel_filter (Optional[List[str]]) – A channel name list (without leading ‘#’ char) which limit the channels from which you’d like to sync. Empty list means no filter.
credentials (Union[SlackSource.DefaultOAuth20Authorization, SlackSource.APITokenCredentials]) – Choose how to authenticate into Slack
Airbyte Source for Recharge
Documentation can be found at https://docs.airbyte.com/integrations/sources/recharge
name (str) – The name of the destination.
start_date (str) – The date from which you’d like to replicate data for Recharge API, in the format YYYY-MM-DDT00:00:00Z. Any data before this date will not be replicated.
access_token (str) – The value of the Access Token generated. See the docs for more information.
Airbyte Source for Openweather
name (str) – The name of the destination.
lat (str) – Latitude for which you want to get weather condition from. (min -90, max 90)
lon (str) – Longitude for which you want to get weather condition from. (min -180, max 180)
appid (str) – Your OpenWeather API Key. See here. The key is case sensitive.
units (Optional[str]) – Units of measurement. standard, metric and imperial units are available. If you do not use the units parameter, standard units will be applied by default.
lang (Optional[str]) – You can use lang parameter to get the output in your language. The contents of the description field will be translated. See here for the list of supported languages.
Airbyte Source for Retently
name (str) – The name of the destination.
credentials (Union[RetentlySource.AuthenticateViaRetentlyOAuth, RetentlySource.AuthenticateWithAPIToken]) – Choose how to authenticate to Retently
Airbyte Source for Yandex Metrica
name (str) – The name of the destination.
auth_token (str) – Your Yandex Metrica API access token
counter_id (str) – Counter ID
start_date (str) – UTC date and time in the format YYYY-MM-DD.
end_date (str) – UTC date and time in the format YYYY-MM-DD.
Airbyte Source for Talkdesk Explore
name (str) – The name of the destination.
start_date (str) – The date from which you’d like to replicate data for Talkdesk Explore API, in the format YYYY-MM-DDT00:00:00. All data generated after this date will be replicated.
timezone (Optional[str]) – Timezone to use when generating reports. Only IANA timezones are supported (https://nodatime.org/TimeZones)
auth_url (str) – Talkdesk Auth URL. Only ‘client_credentials’ auth type supported at the moment.
api_key (str) – Talkdesk API key.
Airbyte Source for Chargify
Documentation can be found at https://docs.airbyte.com/integrations/sources/chargify
name (str) – The name of the destination.
api_key (str) – Chargify API Key.
domain (str) – Chargify domain. Normally this domain follows the following format companyname.chargify.com
Airbyte Source for Rki Covid
Documentation can be found at https://docs.airbyte.com/integrations/sources/rki-covid
name (str) – The name of the destination.
start_date (str) – UTC date in the format 2017-01-25. Any data before this date will not be replicated.
Airbyte Source for Postgres
Documentation can be found at https://docs.airbyte.com/integrations/sources/postgres
name (str) – The name of the destination.
host (str) – Hostname of the database.
port (int) – Port of the database.
database (str) – Name of the database.
schemas (Optional[List[str]]) – The list of schemas (case sensitive) to sync from. Defaults to public.
username (str) – Username to access the database.
password (Optional[str]) – Password associated with the username.
jdbc_url_params (Optional[str]) – Additional properties to pass to the JDBC URL string when connecting to the database formatted as ‘key=value’ pairs separated by the symbol ‘&’. (Eg. key1=value1&key2=value2&key3=value3). For more information read about JDBC URL parameters.
ssl (Optional[bool]) – Encrypt data using SSL. When activating SSL, please select one of the connection modes.
ssl_mode (Union[PostgresSource.Disable, PostgresSource.Allow, PostgresSource.Prefer, PostgresSource.Require, PostgresSource.VerifyCa, PostgresSource.VerifyFull]) – SSL connection modes. disable - Disables encryption of communication between Airbyte and source database allow - Enables encryption only when required by the source database prefer - allows unencrypted connection only if the source database does not support encryption require - Always require encryption. If the source database server does not support encryption, connection will fail verify-ca - Always require encryption and verifies that the source database server has a valid SSL certificate verify-full - This is the most secure mode. Always require encryption and verifies the identity of the source database server Read more in the docs.
replication_method (Union[PostgresSource.Standard, PostgresSource.LogicalReplicationCDC]) – Replication method for extracting data from the database.
tunnel_method (Union[PostgresSource.NoTunnel, PostgresSource.SSHKeyAuthentication, PostgresSource.PasswordAuthentication]) – Whether to initiate an SSH tunnel before connecting to the database, and if so, which kind of authentication to use.
Airbyte Source for Trello
Documentation can be found at https://docs.airbyte.com/integrations/sources/trello
name (str) – The name of the destination.
token (str) – Trello v API token. See the docs for instructions on how to generate it.
key (str) – Trello API key. See the docs for instructions on how to generate it.
start_date (str) – UTC date and time in the format 2017-01-25T00:00:00Z. Any data before this date will not be replicated.
board_ids (Optional[List[str]]) – IDs of the boards to replicate data from. If left empty, data from all boards to which you have access will be replicated.
Airbyte Source for Paystack
Documentation can be found at https://docs.airbyte.com/integrations/sources/paystack
name (str) – The name of the destination.
secret_key (str) – The Paystack API key (usually starts with ‘sk_live\_’; find yours here).
start_date (str) – UTC date and time in the format 2017-01-25T00:00:00Z. Any data before this date will not be replicated.
lookback_window_days (Optional[int]) – When set, the connector will always reload data from the past N days, where N is the value set here. This is useful if your data is updated after creation.
Airbyte Source for S3
Documentation can be found at https://docs.airbyte.com/integrations/sources/s3
name (str) – The name of the destination.
dataset (str) – The name of the stream you would like this source to output. Can contain letters, numbers, or underscores.
path_pattern (str) – A regular expression which tells the connector which files to replicate. All files which match this pattern will be replicated. Use | to separate multiple patterns. See this page to understand pattern syntax (GLOBSTAR and SPLIT flags are enabled). Use pattern ** to pick up all files.
format (Union[S3Source.CSV, S3Source.Parquet, S3Source.Avro, S3Source.Jsonl]) – The format of the files you’d like to replicate
schema (Optional[str]) – Optionally provide a schema to enforce, as a valid JSON string. Ensure this is a mapping of { “column” : “type” }, where types are valid JSON Schema datatypes. Leave as {} to auto-infer the schema.
provider (S3Source.S3AmazonWebServices) – Use this to load files from S3 or S3-compatible services
Airbyte Source for Snowflake
Documentation can be found at https://docs.airbyte.com/integrations/sources/snowflake
name (str) – The name of the destination.
host (str) – The host domain of the snowflake instance (must include the account, region, cloud environment, and end with snowflakecomputing.com).
role (str) – The role you created for Airbyte to access Snowflake.
warehouse (str) – The warehouse you created for Airbyte to access data.
database (str) – The database you created for Airbyte to access data.
schema (str) – The source Snowflake schema tables.
jdbc_url_params (Optional[str]) – Additional properties to pass to the JDBC URL string when connecting to the database formatted as ‘key=value’ pairs separated by the symbol ‘&’. (example: key1=value1&key2=value2&key3=value3).
Airbyte Source for Amplitude
Documentation can be found at https://docs.airbyte.com/integrations/sources/amplitude
name (str) – The name of the destination.
api_key (str) – Amplitude API Key. See the setup guide for more information on how to obtain this key.
secret_key (str) – Amplitude Secret Key. See the setup guide for more information on how to obtain this key.
start_date (str) – UTC date and time in the format 2021-01-25T00:00:00Z. Any data before this date will not be replicated.
Airbyte Source for Posthog
Documentation can be found at https://docs.airbyte.com/integrations/sources/posthog
name (str) – The name of the destination.
start_date (str) – The date from which you’d like to replicate the data. Any data before this date will not be replicated.
api_key (str) – API Key. See the docs for information on how to generate this key.
base_url (Optional[str]) – Base PostHog url. Defaults to PostHog Cloud (https://app.posthog.com).
Airbyte Source for Paypal Transaction
Documentation can be found at https://docs.airbyte.com/integrations/sources/paypal-transactions
name (str) – The name of the destination.
client_id (Optional[str]) – The Client ID of your Paypal developer application.
client_secret (Optional[str]) – The Client Secret of your Paypal developer application.
refresh_token (Optional[str]) – The key to refresh the expired access token.
start_date (str) – Start Date for data extraction in ISO format. Date must be in range from 3 years till 12 hrs before present time.
is_sandbox (bool) – Determines whether to use the sandbox or production environment.
Airbyte Source for Mssql
Documentation can be found at https://docs.airbyte.com/integrations/destinations/mssql
name (str) – The name of the destination.
host (str) – The hostname of the database.
port (int) – The port of the database.
database (str) – The name of the database.
schemas (Optional[List[str]]) – The list of schemas to sync from. Defaults to user. Case sensitive.
username (str) – The username which is used to access the database.
password (Optional[str]) – The password associated with the username.
jdbc_url_params (Optional[str]) – Additional properties to pass to the JDBC URL string when connecting to the database formatted as ‘key=value’ pairs separated by the symbol ‘&’. (example: key1=value1&key2=value2&key3=value3).
ssl_method (Union[MssqlSource.Unencrypted, MssqlSource.EncryptedTrustServerCertificate, MssqlSource.EncryptedVerifyCertificate]) – The encryption method which is used when communicating with the database.
replication_method (Union[MssqlSource.Standard, MssqlSource.LogicalReplicationCDC]) – The replication method used for extracting data from the database. STANDARD replication requires no setup on the DB side but will not be able to represent deletions incrementally. CDC uses {TBC} to detect inserts, updates, and deletes. This needs to be configured on the source database itself.
Airbyte Source for Zoho Crm
Documentation can be found at https://docs.airbyte.com/integrations/sources/zoho-crm
name (str) – The name of the destination.
client_id (str) – OAuth2.0 Client ID
client_secret (str) – OAuth2.0 Client Secret
refresh_token (str) – OAuth2.0 Refresh Token
dc_region (str) – Please choose the region of your Data Center location. More info by this Link
environment (str) – Please choose the environment
start_datetime (Optional[str]) – ISO 8601, for instance: YYYY-MM-DD, YYYY-MM-DD HH:MM:SS+HH:MM
edition (str) – Choose your Edition of Zoho CRM to determine API Concurrency Limits
Airbyte Source for Redshift
Documentation can be found at https://docs.airbyte.com/integrations/destinations/redshift
name (str) – The name of the destination.
host (str) – Host Endpoint of the Redshift Cluster (must include the cluster-id, region and end with .redshift.amazonaws.com).
port (int) – Port of the database.
database (str) – Name of the database.
schemas (Optional[List[str]]) – The list of schemas to sync from. Specify one or more explicitly or keep empty to process all schemas. Schema names are case sensitive.
username (str) – Username to use to access the database.
password (str) – Password associated with the username.
jdbc_url_params (Optional[str]) – Additional properties to pass to the JDBC URL string when connecting to the database formatted as ‘key=value’ pairs separated by the symbol ‘&’. (example: key1=value1&key2=value2&key3=value3).
Airbyte Source for Asana
name (str) – The name of the destination.
credentials (Union[AsanaSource.PATCredentials, AsanaSource.OAuthCredentials]) – Choose how to authenticate to Github
Airbyte Source for Smartsheets
Documentation can be found at https://docs.airbyte.com/integrations/sources/smartsheets
name (str) – The name of the destination.
access_token (str) – The access token to use for accessing your data from Smartsheets. This access token must be generated by a user with at least read access to the data you’d like to replicate. Generate an access token in the Smartsheets main menu by clicking Account > Apps & Integrations > API Access. See the setup guide for information on how to obtain this token.
spreadsheet_id (str) – The spreadsheet ID. Find it by opening the spreadsheet then navigating to File > Properties
start_datetime (Optional[str]) – Only rows modified after this date/time will be replicated. This should be an ISO 8601 string, for instance: 2000-01-01T13:00:00
Airbyte Source for Mailchimp
Documentation can be found at https://docs.airbyte.com/integrations/sources/mailchimp
name (str) – The name of the destination.
Airbyte Source for Sentry
Documentation can be found at https://docs.airbyte.com/integrations/sources/sentry
name (str) – The name of the destination.
auth_token (str) – Log into Sentry and then create authentication tokens.For self-hosted, you can find or create authentication tokens by visiting “{instance_url_prefix}/settings/account/api/auth-tokens/”
hostname (Optional[str]) – Host name of Sentry API server.For self-hosted, specify your host name here. Otherwise, leave it empty.
organization (str) – The slug of the organization the groups belong to.
project (str) – The name (slug) of the Project you want to sync.
discover_fields (Optional[List[str]]) – Fields to retrieve when fetching discover events
Airbyte Source for Mailgun
Documentation can be found at https://docs.airbyte.com/integrations/sources/mailgun
name (str) – The name of the destination.
private_key (str) – Primary account API key to access your Mailgun data.
domain_region (Optional[str]) – Domain region code. ‘EU’ or ‘US’ are possible values. The default is ‘US’.
start_date (Optional[str]) – UTC date and time in the format 2020-10-01 00:00:00. Any data before this date will not be replicated. If omitted, defaults to 3 days ago.
Airbyte Source for Onesignal
Documentation can be found at https://docs.airbyte.com/integrations/sources/onesignal
name (str) – The name of the destination.
user_auth_key (str) – OneSignal User Auth Key, see the docs for more information on how to obtain this key.
start_date (str) – The date from which you’d like to replicate data for OneSignal API, in the format YYYY-MM-DDT00:00:00Z. All data generated after this date will be replicated.
outcome_names (str) – Comma-separated list of names and the value (sum/count) for the returned outcome data. See the docs for more details
Airbyte Source for Python Http Tutorial
Documentation can be found at https://docs.airbyte.com/integrations/sources/exchangeratesapi
name (str) – The name of the destination.
access_key (Optional[str]) – API access key used to retrieve data from the Exchange Rates API.
start_date (str) – UTC date and time in the format 2017-01-25. Any data before this date will not be replicated.
base (str) – ISO reference currency. See here.
Airbyte Source for Airtable
Documentation can be found at https://docs.airbyte.com/integrations/sources/airtable
name (str) – The name of the destination.
api_key (str) – The API Key for the Airtable account. See the Support Guide for more information on how to obtain this key.
base_id (str) – The Base ID to integrate the data from. You can find the Base ID following the link Airtable API, log in to your account, select the base you need and find Base ID in the docs.
tables (List[str]) – The list of Tables to integrate.
Airbyte Source for Mongodb V2
Documentation can be found at https://docs.airbyte.com/integrations/sources/mongodb-v2
name (str) – The name of the destination.
instance_type (Union[MongodbV2Source.StandaloneMongoDbInstance, MongodbV2Source.ReplicaSet, MongodbV2Source.MongoDBAtlas]) – The MongoDb instance to connect to. For MongoDB Atlas and Replica Set TLS connection is used by default.
database (str) – The database you want to replicate.
user (Optional[str]) – The username which is used to access the database.
password (Optional[str]) – The password associated with this username.
auth_source (Optional[str]) – The authentication source where the user information is stored.
Airbyte Source for File Secure
Documentation can be found at https://docs.airbyte.com/integrations/sources/file
name (str) – The name of the destination.
dataset_name (str) – The Name of the final table to replicate this file into (should include letters, numbers dash and underscores only).
format (str) – The Format of the file which should be replicated (Warning: some formats may be experimental, please refer to the docs).
reader_options (Optional[str]) – This should be a string in JSON format. It depends on the chosen file format to provide additional options and tune its behavior.
url (str) – The URL path to access the file which should be replicated.
provider (Union[FileSecureSource.HTTPSPublicWeb, FileSecureSource.GCSGoogleCloudStorage, FileSecureSource.S3AmazonWebServices, FileSecureSource.AzBlobAzureBlobStorage, FileSecureSource.SSHSecureShell, FileSecureSource.SCPSecureCopyProtocol, FileSecureSource.SFTPSecureFileTransferProtocol]) – The storage Provider or Location of the file(s) which should be replicated.
Airbyte Source for Zendesk Support
Documentation can be found at https://docs.airbyte.com/integrations/sources/zendesk-support
name (str) – The name of the destination.
start_date (str) – The date from which you’d like to replicate data for Zendesk Support API, in the format YYYY-MM-DDT00:00:00Z. All data generated after this date will be replicated.
subdomain (str) – This is your Zendesk subdomain that can be found in your account URL. For example, in https://{MY_SUBDOMAIN}.zendesk.com/, where MY_SUBDOMAIN is the value of your subdomain.
credentials (Union[ZendeskSupportSource.OAuth20, ZendeskSupportSource.APIToken]) – Zendesk service provides two authentication methods. Choose between: OAuth2.0 or API token.
Airbyte Source for Tempo
Documentation can be found at https://docs.airbyte.com/integrations/sources/
name (str) – The name of the destination.
api_token (str) – Tempo API Token. Go to Tempo>Settings, scroll down to Data Access and select API integration.
Airbyte Source for Braintree
Documentation can be found at https://docs.airbyte.com/integrations/sources/braintree
name (str) – The name of the destination.
merchant_id (str) – The unique identifier for your entire gateway account. See the docs for more information on how to obtain this ID.
public_key (str) – Braintree Public Key. See the docs for more information on how to obtain this key.
private_key (str) – Braintree Private Key. See the docs for more information on how to obtain this key.
start_date (Optional[str]) – UTC date and time in the format 2017-01-25T00:00:00Z. Any data before this date will not be replicated.
environment (str) – Environment specifies where the data will come from.
Airbyte Source for Salesloft
Documentation can be found at https://docs.airbyte.com/integrations/sources/salesloft
name (str) – The name of the destination.
client_id (str) – The Client ID of your Salesloft developer application.
client_secret (str) – The Client Secret of your Salesloft developer application.
refresh_token (str) – The token for obtaining a new access token.
start_date (str) – The date from which you’d like to replicate data for Salesloft API, in the format YYYY-MM-DDT00:00:00Z. All data generated after this date will be replicated.
Airbyte Source for Linnworks
Documentation can be found at https://docs.airbyte.com/integrations/sources/linnworks
name (str) – The name of the destination.
application_id (str) – Linnworks Application ID
application_secret (str) – Linnworks Application Secret
start_date (str) – UTC date and time in the format 2017-01-25T00:00:00Z. Any data before this date will not be replicated.
Airbyte Source for Chargebee
Documentation can be found at https://apidocs.chargebee.com/docs/api
name (str) – The name of the destination.
site (str) – The site prefix for your Chargebee instance.
site_api_key (str) – Chargebee API Key. See the docs for more information on how to obtain this key.
start_date (str) – UTC date and time in the format 2021-01-25T00:00:00Z. Any data before this date will not be replicated.
product_catalog (str) – Product Catalog version of your Chargebee site. Instructions on how to find your version you may find here under API Version section.
Airbyte Source for Google Analytics Data Api
Documentation can be found at https://docs.airbyte.com/integrations/sources/google-analytics-v4
name (str) – The name of the destination.
property_id (str) – A Google Analytics GA4 property identifier whose events are tracked. Specified in the URL path and not the body
credentials (Union[GoogleAnalyticsDataApiSource.AuthenticateViaGoogleOauth, GoogleAnalyticsDataApiSource.ServiceAccountKeyAuthentication]) – Credentials for the service
date_ranges_start_date (str) – The start date. One of the values Ndaysago, yesterday, today or in the format YYYY-MM-DD
custom_reports (Optional[str]) – A JSON array describing the custom reports you want to sync from Google Analytics. See the docs for more information about the exact format you can use to fill out this field.
window_in_days (Optional[int]) – The time increment used by the connector when requesting data from the Google Analytics API. More information is available in the the docs. The bigger this value is, the faster the sync will be, but the more likely that sampling will be applied to your data, potentially causing inaccuracies in the returned results. We recommend setting this to 1 unless you have a hard requirement to make the sync faster at the expense of accuracy. The minimum allowed value for this field is 1, and the maximum is 364.
Airbyte Source for Outreach
Documentation can be found at https://docs.airbyte.com/integrations/sources/outreach
name (str) – The name of the destination.
client_id (str) – The Client ID of your Outreach developer application.
client_secret (str) – The Client Secret of your Outreach developer application.
refresh_token (str) – The token for obtaining the new access token.
redirect_uri (str) – A Redirect URI is the location where the authorization server sends the user once the app has been successfully authorized and granted an authorization code or access token.
start_date (str) – The date from which you’d like to replicate data for Outreach API, in the format YYYY-MM-DDT00:00:00Z. All data generated after this date will be replicated.
Airbyte Source for Lemlist
Documentation can be found at https://docs.airbyte.com/integrations/sources/lemlist
name (str) – The name of the destination.
api_key (str) – Lemlist API key.
Airbyte Source for Apify Dataset
Documentation can be found at https://docs.airbyte.com/integrations/sources/apify-dataset
name (str) – The name of the destination.
datasetId (str) – ID of the dataset you would like to load to Airbyte.
clean (Optional[bool]) – If set to true, only clean items will be downloaded from the dataset. See description of what clean means in Apify API docs. If not sure, set clean to false.
Airbyte Source for Recurly
Documentation can be found at https://docs.airbyte.com/integrations/sources/recurly
name (str) – The name of the destination.
api_key (str) – Recurly API Key. See the docs for more information on how to generate this key.
begin_time (Optional[str]) – ISO8601 timestamp from which the replication from Recurly API will start from.
end_time (Optional[str]) – ISO8601 timestamp to which the replication from Recurly API will stop. Records after that date won’t be imported.
Airbyte Source for Zendesk Talk
Documentation can be found at https://docs.airbyte.com/integrations/sources/zendesk-talk
name (str) – The name of the destination.
subdomain (str) – This is your Zendesk subdomain that can be found in your account URL. For example, in https://{MY_SUBDOMAIN}.zendesk.com/, where MY_SUBDOMAIN is the value of your subdomain.
credentials (Union[ZendeskTalkSource.APIToken, ZendeskTalkSource.OAuth20]) – Zendesk service provides two authentication methods. Choose between: OAuth2.0 or API token.
start_date (str) – The date from which you’d like to replicate data for Zendesk Talk API, in the format YYYY-MM-DDT00:00:00Z. All data generated after this date will be replicated.
Airbyte Source for Sftp
Documentation can be found at https://docs.airbyte.com/integrations/source/sftp
name (str) – The name of the destination.
user (str) – The server user
host (str) – The server host address
port (int) – The server port
credentials (Union[SftpSource.PasswordAuthentication, SftpSource.SSHKeyAuthentication]) – The server authentication method
file_types (Optional[str]) – Coma separated file types. Currently only ‘csv’ and ‘json’ types are supported.
folder_path (Optional[str]) – The directory to search files for sync
file_pattern (Optional[str]) – The regular expression to specify files for sync in a chosen Folder Path
Airbyte Source for Whisky Hunter
Documentation can be found at https://docs.airbyte.io/integrations/sources/whisky-hunter
name (str) – The name of the destination.
Airbyte Source for Freshdesk
Documentation can be found at https://docs.airbyte.com/integrations/sources/freshdesk
name (str) – The name of the destination.
domain (str) – Freshdesk domain
api_key (str) – Freshdesk API Key. See the docs for more information on how to obtain this key.
requests_per_minute (Optional[int]) – The number of requests per minute that this source allowed to use. There is a rate limit of 50 requests per minute per app per account.
start_date (Optional[str]) – UTC date and time. Any data created after this date will be replicated. If this parameter is not set, all data will be replicated.
Airbyte Source for Gocardless
Documentation can be found at https://docs.airbyte.com/integrations/sources/gocardless
name (str) – The name of the destination.
access_token (str) – Gocardless API TOKEN
gocardless_environment (str) – Environment you are trying to connect to.
gocardless_version (str) – GoCardless version. This is a date. You can find the latest here: https://developer.gocardless.com/api-reference/#api-usage-making-requests
start_date (str) – UTC date and time in the format 2017-01-25T00:00:00Z. Any data before this date will not be replicated.
Airbyte Source for Zuora
Documentation can be found at https://docs.airbyte.com/integrations/sources/zuora
name (str) – The name of the destination.
start_date (str) – Start Date in format: YYYY-MM-DD
window_in_days (Optional[str]) – The amount of days for each data-chunk begining from start_date. Bigger the value - faster the fetch. (0.1 - as for couple of hours, 1 - as for a Day; 364 - as for a Year).
tenant_endpoint (str) – Please choose the right endpoint where your Tenant is located. More info by this Link
data_query (str) – Choose between Live, or Unlimited - the optimized, replicated database at 12 hours freshness for high volume extraction Link
client_id (str) – Your OAuth user Client ID
client_secret (str) – Your OAuth user Client Secret
Airbyte Source for Marketo
Documentation can be found at https://docs.airbyte.com/integrations/sources/marketo
name (str) – The name of the destination.
domain_url (str) – Your Marketo Base URL. See the docs for info on how to obtain this.
client_id (str) – The Client ID of your Marketo developer application. See the docs for info on how to obtain this.
client_secret (str) – The Client Secret of your Marketo developer application. See the docs for info on how to obtain this.
start_date (str) – UTC date and time in the format 2017-01-25T00:00:00Z. Any data before this date will not be replicated.
Airbyte Source for Drift
Documentation can be found at https://docs.airbyte.com/integrations/sources/drift
name (str) – The name of the destination.
Airbyte Source for Pokeapi
Documentation can be found at https://docs.airbyte.com/integrations/sources/pokeapi
name (str) – The name of the destination.
pokemon_name (str) – Pokemon requested from the API.
Airbyte Source for Netsuite
name (str) – The name of the destination.
realm (str) – Netsuite realm e.g. 2344535, as for production or 2344535_SB1, as for the sandbox
consumer_key (str) – Consumer key associated with your integration
consumer_secret (str) – Consumer secret associated with your integration
token_key (str) – Access token key
token_secret (str) – Access token secret
object_types (Optional[List[str]]) – The API names of the Netsuite objects you want to sync. Setting this speeds up the connection setup process by limiting the number of schemas that need to be retrieved from Netsuite.
start_datetime (str) – Starting point for your data replication, in format of “YYYY-MM-DDTHH:mm:ssZ”
window_in_days (Optional[int]) – The amount of days used to query the data with date chunks. Set smaller value, if you have lots of data.
Airbyte Source for Hubplanner
Documentation can be found at https://docs.airbyte.com/integrations/sources/hubplanner
name (str) – The name of the destination.
api_key (str) – Hubplanner API key. See https://github.com/hubplanner/API#authentication for more details.
Airbyte Source for Dv 360
name (str) – The name of the destination.
credentials (Dv360Source.Oauth2Credentials) – Oauth2 credentials
partner_id (int) – Partner ID
start_date (str) – UTC date and time in the format 2017-01-25. Any data before this date will not be replicated
end_date (Optional[str]) – UTC date and time in the format 2017-01-25. Any data after this date will not be replicated.
filters (Optional[List[str]]) – filters for the dimensions. each filter object had 2 keys: ‘type’ for the name of the dimension to be used as. and ‘value’ for the value of the filter
Airbyte Source for Notion
Documentation can be found at https://docs.airbyte.com/integrations/sources/notion
name (str) – The name of the destination.
start_date (str) – UTC date and time in the format 2017-01-25T00:00:00.000Z. Any data before this date will not be replicated.
credentials (Union[NotionSource.OAuth20, NotionSource.AccessToken]) – Pick an authentication method.
Airbyte Source for Zendesk Sunshine
Documentation can be found at https://docs.airbyte.com/integrations/sources/zendesk_sunshine
name (str) – The name of the destination.
subdomain (str) – The subdomain for your Zendesk Account.
start_date (str) – The date from which you’d like to replicate data for Zendesk Sunshine API, in the format YYYY-MM-DDT00:00:00Z.
Airbyte Source for Pinterest
Documentation can be found at https://docs.airbyte.com/integrations/sources/pinterest
name (str) – The name of the destination.
start_date (str) – A date in the format YYYY-MM-DD. If you have not set a date, it would be defaulted to latest allowed date by api (914 days from today).
Airbyte Source for Metabase
Documentation can be found at https://docs.airbyte.com/integrations/sources/metabase
name (str) – The name of the destination.
instance_api_url (str) – URL to your metabase instance API
session_token (Optional[str]) – To generate your session token, you need to run the following command: ` curl -X POST \\ -H "Content-Type: application/json" \\ -d '{"username": "person@metabase.com", "password": "fakepassword"}' \\ http://localhost:3000/api/session ` Then copy the value of the id field returned by a successful call to that API. Note that by default, sessions are good for 14 days and needs to be regenerated.
Airbyte Source for Hubspot
Documentation can be found at https://docs.airbyte.com/integrations/sources/hubspot
name (str) – The name of the destination.
start_date (str) – UTC date and time in the format 2017-01-25T00:00:00Z. Any data before this date will not be replicated.
credentials (Union[HubspotSource.OAuth, HubspotSource.APIKey, HubspotSource.PrivateAPP]) – Choose how to authenticate to HubSpot.
Airbyte Source for Harvest
Documentation can be found at https://docs.airbyte.com/integrations/sources/harvest
name (str) – The name of the destination.
account_id (str) – Harvest account ID. Required for all Harvest requests in pair with Personal Access Token
replication_start_date (str) – UTC date and time in the format 2017-01-25T00:00:00Z. Any data before this date will not be replicated.
credentials (Union[HarvestSource.AuthenticateViaHarvestOAuth, HarvestSource.AuthenticateWithPersonalAccessToken]) – Choose how to authenticate to Harvest.
Airbyte Source for Github
Documentation can be found at https://docs.airbyte.com/integrations/sources/github
name (str) – The name of the destination.
credentials (Union[GithubSource.OAuthCredentials, GithubSource.PATCredentials]) – Choose how to authenticate to GitHub
start_date (str) – The date from which you’d like to replicate data from GitHub in the format YYYY-MM-DDT00:00:00Z. For the streams which support this configuration, only data generated on or after the start date will be replicated. This field doesn’t apply to all streams, see the docs for more info
repository (str) – Space-delimited list of GitHub organizations/repositories, e.g. airbytehq/airbyte for single repository, airbytehq/* for get all repositories from organization and airbytehq/airbyte airbytehq/another-repo for multiple repositories.
branch (Optional[str]) – Space-delimited list of GitHub repository branches to pull commits for, e.g. airbytehq/airbyte/master. If no branches are specified for a repository, the default branch will be pulled.
page_size_for_large_streams (Optional[int]) – The Github connector contains several streams with a large amount of data. The page size of such streams depends on the size of your repository. We recommended that you specify values between 10 and 30.
Airbyte Source for E2e Test
Documentation can be found at https://docs.airbyte.com/integrations/sources/e2e-test
name (str) – The name of the destination.
max_messages (int) – Number of records to emit per stream. Min 1. Max 100 billion.
seed (Optional[int]) – When the seed is unspecified, the current time millis will be used as the seed. Range: [0, 1000000].
message_interval_ms (Optional[int]) – Interval between messages in ms. Min 0 ms. Max 60000 ms (1 minute).
Airbyte Source for Mysql
Documentation can be found at https://docs.airbyte.com/integrations/sources/mysql
name (str) – The name of the destination.
host (str) – The host name of the database.
port (int) – The port to connect to.
database (str) – The database name.
username (str) – The username which is used to access the database.
password (Optional[str]) – The password associated with the username.
jdbc_url_params (Optional[str]) – Additional properties to pass to the JDBC URL string when connecting to the database formatted as ‘key=value’ pairs separated by the symbol ‘&’. (example: key1=value1&key2=value2&key3=value3). For more information read about JDBC URL parameters.
ssl (Optional[bool]) – Encrypt data using SSL.
ssl_mode (Union[MysqlSource.Preferred, MysqlSource.Required, MysqlSource.VerifyCA, MysqlSource.VerifyIdentity]) – SSL connection modes. preferred - Automatically attempt SSL connection. If the MySQL server does not support SSL, continue with a regular connection.required - Always connect with SSL. If the MySQL server doesn`t support SSL, the connection will not be established. Certificate Authority (CA) and Hostname are not verified.verify-ca - Always connect with SSL. Verifies CA, but allows connection even if Hostname does not match.Verify Identity - Always connect with SSL. Verify both CA and Hostname.Read more in the docs.
replication_method (Union[MysqlSource.Standard, MysqlSource.LogicalReplicationCDC]) – Replication method to use for extracting data from the database.
Airbyte Source for My Hours
Documentation can be found at https://docs.airbyte.com/integrations/sources/my-hours
name (str) – The name of the destination.
email (str) – Your My Hours username
password (str) – The password associated to the username
start_date (str) – Start date for collecting time logs
logs_batch_size (Optional[int]) – Pagination size used for retrieving logs in days
Airbyte Source for Kyriba
name (str) – The name of the destination.
domain (str) – Kyriba domain
username (str) – Username to be used in basic auth
password (str) – Password to be used in basic auth
start_date (str) – The date the sync should start from.
end_date (Optional[str]) – The date the sync should end. If let empty the sync will run to the current date.
Airbyte Source for Google Search Console
Documentation can be found at https://docs.airbyte.com/integrations/sources/google-search-console
name (str) – The name of the destination.
site_urls (List[str]) – The URLs of the website property attached to your GSC account. Read more here.
start_date (str) – UTC date in the format 2017-01-25. Any data before this date will not be replicated.
end_date (Optional[str]) – UTC date in the format 2017-01-25. Any data after this date will not be replicated. Must be greater or equal to the start date field.
custom_reports (Optional[str]) – A JSON array describing the custom reports you want to sync from Google Search Console. See the docs for more information about the exact format you can use to fill out this field.
Airbyte Source for Facebook Marketing
Documentation can be found at https://docs.airbyte.com/integrations/sources/facebook-marketing
name (str) – The name of the destination.
account_id (str) – The Facebook Ad account ID to use when pulling data from the Facebook Marketing API.
start_date (str) – The date from which you’d like to replicate data for all incremental streams, in the format YYYY-MM-DDT00:00:00Z. All data generated after this date will be replicated.
end_date (Optional[str]) – The date until which you’d like to replicate data for all incremental streams, in the format YYYY-MM-DDT00:00:00Z. All data generated between start_date and this date will be replicated. Not setting this option will result in always syncing the latest data.
access_token (str) – The value of the access token generated. See the docs for more information
include_deleted (Optional[bool]) – Include data from deleted Campaigns, Ads, and AdSets
fetch_thumbnail_images (Optional[bool]) – In each Ad Creative, fetch the thumbnail_url and store the result in thumbnail_data_url
custom_insights (Optional[List[FacebookMarketingSource.InsightConfig]]) – A list which contains insights entries, each entry must have a name and can contains fields, breakdowns or action_breakdowns)
page_size (Optional[int]) – Page size used when sending requests to Facebook API to specify number of records per page when response has pagination. Most users do not need to set this field unless they specifically need to tune the connector to address specific issues or use cases.
insights_lookback_window (Optional[int]) – The attribution window
max_batch_size (Optional[int]) – Maximum batch size used when sending batch requests to Facebook API. Most users do not need to set this field unless they specifically need to tune the connector to address specific issues or use cases.
Airbyte Source for Surveymonkey
Documentation can be found at https://docs.airbyte.com/integrations/sources/surveymonkey
name (str) – The name of the destination.
access_token (str) – Access Token for making authenticated requests. See the docs for information on how to generate this key.
start_date (str) – UTC date and time in the format 2017-01-25T00:00:00Z. Any data before this date will not be replicated.
survey_ids (Optional[List[str]]) – IDs of the surveys from which you’d like to replicate data. If left empty, data from all boards to which you have access will be replicated.
Airbyte Source for Pardot
name (str) – The name of the destination.
pardot_business_unit_id (str) – Pardot Business ID, can be found at Setup > Pardot > Pardot Account Setup
client_id (str) – The Consumer Key that can be found when viewing your app in Salesforce
client_secret (str) – The Consumer Secret that can be found when viewing your app in Salesforce
refresh_token (str) – Salesforce Refresh Token used for Airbyte to access your Salesforce account. If you don’t know what this is, follow this guide to retrieve it.
start_date (Optional[str]) – UTC date and time in the format 2017-01-25T00:00:00Z. Any data before this date will not be replicated. Leave blank to skip this filter
is_sandbox (Optional[bool]) – Whether or not the the app is in a Salesforce sandbox. If you do not know what this, assume it is false.
Airbyte Source for Flexport
Documentation can be found at https://docs.airbyte.com/integrations/sources/flexport
name (str) – The name of the destination.
Airbyte Source for Kafka
Documentation can be found at https://docs.airbyte.com/integrations/sources/kafka
name (str) – The name of the destination.
MessageFormat (Union[KafkaSource.JSON, KafkaSource.AVRO]) – The serialization used based on this
bootstrap_servers (str) – A list of host/port pairs to use for establishing the initial connection to the Kafka cluster. The client will make use of all servers irrespective of which servers are specified here for bootstrapping—this list only impacts the initial hosts used to discover the full set of servers. This list should be in the form host1:port1,host2:port2,…. Since these servers are just used for the initial connection to discover the full cluster membership (which may change dynamically), this list need not contain the full set of servers (you may want more than one, though, in case a server is down).
subscription (Union[KafkaSource.ManuallyAssignAListOfPartitions, KafkaSource.SubscribeToAllTopicsMatchingSpecifiedPattern]) – You can choose to manually assign a list of partitions, or subscribe to all topics matching specified pattern to get dynamically assigned partitions.
test_topic (Optional[str]) – The Topic to test in case the Airbyte can consume messages.
group_id (Optional[str]) – The Group ID is how you distinguish different consumer groups.
max_poll_records (Optional[int]) – The maximum number of records returned in a single call to poll(). Note, that max_poll_records does not impact the underlying fetching behavior. The consumer will cache the records from each fetch request and returns them incrementally from each poll.
polling_time (Optional[int]) – Amount of time Kafka connector should try to poll for messages.
protocol (Union[KafkaSource.PLAINTEXT, KafkaSource.SASLPLAINTEXT, KafkaSource.SASLSSL]) – The Protocol used to communicate with brokers.
client_id (Optional[str]) – An ID string to pass to the server when making requests. The purpose of this is to be able to track the source of requests beyond just ip/port by allowing a logical application name to be included in server-side request logging.
enable_auto_commit (Optional[bool]) – If true, the consumer’s offset will be periodically committed in the background.
auto_commit_interval_ms (Optional[int]) – The frequency in milliseconds that the consumer offsets are auto-committed to Kafka if enable.auto.commit is set to true.
client_dns_lookup (Optional[str]) – Controls how the client uses DNS lookups. If set to use_all_dns_ips, connect to each returned IP address in sequence until a successful connection is established. After a disconnection, the next IP is used. Once all IPs have been used once, the client resolves the IP(s) from the hostname again. If set to resolve_canonical_bootstrap_servers_only, resolve each bootstrap address into a list of canonical names. After the bootstrap phase, this behaves the same as use_all_dns_ips. If set to default (deprecated), attempt to connect to the first IP address returned by the lookup, even if the lookup returns multiple IP addresses.
retry_backoff_ms (Optional[int]) – The amount of time to wait before attempting to retry a failed request to a given topic partition. This avoids repeatedly sending requests in a tight loop under some failure scenarios.
request_timeout_ms (Optional[int]) – The configuration controls the maximum amount of time the client will wait for the response of a request. If the response is not received before the timeout elapses the client will resend the request if necessary or fail the request if retries are exhausted.
receive_buffer_bytes (Optional[int]) – The size of the TCP receive buffer (SO_RCVBUF) to use when reading data. If the value is -1, the OS default will be used.
auto_offset_reset (Optional[str]) – What to do when there is no initial offset in Kafka or if the current offset does not exist any more on the server - earliest: automatically reset the offset to the earliest offset, latest: automatically reset the offset to the latest offset, none: throw exception to the consumer if no previous offset is found for the consumer’s group, anything else: throw exception to the consumer.
repeated_calls (Optional[int]) – The number of repeated calls to poll() if no messages were received.
max_records_process (Optional[int]) – The Maximum to be processed per execution
Airbyte Destination for Dynamodb
Documentation can be found at https://docs.airbyte.com/integrations/destinations/dynamodb
name (str) – The name of the destination.
dynamodb_endpoint (Optional[str]) – This is your DynamoDB endpoint url.(if you are working with AWS DynamoDB, just leave empty).
dynamodb_table_name_prefix (str) – The prefix to use when naming DynamoDB tables.
dynamodb_region (str) – The region of the DynamoDB.
access_key_id (str) – The access key id to access the DynamoDB. Airbyte requires Read and Write permissions to the DynamoDB.
secret_access_key (str) – The corresponding secret to the access key id.
Airbyte Destination for Bigquery
Documentation can be found at https://docs.airbyte.com/integrations/destinations/bigquery
name (str) – The name of the destination.
project_id (str) – The GCP project ID for the project containing the target BigQuery dataset. Read more here.
dataset_location (str) – The location of the dataset. Warning: Changes made after creation will not be applied. Read more here.
dataset_id (str) – The default BigQuery Dataset ID that tables are replicated to if the source does not specify a namespace. Read more here.
loading_method (Union[BigqueryDestination.StandardInserts, BigqueryDestination.GCSStaging]) – Loading method used to send select the way data will be uploaded to BigQuery. Standard Inserts - Direct uploading using SQL INSERT statements. This method is extremely inefficient and provided only for quick testing. In almost all cases, you should use staging. GCS Staging - Writes large batches of records to a file, uploads the file to GCS, then uses COPY INTO table to upload the file. Recommended for most workloads for better speed and scalability. Read more about GCS Staging here.
credentials_json (Optional[str]) – The contents of the JSON service account key. Check out the docs if you need help generating this key. Default credentials will be used if this field is left empty.
transformation_priority (Optional[str]) – Interactive run type means that the query is executed as soon as possible, and these queries count towards concurrent rate limit and daily limit. Read more about interactive run type here. Batch queries are queued and started as soon as idle resources are available in the BigQuery shared resource pool, which usually occurs within a few minutes. Batch queries don`t count towards your concurrent rate limit. Read more about batch queries here. The default “interactive” value is used if not set explicitly.
big_query_client_buffer_size_mb (Optional[int]) – Google BigQuery client’s chunk (buffer) size (MIN=1, MAX = 15) for each table. The size that will be written by a single RPC. Written data will be buffered and only flushed upon reaching this size or closing the channel. The default 15MB value is used if not set explicitly. Read more here.
Airbyte Destination for Rabbitmq
Documentation can be found at https://docs.airbyte.com/integrations/destinations/rabbitmq
name (str) – The name of the destination.
ssl (Optional[bool]) – SSL enabled.
host (str) – The RabbitMQ host name.
port (Optional[int]) – The RabbitMQ port.
virtual_host (Optional[str]) – The RabbitMQ virtual host name.
username (Optional[str]) – The username to connect.
password (Optional[str]) – The password to connect.
exchange (Optional[str]) – The exchange name.
routing_key (str) – The routing key.
Airbyte Destination for Kvdb
Documentation can be found at https://kvdb.io/docs/api/
name (str) – The name of the destination.
bucket_id (str) – The ID of your KVdb bucket.
secret_key (str) – Your bucket Secret Key.
Airbyte Destination for Clickhouse
Documentation can be found at https://docs.airbyte.com/integrations/destinations/clickhouse
name (str) – The name of the destination.
host (str) – Hostname of the database.
port (int) – HTTP port of the database.
database (str) – Name of the database.
username (str) – Username to use to access the database.
password (Optional[str]) – Password associated with the username.
jdbc_url_params (Optional[str]) – Additional properties to pass to the JDBC URL string when connecting to the database formatted as ‘key=value’ pairs separated by the symbol ‘&’. (example: key1=value1&key2=value2&key3=value3).
ssl (Optional[bool]) – Encrypt data using SSL.
Airbyte Destination for Amazon Sqs
Documentation can be found at https://docs.airbyte.com/integrations/destinations/amazon-sqs
name (str) – The name of the destination.
queue_url (str) – URL of the SQS Queue
region (str) – AWS Region of the SQS Queue
message_delay (Optional[int]) – Modify the Message Delay of the individual message from the Queue’s default (seconds).
access_key (Optional[str]) – The Access Key ID of the AWS IAM Role to use for sending messages
secret_key (Optional[str]) – The Secret Key of the AWS IAM Role to use for sending messages
message_body_key (Optional[str]) – Use this property to extract the contents of the named key in the input record to use as the SQS message body. If not set, the entire content of the input record data is used as the message body.
message_group_id (Optional[str]) – The tag that specifies that a message belongs to a specific message group. This parameter applies only to, and is REQUIRED by, FIFO queues.
Airbyte Destination for Mariadb Columnstore
Documentation can be found at https://docs.airbyte.com/integrations/destinations/mariadb-columnstore
name (str) – The name of the destination.
host (str) – The Hostname of the database.
port (int) – The Port of the database.
database (str) – Name of the database.
username (str) – The Username which is used to access the database.
password (Optional[str]) – The Password associated with the username.
jdbc_url_params (Optional[str]) – Additional properties to pass to the JDBC URL string when connecting to the database formatted as ‘key=value’ pairs separated by the symbol ‘&’. (example: key1=value1&key2=value2&key3=value3).
Airbyte Destination for Kinesis
Documentation can be found at https://docs.airbyte.com/integrations/destinations/kinesis
name (str) – The name of the destination.
endpoint (str) – AWS Kinesis endpoint.
region (str) – AWS region. Your account determines the Regions that are available to you.
shardCount (int) – Number of shards to which the data should be streamed.
accessKey (str) – Generate the AWS Access Key for current user.
privateKey (str) – The AWS Private Key - a string of numbers and letters that are unique for each account, also known as a “recovery phrase”.
bufferSize (int) – Buffer size for storing kinesis records before being batch streamed.
Airbyte Destination for Azure Blob Storage
Documentation can be found at https://docs.airbyte.com/integrations/destinations/azureblobstorage
name (str) – The name of the destination.
azure_blob_storage_endpoint_domain_name (Optional[str]) – This is Azure Blob Storage endpoint domain name. Leave default value (or leave it empty if run container from command line) to use Microsoft native from example.
azure_blob_storage_container_name (Optional[str]) – The name of the Azure blob storage container. If not exists - will be created automatically. May be empty, then will be created automatically airbytecontainer+timestamp
azure_blob_storage_account_name (str) – The account’s name of the Azure Blob Storage.
azure_blob_storage_account_key (str) – The Azure blob storage account key.
azure_blob_storage_output_buffer_size (Optional[int]) – The amount of megabytes to buffer for the output stream to Azure. This will impact memory footprint on workers, but may need adjustment for performance and appropriate block size in Azure.
format (Union[AzureBlobStorageDestination.CSVCommaSeparatedValues, AzureBlobStorageDestination.JSONLinesNewlineDelimitedJSON]) – Output data format
Airbyte Destination for Kafka
Documentation can be found at https://docs.airbyte.com/integrations/destinations/kafka
name (str) – The name of the destination.
bootstrap_servers (str) – A list of host/port pairs to use for establishing the initial connection to the Kafka cluster. The client will make use of all servers irrespective of which servers are specified here for bootstrapping—this list only impacts the initial hosts used to discover the full set of servers. This list should be in the form host1:port1,host2:port2,…. Since these servers are just used for the initial connection to discover the full cluster membership (which may change dynamically), this list need not contain the full set of servers (you may want more than one, though, in case a server is down).
topic_pattern (str) – Topic pattern in which the records will be sent. You can use patterns like ‘{namespace}’ and/or ‘{stream}’ to send the message to a specific topic based on these values. Notice that the topic name will be transformed to a standard naming convention.
test_topic (Optional[str]) – Topic to test if Airbyte can produce messages.
sync_producer (Optional[bool]) – Wait synchronously until the record has been sent to Kafka.
protocol (Union[KafkaDestination.PLAINTEXT, KafkaDestination.SASLPLAINTEXT, KafkaDestination.SASLSSL]) – Protocol used to communicate with brokers.
client_id (Optional[str]) – An ID string to pass to the server when making requests. The purpose of this is to be able to track the source of requests beyond just ip/port by allowing a logical application name to be included in server-side request logging.
acks (str) – The number of acknowledgments the producer requires the leader to have received before considering a request complete. This controls the durability of records that are sent.
enable_idempotence (bool) – When set to ‘true’, the producer will ensure that exactly one copy of each message is written in the stream. If ‘false’, producer retries due to broker failures, etc., may write duplicates of the retried message in the stream.
compression_type (str) – The compression type for all data generated by the producer.
batch_size (int) – The producer will attempt to batch records together into fewer requests whenever multiple records are being sent to the same partition.
linger_ms (str) – The producer groups together any records that arrive in between request transmissions into a single batched request.
max_in_flight_requests_per_connection (int) – The maximum number of unacknowledged requests the client will send on a single connection before blocking. Can be greater than 1, and the maximum value supported with idempotency is 5.
client_dns_lookup (str) – Controls how the client uses DNS lookups. If set to use_all_dns_ips, connect to each returned IP address in sequence until a successful connection is established. After a disconnection, the next IP is used. Once all IPs have been used once, the client resolves the IP(s) from the hostname again. If set to resolve_canonical_bootstrap_servers_only, resolve each bootstrap address into a list of canonical names. After the bootstrap phase, this behaves the same as use_all_dns_ips. If set to default (deprecated), attempt to connect to the first IP address returned by the lookup, even if the lookup returns multiple IP addresses.
buffer_memory (str) – The total bytes of memory the producer can use to buffer records waiting to be sent to the server.
max_request_size (int) – The maximum size of a request in bytes.
retries (int) – Setting a value greater than zero will cause the client to resend any record whose send fails with a potentially transient error.
socket_connection_setup_timeout_ms (str) – The amount of time the client will wait for the socket connection to be established.
socket_connection_setup_timeout_max_ms (str) – The maximum amount of time the client will wait for the socket connection to be established. The connection setup timeout will increase exponentially for each consecutive connection failure up to this maximum.
max_block_ms (str) – The configuration controls how long the KafkaProducer’s send(), partitionsFor(), initTransactions(), sendOffsetsToTransaction(), commitTransaction() and abortTransaction() methods will block.
request_timeout_ms (int) – The configuration controls the maximum amount of time the client will wait for the response of a request. If the response is not received before the timeout elapses the client will resend the request if necessary or fail the request if retries are exhausted.
delivery_timeout_ms (int) – An upper bound on the time to report success or failure after a call to ‘send()’ returns.
send_buffer_bytes (int) – The size of the TCP send buffer (SO_SNDBUF) to use when sending data. If the value is -1, the OS default will be used.
receive_buffer_bytes (int) – The size of the TCP receive buffer (SO_RCVBUF) to use when reading data. If the value is -1, the OS default will be used.
Airbyte Destination for Elasticsearch
Documentation can be found at https://docs.airbyte.com/integrations/destinations/elasticsearch
name (str) – The name of the destination.
endpoint (str) – The full url of the Elasticsearch server
upsert (Optional[bool]) – If a primary key identifier is defined in the source, an upsert will be performed using the primary key value as the elasticsearch doc id. Does not support composite primary keys.
authenticationMethod (Union[ElasticsearchDestination.None\_, ElasticsearchDestination.ApiKeySecret, ElasticsearchDestination.UsernamePassword]) – The type of authentication to be used
Airbyte Destination for Mysql
Documentation can be found at https://docs.airbyte.com/integrations/destinations/mysql
name (str) – The name of the destination.
host (str) – Hostname of the database.
port (int) – Port of the database.
database (str) – Name of the database.
username (str) – Username to use to access the database.
password (Optional[str]) – Password associated with the username.
ssl (Optional[bool]) – Encrypt data using SSL.
jdbc_url_params (Optional[str]) – Additional properties to pass to the JDBC URL string when connecting to the database formatted as ‘key=value’ pairs separated by the symbol ‘&’. (example: key1=value1&key2=value2&key3=value3).
Airbyte Destination for Sftp Json
Documentation can be found at https://docs.airbyte.com/integrations/destinations/sftp-json
name (str) – The name of the destination.
host (str) – Hostname of the SFTP server.
port (Optional[int]) – Port of the SFTP server.
username (str) – Username to use to access the SFTP server.
password (str) – Password associated with the username.
destination_path (str) – Path to the directory where json files will be written.
Airbyte Destination for Gcs
Documentation can be found at https://docs.airbyte.com/integrations/destinations/gcs
name (str) – The name of the destination.
gcs_bucket_name (str) – You can find the bucket name in the App Engine Admin console Application Settings page, under the label Google Cloud Storage Bucket. Read more here.
gcs_bucket_path (str) – GCS Bucket Path string Subdirectory under the above bucket to sync the data into.
gcs_bucket_region (Optional[str]) – Select a Region of the GCS Bucket. Read more here.
credential (GcsDestination.HMACKey) – An HMAC key is a type of credential and can be associated with a service account or a user account in Cloud Storage. Read more here.
format (Union[GcsDestination.AvroApacheAvro, GcsDestination.CSVCommaSeparatedValues, GcsDestination.JSONLinesNewlineDelimitedJSON, GcsDestination.ParquetColumnarStorage]) – Output data format. One of the following formats must be selected - AVRO format, PARQUET format, CSV format, or JSONL format.
Airbyte Destination for Cassandra
Documentation can be found at https://docs.airbyte.com/integrations/destinations/cassandra
name (str) – The name of the destination.
keyspace (str) – Default Cassandra keyspace to create data in.
username (str) – Username to use to access Cassandra.
password (str) – Password associated with Cassandra.
address (str) – Address to connect to.
port (int) – Port of Cassandra.
datacenter (Optional[str]) – Datacenter of the cassandra cluster.
replication (Optional[int]) – Indicates to how many nodes the data should be replicated to.
Airbyte Destination for Firebolt
Documentation can be found at https://docs.airbyte.com/integrations/destinations/firebolt
name (str) – The name of the destination.
username (str) – Firebolt email address you use to login.
password (str) – Firebolt password.
account (Optional[str]) – Firebolt account to login.
host (Optional[str]) – The host name of your Firebolt database.
database (str) – The database to connect to.
engine (Optional[str]) – Engine name or url to connect to.
loading_method (Union[FireboltDestination.SQLInserts, FireboltDestination.ExternalTableViaS3]) – Loading method used to select the way data will be uploaded to Firebolt
Airbyte Destination for Google Sheets
Documentation can be found at https://docs.airbyte.com/integrations/destinations/google-sheets
name (str) – The name of the destination.
spreadsheet_id (str) – The link to your spreadsheet. See this guide for more details.
credentials (GoogleSheetsDestination.AuthenticationViaGoogleOAuth) – Google API Credentials for connecting to Google Sheets and Google Drive APIs
Airbyte Destination for Databricks
Documentation can be found at https://docs.airbyte.com/integrations/destinations/databricks
name (str) – The name of the destination.
accept_terms (bool) – You must agree to the Databricks JDBC Driver Terms & Conditions to use this connector.
databricks_server_hostname (str) – Databricks Cluster Server Hostname.
databricks_http_path (str) – Databricks Cluster HTTP Path.
databricks_port (Optional[str]) – Databricks Cluster Port.
databricks_personal_access_token (str) – Databricks Personal Access Token for making authenticated requests.
database_schema (Optional[str]) – The default schema tables are written to if the source does not specify a namespace. Unless specifically configured, the usual value for this field is “public”.
data_source (Union[DatabricksDestination.AmazonS3, DatabricksDestination.AzureBlobStorage]) – Storage on which the delta lake is built.
purge_staging_data (Optional[bool]) – Default to ‘true’. Switch it to ‘false’ for debugging purpose.
Airbyte Destination for Bigquery Denormalized
Documentation can be found at https://docs.airbyte.com/integrations/destinations/bigquery
name (str) – The name of the destination.
project_id (str) – The GCP project ID for the project containing the target BigQuery dataset. Read more here.
dataset_id (str) – The default BigQuery Dataset ID that tables are replicated to if the source does not specify a namespace. Read more here.
loading_method (Union[BigqueryDenormalizedDestination.StandardInserts, BigqueryDenormalizedDestination.GCSStaging]) – Loading method used to send select the way data will be uploaded to BigQuery. Standard Inserts - Direct uploading using SQL INSERT statements. This method is extremely inefficient and provided only for quick testing. In almost all cases, you should use staging. GCS Staging - Writes large batches of records to a file, uploads the file to GCS, then uses COPY INTO table to upload the file. Recommended for most workloads for better speed and scalability. Read more about GCS Staging here.
credentials_json (Optional[str]) – The contents of the JSON service account key. Check out the docs if you need help generating this key. Default credentials will be used if this field is left empty.
dataset_location (Optional[str]) – The location of the dataset. Warning: Changes made after creation will not be applied. The default “US” value is used if not set explicitly. Read more here.
big_query_client_buffer_size_mb (Optional[int]) – Google BigQuery client’s chunk (buffer) size (MIN=1, MAX = 15) for each table. The size that will be written by a single RPC. Written data will be buffered and only flushed upon reaching this size or closing the channel. The default 15MB value is used if not set explicitly. Read more here.
Airbyte Destination for Sqlite
Documentation can be found at https://docs.airbyte.com/integrations/destinations/sqlite
name (str) – The name of the destination.
destination_path (str) – Path to the sqlite.db file. The file will be placed inside that local mount. For more information check out our docs
Airbyte Destination for Mongodb
Documentation can be found at https://docs.airbyte.com/integrations/destinations/mongodb
name (str) – The name of the destination.
instance_type (Union[MongodbDestination.StandaloneMongoDbInstance, MongodbDestination.ReplicaSet, MongodbDestination.MongoDBAtlas]) – MongoDb instance to connect to. For MongoDB Atlas and Replica Set TLS connection is used by default.
database (str) – Name of the database.
auth_type (Union[MongodbDestination.None\_, MongodbDestination.LoginPassword]) – Authorization type.
Airbyte Destination for Rockset
Documentation can be found at https://docs.airbyte.com/integrations/destinations/rockset
name (str) – The name of the destination.
api_key (str) – Rockset api key
workspace (str) – The Rockset workspace in which collections will be created + written to.
api_server (Optional[str]) – Rockset api URL
Airbyte Destination for Oracle
Documentation can be found at https://docs.airbyte.com/integrations/destinations/oracle
name (str) – The name of the destination.
host (str) – The hostname of the database.
port (int) – The port of the database.
sid (str) – The System Identifier uniquely distinguishes the instance from any other instance on the same computer.
username (str) – The username to access the database. This user must have CREATE USER privileges in the database.
password (Optional[str]) – The password associated with the username.
jdbc_url_params (Optional[str]) – Additional properties to pass to the JDBC URL string when connecting to the database formatted as ‘key=value’ pairs separated by the symbol ‘&’. (example: key1=value1&key2=value2&key3=value3).
schema (Optional[str]) – The default schema is used as the target schema for all statements issued from the connection that do not explicitly specify a schema name. The usual value for this field is “airbyte”. In Oracle, schemas and users are the same thing, so the “user” parameter is used as the login credentials and this is used for the default Airbyte message schema.
encryption (Union[OracleDestination.Unencrypted, OracleDestination.NativeNetworkEncryptionNNE, OracleDestination.TLSEncryptedVerifyCertificate]) – The encryption method which is used when communicating with the database.
Airbyte Destination for Csv
Documentation can be found at https://docs.airbyte.com/integrations/destinations/local-csv
name (str) – The name of the destination.
destination_path (str) – Path to the directory where csv files will be written. The destination uses the local mount “/local” and any data files will be placed inside that local mount. For more information check out our docs
Airbyte Destination for S3
Documentation can be found at https://docs.airbyte.com/integrations/destinations/s3
name (str) – The name of the destination.
access_key_id (Optional[str]) – The access key ID to access the S3 bucket. Airbyte requires Read and Write permissions to the given bucket. Read more here.
secret_access_key (Optional[str]) – The corresponding secret to the access key ID. Read more here
s3_bucket_name (str) – The name of the S3 bucket. Read more here.
s3_bucket_path (str) – Directory under the S3 bucket where data will be written. Read more here
s3_bucket_region (str) – The region of the S3 bucket. See here for all region codes.
format (Union[S3Destination.AvroApacheAvro, S3Destination.CSVCommaSeparatedValues, S3Destination.JSONLinesNewlineDelimitedJSON, S3Destination.ParquetColumnarStorage]) – Format of the data output. See here for more details
s3_endpoint (Optional[str]) – Your S3 endpoint url. Read more here
s3_path_format (Optional[str]) – Format string on how data will be organized inside the S3 bucket directory. Read more here
file_name_pattern (Optional[str]) – The pattern allows you to set the file-name format for the S3 staging file(s)
Airbyte Destination for Aws Datalake
Documentation can be found at https://docs.airbyte.com/integrations/destinations/aws-datalake
name (str) – The name of the destination.
aws_account_id (Optional[str]) – target aws account id
region (str) – Region name
credentials (Union[AwsDatalakeDestination.IAMRole, AwsDatalakeDestination.IAMUser]) – Choose How to Authenticate to AWS.
bucket_name (str) – Name of the bucket
bucket_prefix (str) – S3 prefix
lakeformation_database_name (Optional[str]) – Which database to use
Airbyte Destination for Mssql
Documentation can be found at https://docs.airbyte.com/integrations/destinations/mssql
name (str) – The name of the destination.
host (str) – The host name of the MSSQL database.
port (int) – The port of the MSSQL database.
database (str) – The name of the MSSQL database.
schema (str) – The default schema tables are written to if the source does not specify a namespace. The usual value for this field is “public”.
username (str) – The username which is used to access the database.
password (Optional[str]) – The password associated with this username.
jdbc_url_params (Optional[str]) – Additional properties to pass to the JDBC URL string when connecting to the database formatted as ‘key=value’ pairs separated by the symbol ‘&’. (example: key1=value1&key2=value2&key3=value3).
ssl_method (Union[MssqlDestination.Unencrypted, MssqlDestination.EncryptedTrustServerCertificate, MssqlDestination.EncryptedVerifyCertificate]) – The encryption method which is used to communicate with the database.
Airbyte Destination for Pubsub
Documentation can be found at https://docs.airbyte.com/integrations/destinations/pubsub
name (str) – The name of the destination.
project_id (str) – The GCP project ID for the project containing the target PubSub.
topic_id (str) – The PubSub topic ID in the given GCP project ID.
credentials_json (str) – The contents of the JSON service account key. Check out the docs if you need help generating this key.
Airbyte Destination for R2
Documentation can be found at https://docs.airbyte.com/integrations/destinations/r2
name (str) – The name of the destination.
account_id (str) – Cloudflare account ID
access_key_id (str) – The access key ID to access the R2 bucket. Airbyte requires Read and Write permissions to the given bucket. Read more here.
secret_access_key (str) – The corresponding secret to the access key ID. Read more here
s3_bucket_name (str) – The name of the R2 bucket. Read more here.
s3_bucket_path (str) – Directory under the R2 bucket where data will be written.
format (Union[R2Destination.AvroApacheAvro, R2Destination.CSVCommaSeparatedValues, R2Destination.JSONLinesNewlineDelimitedJSON]) – Format of the data output. See here for more details
s3_path_format (Optional[str]) – Format string on how data will be organized inside the R2 bucket directory. Read more here
file_name_pattern (Optional[str]) – The pattern allows you to set the file-name format for the R2 staging file(s)
Airbyte Destination for Jdbc
Documentation can be found at https://docs.airbyte.com/integrations/destinations/postgres
name (str) – The name of the destination.
username (str) – The username which is used to access the database.
password (Optional[str]) – The password associated with this username.
jdbc_url (str) – JDBC formatted url. See the standard here.
schema (Optional[str]) – If you leave the schema unspecified, JDBC defaults to a schema named “public”.
Airbyte Destination for Keen
Documentation can be found at https://docs.airbyte.com/integrations/destinations/keen
name (str) – The name of the destination.
project_id (str) – To get Keen Project ID, navigate to the Access tab from the left-hand, side panel and check the Project Details section.
api_key (str) – To get Keen Master API Key, navigate to the Access tab from the left-hand, side panel and check the Project Details section.
infer_timestamp (Optional[bool]) – Allow connector to guess keen.timestamp value based on the streamed data.
Airbyte Destination for Tidb
Documentation can be found at https://docs.airbyte.com/integrations/destinations/tidb
name (str) – The name of the destination.
host (str) – Hostname of the database.
port (int) – Port of the database.
database (str) – Name of the database.
username (str) – Username to use to access the database.
password (Optional[str]) – Password associated with the username.
ssl (Optional[bool]) – Encrypt data using SSL.
jdbc_url_params (Optional[str]) – Additional properties to pass to the JDBC URL string when connecting to the database formatted as ‘key=value’ pairs separated by the symbol ‘&’. (example: key1=value1&key2=value2&key3=value3).
Airbyte Destination for Firestore
Documentation can be found at https://docs.airbyte.com/integrations/destinations/firestore
name (str) – The name of the destination.
project_id (str) – The GCP project ID for the project containing the target BigQuery dataset.
credentials_json (Optional[str]) – The contents of the JSON service account key. Check out the docs if you need help generating this key. Default credentials will be used if this field is left empty.
Airbyte Destination for Scylla
Documentation can be found at https://docs.airbyte.com/integrations/destinations/scylla
name (str) – The name of the destination.
keyspace (str) – Default Scylla keyspace to create data in.
username (str) – Username to use to access Scylla.
password (str) – Password associated with Scylla.
address (str) – Address to connect to.
port (int) – Port of Scylla.
replication (Optional[int]) – Indicates to how many nodes the data should be replicated to.
Airbyte Destination for Redis
Documentation can be found at https://docs.airbyte.com/integrations/destinations/redis
name (str) – The name of the destination.
host (str) – Redis host to connect to.
port (int) – Port of Redis.
username (str) – Username associated with Redis.
password (str) – Password associated with Redis.
cache_type (str) – Redis cache type to store data in.
Airbyte Destination for Mqtt
Documentation can be found at https://docs.airbyte.com/integrations/destinations/mqtt
name (str) – The name of the destination.
broker_host (str) – Host of the broker to connect to.
broker_port (int) – Port of the broker.
use_tls (bool) – Whether to use TLS encryption on the connection.
username (Optional[str]) – User name to use for the connection.
password (Optional[str]) – Password to use for the connection.
topic_pattern (str) – Topic pattern in which the records will be sent. You can use patterns like ‘{namespace}’ and/or ‘{stream}’ to send the message to a specific topic based on these values. Notice that the topic name will be transformed to a standard naming convention.
topic_test (Optional[str]) – Topic to test if Airbyte can produce messages.
client (Optional[str]) – A client identifier that is unique on the server being connected to.
publisher_sync (bool) – Wait synchronously until the record has been sent to the broker.
connect_timeout (int) – Maximum time interval (in seconds) the client will wait for the network connection to the MQTT server to be established.
automatic_reconnect (bool) – Whether the client will automatically attempt to reconnect to the server if the connection is lost.
clean_session (bool) – Whether the client and server should remember state across restarts and reconnects.
message_retained (bool) – Whether or not the publish message should be retained by the messaging engine.
message_qos (str) – Quality of service used for each message to be delivered.
Airbyte Destination for Redshift
Documentation can be found at https://docs.airbyte.com/integrations/destinations/redshift
name (str) – The name of the destination.
host (str) – Host Endpoint of the Redshift Cluster (must include the cluster-id, region and end with .redshift.amazonaws.com)
port (int) – Port of the database.
username (str) – Username to use to access the database.
password (str) – Password associated with the username.
database (str) – Name of the database.
schema (str) – The default schema tables are written to if the source does not specify a namespace. Unless specifically configured, the usual value for this field is “public”.
jdbc_url_params (Optional[str]) – Additional properties to pass to the JDBC URL string when connecting to the database formatted as ‘key=value’ pairs separated by the symbol ‘&’. (example: key1=value1&key2=value2&key3=value3).
uploading_method (Union[RedshiftDestination.Standard, RedshiftDestination.S3Staging]) – The method how the data will be uploaded to the database.
Airbyte Destination for Pulsar
Documentation can be found at https://docs.airbyte.com/integrations/destinations/pulsar
name (str) – The name of the destination.
brokers (str) – A list of host/port pairs to use for establishing the initial connection to the Pulsar cluster.
use_tls (bool) – Whether to use TLS encryption on the connection.
topic_type (str) – It identifies type of topic. Pulsar supports two kind of topics: persistent and non-persistent. In persistent topic, all messages are durably persisted on disk (that means on multiple disks unless the broker is standalone), whereas non-persistent topic does not persist message into storage disk.
topic_tenant (str) – The topic tenant within the instance. Tenants are essential to multi-tenancy in Pulsar, and spread across clusters.
topic_namespace (str) – The administrative unit of the topic, which acts as a grouping mechanism for related topics. Most topic configuration is performed at the namespace level. Each tenant has one or multiple namespaces.
topic_pattern (str) – Topic pattern in which the records will be sent. You can use patterns like ‘{namespace}’ and/or ‘{stream}’ to send the message to a specific topic based on these values. Notice that the topic name will be transformed to a standard naming convention.
topic_test (Optional[str]) – Topic to test if Airbyte can produce messages.
producer_name (Optional[str]) – Name for the producer. If not filled, the system will generate a globally unique name which can be accessed with.
producer_sync (Optional[bool]) – Wait synchronously until the record has been sent to Pulsar.
compression_type (str) – Compression type for the producer.
send_timeout_ms (int) – If a message is not acknowledged by a server before the send-timeout expires, an error occurs (in ms).
max_pending_messages (int) – The maximum size of a queue holding pending messages.
max_pending_messages_across_partitions (int) – The maximum number of pending messages across partitions.
batching_enabled (bool) – Control whether automatic batching of messages is enabled for the producer.
batching_max_messages (int) – Maximum number of messages permitted in a batch.
batching_max_publish_delay (int) – Time period in milliseconds within which the messages sent will be batched.
block_if_queue_full (bool) – If the send operation should block when the outgoing message queue is full.
Airbyte Destination for Snowflake
Documentation can be found at https://docs.airbyte.com/integrations/destinations/snowflake
name (str) – The name of the destination.
host (str) – Enter your Snowflake account’s locator (in the format …snowflakecomputing.com)
role (str) – Enter the role that you want to use to access Snowflake
warehouse (str) – Enter the name of the warehouse that you want to sync data into
database (str) – Enter the name of the database you want to sync data into
schema (str) – Enter the name of the default schema
username (str) – Enter the name of the user you want to use to access the database
jdbc_url_params (Optional[str]) – Enter the additional properties to pass to the JDBC URL string when connecting to the database (formatted as key=value pairs separated by the symbol &). Example: key1=value1&key2=value2&key3=value3
loading_method (Union[SnowflakeDestination.SelectAnotherOption, SnowflakeDestination.RecommendedInternalStaging, SnowflakeDestination.AWSS3Staging, SnowflakeDestination.GoogleCloudStorageStaging, SnowflakeDestination.AzureBlobStorageStaging]) – Select a data staging method
Airbyte Destination for Postgres
Documentation can be found at https://docs.airbyte.com/integrations/destinations/postgres
name (str) – The name of the destination.
host (str) – Hostname of the database.
port (int) – Port of the database.
database (str) – Name of the database.
schema (str) – The default schema tables are written to if the source does not specify a namespace. The usual value for this field is “public”.
username (str) – Username to use to access the database.
password (Optional[str]) – Password associated with the username.
ssl (Optional[bool]) – Encrypt data using SSL. When activating SSL, please select one of the connection modes.
ssl_mode (Union[PostgresDestination.Disable, PostgresDestination.Allow, PostgresDestination.Prefer, PostgresDestination.Require, PostgresDestination.VerifyCa, PostgresDestination.VerifyFull]) – SSL connection modes. disable - Chose this mode to disable encryption of communication between Airbyte and destination database allow - Chose this mode to enable encryption only when required by the source database prefer - Chose this mode to allow unencrypted connection only if the source database does not support encryption require - Chose this mode to always require encryption. If the source database server does not support encryption, connection will fail verify-ca - Chose this mode to always require encryption and to verify that the source database server has a valid SSL certificate verify-full - This is the most secure mode. Chose this mode to always require encryption and to verify the identity of the source database server See more information - in the docs.
jdbc_url_params (Optional[str]) – Additional properties to pass to the JDBC URL string when connecting to the database formatted as ‘key=value’ pairs separated by the symbol ‘&’. (example: key1=value1&key2=value2&key3=value3).
Airbyte Destination for Scaffold Destination Python
Documentation can be found at https://docs.airbyte.com/integrations/destinations/scaffold-destination-python
name (str) – The name of the destination.
TODO (Optional[str]) – FIX ME
Airbyte Destination for Local Json
Documentation can be found at https://docs.airbyte.com/integrations/destinations/local-json
name (str) – The name of the destination.
destination_path (str) – Path to the directory where json files will be written. The files will be placed inside that local mount. For more information check out our docs
Airbyte Destination for Meilisearch
Documentation can be found at https://docs.airbyte.com/integrations/destinations/meilisearch
name (str) – The name of the destination.
host (str) – Hostname of the MeiliSearch instance.
api_key (Optional[str]) – MeiliSearch API Key. See the docs for more information on how to obtain this key.