Skip to main content

How to configure Conduit

Conduit accepts CLI flags, environment variables and a configuration file to configure its behavior. Each CLI flag has a corresponding environment variable and a corresponding field in the configuration file. Conduit uses the value for each configuration option based on the following priorities:

CLI flags​

CLI flags (highest priority) - if a CLI flag is provided it will always be respected, regardless of the environment variable or configuration file. To see a full list of available flags run conduit run --help:

Starts the Conduit server and runs the configured pipelines.

Usage:
  conduit run [flags]

Flags:
      --api.enabled                                            enable HTTP and gRPC API (default true)
      --api.grpc.address string                                address for serving the gRPC API (default ":8084")
      --api.http.address string                                address for serving the HTTP API (default ":8080")
      --config.path string                                     global conduit configuration file (default "/Users/username/repo/conduit/conduit.yaml")
      --connectors.path string                                 path to standalone connectors' directory (default "/Users/username/repo/conduit/connectors")
      --db.badger.path string                                  path to badger DB (default "/Users/username/repo/conduit/conduit.db")
      --db.postgres.connection-string string                   postgres connection string, may be a database URL or in PostgreSQL keyword/value format
      --db.postgres.table string                               postgres table in which to store data (will be created if it does not exist) (default "conduit_kv_store")
      --db.sqlite.path string                                  path to sqlite3 DB (default "/Users/username/repo/conduit/conduit.db")
      --db.sqlite.table string                                 sqlite3 table in which to store data (will be created if it does not exist) (default "conduit_kv_store")
      --db.type string                                         database type; accepts badger,postgres,inmemory,sqlite (default "badger")
      --dev.blockprofile string                                write block profile to file
      --dev.cpuprofile string                                  write CPU profile to file
      --dev.memprofile string                                  write memory profile to file
  -h, --help                                                   help for run
      --log.format string                                      sets the format of the logging; accepts json, cli (default "cli")
      --log.level string                                       sets logging level; accepts debug, info, warn, error, trace (default "info")
      --pipelines.error-recovery.backoff-factor int            backoff factor applied to the last delay (default 2)
      --pipelines.error-recovery.max-delay duration            maximum delay before restart (default 10m0s)
      --pipelines.error-recovery.max-retries int               maximum number of retries (default -1)
      --pipelines.error-recovery.max-retries-window duration   amount of time running without any errors after which a pipeline is considered healthy (default 5m0s)
      --pipelines.error-recovery.min-delay duration            minimum delay before restart (default 1s)
      --pipelines.exit-on-degraded                             exit Conduit if a pipeline is degraded
      --pipelines.path string                                  path to pipelines' directory (default "/Users/username/repo/conduit/pipelines")
      --preview.pipeline-arch-v2                               enables experimental pipeline architecture v2 (note that the new architecture currently supports only 1 source and 1 destination per pipeline)
      --processors.path string                                 path to standalone processors' directory (default "/Users/username/repo/conduit/processors")
      --schema-registry.confluent.connection-string string     confluent schema registry connection string
      --schema-registry.type string                            schema registry type; accepts builtin,confluent (default "builtin")

To know the current configurations that conduit will use when you run it, run conduit config:

./conduit config
config.path: /Users/username/repo/conduit/conduit.yaml
db.type: badger
db.badger.path: /Users/username/repo/conduit/conduit.db
db.postgres.table: conduit_kv_store
db.sqlite.path: /Users/username/repo/conduit/conduit.db
db.sqlite.table: conduit_kv_store
api.enabled: true
api.http.address: :8080
api.grpc.address: :8084
log.level: info
log.format: cli
connectors.path: /Users/username/repo/conduit/connectors
processors.path: /Users/username/repo/conduit/processors
pipelines.path: /Users/username/repo/conduit/pipelines
pipelines.exit-on-degraded: false
pipelines.error-recovery.min-delay: 1s
pipelines.error-recovery.max-delay: 10m0s
pipelines.error-recovery.backoff-factor: 2
pipelines.error-recovery.max-retries: -1
pipelines.error-recovery.max-retries-window: 5m0s
schema-registry.type: builtin
preview.pipeline-arch-v2: false

Environment variables​

Environment variables (lower priority) - an environment variable is only used if no CLI flag is provided for the same option. Environment variables have the prefix CONDUIT and contain underscores instead of dots and hyphens (e.g. the flag --db.postgres.connection-string corresponds to CONDUIT_DB_POSTGRES_CONNECTION_STRING).

Configuration file​

Configuration file (lowest priority) - Conduit by default loads the file conduit.yaml placed in the same folder as Conduit. The path to the file can be customized using the CLI flag --config.path. It is not required to provide a configuration file and any value in the configuration file can be overridden by an environment variable or a flag. The file content should be a YAML document where keys can be hierarchically split on .. For example:

db:
  type: postgres # corresponds to flag -db.type and env variable CONDUIT_DB_TYPE
  postgres:
    connection-string: postgres://localhost:5432/conduitdb # -db.postgres.connection-string or CONDUIT_DB_POSTGRES_CONNECTION_STRING

scarf pixel conduit-site-docs-using