Connectors and Data Sync

Use the connector command group to manage external connectors, upload and preview connector files, browse relational and structured source schemas, check data source capabilities, read Kafka Topic samples, and operate import and export tasks. Complete Configuration and Authentication before running these commands.

Manage connectors and upload files

List existing connectors or validate a new connector configuration before creating sync tasks, and upload local files when staging file-based imports:

moi-cli -auth pat -o connector list
moi-cli -auth pat -o connector get --connector-id "$CONNECTOR_ID"
moi-cli -auth pat -o connector local-upload --file-path ./customers.csv

Common connector and file management subcommands:

Subcommand and key flags

Description

connector list / connector get --connector-id <id>

List workspace connectors or inspect a single connector configuration.

connector validate --name <name> --source-type <type> --usage-type <usage> --config-file <path>

Validate connectivity and credentials without saving a connector.

connector create --name <name> --source-type <type> --usage-type <usage> --config-file <path>

Create an external connector; maintain it later with update --connector-id <id> or delete --connector-id <id>.

connector probe-rest-draft --config-file <path> / probe-rest --connector-id <id> / preview-rest --connector-id <id> --json-file <path>

Check REST API connector reachability and preview response payloads.

connector local-upload --file-path <path>

Upload local files to connector staging storage (accepts repeated --file-path flags).

connector upload --volume-id <vol-id> --file-path <path>

Upload local files directly for a target volume import (accepts repeated --file-path flags).

connector files / preview-file / file-download --conn-file-id <id> / file-stream --conn-file-id <id> --output <path> / file-delete --conn-file-id <id>

List, preview, download, stream, or delete connector files.

Browse structured sources and generate column mappings

connector exposes two separate source-browsing command families backed by distinct server protocols whose request and response shapes are independent:

Category

Subcommands and key flags

Protocol and response shape

Database browsing (db-*)

db-list-databases --connector-id <id>, db-list-schemas --connector-id <id> --database <db>, db-list-tables --connector-id <id> --database <db>, db-source-schema --connector-id <id> --database <db> --table <tbl>

Calls /connectors/db: database, schema, and table listings return string arrays, and schema inspection returns columns.

Structured source browsing (structured-*)

structured-sources, structured-source-databases --connector-id <id> --source-type <type>, structured-source-schemas --connector-id <id> --source-type <type> --database <db>, structured-source-objects --json-file <path>, structured-source-schema --json-file <path>

Calls /connectors/structured/v1: uses source_type enums and returns structured object descriptors and field snapshots.

Backfill start resolution

structured-backfill-start-resolve --json-file <path>

Resolves the backfill starting offset/position; confirm a valid response before creating a backfill task.

Target checks & column mapping

mo-database-list --connector-id <id>, mo-database-table-list --connector-id <id> --database <db>, mo-check-table-name --connector-id <id> --database <db> --table <tbl>, mo-column-mapping --connector-id <id> --database <db> --table <tbl> --json-file <path>

Check target MatrixOne databases/tables and generate or validate column mappings from a JSON payload.

The typed CLI output of structured-source-schema omits mapping_draft, target_type_catalog, field_compatibilities, mapping_hash, and flatten_hash; when you need full draft mappings and hashes, call the HTTP API Query the connector table structure.

Check data source capabilities and read samples

For MySQL or Doris connectors, run structured-source-capability with SOURCE_TYPE_MYSQL or SOURCE_TYPE_DORIS (pass --schema when the source requires a schema name):

moi-cli -auth pat -o connector structured-source-capability \
  --workspace-id "$WORKSPACE_ID" \
  --connector-id "$CONNECTOR_ID" \
  --source-type "$SOURCE_TYPE" \
  --database "$SOURCE_DATABASE" \
  --table "$SOURCE_TABLE"

For Kafka connectors, run structured-source-sample with --source-type SOURCE_TYPE_KAFKA and --limit between 1 and 100 to inspect live records from a readable Kafka Topic:

moi-cli -auth pat -o connector structured-source-sample \
  --workspace-id "$WORKSPACE_ID" \
  --connector-id "$CONNECTOR_ID" \
  --source-type SOURCE_TYPE_KAFKA \
  --limit 5 \
  --topic-mode single \
  --topics-json '["orders"]'

Sampling reads source records to verify key/value formats and field names without writing rows to any target table.

Create and manage import and export tasks

Pass complex import and export task definitions via --json-file (inspect wire fields offline first with moi-cli schema connector create-task or moi-cli schema connector create-export-task):

Direction

Subcommands and key flags

Description

Import creation & lookup

connector tasks / get-task --task-id <id> / create-task --json-file <path>

List workspace import tasks, inspect task details, or create an import task.

Import lifecycle

connector update-task --task-id <id> --json-file <path> / update-task-from-spec --task-id <id> --json-file <path> / pause-task --task-id <id> / resume-task --task-id <id> / delete-task --task-id <id>

Update configuration, pause, resume, or delete an import task.

Import file retries

connector task-files / retry-task-files --task-id <id> --ids <file-ids>

List import task files and retry failed file IDs.

Export lifecycle

connector export-tasks / create-export-task --json-file <path> / export-task-info --task-id <id> / export-task-state --task-id <id> / export-task-files --task-id <id> / retry-export-task --task-id <id> --file-ids <ids> / delete-export-task --task-id <id>

Create export tasks, track export state and file records, retry failed files, or delete export task records.

Manual runs and statistics

For an enabled scheduled structured import task that is in the created or uploading state and already has a server workflow job, trigger an immediate one-off execution and inspect its run history (run-task-now supports Kafka, Doris, REST API, and MySQL tasks without standalone compute configurations):

moi-cli -auth pat -o connector run-task-now --workspace-id "$WORKSPACE_ID" --task-id "$TASK_ID"
moi-cli -auth pat -o connector task-runs --workspace-id "$WORKSPACE_ID" --task-id "$TASK_ID"

A successful run-task-now response confirms that the server accepted the trigger; verify actual completion, error messages, and target rows via task-runs.

Warning

connector run-task-full-refresh --workspace-id "$WORKSPACE_ID" --task-id "$TASK_ID" applies only to enabled scheduled incremental Doris tasks or MySQL tasks (without standalone compute configurations) in that ready state. It executes a single full_refresh run without modifying the task’s saved incremental schedule. Confirm how target table data is written beforehand, and check task-runs before retrying if the outcome is unknown.

Use connector task-stats and connector export-task-stats for workspace-level aggregate counts; always check task-runs or export-task-state to verify an individual task execution.

Limitations and notes

  • In 0.1.0 offline packages, db-list-tables and db-source-schema do not expose --schema; obtain a deployment-matched CLI package from your deployment administrator or call the corresponding HTTP endpoint when schema filtering on /connectors/db is needed.

  • After a network timeout on task creation, retry, or resume, check connector tasks, get-task, or export-tasks first to confirm whether the task is already created or running before re-submitting.

  • connector delete-export-task deletes only the task record in MatrixOne Intelligence and does not roll back data already written to external destination systems.

Last updated on