Connectors and Data Sync¶
Use the connector command group to manage external connectors, upload and preview connector files, browse relational and structured source schemas, check data source capabilities, read Kafka Topic samples, and operate import and export tasks. Complete Configuration and Authentication before running these commands.
Manage connectors and upload files¶
List existing connectors or validate a new connector configuration before creating sync tasks, and upload local files when staging file-based imports:
moi-cli -auth pat -o connector list
moi-cli -auth pat -o connector get --connector-id "$CONNECTOR_ID"
moi-cli -auth pat -o connector local-upload --file-path ./customers.csv
Common connector and file management subcommands:
Subcommand and key flags |
Description |
|---|---|
|
List workspace connectors or inspect a single connector configuration. |
|
Validate connectivity and credentials without saving a connector. |
|
Create an external connector; maintain it later with |
|
Check REST API connector reachability and preview response payloads. |
|
Upload local files to connector staging storage (accepts repeated |
|
Upload local files directly for a target volume import (accepts repeated |
|
List, preview, download, stream, or delete connector files. |
Browse structured sources and generate column mappings¶
connector exposes two separate source-browsing command families backed by distinct server protocols whose request and response shapes are independent:
Category |
Subcommands and key flags |
Protocol and response shape |
|---|---|---|
Database browsing ( |
|
Calls |
Structured source browsing ( |
|
Calls |
Backfill start resolution |
|
Resolves the backfill starting offset/position; confirm a valid response before creating a backfill task. |
Target checks & column mapping |
|
Check target MatrixOne databases/tables and generate or validate column mappings from a JSON payload. |
The typed CLI output of structured-source-schema omits mapping_draft, target_type_catalog, field_compatibilities, mapping_hash, and flatten_hash; when you need full draft mappings and hashes, call the HTTP API Query the connector table structure.
Check data source capabilities and read samples¶
For MySQL or Doris connectors, run structured-source-capability with SOURCE_TYPE_MYSQL or SOURCE_TYPE_DORIS (pass --schema when the source requires a schema name):
moi-cli -auth pat -o connector structured-source-capability \
--workspace-id "$WORKSPACE_ID" \
--connector-id "$CONNECTOR_ID" \
--source-type "$SOURCE_TYPE" \
--database "$SOURCE_DATABASE" \
--table "$SOURCE_TABLE"
For Kafka connectors, run structured-source-sample with --source-type SOURCE_TYPE_KAFKA and --limit between 1 and 100 to inspect live records from a readable Kafka Topic:
moi-cli -auth pat -o connector structured-source-sample \
--workspace-id "$WORKSPACE_ID" \
--connector-id "$CONNECTOR_ID" \
--source-type SOURCE_TYPE_KAFKA \
--limit 5 \
--topic-mode single \
--topics-json '["orders"]'
Sampling reads source records to verify key/value formats and field names without writing rows to any target table.
Create and manage import and export tasks¶
Pass complex import and export task definitions via --json-file (inspect wire fields offline first with moi-cli schema connector create-task or moi-cli schema connector create-export-task):
Direction |
Subcommands and key flags |
Description |
|---|---|---|
Import creation & lookup |
|
List workspace import tasks, inspect task details, or create an import task. |
Import lifecycle |
|
Update configuration, pause, resume, or delete an import task. |
Import file retries |
|
List import task files and retry failed file IDs. |
Export lifecycle |
|
Create export tasks, track export state and file records, retry failed files, or delete export task records. |
Manual runs and statistics¶
For an enabled scheduled structured import task that is in the created or uploading state and already has a server workflow job, trigger an immediate one-off execution and inspect its run history (run-task-now supports Kafka, Doris, REST API, and MySQL tasks without standalone compute configurations):
moi-cli -auth pat -o connector run-task-now --workspace-id "$WORKSPACE_ID" --task-id "$TASK_ID"
moi-cli -auth pat -o connector task-runs --workspace-id "$WORKSPACE_ID" --task-id "$TASK_ID"
A successful run-task-now response confirms that the server accepted the trigger; verify actual completion, error messages, and target rows via task-runs.
Warning
connector run-task-full-refresh --workspace-id "$WORKSPACE_ID" --task-id "$TASK_ID" applies only to enabled scheduled incremental Doris tasks or MySQL tasks (without standalone compute configurations) in that ready state. It executes a single full_refresh run without modifying the task’s saved incremental schedule. Confirm how target table data is written beforehand, and check task-runs before retrying if the outcome is unknown.
Use connector task-stats and connector export-task-stats for workspace-level aggregate counts; always check task-runs or export-task-state to verify an individual task execution.
Limitations and notes¶
In
0.1.0offline packages,db-list-tablesanddb-source-schemado not expose--schema; obtain a deployment-matched CLI package from your deployment administrator or call the corresponding HTTP endpoint when schema filtering on/connectors/dbis needed.After a network timeout on task creation, retry, or resume, check
connector tasks,get-task, orexport-tasksfirst to confirm whether the task is already created or running before re-submitting.connector delete-export-taskdeletes only the task record in MatrixOne Intelligence and does not roll back data already written to external destination systems.