Airbyte connector protocol and destination acceptance¶
Route, versions and known boundary¶
Reference images are airbyte/source-mysql:3.53.7 and airbyte/destination-mysql:1.1.1. Prior validation ran official Docker source/destination in a local protocol chain, rather than the full Airbyte control platform, scheduler or Kubernetes HA. Use a real MySQL source; do not assume MatrixOne exposes MySQL binlog.
The official MySQL destination was blocked on MatrixOne by LOAD DATA ... SET NULLIF(@user_variable,...), reported in matrixone#29849. A passing SQL effect adapter must be labeled separately and does not make the official destination pass. Check fixes in your release before running the official probes below.
Prepare source events¶
Provide an accessible MySQL source with read/metadata permissions. Standard reads work for this isolated example; CDC needs separate binlog permissions and retention. Execute this fixture on the MySQL source. The exact payload is a string envelope; deleted is an explicit marker, not automatic discovery of physical deletes by table polling. This example uses its own doc_airbyte database.
CREATE DATABASE IF NOT EXISTS doc_airbyte;
USE doc_airbyte;
CREATE TABLE IF NOT EXISTS airbyte_events (event_seq BIGINT PRIMARY KEY, payload TEXT, deleted BOOLEAN);
INSERT INTO airbyte_events VALUES (1,'{"id":"9007199254740993","amount":"0.000000000000000001","binary_hex":"00ff80"}',FALSE)
ON DUPLICATE KEY UPDATE payload=VALUES(payload),deleted=VALUES(deleted);
SELECT * FROM airbyte_events ORDER BY event_seq;
Configuration files¶
Run this configuration generator in the isolated local environment after setting SOURCE_HOST/PORT/USER/PASSWORD and MO_*. SOURCE_HOST must be reachable from the connector container. The commands use Linux host networking; adapt other container networks. Configuration files contain secrets: restrict permissions and exclude them from source control. The SSL settings below are local-only; configure TLS for your connector release before remote use.
import json, os
from pathlib import Path
source = {"host":os.environ["SOURCE_HOST"],"port":int(os.environ["SOURCE_PORT"]),
"database":"doc_airbyte","username":os.environ["SOURCE_USER"],
"password":os.environ["SOURCE_PASSWORD"],"replication_method":{"method":"STANDARD"},
"ssl_mode":{"mode":"preferred"},"tunnel_method":{"tunnel_method":"NO_TUNNEL"},
"table_filters":[{"database_name":"doc_airbyte","table_name_patterns":["airbyte_events"]}]}
destination = {"host":os.environ["MO_HOST"],"port":int(os.environ["MO_PORT"]),
"database":"doc_airbyte","raw_data_schema":"doc_airbyte",
"username":os.environ["MO_USER"],"password":os.environ["MO_PASSWORD"],
"ssl":False,"tunnel_method":{"tunnel_method":"NO_TUNNEL"}}
for name, data in (("source.json",source),("destination.json",destination)):
p = Path(name)
p.touch(mode=0o600,exist_ok=False)
p.write_text(json.dumps(data))
docker run --rm --network host -v "$PWD:/config:ro" airbyte/source-mysql:3.53.7 check --config /config/source.json > check.jsonl 2> check.stderr
docker run --rm --network host -v "$PWD:/config:ro" airbyte/source-mysql:3.53.7 discover --config /config/source.json > discover.jsonl 2> discover.stderr
Generate a configured catalog from the actual DISCOVER response rather than guessing the schema. Save and run:
import json
from pathlib import Path
messages = []
for line in Path("discover.jsonl").read_text().splitlines():
try:
messages.append(json.loads(line))
except ValueError:
continue
catalog = next(x["catalog"] for x in messages if x.get("type")=="CATALOG")
stream = next(x for x in catalog["streams"] if x["name"]=="airbyte_events")
configured = {"streams":[{"stream":stream,"sync_mode":"full_refresh","destination_sync_mode":"append"}]}
Path("catalog.json").write_text(json.dumps(configured))
docker run --rm --network host -v "$PWD:/config:ro" airbyte/source-mysql:3.53.7 read --config /config/source.json --catalog /config/catalog.json > read.jsonl 2> read.stderr
Preserve RECORD/STATE from the output, excluding LOG and other messages, then run the official destination against an authorized dedicated MatrixOne database. This is a compatibility probe and can fail on affected releases. Retain full TRACE/generated SQL on failure; a source STATE is not a destination ACK.
import json
from pathlib import Path
with Path("protocol.jsonl").open("w") as out:
for line in Path("read.jsonl").read_text().splitlines():
try:
x = json.loads(line)
except ValueError:
continue
if x.get("type") in ("RECORD","STATE"):
out.write(json.dumps(x,ensure_ascii=False)+"\n")
docker run --rm -i --network host -v "$PWD:/config:ro" airbyte/destination-mysql:1.1.1 write --config /config/destination.json --catalog /config/catalog.json < protocol.jsonl > write.jsonl 2> write.stderr
Acceptance, deletes and checkpoints¶
Check CHECK SUCCESS, DISCOVER catalog, READ RECORD/STATE, WRITE ACK/TRACE and fresh destination reads separately. Decode the string envelope and compare fields rather than only RECORD counts. The destination may create raw tables: inspect generated names/schema instead of assuming an expanded business table.
Replaying full_refresh/append can duplicate rows. Standard reads do not automatically propagate old-key UPDATE/DELETE. Qualify incremental/CDC cursors, state resume, delete events and destination deduplication separately. Advance pipeline checkpoints only when the destination has durably acknowledged the corresponding batch. Local connector protocol execution does not atomically commit platform scheduler state. Rediscover after schema changes, compare catalogs and migrate destination mappings.