Local testing with the official Fivetran Connector SDK¶
Role and installation¶
Reference versions are official Connector SDK 2.12.1 and local tester 2.26.0908.001. The validated route is MatrixOne/MySQL source → SDK operations → local tester/DuckDB destination. It does not certify the hosted Fivetran service or an official MatrixOne destination. The selected scope is the SDK’s local chain.
Install fivetran-connector-sdk pymysql, complete the shared connection and save mo_common.py in a dedicated project directory. Enable the failure injection below only in that test directory. Retain the SDK’s local files/state and warehouse.
Connector and exact payload¶
Save as connector.py. The keyed row uses an explicit STRING payload for decimal, binary and label; id fits the SDK LONG signed range. Decimal is text and binary is hex. This exact envelope does not establish lossless default inference for arbitrary large numbers.
This minimal example assumes the two fixed fixture rows and no concurrent source changes. The first process fails after the second upsert and before its checkpoint; a marker makes the injection occur once. The next process resumes the prior checkpoint and replays the same keyed upsert.
import json
from pathlib import Path
from fivetran_connector_sdk import Connector, Operations as op
from mo_common import connect
def schema(configuration):
return [{"table":"matrixone_rows","primary_key":["id"],
"columns":{"id":"LONG","payload":"STRING"}}]
def update(configuration, state):
last = int(state.get("last_id", -1))
with connect() as d:
with d.cursor() as c:
c.execute("SELECT id,CAST(amount AS CHAR),HEX(payload),label,rev FROM lake_demo "
"WHERE id IN (2,9007199254740993) AND id>%s ORDER BY id", (last,))
rows = c.fetchall()
for index, row in enumerate(rows):
payload = json.dumps({"amount_text":row[1],"payload_hex":row[2],
"label":row[3],"rev":row[4]},ensure_ascii=False,separators=(",",":"))
op.upsert(table="matrixone_rows",data={"id":row[0],"payload":payload})
marker = Path("injected-once")
if configuration.get("inject_once") == "true" and index == 1 and not marker.exists():
marker.write_text("uncheckpointed upsert")
raise RuntimeError("Deliberate failure before checkpoint")
op.checkpoint(state={"last_id":row[0]})
connector = Connector(update=update,schema=schema)
{"inject_once":"true"}
Fresh-process resume and operation signatures¶
Save the JSON as configuration.json and run the CLI twice, using a fresh process for the second run. Inspect local files/state.json, tester output and the warehouse: no duplicate business key after replay, complete payload and final last_id 9007199254740993. The first run is expected to fail; verify that failure rather than ignoring its exit code.
fivetran debug --configuration configuration.json
# Expected injected failure; start a new process for resume:
fivetran debug --configuration configuration.json
The update/delete APIs use different parameter names from upsert. These are operation fragments for an incremental event handler; do not insert them unconditionally into the replay loop above:
op.update(table="matrixone_rows",modified={"id":2,"payload":"updated"})
op.delete(table="matrixone_rows",keys={"id":2})
op.checkpoint(state={"event_seq":10})
Delete, schema and recovery boundaries¶
The id cursor is a static-reading example. It misses UPDATE/DELETE on old keys and is unsuitable for production incremental sync with concurrent commits or late rows. Use qualified CDC or a transactional outbox with a committed/retained event sequence. Deletes carry keys, updates carry versions and checkpoints refer to processed event watermarks.
Update the SDK schema/exact envelope after adding source fields and check compatibility when resuming old state. Repeated debug in one process encountered operation-stream reuse issues; this tutorial uses fresh CLI processes. Its scope is local resume and idempotent effects, without SaaS, cross-database atomic commit or production throughput claims.