Kafka Connect, Debezium and JDBC CDC sinks¶
Components and roles¶
Reference versions are Kafka 3.9.1, Debezium MySQL source/JDBC sink 3.1.2.Final and Confluent JDBC sink 10.8.4. Start with the validated Confluent JDBC route below. Debezium JDBC is a different sink using settings such as primary.key.mode/collection.name.format; changing only connector.class in Confluent JSON is incorrect.
Prepare Kafka and a distributed Connect worker. Install the source, JDBC sink and MySQL JDBC driver in plugin.path and verify the REST plugin list. Use a real MySQL8.4 binlog source with ROW/full-row images, permissions and retention; MatrixOne is the destination. This does not establish MatrixOne as a MySQL binlog source.
Tables, converters and configuration¶
Complete the shared fixtures, prepare the same structure/initial values in MySQL source doc_lake.lake_demo and pre-create destination kc_demo. Use JSON converters with schemas enabled for both key and value (schemas.enable=true). Retain logical Decimal for exact values instead of untyped numeric JSON with schemas disabled.
The generator reads SOURCE_, MO_ and KAFKA_BOOTSTRAP and creates sensitive configuration files. It uses isolated local networking: add Kafka SASL/TLS and JDBC TLS for remote use. Disable auto.create/auto.evolve and migrate DDL explicitly.
CREATE DATABASE IF NOT EXISTS doc_lake;
USE doc_lake;
CREATE TABLE IF NOT EXISTS kc_demo (id BIGINT PRIMARY KEY, amount DECIMAL(38,18), payload VARBINARY(256), label VARCHAR(256), rev BIGINT);
SELECT id,CAST(amount AS CHAR),HEX(payload),label,rev FROM kc_demo ORDER BY id;
import json, os
from pathlib import Path
source = {"connector.class":"io.debezium.connector.mysql.MySqlConnector","tasks.max":"1",
"database.hostname":os.environ["SOURCE_HOST"],"database.port":os.environ["SOURCE_PORT"],
"database.user":os.environ["SOURCE_USER"],"database.password":os.environ["SOURCE_PASSWORD"],
"database.server.id":"7722","topic.prefix":"lakecdc","database.include.list":"doc_lake",
"table.include.list":"doc_lake.lake_demo","schema.history.internal.kafka.bootstrap.servers":os.environ["KAFKA_BOOTSTRAP"],
"schema.history.internal.kafka.topic":"lakecdc-schema","snapshot.mode":"initial",
"decimal.handling.mode":"precise","bigint.unsigned.handling.mode":"precise",
"time.precision.mode":"adaptive_time_microseconds","include.schema.changes":"false"}
sink = {"connector.class":"io.confluent.connect.jdbc.JdbcSinkConnector","tasks.max":"1",
"topics":"lakecdc.doc_lake.lake_demo",
"connection.url":f"jdbc:mysql://{os.environ['MO_HOST']}:{os.environ['MO_PORT']}/{os.environ['MO_DATABASE']}?useUnicode=true&characterEncoding=UTF-8",
"connection.user":os.environ["MO_USER"],"connection.password":os.environ["MO_PASSWORD"],
"table.name.format":"kc_demo","insert.mode":"upsert","pk.mode":"record_key","pk.fields":"id",
"delete.enabled":"true","auto.create":"false","auto.evolve":"false","batch.size":"20",
"transforms":"unwrap","transforms.unwrap.type":"io.debezium.transforms.ExtractNewRecordState",
"transforms.unwrap.delete.tombstone.handling.mode":"tombstone","errors.tolerance":"none"}
for name, data in (("source-connect.json",source),("sink-connect.json",sink)):
p = Path(name)
p.touch(mode=0o600,exist_ok=False)
p.write_text(json.dumps(data))
curl --fail "$CONNECT_URL/connector-plugins"
curl --fail -X PUT -H 'Content-Type: application/json' --data-binary @source-connect.json "$CONNECT_URL/connectors/lake-source/config"
curl --fail -X PUT -H 'Content-Type: application/json' --data-binary @sink-connect.json "$CONNECT_URL/connectors/lake-sink/config"
curl --fail "$CONNECT_URL/connectors/lake-source/status"
curl --fail "$CONNECT_URL/connectors/lake-sink/status"
Deletes and schema validation¶
Wait for the two snapshot keys to match field by field, then update label/rev, delete id 2 and change a primary key on the MySQL source. Wait for destination values and retain event keys, op, tombstones and sink SQL. delete.enabled=true needs correct record keys; dropping tombstones or retaining only non-null values can lose deletes. Final-state UPSERT does not handle non-idempotent increments without event deduplication.
For an added nullable column, migrate destination DDL before source DDL and verify event schema. Test renames, dropped columns, narrowing, NOT NULL and default changes separately. Debezium JDBC automatic evolution historically generated parenthesized ALTER ADD COLUMN, reported in matrixone#29851. Pre-created tables with automatic DDL disabled are a distinct route, not a passing automatic evolution result.
Offsets, restart and pressure¶
Exercise worker restart, broker restart on the same disk, rebalance and replay after SQL commit but before offset recording separately. Compare full source/destination fields, duplicates and committed next offsets. Preserve watermarks, original events and SQL effects before resetting a consumer offset. Retain topic/schema history/Connect internal topics rather than clearing evidence to obtain a passing rerun.
The validated boundary is at-least-once delivery plus primary-key idempotent final state. Single-broker same-disk recovery does not establish replicated disaster recovery. Kafka transactions/read-committed views do not atomically commit external SQL and offsets. Report batches, tasks, partitions, actual connections, throughput and lag, without unsupported cross-system exactly-once claims.