Job acceptance for DataX, SeaTunnel, Spark, Flink and Trino¶
Continue with native DataX source scans, native Spark JDBC reads/writes and native Flink binlog, deletes and checkpoint recovery.
Start from a complete job¶
Existing DataX, SeaTunnel, Spark and Flink guides cover installation, connectors and job launch. Run the actual engine rather than substituting Python/JDBC probes. This page applies the earlier type, adapter and recovery acceptance rules to those jobs; prepared but unexecuted heavy runners do not count as passes.
Complete the shared fixtures and record engine/plugin/JDBC/MatrixOne versions. Distinguish MySQL source CDC from MatrixOne SQL sink, pre-create destinations and control DDL separately. Fix the source key range or snapshot before launch: completed partitions need not represent one point in time.
DataX and SeaTunnel: batch contract¶
Use explicit DataX column order and writer writeMode with a plugin actually installed. Qualify insert, replace and update separately. Configure real SeaTunnel Jdbc source/sink and BATCH/STREAMING mode with matching SQL parameter/schema order; shared JDBC does not qualify both modes.
Extend fixtures to negative/over 24-hour TIME, 65-digit Decimal, UInt64/BIT64, empty ENUM/SET, NULL/empty bytes/empty strings and nested JSON. For types a plugin cannot represent, use a reversible CAST/HEX/string contract and label the adapter. Historical file-loading probes include empty ENUM/SET #29840 and decimal parsing #29841.
Check fresh all-field reads, partition coverage and duplicate keys beyond read/write/failed counts. Append retries do not automatically deduplicate, and incremental deletes need tombstones/events. Bound channels/parallelism, batches and actual connections; recover committed partitions explicitly when another partition fails.
Spark: explicit schema and pre-created destination¶
This supplementary load uses the existing lake_demo. Install matching Spark/PySpark and a JDBC driver, then set MO_JDBC_JAR and the shared environment. Read JDBC URL/credentials from the runtime; local execution does not qualify remote TLS. DecimalType(38,18) does not carry SQL precision 65 automatically. Qualify TIME/date ranges and default JDBC reflection separately. Related Spark documentation changes are tracked in matrixone-doc#425.
import os
from decimal import Decimal
from pyspark.sql import SparkSession
from pyspark.sql.types import StructType,StructField,LongType,DecimalType,BinaryType,StringType
spark = SparkSession.builder.appName("matrixone_lake_demo").config("spark.jars",os.environ["MO_JDBC_JAR"]).getOrCreate()
try:
schema = StructType([StructField("id",LongType(),False),StructField("amount",DecimalType(38,18)),
StructField("payload",BinaryType()),StructField("label",StringType()),StructField("rev",LongType())])
frame = spark.createDataFrame([(701,Decimal("12345678901234567890.123456789012345678"),
bytearray(b"\x00\xff\x80"),"Spark😀",0)],schema)
url = f"jdbc:mysql://{os.environ['MO_HOST']}:{os.environ['MO_PORT']}/{os.environ['MO_DATABASE']}?useUnicode=true&characterEncoding=UTF-8"
frame.coalesce(1).write.format("jdbc").option("url",url).option("dbtable","lake_demo").option("driver","com.mysql.cj.jdbc.Driver").option("user",os.environ["MO_USER"]).option("password",os.environ["MO_PASSWORD"]).option("batchsize","31").mode("append").save()
finally:
spark.stop()
Spark write partitions, task retries and speculative execution can replay effects; append is not a globally atomic write. Read partition boundaries need complete non-overlapping coverage, with separate checks for NULL partition keys and concurrent source changes. Record stages/tasks, SQL batches and destination keys around failures.
Flink CDC: checkpoints and external databases¶
Use the existing guide to configure an actual MySQL CDC source, Flink checkpoint storage and real JDBC sink. Keyed updates/deletes need the correct changelog; append-only tables cannot express deletion. Ordinary JDBC sink checkpoints do not establish external SQL exactly-once. Before selecting an XA route, qualify the MatrixOne release, driver and sink’s XA/prepare/commit/recover protocol rather than inferring target support from Flink support.
Exercise failures before/after checkpoints, task recovery, old-checkpoint replay, source binlog truncation, DDL evolution and bad records. Retain checkpoints, source/destination watermarks, keys and full-field oracles; clearing state can hide recovery defects.
Trino: federation and pushdown¶
For a Trino MySQL catalog accessing MatrixOne, record server/plugin/JDBC versions, connection-url and credential injection, and confirm that real coordinator/workers execute queries. Check metadata, type mapping and predicate/aggregate/join pushdown separately using EXPLAIN and actual database SQL. A reachable port is not Trino runtime validation.
Qualify UInt64, Decimal65, TIME and wide dates in the federated type system; label CAST workarounds as adapters. Cross-catalog joins do not automatically share source snapshots, NULL or collation semantics. Read federation does not establish federated write transactions, and a prepared runner is not certification. Retain plans, remote scanned rows/bytes, connections, spills, latency and backlog when measuring load.