Native DataX source scans and exact-value writes¶
Pin the runtime¶
This round used DataX 3 mysqlreader/mysqlwriter, bundled Connector/J 5.1.47, Groovy 2.1.9 and Temurin Java 8. The existing DataX tutorial uses a separate matrixonewriter plugin. This page tests bundled mysqlwriter; qualify each path separately.
Under Java 21, native jobs failed during Groovy initialization against both destinations. A minimal Groovy probe reproduced module access rejection for java.lang.Object.finalize. Twenty additional tinyint append cases passed under Java 8; the original twenty failures remain recorded. “JDK 8+” is not evidence for every newer JDK.
The bundled Connector/J 5.1.47 also needs an authentication method compatible with the MySQL source. If the source uses an authentication plugin that this driver cannot handle, qualify an updated Connector/J or an isolated test account with a compatible method; do not change the server-wide authentication setting to make one test pass. Record the driver and authentication method with the job result.
Actual source and destination tables¶
Use a dedicated test database and the connection fixtures. The source is MySQL and the destination is MatrixOne. Create this table on both; insert the fixture only on the source. Set SRC_JDBC_URL/SRC_USER/SRC_PASSWORD and MO_JDBC_URL/MO_USER/MO_PASSWORD for the respective databases, fixing UTF-8 and UTC. Local plaintext examples do not qualify production TLS.
CREATE TABLE native_demo (
id BIGINT PRIMARY KEY,
amount DECIMAL(38,18), biguint BIGINT UNSIGNED,
payload VARBINARY(256), ts DATETIME(6), elapsed TIME(6), label VARCHAR(256)
);
INSERT INTO native_demo VALUES
(1,12345678901234567890.123456789012345678,18446744073709551615,
X'00FF80','2024-02-29 12:00:00.123456','-00:00:00.000001','中文😀'),
(2,NULL,NULL,NULL,NULL,NULL,NULL),
(3,0,9007199254740993,X'','1970-01-01 00:00:00','37:00:00.999999','');
Generate and run the job¶
First write a text staging table, then explicitly convert into the final table in SQL; decode bytes using Base64. Remove TO_BASE64 line breaks and preserve NULL versus empty bytes. This is a reversible adapter; it does not establish default DateColumn fidelity for microseconds or extended TIME. Save the following as make_datax_job.py.
CREATE TABLE native_demo_stage (
id BIGINT PRIMARY KEY, amount VARCHAR(128), biguint VARCHAR(32),
payload VARBINARY(256), ts VARCHAR(32), elapsed VARCHAR(32), label VARCHAR(256)
);
import json, os
from pathlib import Path
query = """SELECT id, CAST(amount AS CHAR) AS amount,
CAST(biguint AS CHAR) AS biguint,
REPLACE(TO_BASE64(payload), CHAR(10), '') AS payload,
CAST(ts AS CHAR) AS ts, CAST(elapsed AS CHAR) AS elapsed, label
FROM native_demo WHERE id BETWEEN 1 AND 3"""
code = """if(record.getColumn(3).getRawData()!=null) {
record.setColumn(3,new BytesColumn(
java.util.Base64.getDecoder().decode(record.getColumn(3).asString())));
}
return record;"""
job = {"job": {"setting": {"speed": {"channel": 1},
"errorLimit": {"record": 0, "percentage": 0}},
"content": [{
"reader": {"name": "mysqlreader", "parameter": {
"username": os.environ["SRC_USER"], "password": os.environ["SRC_PASSWORD"],
"connection": [{"jdbcUrl": [os.environ["SRC_JDBC_URL"]], "querySql": [query]}]}},
"writer": {"name": "mysqlwriter", "parameter": {
"username": os.environ["MO_USER"], "password": os.environ["MO_PASSWORD"],
"column": ["id", "amount", "biguint", "payload", "ts", "elapsed", "label"],
"writeMode": "insert", "batchSize": 17,
"connection": [{"jdbcUrl": os.environ["MO_JDBC_URL"], "table": ["native_demo_stage"]}]}},
"transformer": [{"name": "dx_groovy", "parameter": {
"code": code, "extraPackage": []}}]
}]}}
path = Path("native-datax.json")
path.write_text(json.dumps(job, ensure_ascii=False), encoding="utf-8")
path.chmod(0o600)
If Groovy cannot run in the selected Java runtime, a separate text-staging adapter can keep Base64 as text and decode it in MatrixOne SQL. Create a second staging table with the same columns except payload_b64 VARCHAR(512) in place of payload VARBINARY(256). In the generated job, remove transformer, replace the fourth writer column with payload_b64, and set the writer table to the new staging table. Keep the reader’s REPLACE(TO_BASE64(payload), CHAR(10), '') expression. After DataX reports zero dirty records, use FROM_BASE64(payload_b64) in the final INSERT ... SELECT, alongside the explicit decimal and temporal casts shown below. Verify NULL, empty bytes and X'00FF80' separately. This avoids the Groovy step; it remains an explicit adapter and does not qualify raw binary or temporal binding.
export JAVA_HOME=/path/to/verified/java8
export PATH="$JAVA_HOME/bin:$PATH"
java -version
python make_datax_job.py
python "$DATAX_HOME/bin/datax.py" -j '-Xms128m -Xmx512m' native-datax.json
Acceptance and failure handling¶
After DataX fills staging, execute this conversion and verify the final table. Directly writing CAST text into a DATETIME destination still loses microseconds; record that failed default binding separately. Expect three successful rows and zero dirty records. Open fresh source/destination connections and compare each field against the original fixture using this query. Matching counts do not prove fidelity. Freeze source ranges before increasing channel, batchSize or splitPk; this round’s querySql scans do not qualify splitPk partitioning.
INSERT INTO native_demo
SELECT id, CAST(amount AS DECIMAL(38,18)), CAST(biguint AS UNSIGNED),
payload, CAST(ts AS DATETIME(6)), CAST(elapsed AS TIME(6)), label
FROM native_demo_stage;
SELECT id, CAST(amount AS CHAR), CAST(biguint AS CHAR), HEX(payload),
CAST(ts AS CHAR), CAST(elapsed AS CHAR), label
FROM native_demo ORDER BY id;
Rerunning insert can conflict with primary keys. Offline append does not propagate deletes or coordinate transactions across tasks. Handle deletion, target DDL and recovery of committed chunks through a separate staging/merge and idempotency protocol. Do not hide dirty records to make a job pass.
Measured scope of this round¶
On 2026-10-13, MatrixOne commit 652c643666428f1c434e03369834f2a9bead7c27 was tested against an isolated MySQL 8.4 source/control. Each scan case used 256 rows, batch=17 and long Unicode labels, with original all-field oracles, fresh-connection reads and typed archives. Each table entry covers both MatrixOne and MySQL destinations.
Type |
Path |
Result at both destinations |
|---|---|---|
int64 |
text_adapter |
Passed |
uint64 |
raw |
Passed |
uint64 |
text_adapter |
Passed |
decimal38_18 |
text_adapter |
Passed |
decimal65_30 |
text_adapter |
Passed |
varchar_utf8 |
text_adapter |
Passed |
varbinary |
text_adapter |
Passed |
datetime6 |
raw |
Value mismatch |
datetime6 |
text_adapter |
Value mismatch |
time6 |
raw |
Execution failed |
time6 |
text_adapter |
Execution failed |
Of 22 real-table scan cases, 14 passed, four DATETIME cases failed full-value checks and four TIME cases failed execution. Native DATETIME .123456 became .123000; source text CAST directly into a temporal destination became .000000. Both destinations behaved alike. Four additional VARCHAR staging + SQL CAST cases all passed, including year 1000/9999 bounds, negative microseconds and ±838-hour TIME. Those supplementary cases used 32 rows each and are separate from the 256-row scans.
Temporary-filesystem, bounded-window results do not certify disk throughput, long soak, HA, distributed scale or cross-system EOS. Evidence and documentation work are tracked in documentation issue #428.