Native DataX source scans and exact-value writes

Pin the runtime

This round used DataX 3 mysqlreader/mysqlwriter, bundled Connector/J 5.1.47, Groovy 2.1.9 and Temurin Java 8. The existing DataX tutorial uses a separate matrixonewriter plugin. This page tests bundled mysqlwriter; qualify each path separately.

Under Java 21, native jobs failed during Groovy initialization against both destinations. A minimal Groovy probe reproduced module access rejection for java.lang.Object.finalize. Twenty additional tinyint append cases passed under Java 8; the original twenty failures remain recorded. “JDK 8+” is not evidence for every newer JDK.

The bundled Connector/J 5.1.47 also needs an authentication method compatible with the MySQL source. If the source uses an authentication plugin that this driver cannot handle, qualify an updated Connector/J or an isolated test account with a compatible method; do not change the server-wide authentication setting to make one test pass. Record the driver and authentication method with the job result.

Actual source and destination tables

Use a dedicated test database and the connection fixtures. The source is MySQL and the destination is MatrixOne. Create this table on both; insert the fixture only on the source. Set SRC_JDBC_URL/SRC_USER/SRC_PASSWORD and MO_JDBC_URL/MO_USER/MO_PASSWORD for the respective databases, fixing UTF-8 and UTC. Local plaintext examples do not qualify production TLS.

CREATE TABLE native_demo (
  id BIGINT PRIMARY KEY,
  amount DECIMAL(38,18), biguint BIGINT UNSIGNED,
  payload VARBINARY(256), ts DATETIME(6), elapsed TIME(6), label VARCHAR(256)
);
INSERT INTO native_demo VALUES
  (1,12345678901234567890.123456789012345678,18446744073709551615,
   X'00FF80','2024-02-29 12:00:00.123456','-00:00:00.000001','中文😀'),
  (2,NULL,NULL,NULL,NULL,NULL,NULL),
  (3,0,9007199254740993,X'','1970-01-01 00:00:00','37:00:00.999999','');

Generate and run the job

First write a text staging table, then explicitly convert into the final table in SQL; decode bytes using Base64. Remove TO_BASE64 line breaks and preserve NULL versus empty bytes. This is a reversible adapter; it does not establish default DateColumn fidelity for microseconds or extended TIME. Save the following as make_datax_job.py.

CREATE TABLE native_demo_stage (
  id BIGINT PRIMARY KEY, amount VARCHAR(128), biguint VARCHAR(32),
  payload VARBINARY(256), ts VARCHAR(32), elapsed VARCHAR(32), label VARCHAR(256)
);
import json, os
from pathlib import Path

query = """SELECT id, CAST(amount AS CHAR) AS amount,
  CAST(biguint AS CHAR) AS biguint,
  REPLACE(TO_BASE64(payload), CHAR(10), '') AS payload,
  CAST(ts AS CHAR) AS ts, CAST(elapsed AS CHAR) AS elapsed, label
  FROM native_demo WHERE id BETWEEN 1 AND 3"""
code = """if(record.getColumn(3).getRawData()!=null) {
  record.setColumn(3,new BytesColumn(
    java.util.Base64.getDecoder().decode(record.getColumn(3).asString())));
}
return record;"""
job = {"job": {"setting": {"speed": {"channel": 1},
                          "errorLimit": {"record": 0, "percentage": 0}},
  "content": [{
    "reader": {"name": "mysqlreader", "parameter": {
      "username": os.environ["SRC_USER"], "password": os.environ["SRC_PASSWORD"],
      "connection": [{"jdbcUrl": [os.environ["SRC_JDBC_URL"]], "querySql": [query]}]}},
    "writer": {"name": "mysqlwriter", "parameter": {
      "username": os.environ["MO_USER"], "password": os.environ["MO_PASSWORD"],
      "column": ["id", "amount", "biguint", "payload", "ts", "elapsed", "label"],
      "writeMode": "insert", "batchSize": 17,
      "connection": [{"jdbcUrl": os.environ["MO_JDBC_URL"], "table": ["native_demo_stage"]}]}},
    "transformer": [{"name": "dx_groovy", "parameter": {
      "code": code, "extraPackage": []}}]
  }]}}
path = Path("native-datax.json")
path.write_text(json.dumps(job, ensure_ascii=False), encoding="utf-8")
path.chmod(0o600)

If Groovy cannot run in the selected Java runtime, a separate text-staging adapter can keep Base64 as text and decode it in MatrixOne SQL. Create a second staging table with the same columns except payload_b64 VARCHAR(512) in place of payload VARBINARY(256). In the generated job, remove transformer, replace the fourth writer column with payload_b64, and set the writer table to the new staging table. Keep the reader’s REPLACE(TO_BASE64(payload), CHAR(10), '') expression. After DataX reports zero dirty records, use FROM_BASE64(payload_b64) in the final INSERT ... SELECT, alongside the explicit decimal and temporal casts shown below. Verify NULL, empty bytes and X'00FF80' separately. This avoids the Groovy step; it remains an explicit adapter and does not qualify raw binary or temporal binding.

export JAVA_HOME=/path/to/verified/java8
export PATH="$JAVA_HOME/bin:$PATH"
java -version
python make_datax_job.py
python "$DATAX_HOME/bin/datax.py" -j '-Xms128m -Xmx512m' native-datax.json

Acceptance and failure handling

After DataX fills staging, execute this conversion and verify the final table. Directly writing CAST text into a DATETIME destination still loses microseconds; record that failed default binding separately. Expect three successful rows and zero dirty records. Open fresh source/destination connections and compare each field against the original fixture using this query. Matching counts do not prove fidelity. Freeze source ranges before increasing channel, batchSize or splitPk; this round’s querySql scans do not qualify splitPk partitioning.

INSERT INTO native_demo
SELECT id, CAST(amount AS DECIMAL(38,18)), CAST(biguint AS UNSIGNED),
       payload, CAST(ts AS DATETIME(6)), CAST(elapsed AS TIME(6)), label
FROM native_demo_stage;
SELECT id, CAST(amount AS CHAR), CAST(biguint AS CHAR), HEX(payload),
       CAST(ts AS CHAR), CAST(elapsed AS CHAR), label
FROM native_demo ORDER BY id;

Rerunning insert can conflict with primary keys. Offline append does not propagate deletes or coordinate transactions across tasks. Handle deletion, target DDL and recovery of committed chunks through a separate staging/merge and idempotency protocol. Do not hide dirty records to make a job pass.

Measured scope of this round

On 2026-10-13, MatrixOne commit 652c643666428f1c434e03369834f2a9bead7c27 was tested against an isolated MySQL 8.4 source/control. Each scan case used 256 rows, batch=17 and long Unicode labels, with original all-field oracles, fresh-connection reads and typed archives. Each table entry covers both MatrixOne and MySQL destinations.

Type

Path

Result at both destinations

int64

text_adapter

Passed

uint64

raw

Passed

uint64

text_adapter

Passed

decimal38_18

text_adapter

Passed

decimal65_30

text_adapter

Passed

varchar_utf8

text_adapter

Passed

varbinary

text_adapter

Passed

datetime6

raw

Value mismatch

datetime6

text_adapter

Value mismatch

time6

raw

Execution failed

time6

text_adapter

Execution failed

Of 22 real-table scan cases, 14 passed, four DATETIME cases failed full-value checks and four TIME cases failed execution. Native DATETIME .123456 became .123000; source text CAST directly into a temporal destination became .000000. Both destinations behaved alike. Four additional VARCHAR staging + SQL CAST cases all passed, including year 1000/9999 bounds, negative microseconds and ±838-hour TIME. Those supplementary cases used 32 rows each and are separate from the 256-row scans.

Temporary-filesystem, bounded-window results do not certify disk throughput, long soak, HA, distributed scale or cross-system EOS. Evidence and documentation work are tracked in documentation issue #428.