Lakehouse ecosystem tool tutorials¶
Start from connections and fixtures, then choose the actual source/destination and execution tool. The tutorials distinguish native APIs/runtimes, explicit SQL/type adapters, blocked official connectors and unqualified routes. Versions or case counts are not production certification. Each page gives usage steps and boundaries for deletes, schema, recovery and fidelity.
Continue with native DataX source scans, native Spark JDBC reads/writes and native Flink binlog, deletes and checkpoint recovery.
Choose a tutorial¶
Tutorial |
Coverage |
|---|---|
Configure the shared connection and exact-value fixtures used by the ecosystem tutorials. |
|
Load pre-created SQL tables with explicit DataFrame/ETL types and bounded batches. |
|
Round-trip exact decimals, binary bytes and microsecond values through files and an explicit SQL sink. |
|
Use a native Iceberg catalog and Arrow schema while keeping SQL effects and commit boundaries explicit. |
|
Configure the official MySQL provider and a transactional task with fresh readback. |
|
Configure a real dbt project, table model and key tests with explicit incremental boundaries. |
|
Configure and distinguish the Shell/Python route and a MYSQL datasource SQL task. |
|
Read MatrixOne as an SDK source and exercise keyed operations and checkpoint resume locally. |
|
Run official check/discover/read/write and distinguish destination STATE ACK from SQL adapters. |
|
Configure ExecuteSQLRecord, Avro and PutDatabaseRecord with explicit failure queues and UPSERT probes. |
|
Configure a MySQL CDC source and MatrixOne JDBC sink with tombstones, types and offset recovery. |
|
Run Kafka input into parameterized SQL with deletes, DLQ and offset replay. |
|
Separate inferred types from explicit CAST/HEX reads and verify metadata and NULL behavior. |
|
Run the official dlt SQLAlchemy destination and define custom Singer target/state boundaries. |
|
Extend existing runtime tutorials with exact types, partitions, deletes, checkpoints and load acceptance. |
Validation and use¶
Retain original inputs/expected values before running the smallest chain. Increase batches, data volume and failure intensity only after complete fresh readback. Requalify the deployed MatrixOne release and each tool version before production; adapter success does not replace a failed official connector. These source changes target latest and do not automatically update historical branches.
Evidence and historical reports: ecosystem validation #29848 and documentation tracker #428.
- Prepare connections, fixtures and acceptance checks
- Batch loading with pandas, PETL, Polars and Dask
- PyArrow IPC and DuckDB Parquet bridges to MatrixOne
- PyIceberg append/scan with an explicit SQL bridge
- Airflow MySqlHook DAG and idempotent retries
- dbt-mysql model builds and data tests
- DolphinScheduler Shell and native SQL tasks
- Local testing with the official Fivetran Connector SDK
- Airbyte connector protocol and destination acceptance
- NiFi Record flows and MatrixOne JDBC writes
- Kafka Connect, Debezium and JDBC CDC sinks
- Exact event loading with Redpanda Connect and kafka-python
- Read fidelity with ConnectorX, Ibis and database drivers
- dlt loading and Singer state-protocol adapters
- Job acceptance for DataX, SeaTunnel, Spark, Flink and Trino
- Native DataX source scans and exact-value writes
- Native Spark JDBC reads, writes, types and partitions
- Native Flink MySQL CDC into MatrixOne and recovery acceptance