Read and resume a Kafka partition¶
The local integration test starts a broker, produces deterministic messages, and checks an offset chain without gaps or duplicates and a transactional checkpoint:
make test-kafka-exttab-e2e-local
For a manual run, first prepare an orders topic on 127.0.0.1:9092 containing JSON objects with id and amount fields. In the same MySQL session, run:
CREATE DATABASE cookbook_kafka;
CREATE EXTERNAL TABLE cookbook_kafka.orders (id BIGINT, amount DECIMAL(12,2))
ENGINE = KAFKA WITH (
'brokers' = '127.0.0.1:9092', 'topic' = 'orders',
'partition' = '0', 'format' = 'jsonl', 'autocommit' = 'false'
);
SELECT id, amount, __mo_message_id FROM cookbook_kafka.orders
WHERE __mo_read_start_id = -1 AND __mo_read_size = 10 AND __mo_read_timeout = 3;
SELECT LAST_KAFKA_MESSAGE_ID();
Record the last offset N. Query again in the same session with __mo_read_start_id = N; the next batch should start at N+1. When copying messages into a target table, commit target rows and the checkpoint in one transaction; see the source Kafka E2E. Clean up with DROP DATABASE cookbook_kafka;; the test script stops its broker.