Check Data Sources and Read Samples

Before configuring structured data imports, use the AI Studio SDK to check CDC capabilities and disabled reasons for MySQL or Doris connector tables, or read a small batch of sample messages from a Kafka Topic to verify fields and message structure.

Task flow

Pre-import checks include two read-only operations: inspecting relational table capabilities and sampling streaming messages. Both run against a configured connector ID and do not write data to destination tables.

Workspace Connector Service + Configured Connector ID
                         │
                         ├── MySQL / Doris connector ──▶ Check table CDC capabilities and disabled reasons
                         │
                         └── Kafka connector + Topic ──▶ Read sample messages and next position
                                                                   │
                                                                   ▼
                                                Confirm read mode and message schema before import

Prerequisites

  1. Complete the endpoint, personal access token, and workspace setup in Use the SDK.

  2. Relational connector and table details (when checking CDC capabilities): obtain a configured MySQL or Doris connector ID (RELATIONAL_CONNECTOR_ID) from Manage Connectors, and confirm the source type (SOURCE_TYPE), database name (SOURCE_DATABASE), table name (SOURCE_TABLE), and optional schema (SOURCE_SCHEMA).

  3. Kafka connector and Topic (when reading samples): obtain a configured Kafka connector ID (CONNECTOR_ID) from Manage Connectors, and confirm a readable KAFKA_TOPIC with your Kafka administrator.

Check relational table capabilities

The source capability check applies to MySQL and Doris connectors, with SOURCE_TYPE set to SOURCE_TYPE_MYSQL or SOURCE_TYPE_DORIS. The call returns CDC availability flags and any disabled reasons so you can choose between full reads and incremental CDC before creating a structured import task.

import os

import moi_product_sdk as sdk

client = sdk.new_with_personal_access_token(
    os.environ["PRODUCT_API_BASE_URL"],
    os.environ["PRODUCT_API_KEY"],
)
workspace = client.workspace(os.environ["WORKSPACE_ID"])
service = workspace.connectors()

capability = service.structured_source_capability(
    os.environ["RELATIONAL_CONNECTOR_ID"],
    os.environ["SOURCE_TYPE"],
    os.environ["SOURCE_DATABASE"],
    os.environ["SOURCE_TABLE"],
    os.environ.get("SOURCE_SCHEMA", ""),
)
print("MySQL CDC:", capability.mysql_cdc_available, "Doris CDC:", capability.doris_cdc_available)
print("Disabled reasons:", list(capability.disabled_reasons))
package main

import (
	"context"
	"fmt"
	"os"

	sdk "github.com/matrixorigin/matrixflow/sdk/go-sdk"
)

func main() {
	ctx := context.Background()
	client, err := sdk.NewWithPersonalAccessToken(
		os.Getenv("PRODUCT_API_BASE_URL"),
		os.Getenv("PRODUCT_API_KEY"),
	)
	if err != nil {
		panic(err)
	}
	workspace, err := client.Workspace(os.Getenv("WORKSPACE_ID"))
	if err != nil {
		panic(err)
	}
	service := workspace.Connectors()

	capability, err := service.StructuredSourceCapability(
		ctx,
		os.Getenv("RELATIONAL_CONNECTOR_ID"),
		os.Getenv("SOURCE_TYPE"),
		os.Getenv("SOURCE_DATABASE"),
		os.Getenv("SOURCE_TABLE"),
		os.Getenv("SOURCE_SCHEMA"),
	)
	if err != nil {
		panic(err)
	}
	fmt.Println("MySQL CDC:", capability.MysqlCDCAvailable, "Doris CDC:", capability.DorisCDCAvailable)
	fmt.Println("Disabled reasons:", capability.DisabledReasons)
}

Read Kafka Topic samples

Reuse the workspace connector service and supply the Kafka connector ID along with the target Kafka Topic to inspect sample messages. Set the source type to SOURCE_TYPE_KAFKA; the sample limit ranges from 1 to 100. The following example reads up to 5 messages from a single Kafka Topic and prints whether more messages exist along with the next read position.

sample = service.structured_source_sample(
    os.environ["CONNECTOR_ID"],
    "SOURCE_TYPE_KAFKA",
    5,
    "single",
    [os.environ["KAFKA_TOPIC"]],
)
for row in sample.rows:
    print(row)
print("More samples:", sample.has_more, "Next position:", sample.next_position)
sample, err := service.StructuredSourceSample(
	ctx,
	os.Getenv("CONNECTOR_ID"),
	"SOURCE_TYPE_KAFKA",
	5,
	"single",
	[]string{os.Getenv("KAFKA_TOPIC")},
	"",
)
if err != nil {
	panic(err)
}
for _, row := range sample.Rows {
	fmt.Println(string(row))
}
fmt.Println("More samples:", sample.HasMore)
fmt.Println("Next position:", string(sample.NextPosition))

Capability checks and message sampling inspect source inputs only; confirm actual import completion through the corresponding import task and destination table records.

Next steps

Last updated on