Check Data Sources and Read Samples¶
Before configuring structured data imports, use the AI Studio SDK to check CDC capabilities and disabled reasons for MySQL or Doris connector tables, or read a small batch of sample messages from a Kafka Topic to verify fields and message structure.
Task flow¶
Pre-import checks include two read-only operations: inspecting relational table capabilities and sampling streaming messages. Both run against a configured connector ID and do not write data to destination tables.
Workspace Connector Service + Configured Connector ID
│
├── MySQL / Doris connector ──▶ Check table CDC capabilities and disabled reasons
│
└── Kafka connector + Topic ──▶ Read sample messages and next position
│
▼
Confirm read mode and message schema before import
Prerequisites¶
Complete the endpoint, personal access token, and workspace setup in Use the SDK.
Relational connector and table details (when checking CDC capabilities): obtain a configured MySQL or Doris connector ID (
RELATIONAL_CONNECTOR_ID) from Manage Connectors, and confirm the source type (SOURCE_TYPE), database name (SOURCE_DATABASE), table name (SOURCE_TABLE), and optional schema (SOURCE_SCHEMA).Kafka connector and Topic (when reading samples): obtain a configured Kafka connector ID (
CONNECTOR_ID) from Manage Connectors, and confirm a readableKAFKA_TOPICwith your Kafka administrator.
Check relational table capabilities¶
The source capability check applies to MySQL and Doris connectors, with SOURCE_TYPE set to SOURCE_TYPE_MYSQL or SOURCE_TYPE_DORIS. The call returns CDC availability flags and any disabled reasons so you can choose between full reads and incremental CDC before creating a structured import task.
import os
import moi_product_sdk as sdk
client = sdk.new_with_personal_access_token(
os.environ["PRODUCT_API_BASE_URL"],
os.environ["PRODUCT_API_KEY"],
)
workspace = client.workspace(os.environ["WORKSPACE_ID"])
service = workspace.connectors()
capability = service.structured_source_capability(
os.environ["RELATIONAL_CONNECTOR_ID"],
os.environ["SOURCE_TYPE"],
os.environ["SOURCE_DATABASE"],
os.environ["SOURCE_TABLE"],
os.environ.get("SOURCE_SCHEMA", ""),
)
print("MySQL CDC:", capability.mysql_cdc_available, "Doris CDC:", capability.doris_cdc_available)
print("Disabled reasons:", list(capability.disabled_reasons))
package main
import (
"context"
"fmt"
"os"
sdk "github.com/matrixorigin/matrixflow/sdk/go-sdk"
)
func main() {
ctx := context.Background()
client, err := sdk.NewWithPersonalAccessToken(
os.Getenv("PRODUCT_API_BASE_URL"),
os.Getenv("PRODUCT_API_KEY"),
)
if err != nil {
panic(err)
}
workspace, err := client.Workspace(os.Getenv("WORKSPACE_ID"))
if err != nil {
panic(err)
}
service := workspace.Connectors()
capability, err := service.StructuredSourceCapability(
ctx,
os.Getenv("RELATIONAL_CONNECTOR_ID"),
os.Getenv("SOURCE_TYPE"),
os.Getenv("SOURCE_DATABASE"),
os.Getenv("SOURCE_TABLE"),
os.Getenv("SOURCE_SCHEMA"),
)
if err != nil {
panic(err)
}
fmt.Println("MySQL CDC:", capability.MysqlCDCAvailable, "Doris CDC:", capability.DorisCDCAvailable)
fmt.Println("Disabled reasons:", capability.DisabledReasons)
}
Read Kafka Topic samples¶
Reuse the workspace connector service and supply the Kafka connector ID along with the target Kafka Topic to inspect sample messages. Set the source type to SOURCE_TYPE_KAFKA; the sample limit ranges from 1 to 100. The following example reads up to 5 messages from a single Kafka Topic and prints whether more messages exist along with the next read position.
sample = service.structured_source_sample(
os.environ["CONNECTOR_ID"],
"SOURCE_TYPE_KAFKA",
5,
"single",
[os.environ["KAFKA_TOPIC"]],
)
for row in sample.rows:
print(row)
print("More samples:", sample.has_more, "Next position:", sample.next_position)
sample, err := service.StructuredSourceSample(
ctx,
os.Getenv("CONNECTOR_ID"),
"SOURCE_TYPE_KAFKA",
5,
"single",
[]string{os.Getenv("KAFKA_TOPIC")},
"",
)
if err != nil {
panic(err)
}
for _, row := range sample.Rows {
fmt.Println(string(row))
}
fmt.Println("More samples:", sample.HasMore)
fmt.Println("Next position:", string(sample.NextPosition))
Capability checks and message sampling inspect source inputs only; confirm actual import completion through the corresponding import task and destination table records.