Data Loading¶

Data loading writes external data or local files to the Catalog, creating data assets that can be reused by queries, workflows, and knowledge bases. Structured records are loaded into tables; unstructured files such as documents, images, audio, and video are loaded into volumes. First create an appropriate connector (for the connection fields of each type, see Reference · Connectors), and then select a loading method based on the data format and source.

Preparation¶

Confirm the data format and target location: structured data goes into tables, and unstructured data goes into volumes.
Connector sources must support the “Load” purpose. Local uploads and web collection do not require a connector to be created in advance.
Confirm that you have access to the source data and the target Catalog directory/database/table/volume.
For data synchronized over the long term, prepare the incremental field, initial historical range, synchronization frequency, and a read window that the source can tolerate.
Select compute resources. Structured loading typically uses both task compute resources and query compute resources; unstructured loading uses task compute resources.

Go to Data Connections > Data Loading, click Load Data, and select structured or unstructured data. Then configure the source, target, policy, and runtime resources as described below.

Data Types and Sources¶

Source Type	Typical Data Sources	Structured Loading (Target: Table)	Unstructured Loading (Target: Volume)
Object storage and HDFS	OSS, S3 / MinIO, COS, OBS, HDFS	Supports CSV / XLS / XLSX	Supports documents and other files
File protocols	FTP, SFTP, SharePoint, SMB / NAS	—	Supported
Databases	MySQL, PostgreSQL, MatrixOne, MongoDB, Oracle, SQL Server, ClickHouse, Doris, Hive	Supported	—
Message queues	Kafka, RabbitMQ, MQTT	Supports real-time loading	—
Enterprise applications and collaboration platforms	SAP, Salesforce, Lark, GitHub, Slack, etc.	Supports loading business objects, messages, or metadata	—
Email	Gmail, Outlook, IMAP, Microsoft 365 Graph, etc.	Supports loading email records into tables	Supports loading emails and attachments into volumes
APIs	REST API, GraphQL	Supports loading records returned by an API into tables	—
Local uploads	CSV, Excel, documents, images, audio, video, etc.	Supports CSV / XLS / XLSX	Supports multiple file types
Web collection	Web page body content, links, and attachments	—	Supported

The capabilities actually available vary with the connector type and product interface. After you select a source, the interface displays only the loading options and data selection controls available for that source.

Spreadsheet and Database Data¶

Structured loading applies to CSV, Excel, database tables, records returned by APIs, enterprise business objects, collaboration platform metadata, and message queue events. The target must be a table in the Catalog.

Data Source¶

You can select a connector or local upload:

Object storage / HDFS: Select a CSV, XLS, or XLSX file in the file browser.
Database: Select the database and table in sequence. For MongoDB, select the database and collection.
REST API / GraphQL, enterprise applications, and collaboration platforms: Select an endpoint, business object, repository, channel, or knowledge base scope, and sample the data first to confirm the returned fields.
Message queue: Select a topic or queue and continuously read records in real-time mode.
Local upload: Upload a CSV or Excel file.

Target Table¶

Use the Catalog selector to determine the loading location:

Target Mode	Use Case	Key Configuration
New table	No target table exists yet, or you want to generate a new table from source fields	Select a directory and database, and define the table name, description, column names, types, primary key, and default values
Existing table	The target table already exists and data needs to be appended, overwritten, or synchronized	Select the directory, database, and table, and map source columns to target columns

When using an existing table, you can typically select “Append after existing data” or “Clear existing data and then append” as the initial loading rule. For tables with a primary key, also select a conflict handling policy: Fail Import, Skip Conflicting Rows, or Replace Conflicting Rows.

File Parsing and Column Mapping¶

Before writing CSV and Excel files to a table, confirm the parsing and mapping:

For CSV, you can set the separator, delimiter, and backslash escaping, and use the preview to verify that headers and data rows are recognized correctly.
For files, you can specify whether to use a header as column names, the row containing the header, and the row where the actual import begins.
Excel supports selecting worksheets. When selecting multiple worksheets at once, select a new or existing table and the corresponding column mappings separately for each sheet.
For mapping, you can use intelligent recommendations or positional mapping and then confirm the result manually. Each source column can map to only one target column; every primary key column in the target table must have a source column.

Synchronization Modes, Incremental Loading, and Backfill¶

Database sources typically support one-time loading or real-time updates; message queues continuously ingest data in real time. Real-time synchronization generally uses a watermark to continuously retrieve new or changed records, and allows you to set an incremental field and lookback window.

When first establishing long-running synchronization, you can enable data backfill to load historical data before the watermark. The system reads from the specified historical starting point in batches and switches to regular synchronization after catching up with current data. You can configure the backfill start time, batch size, interval, and off-peak execution window during creation or in Load Tasks.

Note A full overwrite rereads the source and overwrites the target table every time. It is suitable for small dictionaries or enumeration data. To continuously load historical data and incremental updates, use incremental synchronization together with data backfill.

Documents, Images, Audio, and Video¶

Unstructured loading writes files to Catalog volumes, providing a data foundation for subsequent parsing, workflow processing, or knowledge bases. Sources can include object storage, HDFS, FTP, SFTP, SharePoint, SMB / NAS, email connectors, or local uploads.

Loading Methods and Targets¶

Method	Operation
Connector loading	After selecting a connector, select files and folders in the source browser. You can use breadcrumbs to enter subdirectories and select multiple items
Local upload	Select or drag local files. This is suitable for temporary ingestion or small batches of materials
Web collection	Switch to web collection, and configure the URL, collection areas, and rules. For details, see Web Collection

When selecting a target, browse or create a location by directory → database → volume. Unstructured data cannot be loaded into tables, and volumes should not be used in place of tables for structured data.

Loading Policy¶

Configuration	Description
Loading mode	Load once, or load continuously on a schedule in minutes, hours, days, or other intervals. For daily or longer intervals, you can set a run time
Extraction policy	Preserve the original directory structure, or ignore the directory structure and store files in a flat layout
Duplicate file handling	Detect duplicates by file name and MD5. The handling method is typically skip or overwrite
File types	Select the types of documents, images, audio, video, spreadsheets, and other files to process
Path rules	Use path rules to limit the scope loaded from a connector. These rules apply together with file type conditions

For connector sources, you can also view the name, size, and type in the file selection area, and confirm the number of selected files and folders. After creation, view processing progress, loaded files, and failure information in Load Tasks.

Email¶

Email connectors can write email records to tables as structured data, or write emails and attachments to volumes as unstructured files:

Target	Key Configuration
Table	Select a mailbox and folder, and synchronize email fields as records. This is suitable for search, statistics, or association with other business tables
Volume	Read emails and attachments within an email time window. This is suitable for processing attachments as documents

The email connector handles account authorization and connectivity; the read scope is determined in the load task. Common time windows include the last 7 days, the last 1/3/6 months, all emails, or a custom start and end date. Scheduled loading starts with the initial time range and subsequently reads new emails incrementally.

Web Collection¶

Web collection is a form of unstructured loading suitable for websites composed of announcement sites, list pages, detail pages, and attachments. It saves collected results to a Catalog volume.

On the unstructured loading page, switch to Web Collection, and select the target volume and update method (incremental update or full overwrite).
Enter the task name and web page address, and load the page preview. You can use a site template provided by the product as a starting point.
In the preview, select content areas to collect, file downloads, pagination controls, search areas, query buttons, areas to exclude, or links to follow.
Review the generated collection rules and confirm that pages, attachments, pagination, and navigation to detail pages work as expected.
Set the collection depth, automatic pagination, content to collect, request interval, and concurrency, then select task resources, create the task, and start loading.

Recommendation When collecting public sites, prefer low concurrency and an appropriate request interval, and comply with the target website’s terms of access and robots/anti-crawling restrictions. Validate the rules with a small number of pages before expanding the scope.

Local Uploads¶

Local uploads do not depend on connectors and are suitable for temporary ingestion or small batches:

When uploading CSV, XLS, or XLSX files, use the structured loading flow, select or create a target table, and complete the parsing, schema, and mapping configuration.
When uploading PDF, Word, TXT, images, audio, video, and other files, use the unstructured loading flow, select a target volume, and configure the file scope, deduplication, and resources.

For production data that requires long-term, scheduled synchronization, prefer a connector over repeated manual uploads.

Runtime Resources and Validation¶

Task Type	Compute Resources
Structured loading	Task compute resources handle retrieval, parsing, and mapping; query compute resources write to MOI tables
Unstructured loading and web collection	Task compute resources handle reading, downloading, extraction, deduplication, and writing to volumes

You can start with the platform’s default elastic resources, and then select dedicated resources and high, medium, or low priority based on data volume and timeliness requirements. After creating a task, confirm its status, target location, latest run, and error information in Load Tasks. For structured data, inspect the table schema and sample data. For unstructured data, inspect the files and parsing results in the volume.