Import data¶
Import tasks write external files, database objects, or Kafka messages to Catalog. Unstructured files go to volumes; structured files, database objects, and Kafka messages go to tables. Select the data form, source, and destination first.
Load type |
Data source |
Object written to Catalog |
|---|---|---|
Unstructured |
Local files or files in OSS, S3, or HDFS connectors |
Volumes |
Structured files |
Local |
Table |
Structured databases |
Tables or collections in enabled, available database connectors |
Table |
Kafka |
Topics in a configured connector |
Table |
Create an unstructured import task¶
Use this flow for documents, images, audio, and video.
Open Data Connections > Import Tasks and select Unstructured.
Select a connector source or local upload.
For connector imports, select an OSS, S3, or HDFS connector with the import purpose.
Select a destination Catalog volume, or create one in the picker.
Configure file-handling rules.
Select source files or upload local files or a folder.
Click Create and start importing.
Configure file processing rules¶
Setting |
Description |
|---|---|
Import interval |
Connectors support one-time or recurring imports; local uploads are one-time operations. |
Extraction strategy |
Ignore directory structure is currently available. Preserving directory structure is currently unavailable. |
Deduplication rules |
Identify duplicate files by filename, MD5, or both, then skip or overwrite duplicates. |
Processing scope |
Filter files using file types and path regexes. All types are selected by default; adjust document, audio, video, image, webpage, email, CAD, and other categories. |
For recurring imports using a connector, select folders rather than individual files.
Periodic connector imports require folders rather than individual files. Local uploads allow up to 20 files, each no larger than 200 MB. Use Select folder to upload files from a directory.
Create a structured file import task¶
Structured file import supports CSV, XLS, and XLSX. Each task selects one source file and writes parsed data to one or more Catalog tables.
Open Data Connections > Import Tasks, select Structured, and keep the File type.
Select a connector source or local upload.
Select or upload a
CSV,XLS, orXLSXfile.Under the destination settings, select an existing table or a new table for the file or each selected Excel sheet.
Review parsing and column settings.
Click Create and start importing.
Local structured uploads accept one file at a time, no larger than 200 MB.
Configure file parsing¶
After selecting the file, inspect its preview and settings:
For
CSVfiles, configure the column separator, text delimiter, and escaping;Set whether to use column names, the header row, and the first data row.
For Excel, select one or more worksheets and configure each destination separately.
For a new table, check its name, description, columns, types, primary key, defaults, and column descriptions.
For an existing table, map columns by position or name, then choose append or clear-and-append for the initial import.
When the destination has a primary key, choose whether conflicts fail, skip the conflicting row, or replace it.
Create a structured database import task¶
This flow writes external database tables or MongoDB collections to Catalog tables. Database is available only when the workspace enables the feature and an appropriate connector is available.
Open Data Connections > Import Tasks, select Structured, then Database.
Select the connector, database, and schema when required.
Select the source table or collection.
Select an existing destination table, or choose a database and create a table.
Check source fields, destination fields, and type mappings.
Choose a one-time or periodic import.
For periodic imports, choose incremental synchronization or full overwrite and complete the schedule.
Configure the initial import and primary-key conflict policy.
Click Create and start importing.
Clear-and-append applies only to existing tables. Replacing conflicting rows requires at least one retained primary-key mapping. When importing from an Apache Doris connector, the UI validates available read methods (High-speed read, Parallel read, or Standard read); realtime sync (Snapshot + CDC, starting from Latest or a chosen timestamp) becomes selectable only when the source table passes ROW Binlog, historical value, primary/unique key, and Flight SQL capability checks.
Import tasks currently cover local uploads, OSS/S3/HDFS files, supported relational/Doris/MongoDB databases, Kafka topics, and REST APIs. Web crawling is not an independent import-task type (use workflow parser nodes when crawling webpages).
View and manage import tasks¶
Open Data Connections > Import Tasks to inspect type, source, destination, schedule, status, failure reason, and times. Open a task ID for details.
Task |
Suitable case |
|---|---|
Pause |
Running tasks. |
Resume |
A paused task. |
Edit |
A paused task that the page permits editing. |
Retry failed files |
When task details show failed files, click Retry, select those files, and confirm. |
Delete |
Nonrunning tasks. |
Periodic structured database imports support pause and resume; one-time database imports do not show these controls.
Creation confirms submission. Inspect status in the list or details, then check destination data in Catalog after completion.
Investigate import failures¶
Symptom |
Next step |
|---|---|
Connection or authentication failed |
Open the Connector in use, check parameters, and run Test connection again. |
Cannot find the source file, table, or collection |
Check connectors, paths, databases, Schemas, and object selection. |
File parsing or column mapping fails |
Read the failure reason in task details, then check file format, parsing settings, and destination column mappings. |
Destination write failed |
Check destination volume/table privileges, table names, and initial import rules. |
Import a Kafka topic¶
Kafka connections and consumer tasks are configured separately. See Import data from Kafka for topic discovery, destination mappings, and validation. A topic does not provide every schema-query capability of a database table.
Recover uploads and scheduled imports¶
Follow upload progress and wait when creation is disabled. Upload completion does not establish that destination writes have completed.
For MySQL requiring SSL, use source-compatible connection settings. Test permissions and character data with a small import. After periodic import failure, preserve the task ID, destination, and error. Authorized maintenance may be needed to resolve server-side stale locks. Inspect subsequent imports and existing data afterward; do not delete destination tables or edit internal locks as a recovery procedure.