Import data

Import tasks write external files, database objects, or Kafka messages to Catalog. Unstructured files go to volumes; structured files, database objects, and Kafka messages go to tables. Select the data form, source, and destination first.

Load type

Data source

Object written to Catalog

Unstructured

Local files or files in OSS, S3, or HDFS connectors

Volumes

Structured files

Local CSV, XLS, XLSX, or equivalent files in OSS/S3/HDFS connectors

Table

Structured databases

Tables or collections in enabled, available database connectors

Table

Kafka

Topics in a configured connector

Table

Create an unstructured import task

Use this flow for documents, images, audio, and video.

  1. Open Data Connections > Import Tasks and select Unstructured.

  2. Select a connector source or local upload.

  3. For connector imports, select an OSS, S3, or HDFS connector with the import purpose.

  4. Select a destination Catalog volume, or create one in the picker.

  5. Configure file-handling rules.

  6. Select source files or upload local files or a folder.

  7. Click Create and start importing.

Configure file processing rules

Setting

Description

Import interval

Connectors support one-time or recurring imports; local uploads are one-time operations.

Extraction strategy

Ignore directory structure is currently available. Preserving directory structure is currently unavailable.

Deduplication rules

Identify duplicate files by filename, MD5, or both, then skip or overwrite duplicates.

Processing scope

Filter files using file types and path regexes. All types are selected by default; adjust document, audio, video, image, webpage, email, CAD, and other categories.

For recurring imports using a connector, select folders rather than individual files.

Periodic connector imports require folders rather than individual files. Local uploads allow up to 20 files, each no larger than 200 MB. Use Select folder to upload files from a directory.

Create a structured file import task

Structured file import supports CSV, XLS, and XLSX. Each task selects one source file and writes parsed data to one or more Catalog tables.

  1. Open Data Connections > Import Tasks, select Structured, and keep the File type.

  2. Select a connector source or local upload.

  3. Select or upload a CSV, XLS, or XLSX file.

  4. Under the destination settings, select an existing table or a new table for the file or each selected Excel sheet.

  5. Review parsing and column settings.

  6. Click Create and start importing.

Local structured uploads accept one file at a time, no larger than 200 MB.

Configure file parsing

After selecting the file, inspect its preview and settings:

  • For CSV files, configure the column separator, text delimiter, and escaping;

  • Set whether to use column names, the header row, and the first data row.

  • For Excel, select one or more worksheets and configure each destination separately.

  • For a new table, check its name, description, columns, types, primary key, defaults, and column descriptions.

  • For an existing table, map columns by position or name, then choose append or clear-and-append for the initial import.

  • When the destination has a primary key, choose whether conflicts fail, skip the conflicting row, or replace it.

Create a structured database import task

This flow writes external database tables or MongoDB collections to Catalog tables. Database is available only when the workspace enables the feature and an appropriate connector is available.

  1. Open Data Connections > Import Tasks, select Structured, then Database.

  2. Select the connector, database, and schema when required.

  3. Select the source table or collection.

  4. Select an existing destination table, or choose a database and create a table.

  5. Check source fields, destination fields, and type mappings.

  6. Choose a one-time or periodic import.

  7. For periodic imports, choose incremental synchronization or full overwrite and complete the schedule.

  8. Configure the initial import and primary-key conflict policy.

  9. Click Create and start importing.

Clear-and-append applies only to existing tables. Replacing conflicting rows requires at least one retained primary-key mapping. When importing from an Apache Doris connector, the UI validates available read methods (High-speed read, Parallel read, or Standard read); realtime sync (Snapshot + CDC, starting from Latest or a chosen timestamp) becomes selectable only when the source table passes ROW Binlog, historical value, primary/unique key, and Flight SQL capability checks.

Import tasks currently cover local uploads, OSS/S3/HDFS files, supported relational/Doris/MongoDB databases, Kafka topics, and REST APIs. Web crawling is not an independent import-task type (use workflow parser nodes when crawling webpages).

View and manage import tasks

Open Data Connections > Import Tasks to inspect type, source, destination, schedule, status, failure reason, and times. Open a task ID for details.

Task

Suitable case

Pause

Running tasks.

Resume

A paused task.

Edit

A paused task that the page permits editing.

Retry failed files

When task details show failed files, click Retry, select those files, and confirm.

Delete

Nonrunning tasks.

Periodic structured database imports support pause and resume; one-time database imports do not show these controls.

Creation confirms submission. Inspect status in the list or details, then check destination data in Catalog after completion.

Investigate import failures

Symptom

Next step

Connection or authentication failed

Open the Connector in use, check parameters, and run Test connection again.

Cannot find the source file, table, or collection

Check connectors, paths, databases, Schemas, and object selection.

File parsing or column mapping fails

Read the failure reason in task details, then check file format, parsing settings, and destination column mappings.

Destination write failed

Check destination volume/table privileges, table names, and initial import rules.

Import a Kafka topic

Kafka connections and consumer tasks are configured separately. See Import data from Kafka for topic discovery, destination mappings, and validation. A topic does not provide every schema-query capability of a database table.

Recover uploads and scheduled imports

Follow upload progress and wait when creation is disabled. Upload completion does not establish that destination writes have completed.

For MySQL requiring SSL, use source-compatible connection settings. Test permissions and character data with a small import. After periodic import failure, preserve the task ID, destination, and error. Authorized maintenance may be needed to resolve server-side stale locks. Inspect subsequent imports and existing data afterward; do not delete destination tables or edit internal locks as a recovery procedure.

Last updated on