Import tasks

Import tasks do more than import CSV files. The creation pages provide different flows by data shape, source, and target. Select a flow from the following table, then call its API.

Data shape

Data source

Import target

Task creation method

Unstructured files

Files or directories in an object-storage or distributed-file-system connector

Data volume

Use Create a connector-file import task.

Unstructured files

Local files or directories

Data volume

Use Upload locally and create an import task.

Structured files

CSV, XLS, or XLSX files available on the current creation page

An existing data table, or a new data table in the specified database

Use Create a connector-file import task and provide table_config.

Structured files

Local CSV, XLS, or XLSX files available on the current creation page

An existing data table, or a new data table in the specified database

First upload files, then use Upload locally and create an import task to submit the table configuration.

Structured database data

Tables, collections, or other supported objects in a database connector

An existing data table, or a new data table in the specified database

Use Create a database structured-load task.

Confirm where data will be imported

  • Unstructured imports: The selected data volume is the final write location. Connector-based imports specify it with volume_id; local imports specify it with the multipart VolumeID field. The current request has no target-directory field: files are written to the selected data volume.

  • Structured-file imports: Data is written to the target table in table_config. For an existing table, provide table_id; to create a table, provide database_id, new_table: true, and create_table. A data volume is not the target location for data rows in this task type.

  • Structured database loads: Data is written to the target table in structured_load_config.target. The request cannot include volume_id and has no target-directory field.

A connector file or directory source path is specified in source_config.common_file_task_config.uris. In local uploads, the path in meta describes and filters the uploaded source path; it is not a target directory in the data volume. The current creation page does not offer Keep directory structure, so compressed packages are handled in a flattened form.

Check connector file types

Before creating a task, you can call List files. data.files[].type in the response is the file type recognized by the connector, and data.files[].uri can be used as the source URI for connector-file imports.

  • The current structured file picker allows only CSV (7), XLS (24), and XLSX (25).

  • The current unstructured file filter offers document, audio, video, image, web page, email, CAD, and other categories. Use a type value returned by the file list in file_filter_config.file_types or the local-upload file_types field.

File types indicate only that the type can be recognized from a file name. Whether a task completes still depends on the source file, connector, and import configuration. After submitting a task, get task details, task files, or run records to confirm the actual result.

Local-upload count and size limits

The following are limits of the current creation page’s local file picker. They are not a general promise about server-side processing for direct API calls.

Import type

Maximum selection per attempt

Single-file size

Notes

Unstructured files

Up to 20 files

Up to 200 MiB

You can select local files or a folder. Each file in a selected folder counts toward the 20-file limit.

Structured files

1 file

Up to 200 MiB

The current file picker allows only CSV, XLS, or XLSX.

Configurable options for unstructured imports

After selecting a data volume, unstructured imports can use the following options as needed.

Page feature

Request field

Effect

Connector source / local upload

/task JSON request or /connectors/upload multipart request

Determines whether files are read from a connector or uploaded from the local computer.

One-time / scheduled import

load_mode_config

A connector source can run once or scan a directory every minute, hour, or day. Scheduled tasks can select only directories.

File-type filter

file_filter_config.file_types or file_types

Select document, audio, video, image, web page, email, CAD, and other types. When Other is selected, files are not filtered by type.

Path regular expression

file_filter_config.path_regex or path_regex

Processes only files whose source paths match the regular expression.

Decompression strategy

unzip_keep_structure

The current page handles compressed packages in a flattened form.

Deduplication strategy

dedup.by, dedup.strategy

Detects duplicate files by file name and/or MD5, then skips or overwrites them.
Last updated on