Create task

Create unstructured or structured file import tasks via connectors. A successful response only returns the task ID; subsequent query details, files, and running records confirm the actual processing results.

This interface only applies to Connector Source. Local file upload uses multipart request, see Local upload and create import task; structured data in the database uses config_type: 3, see Create database structured load task.

POST https://moi.matrixorigin.cn/newmoi/task

Before you call

Before creating, confirm the import target:

  • When importing unstructured files, prepare a writable data volume, connector ID, and URI of the file or directory to be processed.

  • When importing structured files, the current creation page can only select CSV, XLS or XLSX from the connector. Prepare the connector ID, file URI, and target table; you can also create the target table in the specified database.

  • The connector must have an “import” purpose, and the caller should have access to the connector and the target resource.

The example below uses:

  • $AI_STUDIO_API_KEY: The actual personal access token, passed through the X-API-Key Header.

  • $WORKSPACE_ID: The workspace ID of the task to be created, passed through the X-Workspace-ID Header.

Request example

Import the connector file into the data volume

The following request reads a file from a directory in the connector and writes it to the data volume specified by <VOLUME_ID>. path_regex is used to match the connector source path and does not control the target directory within the data volume; successful task creation only means that the task has been accepted.

curl -X POST "https://moi.matrixorigin.cn/newmoi/task" \
  -H "X-API-Key: $AI_STUDIO_API_KEY" \
  -H "X-Workspace-ID: $WORKSPACE_ID" \
  -H 'Content-Type: application/json' \
  -d '{
    "source_connector_id":"<CONNECTOR_ID>",
    "name":"<TASK_NAME>",
    "config_type":1,
    "volume_id":"<VOLUME_ID>",
    "source_config":{
      "common_file_task_config":{
        "uris":["<CONNECTOR_DIRECTORY_URI>"],
        "load_mode_config":{"load_interval_type":4},
        "file_filter_config":{"path_regex":".*\\.(pdf|docx)$"},
        "unzip_keep_structure":false,
        "dedup":{"by":["name","md5"],"strategy":"skip"}
      }
    }
  }'

Connector CSV import into existing data table

The following request reads a CSV from the connector and writes the data to an existing table specified by <TARGET_TABLE_ID>. The target of structured file import is a data table, so volume_id is an empty string, and the target directory field is not provided.

curl -X POST "https://moi.matrixorigin.cn/newmoi/task" \
  -H "X-API-Key: $AI_STUDIO_API_KEY" \
  -H "X-Workspace-ID: $WORKSPACE_ID" \
  -H 'Content-Type: application/json' \
  -d '{
    "source_connector_id":"<CONNECTOR_ID>",
    "name":"<TASK_NAME>",
    "config_type":1,
    "volume_id":"",
    "source_config":{
      "common_file_task_config":{
        "uris":["<CONNECTOR_FILE_URI>"],
        "load_mode_config":{"load_interval_type":4},
        "table_config":{
          "new_table":false,
          "table_id":<TARGET_TABLE_ID>,
          "isColumnName":true,
          "columnNameRow":1,
          "rowStart":2,
          "csv":{"separator":","},
          "existed_table":[
            {"tableColumn":"<TARGET_COLUMN>","column":"<FILE_COLUMN>","col_num_in_file":1}
          ],
          "existed_table_opts":{"method":"append"}
        }
      }
    }
  }'

When creating the target table, set new_table to true, use database_id to specify the target database, and fill in create_table. The created table will not be automatically rolled back when the task is deleted.

Request body

Field

Type

Is it required

Description

source_connector_id

string or integer

Yes

The source connector ID.

name

string

Yes

The name of the task.

config_type

integer

yes

File import is fixed to 1.

source_config

object

Yes

File source and processing settings, must contain common_file_task_config.

volume_id

string

Condition required

Fill in the target data volume ID when importing unstructured files. The request has no target directory field and the file is written to the selected data volume; an empty string is left when structured files are written to the data table.

[] after a type denotes an array. [] in a field path denotes each item in an array.

The fields of source_config.common_file_task_config are as follows.

Field

Type

Is it required

Description

uris

string[]

Yes

The URI of the connector file or directory to be imported. Periodic import can only specify directories.

load_mode_config

object

Yes

Import cycle settings. Fill in {"load_interval_type":4} for one-time tasks.

load_mode_config.load_interval_type

integer

Yes

1 means daily, 2 means hourly, 3 means minute, 4 means one-time. Also fill in interval when in days, hours or minutes.

load_mode_config.interval

integer

Condition required

Interval for periodic import: fill in the interval value in minute and hour modes; fill in the execution hour (0 to 23) in daily mode.

file_filter_config

object

No

File filter conditions.

file_filter_config.filename_globs

string[]

No

Filename glob filter criteria.

file_filter_config.min_file_size

integer

No

Minimum file size, in bytes.

file_filter_config.max_file_size

integer

No

Maximum file size, in bytes.

file_filter_config.min_create_time

integer

No

Lower limit of file creation time.

file_filter_config.max_create_time

integer

No

Upper limit of file creation time.

file_filter_config.file_types

integer[]

No

File type codes allowed to be processed.

file_filter_config.path_regex

string

No

Source path regular expression.

unzip_keep_structure

boolean

No

Whether to keep the directory structure when decompressing. The creation page currently handles archives in a flat manner.

dedup

object

No

Duplicate file handling policy when writing to data volumes.

dedup.by

string[]

No

The basis for judging duplication: name, md5 or both.

dedup.strategy

string

no

Strategy when repeating: skip or overwrite.

table_config

object

Condition required

Settings for the target table for structured file import; do not fill in for unstructured file import.

multi_table_config

object

No

Table configuration for a multisheet structured file; contains multi_sheet:true and tables, each of which is a table_config object.

csv

object

No

File-level CSV parsing settings, fields are the same as table_config.csv.

table_config.sheet_name

string

No

The sheet name for XLS or XLSX.

table_config.new_table

boolean

Yes (structured)

Whether to create the target table.

table_config.table_id

integer

Condition required

The target table ID when writing to an existing table.

table_config.database_id

integer

Condition required

The target database ID when creating a new table.

table_config.conn_file_ids

string[]

No

ID of the uploaded temporary file. Connector sources typically use uris; local structured file processes use this field.

table_config.isColumnName

boolean

Yes (structured)

Whether to use the specified row as a column name.

table_config.columnNameRow

integer

Condition required

The row where the column name is located, starting from 1.

table_config.rowStart

integer

Yes (structured)

Data start row, starting from 1.

table_config.csv

object

No

CSV parsing settings.

table_config.csv.separator

string

No

CSV field separator.

table_config.csv.delimiter

string

No

CSV string delimiter.

table_config.csv.isEscape

boolean

No

Whether to enable escape processing.

table_config.conflict

integer

No

Import conflict handling code.

table_config.existed_table

object[]

Condition required

Column mapping when writing to an existing table. Each item contains tableColumn (target column), column (file column) and col_num_in_file (file column number).

table_config.existed_table_opts.method

string

No

The initial writing method of the existing table.

table_config.existed_table_opts.table_name

string

No

Table name already exists.

table_config.create_table

object

Condition required

Create a new table definition; contains name, description and tableColumn.

table_config.create_table.tableColumn

object[]

Condition required

New table column definition; each contains column, dataType, precision, isKey, defaultValue, description and col_num_in_file.

Successful response

When the code in the response is OK, the import task has been accepted, but it does not mean that the data has been imported.

{
  "code": "OK",
  "msg": "OK",
  "data": {
    "task_id": "task_01"
  }
}

The response fields are as follows.

Field

Type

Description

code

string

OK on success.

msg

string

OK on success.

data.task_id

string

New import task ID, used for subsequent query details, files and running records.

Error response

{
  "code": "ErrParamInvalid",
  "msg": "source_config is required",
  "data": null
}

Common HTTP errors

HTTP status code

error code

Common causes

Recommended actions

403

ErrForbidden

The caller does not have permission to create tasks or access dependent objects.

Check workspace, connector and target resource authorizations.

409

IAM_ERROR_CODE_IDEMPOTENCY_CONFLICT

Conflicting requests for the same IAM lifecycle.

Try again with a new request association ID.

503

ErrCoreAuthorizeUnavailable

The service is temporarily unable to complete the authorization check.

Try again later.

200

ErrParamInvalid

Import settings are missing, or the request JSON is invalid.

Correct the request fields according to the error response prompts and try again.

200

ErrServer

The service failed to create the task.

Check both HTTP status and code.

Follow-up operations

Log data.task_id. Successful creation only means that the task has been accepted, but does not mean that the data has been imported. Use this ID to Query task details to track progress and file statistics; if failure occurs, then Query task files or Query running records.

Last updated on