Create task¶
Create unstructured or structured file import tasks via connectors. A successful response only returns the task ID; subsequent query details, files, and running records confirm the actual processing results.
This interface only applies to Connector Source. Local file upload uses multipart request, see Local upload and create import task; structured data in the database uses config_type: 3, see Create database structured load task.
POST https://moi.matrixorigin.cn/newmoi/task
Before you call¶
Before creating, confirm the import target:
When importing unstructured files, prepare a writable data volume, connector ID, and URI of the file or directory to be processed.
When importing structured files, the current creation page can only select CSV, XLS or XLSX from the connector. Prepare the connector ID, file URI, and target table; you can also create the target table in the specified database.
The connector must have an “import” purpose, and the caller should have access to the connector and the target resource.
The example below uses:
$AI_STUDIO_API_KEY: The actual personal access token, passed through theX-API-KeyHeader.$WORKSPACE_ID: The workspace ID of the task to be created, passed through theX-Workspace-IDHeader.
Request example¶
Import the connector file into the data volume¶
The following request reads a file from a directory in the connector and writes it to the data volume specified by <VOLUME_ID>. path_regex is used to match the connector source path and does not control the target directory within the data volume; successful task creation only means that the task has been accepted.
curl -X POST "https://moi.matrixorigin.cn/newmoi/task" \
-H "X-API-Key: $AI_STUDIO_API_KEY" \
-H "X-Workspace-ID: $WORKSPACE_ID" \
-H 'Content-Type: application/json' \
-d '{
"source_connector_id":"<CONNECTOR_ID>",
"name":"<TASK_NAME>",
"config_type":1,
"volume_id":"<VOLUME_ID>",
"source_config":{
"common_file_task_config":{
"uris":["<CONNECTOR_DIRECTORY_URI>"],
"load_mode_config":{"load_interval_type":4},
"file_filter_config":{"path_regex":".*\\.(pdf|docx)$"},
"unzip_keep_structure":false,
"dedup":{"by":["name","md5"],"strategy":"skip"}
}
}
}'
Connector CSV import into existing data table¶
The following request reads a CSV from the connector and writes the data to an existing table specified by <TARGET_TABLE_ID>. The target of structured file import is a data table, so volume_id is an empty string, and the target directory field is not provided.
curl -X POST "https://moi.matrixorigin.cn/newmoi/task" \
-H "X-API-Key: $AI_STUDIO_API_KEY" \
-H "X-Workspace-ID: $WORKSPACE_ID" \
-H 'Content-Type: application/json' \
-d '{
"source_connector_id":"<CONNECTOR_ID>",
"name":"<TASK_NAME>",
"config_type":1,
"volume_id":"",
"source_config":{
"common_file_task_config":{
"uris":["<CONNECTOR_FILE_URI>"],
"load_mode_config":{"load_interval_type":4},
"table_config":{
"new_table":false,
"table_id":<TARGET_TABLE_ID>,
"isColumnName":true,
"columnNameRow":1,
"rowStart":2,
"csv":{"separator":","},
"existed_table":[
{"tableColumn":"<TARGET_COLUMN>","column":"<FILE_COLUMN>","col_num_in_file":1}
],
"existed_table_opts":{"method":"append"}
}
}
}
}'
When creating the target table, set new_table to true, use database_id to specify the target database, and fill in create_table. The created table will not be automatically rolled back when the task is deleted.
Request body¶
Field |
Type |
Is it required |
Description |
|---|---|---|---|
|
string or integer |
Yes |
The source connector ID. |
|
string |
Yes |
The name of the task. |
|
integer |
yes |
File import is fixed to |
|
object |
Yes |
File source and processing settings, must contain |
|
string |
Condition required |
Fill in the target data volume ID when importing unstructured files. The request has no target directory field and the file is written to the selected data volume; an empty string is left when structured files are written to the data table. |
[] after a type denotes an array. [] in a field path denotes each item in an array.
The fields of source_config.common_file_task_config are as follows.
Field |
Type |
Is it required |
Description |
|---|---|---|---|
|
string[] |
Yes |
The URI of the connector file or directory to be imported. Periodic import can only specify directories. |
|
object |
Yes |
Import cycle settings. Fill in |
|
integer |
Yes |
|
|
integer |
Condition required |
Interval for periodic import: fill in the interval value in minute and hour modes; fill in the execution hour ( |
|
object |
No |
File filter conditions. |
|
string[] |
No |
Filename glob filter criteria. |
|
integer |
No |
Minimum file size, in bytes. |
|
integer |
No |
Maximum file size, in bytes. |
|
integer |
No |
Lower limit of file creation time. |
|
integer |
No |
Upper limit of file creation time. |
|
integer[] |
No |
File type codes allowed to be processed. |
|
string |
No |
Source path regular expression. |
|
boolean |
No |
Whether to keep the directory structure when decompressing. The creation page currently handles archives in a flat manner. |
|
object |
No |
Duplicate file handling policy when writing to data volumes. |
|
string[] |
No |
The basis for judging duplication: |
|
string |
no |
Strategy when repeating: |
|
object |
Condition required |
Settings for the target table for structured file import; do not fill in for unstructured file import. |
|
object |
No |
Table configuration for a multisheet structured file; contains |
|
object |
No |
File-level CSV parsing settings, fields are the same as |
|
string |
No |
The sheet name for XLS or XLSX. |
|
boolean |
Yes (structured) |
Whether to create the target table. |
|
integer |
Condition required |
The target table ID when writing to an existing table. |
|
integer |
Condition required |
The target database ID when creating a new table. |
|
string[] |
No |
ID of the uploaded temporary file. Connector sources typically use |
|
boolean |
Yes (structured) |
Whether to use the specified row as a column name. |
|
integer |
Condition required |
The row where the column name is located, starting from |
|
integer |
Yes (structured) |
Data start row, starting from |
|
object |
No |
CSV parsing settings. |
|
string |
No |
CSV field separator. |
|
string |
No |
CSV string delimiter. |
|
boolean |
No |
Whether to enable escape processing. |
|
integer |
No |
Import conflict handling code. |
|
object[] |
Condition required |
Column mapping when writing to an existing table. Each item contains |
|
string |
No |
The initial writing method of the existing table. |
|
string |
No |
Table name already exists. |
|
object |
Condition required |
Create a new table definition; contains |
|
object[] |
Condition required |
New table column definition; each contains |
Successful response¶
When the code in the response is OK, the import task has been accepted, but it does not mean that the data has been imported.
{
"code": "OK",
"msg": "OK",
"data": {
"task_id": "task_01"
}
}
The response fields are as follows.
Field |
Type |
Description |
|---|---|---|
|
string |
|
|
string |
|
|
string |
New import task ID, used for subsequent query details, files and running records. |
Error response¶
{
"code": "ErrParamInvalid",
"msg": "source_config is required",
"data": null
}
Common HTTP errors¶
HTTP status code |
error code |
Common causes |
Recommended actions |
|---|---|---|---|
|
|
The caller does not have permission to create tasks or access dependent objects. |
Check workspace, connector and target resource authorizations. |
|
|
Conflicting requests for the same IAM lifecycle. |
Try again with a new request association ID. |
|
|
The service is temporarily unable to complete the authorization check. |
Try again later. |
|
|
Import settings are missing, or the request JSON is invalid. |
Correct the request fields according to the error response prompts and try again. |
|
|
The service failed to create the task. |
Check both HTTP status and |
Follow-up operations¶
Log data.task_id. Successful creation only means that the task has been accepted, but does not mean that the data has been imported. Use this ID to Query task details to track progress and file statistics; if failure occurs, then Query task files or Query running records.