Upload locally and create an import task

Upload local files and create import tasks. This interface uses multipart/form-data; unlike the connector source’s JSON creation interface. A successful response returns the task ID but does not indicate that all files have been written to the target location or that all parsing has been completed.

POST https://moi.matrixorigin.cn/newmoi/connectors/upload

Before you call

First select the data form and target:

  • Unstructured files are written to the data volume. Prepare a writable data volume ID and specify it via VolumeID.

  • Structured File is written to the data table. Currently, only CSV, XLS or XLSX can be selected for the creation page. First obtain conn_file_id through Upload File, and then specify an existing target table or create a new target table through table_config. The data rows of the structured file are written to the table, not to the data volume specified by VolumeID.

The local file selection restrictions for the currently created page are as follows:

  • Unstructured files: select up to 20 files at a time, with a maximum size of 200 MiB for a single file. You can select files or folders; when you select a folder, each file in the folder counts toward the quantity.

  • Structured files: Only 1 file can be selected at a time, with a maximum size of 200 MiB per file; currently only CSV, XLS or XLSX are allowed.

The example below uses:

  • $AI_STUDIO_API_KEY: The actual personal access token, passed through the X-API-Key Header.

  • $WORKSPACE_ID: The workspace ID of the task to be created, passed through the X-Workspace-ID Header.

Request example

Import unstructured files into data volumes

The following example uploads local files to the <VOLUME_ID> data volume. Path regularization operates on meta.path, so you can use it to exclude unnecessary files; it does not control the target directory within the data volume.

curl -X POST "https://moi.matrixorigin.cn/newmoi/connectors/upload" \
  -H "X-API-Key: $AI_STUDIO_API_KEY" \
  -H "X-Workspace-ID: $WORKSPACE_ID" \
  -F 'file=@./manuals/product.pdf' \
  -F 'file=@./manuals/guide.docx' \
  -F 'VolumeID=<VOLUME_ID>' \
  -F 'meta=[{"filename":"product.pdf","path":"manuals/product.pdf"},{"filename":"guide.docx","path":"manuals/guide.docx"}]' \
  -F 'path_regex=^manuals/.*' \
  -F 'unzip_keep_structure=false' \
  -F 'dedup={"by":["name","md5"],"strategy":"skip"}'

Import local CSV into existing data table

First upload file obtains <CONN_FILE_ID>, and then submits the table configuration. The example uses row 1 as the table header, starts reading data from row 2, and maps the file columns to the existing target table.

curl -X POST "https://moi.matrixorigin.cn/newmoi/connectors/upload" \
  -H "X-API-Key: $AI_STUDIO_API_KEY" \
  -H "X-Workspace-ID: $WORKSPACE_ID" \
  -F 'meta=[{"filename":"orders.csv","path":"orders.csv"}]' \
  -F 'table_config={"new_table":false,"table_id":<TARGET_TABLE_ID>,"conn_file_ids":["<CONN_FILE_ID>"],"isColumnName":true,"columnNameRow":1,"rowStart":2,"csv":{"separator":","},"existed_table":[{"tableColumn":"<TARGET_COLUMN>","column":"<FILE_COLUMN>","col_num_in_file":1}],"existed_table_opts":{"method":"append"}}'

Form fields

Field

Type

Is it required

Description

file

file[]

Condition required

Upload one or more files during unstructured local import. Each file is passed using the field file with the same name. When the structured file process has been uploaded as a temporary file first, this field will not be uploaded again.

meta

JSON array

Yes

Source information corresponding to the order of uploaded files. Each item contains filename and path; path is used to record and match the source path, not the target data volume directory.

VolumeID

string

Conditional required

The target data volume ID for unstructured import. The request does not have a target directory field, and the file is written to this data volume; structured file import does not fill it out.

file_types

JSON integer array

No

File type codes allowed to be processed. The code is the same as type in Query file list. Does not filter by file type when 0 (other) is included.

path_regex

string

No

Only process files whose meta.path matches this regular expression.

unzip_keep_structure

boolean string

No

Whether to keep the directory structure when decompressing, for example false. The current creation page handles compressed packages in a flat manner.

dedup

JSON object

No

Duplicate file handling settings. Object contains by (name, md5 or both) and strategy (skip or overwrite).

table_config

JSON object

Condition required

Target table settings for structured file import. Can contain sheet_name, new_table, table_id, database_id, conn_file_ids, isColumnName, columnNameRow, rowStart, csv, conflict, existed_table, create_table and existed_table_opts. If there is an existing table, fill in new_table:false, table_id, header/starting row and existed_table column mapping; when creating a new table, fill in new_table:true, database_id and create_table. Use multi_sheet:true and tables when supporting multiple sheets. For the meaning of each field, see Create Connector File Import Task.

Successful response

The task has been accepted when the code in the response is OK.

{
  "code": "OK",
  "msg": "OK",
  "data": {
    "task_id": "task_01",
    "file_ids": ["file_01"],
    "success": true,
    "message": "OK",
    "results": [
      {"success": true, "message": "OK"}
    ]
  }
}

The response fields are as follows.

Field

Type

Description

code

string

OK on success.

msg

string

OK on success.

data.task_id

string

ID of the created import task.

data.file_ids

string[]

This newly uploaded and successfully processed file ID; may be empty in the structured temporary file process.

data.success

boolean

Whether the request was created successfully.

data.message

string

Processing result description.

data.results

object[]

Processing results returned by file. Each item contains success and message.

[] after a type denotes an array. [] in a field path denotes each item in an array.

Error response

{
  "code": "ErrParamInvalid",
  "msg": "invalid upload parameters",
  "data": null
}

Common HTTP errors

HTTP status code

error code

Common causes

Recommended actions

400

ErrParamInvalid

The multipart form is invalid, or VolumeID, table_config, and path regularity do not meet the requirements.

Check field names, JSON strings, target resource IDs, and regular expressions.

403

ErrForbidden

The caller does not have permission to write data to the target data volume or target table.

Check workspace, data volumes and data table authorizations.

200

ErrServer

The service failed to create or schedule the task.

Check HTTP status and code at the same time, retry later or query task status.

Follow-up operations

Log data.task_id. Use the ID to Query task details to confirm the target, status and processing statistics; if necessary, Query task files or Query running records.

Last updated on