Data access related API

Connector

Create a Connector

POST /conectors

Input parameters:

Parameters	Required	Meaning
name	Yes	Connector name
source_type	Yes	Connector type, 4 is OSS, 5 is standard S3
config	Yes	Configure the connection information of oss and s3. If it is oss, you need to fill in endpoint, access_key_id, access_key_secret, and bucket_name. If it is s3, in addition to the parameters in oss, you need to fill in region and path_style (path style) information.

body:

{
    "name": "new_connector_name",
    "source_type": 4,
    "config": {
        "oss": { // If source_type is 4, fill in this field
            "endpoint": "example_oss_endpoint",
            "access_key_id": "example_access_key_id",
            "access_key_secret": "example_access_key_secret",
            "bucket_name": "example_bucket_name"
        },
        "s3": { // If source_type is 5, fill in this field
            "endpoint": "example_s3_endpoint",
            "access_key_id": "example_access_key_id",
            "access_key_secret": "example_access_key_secret",
            "bucket_name": "example_bucket_name",
            "region": "example_region",
            "path_style": true //Path style, true is path style, false is virtual hoste
        }
    }
}

Example:

import requests

url = "https://freetier-01.cn-hangzhou.cluster.cn-dev.matrixone.tech/connectors" # Please replace it with the actual API address
headers = {
    "user-id":"0194dfaa-3eda-7ea5-b47c-b4f4f594xxxx",
    "Access-Token": "xxxx",
    "uid": "91a2234e-0a06-4d84-815e-840e379d9e1c-0194dfaa-3eda-7ea5-b47c-b4f4f594xxxx:admin:accountadmin"
}

body = {
    "name": "oss-test2",
    "source_type": 4,
    "config": {
        "oss": {
            "endpoint": "xxxx",
            "access_key_id": "xxxx",
            "access_key_secret": "xxxx",
            "bucket_name": "xxxx"
        }
    }
}

response = requests.post(url, json=body, headers=headers)

if response.status_code == 200:
    print(response.json())
else:
    print(f"Request failed, status code: {response.status_code}, error message: {response.text}")

return:

{'code': 'OK', 'msg': 'OK'}

Verify the connector

POST /connectors/validate

Input parameters:

Parameters	Required	Meaning
source_type	Yes	Connector type, 4 is OSS, 5 is standard S3
connector_id	No	If you fill in connector_id, you don’t need to fill in the config information.
config	No	If you fill in config, you do not need to fill in the connection information of the connector_id information configuration oss and s3. If it is oss, you need to fill in endpoint, access_key_id, access_key_secret, bucket_name. If it is s3, in addition to the parameters in oss, you also need to fill in region and path_style information.

Example:

import requests

url = "https://freetier-01.cn-hangzhou.cluster.cn-dev.matrixone.tech/connectors/validate"
headers = {
    "user-id":"0194dfaa-3eda-7ea5-b47c-b4f4f594xxxx",
    "Access-Token": "xxxx",
    "uid": "2c5a7dc4-032a-42ac-b5ad-4db47f11772b-0194dfaa-3eda-7ea5-b47c-b4f4f594xxxx:admin:accountadmin"
}

body = {
    "source_type": 4,
    "config": {
        "oss": {
            "endpoint": "xxxx",
            "access_key_id": "xxxx",
            "access_key_secret": "xxxx",
            "bucket_name": "xxxx"
        }
    }
}

response = requests.post(url, json=body, headers=headers)

if response.status_code == 200:
    print(response.json())
else:
    print(f"Request failed, status code: {response.status_code}, error message: {response.text}")

Return to the example:

{'code': 'OK', 'msg': 'OK', 'data': {'valid': True}}

Query Connector

GET /connectors/list

Output parameters:

Parameters	Meaning
id	connector-id
source_type	Connector type, 4 is OSS, 5 is standard S3
name	connector name
status	Connection status
created_at	Created time
updated_at	Update time
username	Created by
updated_at	Update time
username	Created by
related_task_ids	associated TaskID
config	Connector configuration information
total	Return number

Example:

import requests
import json

url = "https://freetier-01.cn-hangzhou.cluster.cn-dev.matrixone.tech/connectors/list"
headers = {
    "user-id":"xxxx",
    "uid": "f42d1006-b48f-4b1d-bd44-3a18dd7bb8ec-0194dfaa-3eda-7ea5-b47c-b4f4f594xxxx:admin:accountadmin"
}


response = requests.get(url, headers=headers)

print("Response Body:", json.dumps(response.json(), indent=4, ensure_ascii=False))

return:

Response Body: {
    "code": "OK",
    "msg": "OK",
    "data": {
        "connectors": [
            {
                "id": 100004,
                "source_type": 4,
                "name": "oss-test1",
                "status": "active",
                "created_at": 1738919558,
                "updated_at": 1738919558,
                "username": "admin",
                "related_task_ids": [
                    1889223922712281088
                ],
                "config": {
                    "oss": {
                        "endpoint": "xxxx",
                        "access_key_id": "xxxx","access_key_secret": "xxxx",
                        "bucket_name": "xxxx"
                    }
                }
            },
            {
                "id": 100005,
                "source_type": 5,
                "name": "s3-test",
                "status": "active",
                "created_at": 1738919761,
                "updated_at": 1738919761,
                "username": "admin",
                "related_task_ids": null,
                "config": {
                    "s3": {
                        "endpoint": "xxxx",
                        "access_key_id": "xxxx",
                        "access_key_secret": "xxxx",
                        "bucket_name": "xxxx",
                        "region": "xxxx"
                    }
                }
            }
        ],
        "total": 2
    }
}

Update the connector

PUT /connectors/{id}

Input parameters:

Parameters	Required	Meaning
name	Yes	Connector name
config	Yes	Connector Configuration Information

Example:

import requests
import json

url = "https://freetier-01.cn-hangzhou.cluster.cn-dev.matrixone.tech/connectors/100005"

headers = {
    "user-id":"0194dfaa-3eda-7ea5-b47c-b4f4f594xxxx",
    "Access-Token": "xxxx",
    "uid": "f42d1006-b48f-4b1d-bd44-3a18dd7bb8ec-0194dfaa-3eda-7ea5-b47c-b4f4f594xxxx:admin:accountadmin"
}

body = {
    "name": "s3-test1",
    "config": {
                    "s3": {
                        "endpoint": "xxxx",
                        "access_key_id": "xxxx",
                        "access_key_secret": "xxxx",
                        "bucket_name": "xxxx",
                        "region": "xxxx"
                    }

}
}

response = requests.put(url, json=body, headers=headers)

if response.status_code == 200:
    print(response.json())
else:
    print(f"Request failed, status code: {response.status_code}, error message: {response.text}")

Query the connector source file

GET /connectors/files/list

Input parameters:

Parameters	Required	Meaning
connector_id	Yes	Connector id
file_types	No	File type, 0: empty file type (may be used as default or invalid value); 1: TXT text file type; 2: PDF document file type 3: picture file type; 4: PPT presentation file type; 5: Word document file type; 6: Markdown markup language file type; 7: CSV comma separated value file type; 8: Parquet columnar storage file type; 9: SQL file type; 10: directory type

Output parameters:

Parameters	Meaning
uri	represents the unique resource locator in the source connector.
filename	filename
size	File size
type	File type, 0: Empty file type (may be used as default or invalid value); 1: TXT text file type; 2: PDF document file type 3: Picture file type; 4: PPT presentation file type; 5: Word document file type; 6: Markdown markup language file type; 7: CSV comma separated value file type; 8: Parquet columnar storage file type; 9: SQL file type; 10: Directory type.

Example:

import requests
import json

url = "https://freetier-01.cn-hangzhou.cluster.cn-dev.matrixone.tech/connectors/files/list"

headers = {
    "user-id": "<your_workspace_id>",
    "Access-Token": "<your_access_token>",
    "uid": "<your_uid>",
}

params = {
    "connector_id": 100004,
    "file_types": 2
}

response = requests.get(url, headers=headers, params=params)

try:
    response_json = response.json()
    print("Response Body:", json.dumps(response_json, indent=4, ensure_ascii=False))
except json.JSONDecodeError:
    print("Response is not in JSON format:", response.text)

return:

Response Body: {
    "code": "OK",
    "msg": "OK",
    "data": {
        "files": [
            {
                "uri": "Dream of the Red Chamber (Public Book) Simplified Chinese horizontal row.pdf",
                "filename": "Dream of the Red Chamber (Public Book) Simplified Chinese horizontal row.pdf",
                "size": 6787457,
                "type": 2
            },
            {
                "uri": "file1",
                "filename": "file1",
                "size": 0,
                "type": 10
            }
        ]
    }
}

Data loading

Create a load task

POST /task

Input parameters:

Parameters	Required	Meaning
source_connector_id	Yes	Connector Name
volume_id	Yes	id of the original volume to be loaded
source_config	Yes	Configure for loading task source
common_file_task_config	Yes	Configure for common files
load_mode_config	Yes	Load mode setting, interval is the load cycle, load_interval_type represents the load cycle unit and type, 0: unknown load interval type, can be used as the default invalid value); 1: load by day, which may represent a fixed time load per day; 2: load by hour, which may represent a fixed time load per hour 3: load by minute, which may represent a fixed time load per minute; 4: default type, which is loaded only once.
uris	Yes	File to load
config_type	Yes	Indicates the common file load configuration type, default is 1

Example:

import requests
import json

url = "https://freetier-01.cn-hangzhou.cluster.cn-dev.matrixone.tech/task"
headers = {
    "user-id":"xxxx",
    "Access-Token": "xxxx",
    "uid": "f42d1006-b48f-4b1d-bd44-3a18dd7bb8ec-0194dfaa-3eda-7ea5-b47c-b4f4f594xxxx:admin:accountadmin"
}

body = {
    "source_connector_id":100004,
    "volume_id":"1889578498228068352",
    "source_config":
        {"common_file_task_config":
             {
                 "load_mode_config":
                  {
                      "load_interval_type":4
                   },
              "uris":["Dream of the Red Chamber (Public Book) Simplified Chinese horizontal row.pdf"]
             }
        },
    "config_type":1
}

response = requests.post(url, json=body, headers=headers)
print(response.json())

return:

{'code': 'OK', 'msg': 'OK'}

Load the task list

GET /task/list

Input parameters:

Parameters	Required	Meaning
is_desc	No	Sort
load_interval_types	No	Load interval, 0: unknown interval type; 1: by day; 2: by hour; 3: by minute; 4: only once
order_by	No	Configure for loading task source
page	No	Page
page_size	No	When the page number
status	No	Indicates the common file load configuration type, default is 1

Output parameters:

Parameters	Meaning
id	Load id
source_connector_id	connector id
source_connector_type	Connector type
volume_id	original volume id
volume_name	original volume name
status	Loading task status, 0: unknown state; 1: normal execution or executable state; 2: pausing state; 3: paused state; 4: completed state.
creator	Created by
source_config	Load configuration
start_at	Loading time
end_at	end time
created_at	Created time
updated_at	Update time
total	Return quantity

Example:

import requests
import json

url = "https://freetier-01.cn-hangzhou.cluster.cn-dev.matrixone.tech/task/list"

headers = {
    "user-id": "0194dfaa-3eda-7ea5-b47c-b4f4f594xxxx",
    "Access-Token": "xxxx",
    "uid": "cef0d7a2-1505-4384-a89d-5ee5b8246f95-0194dfaa-3eda-7ea5-b47c-b4f4f594xxxx:admin:accountadmin",
}

response = requests.get(url, headers=headers)

print("Response Body:", json.dumps(response.json(), indent=4, ensure_ascii=False))

return:

Response Body: {
    "code": "OK",
    "msg": "OK",
    "data": {
        "tasks": [
            {
                "id": "1889223922712281088",
                "source_connector_id": 100004,
                "source_connector_type": 4,
                "volume_id": "1889223879880048640",
                "volume_name": "b-vol1",
                "name": "",
                "status": 4,
                "creator": "admin",
                "source_config": {
                    "common_file_task_config": {
                        "uris": [
                            "Dream of the Red Chamber (Public Book) Simplified Chinese Horizontal.pdf"
                        ],
                        "load_mode_config": {
                            "load_interval_type": 4,
                            "interval": 0
                        }
                    }
                },
                "start_at": 1739261063,
                "end_at": 1739261640,
                "created_at": 1739261057,
                "updated_at": 1739261640
            }
        ],
        "total": 1
    }
}

Load task update

POST /task/update

Input parameters:

Parameters	Required	Meaning
task_id	Yes	Task id
load_mode_config	No	Load mode setting, interval is the load cycle, load_interval_type represents the load cycle unit and type, 0: unknown load interval type, can be used as the default invalid value); 1: load by day, which may represent a fixed time load per day; 2: load by hour, which may represent a fixed time load per hour 3: load by minute, which may represent a fixed time load per minute; 4: default type, which is loaded only once.
uris	No	Load file

Example:

import requests
import json
# API URL
url = "https://freetier-01.cn-hangzhou.cluster.cn-dev.matrixone.tech/task/update"

headers = {
    "user-id":"0194dfaa-3eda-7ea5-b47c-b4f4f594xxxx",
    "Access-Token": "xxxx",
    "uid": "f42d1006-b48f-4b1d-bd44-3a18dd7bb8ec-0194dfaa-3eda-7ea5-b47c-b4f4f594xxxx:admin:accountadmin"
}

body = {
    "task_id":"1889698919578107904",
    "load_mode_config":{
                      "load_interval_type":4
                   },
              "uris":["Dream of the Red Chamber (Public Book) Simplified Chinese horizontal row.pdf"]
}

response = requests.post(url, json=body, headers=headers)
print(response.json())

return:

{'code': 'OK', 'msg': 'OK'}

Load task deletion

POST /task/delete

Input parameters:

Parameters	Required	Meaning
task_id	Yes	Task id

Example:

import requests
import json

url = "https://freetier-01.cn-hangzhou.cluster.cn-dev.matrixone.tech/task/delete/"

headers = {
    "user-id": "0194dfaa-3eda-7ea5-b47c-b4f4f594xxxx",
    "Access-Token": "xxxx",
    "uid": "cef0d7a2-1505-4384-a89d-5ee5b8246f95-0194dfaa-3eda-7ea5-b47c-b4f4f594xxxx:admin:accountadmin",
}

body = {
    "task_id": "1234567890"
}

response = requests.post(url, headers=headers, json=body)

if response.status_code == 200:
    print(response.json())
else:
    print(f"Request failed, status code: {response.status_code}, error message: {response.text}")

return:

{'code': 'OK', 'msg': 'OK'}

Loading task pause

POST /task/pause

Input parameters:

Parameters	Required	Meaning
task_id	Yes	Task id

Example:

import requests
import json

url = "https://freetier-01.cn-hangzhou.cluster.cn-dev.matrixone.tech/task/pause"

headers = {
    "user-id": "0194dfaa-3eda-7ea5-b47c-b4f4f594xxxx",
    "Access-Token": "xxxx",
    "uid": "cef0d7a2-1505-4384-a89d-5ee5b8246f95-0194dfaa-3eda-7ea5-b47c-b4f4f594xxxx:admin:accountadmin",
}

body= {
    "task_id": "1889613340219121664"
}

response = requests.post(url, headers=headers, json=body)

if response.status_code == 200:
    print(response.json()) # Print the returned JSON data
else:
    print(f"Request failed, status code: {response.status_code}, error message: {response.text}")

return:

{'code': 'OK', 'msg': 'OK'}

Load task recovery

POST /task/resume

Input parameters:

Parameters	Required	Meaning
task_id	Yes	Task id

Example:

import requests
import json

url = "https://freetier-01.cn-hangzhou.cluster.cn-dev.matrixone.tech/task/resume"

headers = {
    "user-id": "0194dfaa-3eda-7ea5-b47c-b4f4f594xxxx",
    "Access-Token": "xxxx",
    "uid": "cef0d7a2-1505-4384-a89d-5ee5b8246f95-0194dfaa-3eda-7ea5-b47c-b4f4f594xxxx:admin:accountadmin",
}

data = {
    "task_id": "1889613340219121664"
}

response = requests.post(url, headers=headers, json=data)

if response.status_code == 200:
    print(response.json())
else:
    print(f"Request failed, status code: {response.status_code}, error message: {response.text}")

return:

{'code': 'OK', 'msg': 'OK'}

Loading task retry

POST /task/retry

Input parameters:

Parameters	Required	Meaning
task_id	Yes	Task id
ids	Yes	Failed file id

Example:

import requests
import json

url = "https://freetier-01.cn-hangzhou.cluster.cn-dev.matrixone.tech/task/retry"
headers = {
    "user-id":"0194e0c2-7e81-7040-ba44-f1d4f51axxxx",
    "Access-Token": "",
    "uid": "0401ffeb-592c-4472-bed4-fb4631c72688-0194e0c2-7e81-7040-ba44-f1d4f51axxxx:admin:accountadmin"
}

body = {
    "task_id": "1889074091616481280",
    "ids": ["1889074111245824000"]
}

response = requests.post(url, headers=headers,json=body)
print("Response Body:", json.dumps(response.json(), indent=4, ensure_ascii=False))

return:

Response Body: {
    "code": "OK",
    "msg": "OK"
}

Loading task acquisition

GET /task/get

Input parameters:

Parameters	Required	Meaning
task_id	Yes	Task id

Output parameters:

Parameters	Meaning
id	Load id
source_connector_id	connector id
source_connector_type	Connector type
volume_id	original volume id
volume_name	original volume name
status	Loading task status, 0: unknown state; 1: normal execution or executable state; 2: pausing state; 3: paused state; 4: completed state.
creator	Created by
source_config	Load configuration
start_at	Loading time
end_at	end time
created_at	Created time
updated_at	Update time
total	Return quantity

Example:

import requests
import json

url = "https://freetier-01.cn-hangzhou.cluster.cn-dev.matrixone.tech/task/get?task_id=1889613340219121664"

headers = {
    "user-id": "0194dfaa-3eda-7ea5-b47c-b4f4f594xxxx",
    "Access-Token": "xxxx",
    "uid": "cef0d7a2-1505-4384-a89d-5ee5b8246f95-0194dfaa-3eda-7ea5-b47c-b4f4f594xxxx:admin:accountadmin",
}

response = requests.get(url, headers=headers)
print("Response Body:", json.dumps(response.json(), indent=4, ensure_ascii=False))

return:

Response Body: {
    "code": "OK",
    "msg": "OK",
    "data": {
        "task": {
            "id": "1889613340219121664",
            "source_connector_id": 100004,
            "source_connector_type": 4,
            "volume_id": "1889578498228068352",
            "volume_name": "b-vol2",
            "name": "",
            "status": 4,
            "creator": "admin",
            "source_config": {
                "common_file_task_config": {
                    "uris": [
                        "Dream of the Red Chamber (Public Book) Simplified Chinese Horizontal.pdf"
                    ],
                    "load_mode_config": {
                        "load_interval_type": 4,
                        "interval": 0
                    }
                }
            },
            "start_at": 1739355480,
            "end_at": 1739355480,
            "created_at": 1739353902,
            "updated_at": 1739355480
        }
    }
}

Get the file under the load task

GET /task/files

Input parameters:

Parameters	Required	Meaning
task_id	Yes	Task id
page	No	Page
page_size	No	Maximum number of files displayed on the page

Output parameters:

Parameters	Meaning
id	Load id
name	file name
type	File type
status	File load status
size	File size
update_time	Update time
user	Founder
start_at	Loading time
end_at	end time

Example:

import requests
import json

url = "https://freetier-01.cn-hangzhou.cluster.cn-dev.matrixone.tech/task/files?task_id=1889613340219121664"

headers = {
    "user-id": "0194dfaa-3eda-7ea5-b47c-b4f4f594xxxx",
    "Access-Token": "xxxx",
    "uid": "cef0d7a2-1505-4384-a89d-5ee5b8246f95-0194dfaa-3eda-7ea5-b47c-b4f4f594xxxx:admin:accountadmin",
}

params = {
    "task_id": 1889613340219121664
}

response = requests.get(url, headers=headers, params=params)

print("Response Body:", json.dumps(response.json(), indent=4, ensure_ascii=False))

return:

Response Body: {
    "code": "OK",
    "msg": "OK",
    "data": {
        "total": 1,
        "total_success": 1,
        "total_failed": 0,
        "files": [
            {
                "id": "1889613341347389440",
                "name": "Dream of the Red Chamber (Public Book) Simplified Chinese horizontal row.pdf",
                "type": 2,
                "status": 5,
                "size": 6787457,
                "update_time": 1739353902,
                "other_metadata": "",
                "reason": "",
                "user": "admin",
                "start_time": 1739353902,
                "end_time": 0,
                "path": "/b-vol2/Dream of the Red Chamber (Public Book) Simplified Chinese horizontal row.pdf"
            }
        ]
    }
}