Data access related API
Connector
Create a Connector
POST /conectors
Input parameters:
| Parameters | Required | Meaning |
|---|---|---|
| name | Yes | Connector name |
| source_type | Yes | Connector type, 4 is OSS, 5 is standard S3 |
| config | Yes | Configure the connection information of oss and s3. If it is oss, you need to fill in endpoint, access_key_id, access_key_secret, and bucket_name. If it is s3, in addition to the parameters in oss, you need to fill in region and path_style (path style) information. |
body:
{
"name": "new_connector_name",
"source_type": 4,
"config": {
"oss": { // If source_type is 4, fill in this field
"endpoint": "example_oss_endpoint",
"access_key_id": "example_access_key_id",
"access_key_secret": "example_access_key_secret",
"bucket_name": "example_bucket_name"
},
"s3": { // If source_type is 5, fill in this field
"endpoint": "example_s3_endpoint",
"access_key_id": "example_access_key_id",
"access_key_secret": "example_access_key_secret",
"bucket_name": "example_bucket_name",
"region": "example_region",
"path_style": true //Path style, true is path style, false is virtual hoste
}
}
}
Example:
import requests
url = "https://freetier-01.cn-hangzhou.cluster.cn-dev.matrixone.tech/connectors" # Please replace it with the actual API address
headers = {
"user-id":"0194dfaa-3eda-7ea5-b47c-b4f4f594xxxx",
"Access-Token": "xxxx",
"uid": "91a2234e-0a06-4d84-815e-840e379d9e1c-0194dfaa-3eda-7ea5-b47c-b4f4f594xxxx:admin:accountadmin"
}
body = {
"name": "oss-test2",
"source_type": 4,
"config": {
"oss": {
"endpoint": "xxxx",
"access_key_id": "xxxx",
"access_key_secret": "xxxx",
"bucket_name": "xxxx"
}
}
}
response = requests.post(url, json=body, headers=headers)
if response.status_code == 200:
print(response.json())
else:
print(f"Request failed, status code: {response.status_code}, error message: {response.text}")
return:
{'code': 'OK', 'msg': 'OK'}
Verify the connector
POST /connectors/validate
Input parameters:
| Parameters | Required | Meaning |
|---|---|---|
| source_type | Yes | Connector type, 4 is OSS, 5 is standard S3 |
| connector_id | No | If you fill in connector_id, you don’t need to fill in the config information. |
| config | No | If you fill in config, you do not need to fill in the connection information of the connector_id information configuration oss and s3. If it is oss, you need to fill in endpoint, access_key_id, access_key_secret, bucket_name. If it is s3, in addition to the parameters in oss, you also need to fill in region and path_style information. |
Example:
import requests
url = "https://freetier-01.cn-hangzhou.cluster.cn-dev.matrixone.tech/connectors/validate"
headers = {
"user-id":"0194dfaa-3eda-7ea5-b47c-b4f4f594xxxx",
"Access-Token": "xxxx",
"uid": "2c5a7dc4-032a-42ac-b5ad-4db47f11772b-0194dfaa-3eda-7ea5-b47c-b4f4f594xxxx:admin:accountadmin"
}
body = {
"source_type": 4,
"config": {
"oss": {
"endpoint": "xxxx",
"access_key_id": "xxxx",
"access_key_secret": "xxxx",
"bucket_name": "xxxx"
}
}
}
response = requests.post(url, json=body, headers=headers)
if response.status_code == 200:
print(response.json())
else:
print(f"Request failed, status code: {response.status_code}, error message: {response.text}")
Return to the example:
{'code': 'OK', 'msg': 'OK', 'data': {'valid': True}}
Query Connector
GET /connectors/list
Output parameters:
| Parameters | Meaning |
|---|---|
| id | connector-id |
| source_type | Connector type, 4 is OSS, 5 is standard S3 |
| name | connector name |
| status | Connection status |
| created_at | Created time |
| updated_at | Update time |
| username | Created by |
| updated_at | Update time |
| username | Created by |
| related_task_ids | associated TaskID |
| config | Connector configuration information |
| total | Return number |
Example:
import requests
import json
url = "https://freetier-01.cn-hangzhou.cluster.cn-dev.matrixone.tech/connectors/list"
headers = {
"user-id":"xxxx",
"uid": "f42d1006-b48f-4b1d-bd44-3a18dd7bb8ec-0194dfaa-3eda-7ea5-b47c-b4f4f594xxxx:admin:accountadmin"
}
response = requests.get(url, headers=headers)
print("Response Body:", json.dumps(response.json(), indent=4, ensure_ascii=False))
return:
Response Body: {
"code": "OK",
"msg": "OK",
"data": {
"connectors": [
{
"id": 100004,
"source_type": 4,
"name": "oss-test1",
"status": "active",
"created_at": 1738919558,
"updated_at": 1738919558,
"username": "admin",
"related_task_ids": [
1889223922712281088
],
"config": {
"oss": {
"endpoint": "xxxx",
"access_key_id": "xxxx","access_key_secret": "xxxx",
"bucket_name": "xxxx"
}
}
},
{
"id": 100005,
"source_type": 5,
"name": "s3-test",
"status": "active",
"created_at": 1738919761,
"updated_at": 1738919761,
"username": "admin",
"related_task_ids": null,
"config": {
"s3": {
"endpoint": "xxxx",
"access_key_id": "xxxx",
"access_key_secret": "xxxx",
"bucket_name": "xxxx",
"region": "xxxx"
}
}
}
],
"total": 2
}
}
Update the connector
PUT /connectors/{id}
Input parameters:
| Parameters | Required | Meaning |
|---|---|---|
| name | Yes | Connector name |
| config | Yes | Connector Configuration Information |
Example:
import requests
import json
url = "https://freetier-01.cn-hangzhou.cluster.cn-dev.matrixone.tech/connectors/100005"
headers = {
"user-id":"0194dfaa-3eda-7ea5-b47c-b4f4f594xxxx",
"Access-Token": "xxxx",
"uid": "f42d1006-b48f-4b1d-bd44-3a18dd7bb8ec-0194dfaa-3eda-7ea5-b47c-b4f4f594xxxx:admin:accountadmin"
}
body = {
"name": "s3-test1",
"config": {
"s3": {
"endpoint": "xxxx",
"access_key_id": "xxxx",
"access_key_secret": "xxxx",
"bucket_name": "xxxx",
"region": "xxxx"
}
}
}
response = requests.put(url, json=body, headers=headers)
if response.status_code == 200:
print(response.json())
else:
print(f"Request failed, status code: {response.status_code}, error message: {response.text}")
Query the connector source file
GET /connectors/files/list
Input parameters:
| Parameters | Required | Meaning |
|---|---|---|
| connector_id | Yes | Connector id |
| file_types | No | File type, 0: empty file type (may be used as default or invalid value); 1: TXT text file type; 2: PDF document file type 3: picture file type; 4: PPT presentation file type; 5: Word document file type; 6: Markdown markup language file type; 7: CSV comma separated value file type; 8: Parquet columnar storage file type; 9: SQL file type; 10: directory type |
Output parameters:
| Parameters | Meaning |
|---|---|
| uri | represents the unique resource locator in the source connector. |
| filename | filename |
| size | File size |
| type | File type, 0: Empty file type (may be used as default or invalid value); 1: TXT text file type; 2: PDF document file type 3: Picture file type; 4: PPT presentation file type; 5: Word document file type; 6: Markdown markup language file type; 7: CSV comma separated value file type; 8: Parquet columnar storage file type; 9: SQL file type; 10: Directory type. |
Example:
import requests
import json
url = "https://freetier-01.cn-hangzhou.cluster.cn-dev.matrixone.tech/connectors/files/list"
headers = {
"user-id": "<your_workspace_id>",
"Access-Token": "<your_access_token>",
"uid": "<your_uid>",
}
params = {
"connector_id": 100004,
"file_types": 2
}
response = requests.get(url, headers=headers, params=params)
try:
response_json = response.json()
print("Response Body:", json.dumps(response_json, indent=4, ensure_ascii=False))
except json.JSONDecodeError:
print("Response is not in JSON format:", response.text)
return:
Response Body: {
"code": "OK",
"msg": "OK",
"data": {
"files": [
{
"uri": "Dream of the Red Chamber (Public Book) Simplified Chinese horizontal row.pdf",
"filename": "Dream of the Red Chamber (Public Book) Simplified Chinese horizontal row.pdf",
"size": 6787457,
"type": 2
},
{
"uri": "file1",
"filename": "file1",
"size": 0,
"type": 10
}
]
}
}
Data loading
Create a load task
POST /task
Input parameters:
| Parameters | Required | Meaning |
|---|---|---|
| source_connector_id | Yes | Connector Name |
| volume_id | Yes | id of the original volume to be loaded |
| source_config | Yes | Configure for loading task source |
| common_file_task_config | Yes | Configure for common files |
| load_mode_config | Yes | Load mode setting, interval is the load cycle, load_interval_type represents the load cycle unit and type, 0: unknown load interval type, can be used as the default invalid value); 1: load by day, which may represent a fixed time load per day; 2: load by hour, which may represent a fixed time load per hour 3: load by minute, which may represent a fixed time load per minute; 4: default type, which is loaded only once. |
| uris | Yes | File to load |
| config_type | Yes | Indicates the common file load configuration type, default is 1 |
Example:
import requests
import json
url = "https://freetier-01.cn-hangzhou.cluster.cn-dev.matrixone.tech/task"
headers = {
"user-id":"xxxx",
"Access-Token": "xxxx",
"uid": "f42d1006-b48f-4b1d-bd44-3a18dd7bb8ec-0194dfaa-3eda-7ea5-b47c-b4f4f594xxxx:admin:accountadmin"
}
body = {
"source_connector_id":100004,
"volume_id":"1889578498228068352",
"source_config":
{"common_file_task_config":
{
"load_mode_config":
{
"load_interval_type":4
},
"uris":["Dream of the Red Chamber (Public Book) Simplified Chinese horizontal row.pdf"]
}
},
"config_type":1
}
response = requests.post(url, json=body, headers=headers)
print(response.json())
return:
{'code': 'OK', 'msg': 'OK'}
Load the task list
GET /task/list
Input parameters:
| Parameters | Required | Meaning |
|---|---|---|
| is_desc | No | Sort |
| load_interval_types | No | Load interval, 0: unknown interval type; 1: by day; 2: by hour; 3: by minute; 4: only once |
| order_by | No | Configure for loading task source |
| page | No | Page |
| page_size | No | When the page number |
| status | No | Indicates the common file load configuration type, default is 1 |
Output parameters:
| Parameters | Meaning |
|---|---|
| id | Load id |
| source_connector_id | connector id |
| source_connector_type | Connector type |
| volume_id | original volume id |
| volume_name | original volume name |
| status | Loading task status, 0: unknown state; 1: normal execution or executable state; 2: pausing state; 3: paused state; 4: completed state. |
| creator | Created by |
| source_config | Load configuration |
| start_at | Loading time |
| end_at | end time |
| created_at | Created time |
| updated_at | Update time |
| total | Return quantity |
Example:
import requests
import json
url = "https://freetier-01.cn-hangzhou.cluster.cn-dev.matrixone.tech/task/list"
headers = {
"user-id": "0194dfaa-3eda-7ea5-b47c-b4f4f594xxxx",
"Access-Token": "xxxx",
"uid": "cef0d7a2-1505-4384-a89d-5ee5b8246f95-0194dfaa-3eda-7ea5-b47c-b4f4f594xxxx:admin:accountadmin",
}
response = requests.get(url, headers=headers)
print("Response Body:", json.dumps(response.json(), indent=4, ensure_ascii=False))
return:
Response Body: {
"code": "OK",
"msg": "OK",
"data": {
"tasks": [
{
"id": "1889223922712281088",
"source_connector_id": 100004,
"source_connector_type": 4,
"volume_id": "1889223879880048640",
"volume_name": "b-vol1",
"name": "",
"status": 4,
"creator": "admin",
"source_config": {
"common_file_task_config": {
"uris": [
"Dream of the Red Chamber (Public Book) Simplified Chinese Horizontal.pdf"
],
"load_mode_config": {
"load_interval_type": 4,
"interval": 0
}
}
},
"start_at": 1739261063,
"end_at": 1739261640,
"created_at": 1739261057,
"updated_at": 1739261640
}
],
"total": 1
}
}
Load task update
POST /task/update
Input parameters:
| Parameters | Required | Meaning |
|---|---|---|
| task_id | Yes | Task id |
| load_mode_config | No | Load mode setting, interval is the load cycle, load_interval_type represents the load cycle unit and type, 0: unknown load interval type, can be used as the default invalid value); 1: load by day, which may represent a fixed time load per day; 2: load by hour, which may represent a fixed time load per hour 3: load by minute, which may represent a fixed time load per minute; 4: default type, which is loaded only once. |
| uris | No | Load file |
Example:
import requests
import json
# API URL
url = "https://freetier-01.cn-hangzhou.cluster.cn-dev.matrixone.tech/task/update"
headers = {
"user-id":"0194dfaa-3eda-7ea5-b47c-b4f4f594xxxx",
"Access-Token": "xxxx",
"uid": "f42d1006-b48f-4b1d-bd44-3a18dd7bb8ec-0194dfaa-3eda-7ea5-b47c-b4f4f594xxxx:admin:accountadmin"
}
body = {
"task_id":"1889698919578107904",
"load_mode_config":{
"load_interval_type":4
},
"uris":["Dream of the Red Chamber (Public Book) Simplified Chinese horizontal row.pdf"]
}
response = requests.post(url, json=body, headers=headers)
print(response.json())
return:
{'code': 'OK', 'msg': 'OK'}
Load task deletion
POST /task/delete
Input parameters:
| Parameters | Required | Meaning |
|---|---|---|
| task_id | Yes | Task id |
Example:
import requests
import json
url = "https://freetier-01.cn-hangzhou.cluster.cn-dev.matrixone.tech/task/delete/"
headers = {
"user-id": "0194dfaa-3eda-7ea5-b47c-b4f4f594xxxx",
"Access-Token": "xxxx",
"uid": "cef0d7a2-1505-4384-a89d-5ee5b8246f95-0194dfaa-3eda-7ea5-b47c-b4f4f594xxxx:admin:accountadmin",
}
body = {
"task_id": "1234567890"
}
response = requests.post(url, headers=headers, json=body)
if response.status_code == 200:
print(response.json())
else:
print(f"Request failed, status code: {response.status_code}, error message: {response.text}")
return:
{'code': 'OK', 'msg': 'OK'}
Loading task pause
POST /task/pause
Input parameters:
| Parameters | Required | Meaning |
|---|---|---|
| task_id | Yes | Task id |
Example:
import requests
import json
url = "https://freetier-01.cn-hangzhou.cluster.cn-dev.matrixone.tech/task/pause"
headers = {
"user-id": "0194dfaa-3eda-7ea5-b47c-b4f4f594xxxx",
"Access-Token": "xxxx",
"uid": "cef0d7a2-1505-4384-a89d-5ee5b8246f95-0194dfaa-3eda-7ea5-b47c-b4f4f594xxxx:admin:accountadmin",
}
body= {
"task_id": "1889613340219121664"
}
response = requests.post(url, headers=headers, json=body)
if response.status_code == 200:
print(response.json()) # Print the returned JSON data
else:
print(f"Request failed, status code: {response.status_code}, error message: {response.text}")
return:
{'code': 'OK', 'msg': 'OK'}
Load task recovery
POST /task/resume
Input parameters:
| Parameters | Required | Meaning |
|---|---|---|
| task_id | Yes | Task id |
Example:
import requests
import json
url = "https://freetier-01.cn-hangzhou.cluster.cn-dev.matrixone.tech/task/resume"
headers = {
"user-id": "0194dfaa-3eda-7ea5-b47c-b4f4f594xxxx",
"Access-Token": "xxxx",
"uid": "cef0d7a2-1505-4384-a89d-5ee5b8246f95-0194dfaa-3eda-7ea5-b47c-b4f4f594xxxx:admin:accountadmin",
}
data = {
"task_id": "1889613340219121664"
}
response = requests.post(url, headers=headers, json=data)
if response.status_code == 200:
print(response.json())
else:
print(f"Request failed, status code: {response.status_code}, error message: {response.text}")
return:
{'code': 'OK', 'msg': 'OK'}
Loading task retry
POST /task/retry
Input parameters:
| Parameters | Required | Meaning |
|---|---|---|
| task_id | Yes | Task id |
| ids | Yes | Failed file id |
Example:
import requests
import json
url = "https://freetier-01.cn-hangzhou.cluster.cn-dev.matrixone.tech/task/retry"
headers = {
"user-id":"0194e0c2-7e81-7040-ba44-f1d4f51axxxx",
"Access-Token": "",
"uid": "0401ffeb-592c-4472-bed4-fb4631c72688-0194e0c2-7e81-7040-ba44-f1d4f51axxxx:admin:accountadmin"
}
body = {
"task_id": "1889074091616481280",
"ids": ["1889074111245824000"]
}
response = requests.post(url, headers=headers,json=body)
print("Response Body:", json.dumps(response.json(), indent=4, ensure_ascii=False))
return:
Response Body: {
"code": "OK",
"msg": "OK"
}
Loading task acquisition
GET /task/get
Input parameters:
| Parameters | Required | Meaning |
|---|---|---|
| task_id | Yes | Task id |
Output parameters:
| Parameters | Meaning |
|---|---|
| id | Load id |
| source_connector_id | connector id |
| source_connector_type | Connector type |
| volume_id | original volume id |
| volume_name | original volume name |
| status | Loading task status, 0: unknown state; 1: normal execution or executable state; 2: pausing state; 3: paused state; 4: completed state. |
| creator | Created by |
| source_config | Load configuration |
| start_at | Loading time |
| end_at | end time |
| created_at | Created time |
| updated_at | Update time |
| total | Return quantity |
Example:
import requests
import json
url = "https://freetier-01.cn-hangzhou.cluster.cn-dev.matrixone.tech/task/get?task_id=1889613340219121664"
headers = {
"user-id": "0194dfaa-3eda-7ea5-b47c-b4f4f594xxxx",
"Access-Token": "xxxx",
"uid": "cef0d7a2-1505-4384-a89d-5ee5b8246f95-0194dfaa-3eda-7ea5-b47c-b4f4f594xxxx:admin:accountadmin",
}
response = requests.get(url, headers=headers)
print("Response Body:", json.dumps(response.json(), indent=4, ensure_ascii=False))
return:
Response Body: {
"code": "OK",
"msg": "OK",
"data": {
"task": {
"id": "1889613340219121664",
"source_connector_id": 100004,
"source_connector_type": 4,
"volume_id": "1889578498228068352",
"volume_name": "b-vol2",
"name": "",
"status": 4,
"creator": "admin",
"source_config": {
"common_file_task_config": {
"uris": [
"Dream of the Red Chamber (Public Book) Simplified Chinese Horizontal.pdf"
],
"load_mode_config": {
"load_interval_type": 4,
"interval": 0
}
}
},
"start_at": 1739355480,
"end_at": 1739355480,
"created_at": 1739353902,
"updated_at": 1739355480
}
}
}
Get the file under the load task
GET /task/files
Input parameters:
| Parameters | Required | Meaning |
|---|---|---|
| task_id | Yes | Task id |
| page | No | Page |
| page_size | No | Maximum number of files displayed on the page |
Output parameters:
| Parameters | Meaning |
|---|---|
| id | Load id |
| name | file name |
| type | File type |
| status | File load status |
| size | File size |
| update_time | Update time |
| user | Founder |
| start_at | Loading time |
| end_at | end time |
Example:
import requests
import json
url = "https://freetier-01.cn-hangzhou.cluster.cn-dev.matrixone.tech/task/files?task_id=1889613340219121664"
headers = {
"user-id": "0194dfaa-3eda-7ea5-b47c-b4f4f594xxxx",
"Access-Token": "xxxx",
"uid": "cef0d7a2-1505-4384-a89d-5ee5b8246f95-0194dfaa-3eda-7ea5-b47c-b4f4f594xxxx:admin:accountadmin",
}
params = {
"task_id": 1889613340219121664
}
response = requests.get(url, headers=headers, params=params)
print("Response Body:", json.dumps(response.json(), indent=4, ensure_ascii=False))
return:
Response Body: {
"code": "OK",
"msg": "OK",
"data": {
"total": 1,
"total_success": 1,
"total_failed": 0,
"files": [
{
"id": "1889613341347389440",
"name": "Dream of the Red Chamber (Public Book) Simplified Chinese horizontal row.pdf",
"type": 2,
"status": 5,
"size": 6787457,
"update_time": 1739353902,
"other_metadata": "",
"reason": "",
"user": "admin",
"start_time": 1739353902,
"end_time": 0,
"path": "/b-vol2/Dream of the Red Chamber (Public Book) Simplified Chinese horizontal row.pdf"
}
]
}
}