Import data from REST APIs¶
REST API data ingestion loads JSON records from external HTTP/HTTPS endpoints into Catalog tables. Save the base URL and authentication settings in a REST API connector first, then configure the endpoint path, pagination, incremental cursor, and target table mapping in an import task.
1. Create a REST API connector¶
Go to Data Connection > Connectors and click New connector.
Enter a connector name, select REST API as the data source type, and keep the import purpose enabled.
Enter the API Base URL (for example,
https://api.example.com/v1/—must be a complete HTTP/HTTPS URL without query strings or fragments).Choose an Authentication mode and fill in the required credentials:
Authentication mode |
Required fields |
|---|---|
None |
No credentials required |
Bearer Token |
Bearer Token |
API Key |
Key Location (Header or Query), Key Name, and Key Value |
HTTP Basic |
Username and Password |
OAuth 2.0 · Client Credentials |
Token URL, Client ID, Client Secret, Client Authentication, optional Scope, and optional Token Request Parameters |
Expand Advanced options as needed to add Shared Headers, Shared Query Parameters, and Request Timeout (seconds) (an integer between 1 and 60).
To test credentials before saving, enable Configure verification endpoint (optional), specify a lightweight read-only Verification endpoint path (starting with a single
/), Method, optional parameters, and optional Business status field path + Expected value (JSON), then click Test connection.Save the connector once verified.
When editing an existing connector, leave secret fields blank to retain saved credentials.
2. Create a REST API import task¶
Go to Data Connection > Import Tasks, choose Structured, and select REST API as the source.
Select your REST API connector and enter the Endpoint path (starting with a single
/, appended to the connector’s base URL).Expand advanced options to match the endpoint contract:
Record path in the response (optional): provide the JSON path (for example,
$.data.items) when records are nested inside an object;Pagination: choose None, Page number, Offset, Cursor, or Next page URL, and configure the query parameter, starting page/offset, or response path (such as
$.paging.next);Response schema (optional): enable Use the schema returned by the API and enter the schema path (such as
$.schema.fields) and format (Column descriptors or JSON Schema); otherwise field types are inferred from preview records.
Click Preview data (supports
GET,OPTIONS, andPOSTafter confirming the endpoint is read-only). Previewing reads the first page and displays up to 20 records without creating an import task.Choose a target table (new or existing) and review column mappings, target types (
BIGINT,DECIMAL,TEXT,VARCHAR,DATE,TIMESTAMP,BOOL,JSON,BLOB,VARBINARY), and binary encodings (Base64/Hex).Choose one-time or scheduled execution:
For scheduled incremental loads, configure the Incremental query parameter, Cursor field in records, and Initial cursor, confirm inclusive (
>=) semantics, and map at least one primary key plus a non-nullBIGINTorTIMESTAMP(6)version field;A full replacement import replaces all rows in an existing target table after succeeding.
Click Create and start import. Track execution in Import tasks and confirm written rows in Catalog once completed.
Verification stages and completion check¶
Test connection (in Connectors): checks endpoint reachability and credentials against the verification path only.
Preview data (in Import Tasks): verifies pagination, record path, and type mappings against the first page without writing to the target table.
Completion check: After the job completes in Import tasks, open the target Catalog table to verify written rows and confirm scheduled runs advance the watermark cursor.