Data Processing

Create and run workflows, manage custom operators, templates, and WorkItem API services, execute SQL, and manage workbooks.

Common tasks

Task

Page

Deploy workflow definitions and obtain IDs for execution

Create workflow

Start, inspect, or control a workflow run

Run and track workflows

Create custom operators or workflow templates

Custom operators and workflow templates

Publish and invoke WorkItem API services

WorkItem API services

View artifacts, nodes, and data lineage

View workflow artifacts and data lineage

Execute SQL, save workbooks, and read query results

Execute SQL and read query results

Typical workflow

  1. Use Create workflow to deploy DSL definitions to the workspace, saving the returned workflow ID.

  2. Use Run and track workflows to trigger a run, save the run ID, and read results or control execution based on status.

  3. When logic reuse is needed, use Custom operators and workflow templates to extend workflow capabilities.

  4. When exposing SQL or workflow outputs as callable APIs, use WorkItem API services.

Last updated on