Workflows

After data is registered in the Catalog, it is still in its raw form. PDFs, spreadsheets, audio, and video cannot yet be retrieved directly by AI. A workflow parses, splits, vectorizes, and otherwise processes that data into a form that AI can use. A workflow is a definition of how to process a set of data: a group of operators and the order in which data flows between them.

Distinguish these three terms

The term “workflow” is often used interchangeably with two related concepts. Keep them separate:

Term

Meaning

Workflow

A definition that records processing steps and their order. It is stored in the Catalog; until it is started, it takes no action.

Run

A single execution after the definition is started. Each start creates another workflow job.

Canvas

One way to edit a workflow definition. It is not the workflow itself.

When troubleshooting, first determine whether the definition is incorrect or whether a particular run failed.

Where workflows fit

Workflows occupy the stage between raw and usable data:

Raw data registered in the Catalog → Workflow processing → Knowledge bases / agents

Upstream, a workflow retrieves data to process from the Catalog. Downstream, it writes results back to the Catalog and records lineage, where knowledge bases and agents can use them. A workflow does not store data itself; it performs the processing.

Three ways to create the same definition

Processing steps are represented as a data definition rather than a program that only engineers can modify. You can therefore produce the same workflow definition in three ways:

  • Use the canvas to drag operators and connect them.

  • Write the definition file directly.

  • Describe the requirement in natural language, then use the built-in AI to generate and refine the definition.

All three approaches produce the same type of definition file and run the same way. Natural language and platform-provided operators make it possible to compose processing flows for different data types: SQL operators for structured data, parsing and splitting operators for unstructured data, and custom operators for special requirements.

A workflow that starts might still produce incorrect results

The scheduling engine executes a definition as written; it does not understand business intent. Incorrect connections or parameter values are still run. A workflow that can be saved and started does not necessarily produce correct results. Check each workflow job to validate its output. For steps to locate a failure, see Run and debug.

Further reading

Last updated on