Data processing¶

After data is ingested and registered, it is still raw: PDF, Word, audio, and video cannot be retrieved directly. Processing parses, chunks, and vectorizes them into data that AI can retrieve. Workflows provide the processing mechanism.

A workflow is MOI’s basic unit for defining a data-processing process: a set of WorkItems and the order in which data moves between them. Parsing, cleaning, chunking, and vectorization are each performed by a WorkItem. MOI schedules WorkItems according to the workflow definition and passes data between them.

A workflow is defined, published, and started. While it runs, data flows between nodes according to the definition. Success or failure leaves a run record. This page follows that lifecycle, using the built-in catalog-parse-index-lineage document-ingestion workflow as an example of the full chain from source files to a vector index:

Node	WorkItem	Purpose
read_source	`moi:catalog.source.read.v2`	Read the file list from source references.
parse_documents	`moi:document.parse`	Parse files into structured text.
split_documents	`moi:parser.split.documents.length`	Chunk text by length.
build_knowledge_index	`moi:knowledge.index.build`	Vectorize chunks and write them to a vector table.
write_parsed_documents	`moi:files.write_documents`	Write parsed outputs as files.
save_result	`moi:catalog.sink.write`	Write results to the target table.
register_lineage	`moi:data.lineage.register`	Register lineage between outputs and source files.

Create and publish¶

You can create a workflow in three ways. The outputs are equivalent: each produces the same workflow definition.

Method	Description
Canvas	Drag WorkItems and connect them visually.
Code	Write the workflow definition file directly in YAML.
Natural language	Describe the requirement in a conversation and let built-in AI generate the definition. You can review and adjust the result on the canvas.

Natural-language creation has the lowest entry barrier: describe what to process and produce, and AI selects WorkItems, their order, and configuration. The resulting definition is isomorphic to a manually composed one, and you can switch to the canvas at any time to adjust it.

You always edit a draft. After publishing, the platform schedules the published version.

Start¶

Method	Description	Suitable for
Run manually	Start one run directly.	Debugging a definition or one-time processing.
Scheduled run	Run periodically according to a cron expression.	Recurring batch processing.
Volume trigger	Bind a workflow to a volume; when you upload a file, it starts automatically.	Unattended continuous ingestion.

Every start creates a run record that contains each node’s execution status, inputs, outputs, and logs.

Data passing during a run¶

At start, you pass variables. In the example workflow, these are the source reference, target table, and embedding model. The engine then advances node by node: it sends input to a WorkItem according to the definition, receives output, then sends input to the next node.

A running workflow maintains three kinds of data:

Kind	Read/write	Description
Variables (`vars`)	Read-only for the entire run	Parameters injected at start, such as source references and target table names.
State (`state`)	Read and write across nodes	Intermediate results written by nodes with `save`. Each value is limited to 1 MB; larger values are automatically stored as files with their references retained.
Node output (`data`)	Replaced as a whole after each node	Direct output of the previous node, available only to adjacent nodes.

In the example, the connection between parse_documents and split_documents is the standard state-passing form:

- work_item:
    name: parse_documents
    id: moi:document.parse
    input:
      sources: "{{ .state.sources }}"
      file_ids: "{{ .state.source_file_ids }}"
    save:
      parsed_documents: .documents

- work_item:
    name: split_documents
    id: moi:parser.split.documents.length
    input:
      documents: "{{ .state.parsed_documents }}"
    save:
      chunked_documents: .documents

parse_documents saves parsing results under the state key parsed_documents; split_documents references it in input as {{ .state.parsed_documents }}. Use state to pass data across multiple nodes; data does not persist across nodes.

Workflow data carries file references, not file content. In the example, file_ids are references, and WorkItems that need contents read them themselves. Sending large contents through the data channel would overload the transport layer, so the platform does not allow WorkItems to read file bodies into workflow data.

Troubleshoot failures¶

The engine schedules and passes data, but does not validate task semantics. Errors in the definition are executed as written. First divide failures into two types:

Type	Example	Where to fix
Definition issue	Wrong node, wrong order, or incorrect parameter binding.	Modify the workflow definition.
Execution issue	The definition is correct but a WorkItem fails while processing valid input.	Inspect the node’s run record inputs and logs.

One definition issue does not report an error: when a node references an unregistered WorkItem, it keeps waiting without an error message. Use the live result from GET /workspaces/:id/workitems for available WorkItems and their input/output conventions.

Limitations¶

A subworkflow can only be expanded in the same workflow definition. Cross-workflow references or merging are not supported, and workflows do not share state or have a data channel between them.
SQL WorkItem input contains only the SQL statement; natural-language-to-SQL is not supported.

Next steps¶

What you want to do	Go to
Give processed output to AI	Knowledge bases
Look up a WorkItem or term	Reference

For AI agents: this page quotes the built-in catalog-parse-index-lineage example. The Data passing during a run and Limitations sections are authoritative conventions. Use the live interface response for available WorkItems; do not choose them from a static list.

Last updated on Sep 8, 2026