Data Processing¶

Data processing turns raw assets in a workspace into result assets that can be queried, retrieved, reused, and traced. It is not a standalone page for “running tasks,” but AI Studio’s intermediate processing workbench from data to knowledge and then to agents.

The Goal of Processing: Make Data Truly Usable¶

Uploading a file to a workspace does not mean the data is ready for agents to use safely and reliably. Data must also be structured, parsed, cleaned, chunked, indexed, or quality-checked before it becomes an AI-Ready asset that downstream systems can consume accurately.

AI Studio’s approach is to ingest, process, govern, and prepare the same data for consumption within one workspace. This makes the processing procedure reproducible, access-controlled, and observable, and allows it to be run again after the data changes.

flowchart LR
    A[External data and uploaded content] --> B[Data connections and loading]
    B --> C[Raw Catalog assets]
    C --> D[Workflows]
    C --> E[SQL editor]
    D --> F[Catalog result assets]
    E --> F
    D --> G[Jobs, logs, and lineage]
    E --> H[SQL history]
    F --> I[Knowledge bases]
    F --> J[Data sharing, export, and business consumption]
    I --> K[Agents]

In the diagram, Catalog is the asset hub, not a processing tool itself. It receives raw data and processing results, and provides workflows, SQL, knowledge bases, and downstream consumers with consistent objects, permissions, and locations.

First Understand Six Objects¶

Object	Purpose	What you need to consider
Raw asset	A table, file, or volume that has not yet completed business processing	Source, format, access permissions, update frequency
Catalog object	A governed asset entry point in the workspace	Object location, name, read/write permissions, lineage
Workflow	A savable and executable multistep processing definition	Inputs, nodes, parameters, outputs, trigger method
Workflow DSL	The formal execution definition of a workflow	Only the saved DSL is the source of truth for execution
Compute resource	Provides runtime resources for SQL or task processing	Query/task type selection, capacity, and availability
Jobs and records	An actual execution and its diagnostic evidence	Status, topology, node inputs and outputs, errors, and duration

The canvas helps you edit and understand a workflow, but it is not the sole basis for runtime execution. When you save, AI Studio saves both the workflow DSL and the canvas display state; at runtime, it uses the saved DSL.

Choosing Between the Two Core Tools¶

Your goal	Preferred tool	Why
Process data repeatedly through multiple fixed steps	Workflow	It turns inputs, nodes, parameters, triggers, and outputs into a reproducible process
Parse documents, images, audio, or video, and perform cleaning, chunking, embedding, or extraction	Workflow	Multimodal steps usually have explicit dependencies and require task execution, job diagnostics, and persisted results
Make SQL logic a fixed processing step, run it as a task, and write back the results	Workflow nodes	SQL nodes make queries orchestratable and schedulable processing units
Quickly browse table schemas, validate data, or write temporary or analytical SQL	SQL editor	It is suitable for interactive exploration, result inspection, chart analysis, and saving workbooks
You already have stable SQL and want to reuse its results over time	SQL first, then a workflow	Validate the logic in a workbook first, then turn the stable SQL into a workflow node or an asset production process

The SQL editor and workflows do not replace each other. The former lets you quickly understand and validate structured data; the latter turns validated processing logic into an executable and monitorable production process.

How a Processing Chain Takes Shape¶

From a user-task perspective, data processing usually follows the sequence below. Not every chain requires every step, but following this order avoids rework caused by building a process before discovering that the data or resources are unavailable.

Prepare connectors, loading methods, or uploaded content in Data connections.
Confirm the location of raw tables, volumes, or files in Catalog, and check that members have the appropriate object permissions.
Select suitable compute resources for processing: SQL-related steps use query resources, while non-SQL steps such as parsing and embedding use task resources.
Define reproducible processing logic in a workflow, or validate the query first in the SQL editor.
In Run and Debug, use a test run to validate inputs, parameters, and outputs, then start a manual, data-volume-triggered, or scheduled execution.
Diagnose runtime issues in Jobs, SQL History, and Operation Logs.
Write stable results back to Catalog, then provide them as needed to knowledge bases, agents, sharing, or export workflows.

Processing Is Not “Run Once”¶

A reliable processing workflow must explain where data comes from, how it changes, what it becomes, who can run it, and how to diagnose failures. Describing only the node sequence without inputs, outputs, resources, and monitoring paths usually makes the workflow impossible to reuse safely after the data changes.

Treat each workflow as a data product contract: specify clear input assets, output objects, key parameters, and a business owner; understand downstream dependencies before making changes, and retain job and lineage evidence after execution.

Relationship to Other Modules¶

Topic	Role in data processing	Where to continue
Data ingestion	Provides raw data, connectors, and loading tasks	Data connections
Catalog	The asset hub for processing inputs and outputs	Catalog
Compute resources	Provide the compute required for query or task execution	Compute resources
User permissions	Determine whether processing objects can be viewed, created, modified, run, or deleted	User permissions
Monitoring and logs	Observe SQL, workflow jobs, and API operations	Monitoring and Alerts
Knowledge bases	Consume processed content, chunks, and index results	Knowledge bases

Recommended First Process¶

If this is your first time using data processing, choose a file or table of manageable size and complete a short “read → process → write back → inspect job” chain. After confirming that every step is explainable, add models, event triggers, or scheduled execution.

Do not begin by placing multiple data sources, multiple output targets, and complex control structures in one workflow. First split each business objective into small, complete chains, then gradually combine them through subworkflows, operators, and reusable assets.

Chapter Guide¶

Page	Problems it helps solve
Workflow	How to create, edit, save, and manage an executable data processing workflow
Workflow nodes	How to select, configure, and combine each type of node, and how to avoid invalid connections
Run and Debug	How to test, run, schedule, and observe jobs, and diagnose and recover after failures
SQL editor	How to browse databases and tables, run SQL, manage workbooks, and view results and charts
Processing patterns	How to design structured, multimodal, and knowledge-base scenarios as production-ready processing chains