Data Processing Practices¶
Data processing creates value by continuously turning raw data into AI-ready assets that can be queried, retrieved, cited, and used in decisions. A sound design must support data quality, reproducibility, governance, and consumption.
This page presents design methods and common patterns. It helps you turn a business objective into a verifiable, runnable, and maintainable processing chain.
Design backward from consumption¶
First ask who consumes the result, what form they need, and how acceptance is determined. Then work backward through inputs, processing, and execution.
Downstream use → Result asset → Acceptance criteria → Processing steps → Raw data → Access and permissions
Downstream use |
Required result |
Design focus |
|---|---|---|
Analysis or business queries |
Stable schema, clear definitions, repeatable calculation |
SQL transformations, field validation, target-table write strategy |
Knowledge retrieval and agent answers |
Locatable chunks, source metadata, usable index |
Parsing, cleaning, chunking, embedding, knowledge index |
Structured extraction |
Field definitions, format validation, exception samples |
Parsing, extraction, business-rule validation, result table |
External systems or teams |
Stable objects, clear versions and permission boundaries |
Catalog output, lineage, sharing/export strategy |
Periodic refresh |
Rerunnable, observable, recoverable processing |
Scheduling, idempotent writes, job monitoring, failure handling |
Define acceptance first. Completion depends on reliable downstream consumption, not merely a successful job status.
Four levels of AI readiness¶
Level |
Required state |
Processing implementation |
|---|---|---|
Accessible |
Data is a controlled workspace asset with clear access and permissions |
Connections, loading, Catalog objects, object permissions |
Understandable |
Structure, content, origin, and key fields can be explained |
Parsing, structuring, metadata, field/content checks |
Retrievable/computable |
Queries, chunks, or semantics use content correctly |
SQL, cleaning, chunking, embedding, indexing |
Traceable |
Origin, steps, parameters, and run records are recoverable |
Workflows, jobs, lineage, operation and SQL history |
Accessible raw files can still be hard to consume; retrievable results without provenance are hard to trust. Together these levels provide the data foundation for “Control Your Data, Trust Your AI.”
Recommended asset layers¶
Layer |
Contents |
Typical write |
Consumers |
|---|---|---|---|
Raw |
Newly loaded tables, volumes, files, snapshots |
Controlled persistence after loading or reading |
Processing workflows, quality checks |
Preparation |
Cleaned, parsed, normalized intermediate results |
Workflow nodes or SQL |
Later transformation, extraction, review |
Serving |
Stable results for business, knowledge bases, or agents |
Validated tables, indexes, or content assets |
SQL, knowledge bases, agents, sharing/export |
This is a design convention, not a built-in platform hierarchy. Names and locations should reveal whether data is consumable, overwritable, and owned.
Pattern 1: Clean structured data and produce result tables¶
Raw table → Field checks/filtering → SQL transformation and joins → Quality validation → Target result table → Analysis or downstream services
Design steps¶
In Catalog, identify the source table, primary key, time field, incremental range, and sensitive fields.
Validate filters, joins, and aggregations on a small range in SQL Editor.
Put stable SQL in a workflow SQL node and configure parameters, target table, and write mode.
Define checks for NULLs, duplicate keys, outliers, and row counts.
Test a small batch before configuring one-time or periodic execution.
Inspect row counts, duration, target tables, and exceptions in the job and retain traceable evidence.
Choose a write strategy¶
Strategy |
Suitable case |
Risk |
|---|---|---|
Create |
First creation of a result table |
Existing-table handling and schema evolution |
Replace |
Complete snapshots accepted by downstream consumers |
Accidental replacement loses history; coordinate read timing |
Append |
Independent incremental batches |
Reruns can duplicate data; use business keys or batch markers |
For periodic jobs, retain batch, date, or source identifiers and define deduplication or overwrite behavior explicitly.
Pattern 2: Prepare documents for a knowledge base¶
File volume → Document parsing → Cleaning → Chunking → Embedding/indexing → Knowledge base → Agent retrieval and answers
Questions for each step¶
Step |
Confirm |
|---|---|
File preparation |
Correct volume, stable names, provenance, permission boundary |
Parsing |
Correct recognition of content, tables, images, and layout; exception recording |
Cleaning |
Removal of distracting headers, footers, duplicates, garbling, and meaningless content |
Chunking |
Complete semantics and retained document, page, and section location |
Embedding/indexing |
Model and index match the knowledge-base scenario |
Validation |
Typical questions retrieve correct chunks and answers trace to sources |
Choose chunking from downstream questions; legal clauses, manuals, and meeting notes need different boundaries. For important documents, validate a small sample manually before enabling volume triggers or bulk runs. See Knowledge Bases.
Pattern 3: Extract structured business data¶
Files/content → Parsing → Cleaning and structure recognition → Field extraction → Format and business validation → Result table + exception queue
Design considerations¶
Focus |
Practice |
|---|---|
Define the target schema first |
Specify names, types, required fields, formats, and business meaning before using a model. |
Preserve provenance |
Retain file, page, chunk, or timestamp for every result. |
Separate missing from incorrect |
Route NULLs, format errors, and conflicts to review instead of the normal table. |
Add rule validation |
Apply deterministic checks to dates, amounts, IDs, enums, and field relationships. |
Sample acceptance |
Measure accuracy, completeness, and exception rates against known answers. |
Treat model extraction as uncertain. Critical fields such as amounts, contract parties, and compliance classes require explicit validation and exception paths before downstream writes.
Pattern 4: Process multimedia¶
Media files → Type-specific parsing → Content/time segments → Cleaning or extraction → Table/volume/index results → Business or knowledge consumption
Type |
Focus |
|---|---|
Image |
OCR/region quality, source, key-region location, exception handling |
Audio |
Transcription language, timestamps, speaker/paragraph boundaries, fallback for poor audio |
Video |
Time segments, audio/video relationships, cost, consumption scenario |
Mixed documents |
Relationships among tables, images, and text; page-level provenance |
Separate large-file processing, result validation, and index building so one parse failure does not invalidate successful files and failed ranges can be rerun independently.
Pattern 5: Event-driven and periodic refresh¶
Mode |
Suitable work |
Design focus |
|---|---|---|
Manual one-time |
Validation, backfill, temporary replay |
Record parameters and scope |
Volume trigger |
Process new files on arrival |
Clear input volume, independent per-file handling, locatable failures |
Periodic |
Daily/weekly/monthly refresh |
Idempotent writes, batch boundaries, late data, rerun strategy |
Determine whether data can arrive twice, how reruns avoid duplicates, and who restores a failure from which job record. See Run and Debug.
Processing-quality acceptance matrix¶
Dimension |
Evidence |
Example question |
|---|---|---|
Completeness |
Input, output, skipped, and failed counts |
Was every file processed without silent omissions? |
Correctness |
Samples, field rules, reconciliation |
Are amounts, dates, keys, chunks, and classes correct? |
Consistency |
Target structure, definitions, write mode |
Are fields comparable across batches? |
Traceability |
Source, node, parameters, job, lineage |
Can a result be traced to its file or source table? |
Rerunnability |
Idempotency, batch ID, recovery method |
Will rerun duplicate, overwrite, or create unexplained differences? |
Cost/timeliness |
Duration, resource size, input scale, frequency |
Are resources sufficient, and is needless full recomputation avoided? |
Security |
Permissions, sensitive fields, download/external boundaries |
Who can see raw data, results, and model output? |
Turn this matrix into the team’s minimum acceptance template.
Production checklist¶
Data and assets¶
Are inputs, outputs, and naming clear?
Is raw data retained and separated from temporary results?
Does the write mode support reruns?
Does Catalog lineage explain upstream and downstream impact?
Logic and models¶
Does each node have one explainable responsibility?
Are model inputs, prompts, outputs, and exception handling explicit?
Was SQL validated in a workbook without unnecessary full scans?
Are critical quality rules reapplied after model output?
Execution and recovery¶
Did tests cover typical, boundary, and exceptional inputs?
Does the trigger match data arrival and latency requirements?
Who can inspect, modify, and rerun a failed job?
Is there a plan for duplicates, late data, and partial success?
Governance and collaboration¶
Can others understand the workflow name, description, I/O, and owner?
Do roles follow least privilege for viewing, editing, running, and deleting?
Do downloads, shares, and exports respect data permissions?
Do important changes receive impact analysis and sample validation before execution?
Close the loop with agents¶
Processed assets feed knowledge bases and agents, but processing does not end when an agent is created. Retrieval, tool calls, answer quality, and feedback expose missing content, coarse chunks, incomplete fields, and stale updates. Feed those findings into the next processing iteration instead of changing only the prompt.
This feedback loop keeps data, knowledge, and agents improving around the same controlled assets.