# Catalog Catalog is the workspace's **unified data-asset center**. Raw data loaded through connectors, results written by workflows, and accumulated models and operators are all browsed, managed, and reused here. It is the destination for import tasks and the upstream asset source for SQL, workflows, knowledge bases, sharing, and export. Entry point: **Resource Center → Catalog**. --- ## How Catalog works with other features Catalog receives loaded and processed data and makes it available for queries, knowledge bases, sharing, and export. ```text Connectors / load → Catalog → Workflows · SQL · Knowledge bases · Sharing/subscriptions · Export ↑__________________| (results can be written back) ``` | What you want to do | Where to go | | --- | --- | | Bring in external data | [Data connections](../data-sources/index.md): connectors, loading, and import tasks | | Prepare compute for SQL, workflows, or importing data | [Compute instances](compute.md) | | Process tables and write back results | [Data processing](../processing/index.md): workflows, running, and debugging | | Query tables in Catalog | [SQL editor](../processing/sql-editor.md) | | Turn data into searchable, conversational knowledge | [Knowledge bases](../knowledge-bases/index.md) | | Make data available to agents | [Agents](../agents/index.md) (configure a knowledge base first) | | Let another workspace use the data | [Data sharing](share-data.md), [Data subscriptions](subscribe.md) | | Export externally | [Data export](../data-sources/export.md) | | Control who can manage these objects | [User permissions](../permissions/index.md) | | Check task completion and alerts | [Monitoring and alerts](../monitoring/index.md) | | View storage and other costs | [Cost and billing](../../billing/index.md) | Cross-workspace sharing, external-database catalogs, and similar capabilities depend on console availability. Follow the procedures below when their entries are enabled. --- ## Object-tree structure From top to bottom: ```text Workspace → Catalog → Database → Volume / Table / Model / Operator ``` | Level | Meaning | | --- | --- | | Catalog | Top-level group: an **MOI catalog** whose data is stored in the platform, or an **external database catalog** whose data remains at the source while the platform registers metadata | | Database | Container for a group of related objects | | Table | Structured data | | Volume | File-based, unstructured data | | Model | Model-service integration | | Operator | Reusable processing capability | Catalog also contains read-only **subscribed libraries** created through [data subscriptions](subscribe.md) and built-in read-only **system catalogs**. --- ## Browse and preview 1. Expand the object tree from the workspace root: catalog → database → volume/table/model/operator. 2. Search by name or filter by type, such as volume, table, operator, or model. 3. Select a node; the details pane and breadcrumbs change accordingly. External catalogs, subscribed libraries, and built-in system objects generally use different icons or labels such as "System Default." ### Common details-page content | Object | Main details | | --- | --- | | Catalog | Database list and size information; external catalogs also show connection and synchronization status | | Database | Volume, table, operator, and model lists | | Table | Row count and size; columns, statistics, SQL, data sampling, **lineage**, and other tabs | | Volume | File list; upload, download, **file lineage**, and more | | Model | Category, version, and integration method such as API or HuggingFace | | Operator | Basic information; an **API Service** tab if an API has been published | **Data sampling** for a table is available only when processing has completed or the object is otherwise ready. --- ## Manage objects | Object | Common operations | Example creation entry | | --- | --- | --- | | Catalog | Create / rename / delete | "Create Catalog" in Catalog | | Database | Create / rename / delete | "Create Database" in catalog details | | Table | Create / rename / delete | "Create → Table" in a database | | Volume | Create; upload/download files | "Create → Volume" in a database | | Model | Create / manage | "Create → Model" in a database | | Operator | Create / manage; publish as an API | "Create → Operator" in a database | Before deletion, confirm that downstream workflows, knowledge bases, export tasks, and other consumers have no hard dependency. For impact analysis, first inspect the **Lineage** tab in table details. ### Create a catalog Choose one of two types: | | MOI catalog | External database catalog | | --- | --- | --- | | Data location | Inside the platform | Remains in the source database; the platform registers metadata | | Creation method | Create databases, tables, and volumes | Select an existing **database connector** and synchronize its hierarchy | | Writability | Supports create, delete, and update | Generally read-only; structure follows the upstream source | | Query | Uses platform storage and compute | Pushes SQL down to the source and does not require a platform-side data copy | **To create an external catalog when the entry is available:** 1. Create a catalog and select **External Database Catalog**. 2. Select a configured database connector. Maintain connection information under [Connectors](../data-sources/connectors.md), not here. 3. Enter a catalog name and description. 4. After creation, synchronize database, table, schema, and other metadata. Hierarchy terms differ among source databases: PostgreSQL includes schemas, while MongoDB uses collections, for example. Federated queries do not mount platform storage but may still depend on source connectivity. For high-stability requirements, load the data into an MOI catalog instead. ### Create a table Define columns individually in the UI, or paste/upload `CREATE TABLE` SQL. ### Volumes and files - Uploading creates a [import task](../data-sources/load-tasks.md) that the platform uses to parse and ingest data. - Compressed archives generally cannot be uploaded directly; extract them first. - Files with the same name as an existing volume file may be skipped. - The file list can download original files or parsed artifacts, subject to the UI. ### Models Models wrap LLM, embedding, and similar integrations for use by workflow operators, knowledge bases, and other features. Common integration methods include: - **API integration:** provider, base address, model ID, API Key, and other settings; connection testing is supported - **HuggingFace:** model ID, token, and other settings Also specify a model category, such as large language model, vision, OCR, text embedding, or reranking. For the billing boundary among platform **Credit, Genesis, and BYOM**, see [Credit and billable items](../../billing/credit-and-pricing.md). ### Operators An operator is an atomic "input + parameters → processing → output" capability that can be reused as a custom node in a [workflow](../processing/workflow.md). When creating one, configure a Chinese name and English identifier, language such as Python or SQL, input/output descriptions, parameter list, implementation code, and more. A code node in a workflow can also be frozen as an operator and added to Catalog. #### Publish as an API service (from operator details) Both system and custom operators can **publish an API service** from operator details for synchronous calls by external systems: callers submit data and receive an immediate result. Prefer the workflow job mechanism for long-running batch processing. Publishing generally configures: | Item | Description | | --- | --- | | Operator version | Pin a version so platform upgrades do not change behavior | | Runtime resource | Non-model operators use a [task compute instance](compute.md); model operators may not require one and can use a model service | | Call-rate limit | Maximum calls per unit of time | | Authentication | API Key / OAuth / public access (use public access cautiously in production) | | Timeout and concurrency | Runtime settings | After publication, an **API Service** tab may appear in details for enabling/disabling the service and viewing endpoint documentation. For development-side calls, see [Development](../../../developer/index.md). ### System catalogs System operator libraries, system model libraries, sample libraries, metadata libraries, and similar assets are **read-only** and cannot be created, deleted, or modified. System operators and model versions used by workflows often come from here. Create custom assets in an ordinary MOI catalog. ### Subscribed libraries These read-only libraries are created when another workspace [publishes](share-data.md) and this workspace [subscribes](subscribe.md). They are placed at the Catalog location selected during subscription. Their contents update with the source and become inaccessible when the subscription is canceled or publication is revoked. --- ## Data lineage Lineage answers "where did this data come from, and who uses it?" The platform derives it automatically from **declared processing relationships** such as loads, workflow reads/writes, exports, and knowledge-base associations; manual registration is generally unnecessary. ### Structured tables (Table details · Lineage) There are three levels, from coarse to fine: | Level | What it shows | | --- | --- | | Table-level graph | Upstream sources and downstream consumers of the current table | | Column-level mapping | Which upstream columns and logic produced each column | | Row-level trace | Source rows that contributed to a result, traced by grouping/association keys when applicable | Common graph nodes include external sources, processing nodes such as SQL, Python, and loading, Catalog tables, and consumers such as knowledge bases and export tasks. Switch between "Upstream · How it was produced" and "Downstream · Who uses it." Before changing or deleting a table, inspect downstream impact. A table not involved in platform processing may show no lineage. Column mappings may label pass-through, expression, aggregation, JOIN-key, and other transformation types. A processing node can link to its [workflow](../processing/workflow.md). ### Unstructured files (Volume · File lineage) Documents, images, audio, and video use page-level or source-unit-level **forward tracking**. Select a page or segment and follow artifacts through parsing → cleaning → chunking → embedding → knowledge base and other steps. Open **Lineage** from the file list in volume details. Typical minimum units are page/block for documents, whole image/region for images, and time slice for audio/video. Select a chunk to compare its originating layout block and inspect processing parameters at each step. This corresponds to knowledge-base chunk maintenance; see [Chunks and semantics](../knowledge-bases/chunks-semantics.md). ### Difference from job monitoring Lineage describes **data dependencies and origins**. To determine whether a job succeeded, see jobs and logs under [Monitoring and alerts](../monitoring/index.md). For workflow runtime debugging, see [Run and debug](../processing/run-debug.md). --- ## Versions and connection strings | Capability | Description | | --- | --- | | Data versions | Some objects support historical versions, comparisons, and rollback under version management in object details | | SQL connection string | Host, port, database/table path, and authentication for direct BI/client/application connections | Connection-string queries consume **query compute instances**. For sustained workloads, specify a dedicated query instance; see [Compute instances](compute.md). For development-side connection instructions, see [Development](../../../developer/index.md). --- ## Storage and costs Raw data, processing results, vectors, and other data in Catalog generally count toward Credit billable items such as **data storage**. For usage and bills, see [View cost and usage](../../billing/cost-and-usage.md) and [Credit and billable items](../../billing/credit-and-pricing.md).