Data Connections

Data connections address two needs: bringing data from external systems into the workspace and delivering processed results back to external systems. Connectors retain connectivity and read/write capabilities, which are then used by loading, workflow, or export tasks.

Data Flow

External data sources
    ↓
Connectors (address, credentials, authentication, and read/write capabilities)
    ↓
Data loading ─────────────→ Catalog tables / volumes
                              ↓
                       Workflow processing, queries, and knowledge-base use
                              ↓
Data export ─────────────→ External object storage, databases, or model platforms

Capability	Purpose	Use Case
Connectors	Retain connection information and read/write capabilities for external data sources	When initially connecting a database, object storage service, API, or enterprise application
Data Loading	Materialize external data into Catalog tables or volumes	When creating a reusable, governable data foundation
Load Tasks	Manage created synchronization tasks, schedules, and run status	When adjusting schedules, backfilling historical data, or troubleshooting failures
Data Export	Send files or table data from the Catalog to external systems	When delivering model training data, creating backups, or making results available to external systems

Preparation

1. Confirm that you have access to the target workspace, Catalog location, and external data source. Credentials stored by connectors should use an account permitted by your organization and granted the minimum required privileges.

2. Prepare the data source address, authentication information, and required network connectivity. For databases and APIs, also confirm that access from the network where MOI is located is allowed.

3. Determine the data target: load structured records into tables, and load unstructured files such as documents, images, audio, and video into volumes.

4. If a task needs to run over the long term, determine the synchronization frequency, incremental field, available compute resources, and the read load that the source can tolerate in advance.

Choose the Right Entry Point

External data can enter the Catalog through either a standalone load task or temporary reads by an “External Data” data-reading node in a workflow. Both use the same connectors, but their outputs and lifecycles differ.

Consideration	Data Load Task	Read External Data in a Workflow
Nature	A standalone data synchronization pipeline that can run over the long term	A data retrieval step within a workflow
Trigger	One-time, scheduled, or real-time synchronization	Triggered by a workflow run or workflow schedule
Output	A reusable table or volume in the Catalog	The current workflow’s data stream, which can be persisted as required by the process
Best for	Reuse by multiple processes, continuous freshness, or creating a data asset	Serving only one process, one-time validation, or temporary data retrieval

If data will be reused by multiple workflows, queries, or knowledge bases, or needs to remain current independently of a workflow, create a load task. If the data serves only the current process, reading external data directly in the workflow is more lightweight. After data enters the Catalog, use Data Processing to clean, transform, or analyze it.

Connectors and External Databases

Database connectors maintain only instance- or service-level connection information. You typically select a specific database, schema, and table only when loading or querying data. This allows the same connector to be reused by multiple tasks without storing credentials separately for each database.

For the complete connection fields for all data sources (41 types), see Reference · Connectors. The Connectors page in this guide covers only creation, testing, permissions, and maintenance.

To keep data in the source database and read it through federated queries, reuse an existing database connector. The connector page is used to maintain the connection itself. Mounting external catalogs, refreshing metadata, and SQL pushdown do not have separate operation pages under Data Connections.

Pages in This Guide

Page	Content
Connectors	Creation, testing, editing, deletion, and permissions; links to the reference for complete field details
Data Loading	Loading methods for spreadsheets and databases, files, email, web collection, and local uploads
Load Tasks	Task list, details, editing, scheduling, incremental synchronization, backfill, and run control
Data Export	Exporting data to object storage, MatrixOne, or model/dataset platforms

Related Documentation

• Reference · Connectors — Field dictionary for 41 data source types in 9 categories

• Platform Management · Catalog — Select target tables and volumes for loading, and view data assets

• Data Processing — Orchestrate and run workflows using Catalog data

• Knowledge Bases — Use connected files or tables as knowledge base data sources

• User Permissions — Authorize access to workspace resources and connectors

Documentation Tree

Documentation tree