# Extract key information from S3 engineering documents in bulk

## Scenario

In engineering investigation and geological survey projects, business teams often accumulate large collections of borehole logs, PDFs, scans, and drawings in MinIO or another S3-compatible object store. Project registers, quality checks, and reporting need fields that can be queried directly—not a collection of files that someone must open one by one.

Without automation, a document specialist must download each file, find the borehole identifier and depth, copy the values into a business system, and record the processing status. As file volume grows, omissions and transcription errors become more likely, and it becomes difficult to answer, “Has this file been processed, and where is its result?”

The purpose of this scenario is to process a large collection of engineering documents that already exists in an S3 folder and convert every document into a verifiable, ingestible structured result rather than stopping after text recognition. MOI uses a Standard S3 connector for batch import, and an information-extraction workflow parses all imported files against the same schema.

The fields and results produced in this tutorial can support the following downstream actions. Adapt them to your own business system.

| Extracted result | Typical downstream use |
| --- | --- |
| Borehole identifier `hole_id` | Associate the file with a borehole record and detect missing or duplicate identifiers |
| Borehole depth `depth` | Write to an engineering register, compare with a design value, or aggregate project depth |
| Structured JSON with source association | Load results through an API, generate statistics, and trace a result back to its source file |
| Processing status | Track batch progress and route failures or missing required fields to manual review |

This tutorial walks you through building a batch-processing pipeline. MOI uses a Standard S3 connector to import the engineering files already stored in a MinIO folder into Catalog in one operation. You then run an information-extraction workflow manually, extract fields against a fixed schema, and write structured JSON to a result volume. For downstream integration, you can optionally query the workflow job and download artifacts through the API.

This tutorial uses a borehole log as an example and extracts the borehole identifier `hole_id` and borehole depth `depth`. The same process also applies to contracts, reports, forms, scans, and other documents that require a fixed set of fields.

| Input | Automated pipeline | Final artifact |
| --- | --- | --- |
| A batch of existing PDFs, scans, or drawings in MinIO | Standard S3 connector → one-time batch import → manually run the parsing and information-extraction workflow | JSON or an artifact ZIP in the Catalog result volume |

Each step ends with a **Completion check**. Continue only after you see the expected result.

## What you will complete

1. Create a Standard S3 connector for MinIO and import the files in a selected folder once.
2. Create a workflow from the **Parsing and Information Extraction** template and configure a JSON Schema.
3. Run the workflow manually to process the imported data volume as a batch.
4. Verify the import count, workflow job, and structured results.
5. If needed, use the API to query the job result and download JSON artifacts.
6. Reconcile the entire batch across the pipeline and retry only failed files or failed workflow jobs.

## Before you begin

Prepare the following resources:

- Have an MOI account. Use [MOI Cloud sign in/sign up](https://moi.matrixorigin.cn/).
- An MOI workspace where you can create connectors, import tasks, Catalog volumes, and workflows;
- A MinIO bucket and folder containing the files to process, the MinIO S3 API endpoint, and an Access Key that can read that path;
- A model that can parse documents and extract information;
- A personal access token with access to the target workspace, for use with the optional result-API section;
- The synthetic sample package supplied with this tutorial.

### Download the synthetic sample package

Download and extract the [synthetic S3 engineering-document sample package](../assets/downloads/moi-s3-engineering-demo.zip). Every project name, identifier, date, value, and layout in the package was created from scratch for this tutorial. It contains no real customer or engineering-project information.

| File | Purpose | Expected result |
| --- | --- | --- |
| `input/demo-borehole-log-001.pdf` | First end-to-end validation | Extract `DEMO-ZK-001` and `35.8m` |
| `input/demo-borehole-log-002.pdf` | Validate removal of OCR-inserted spaces | Normalize the identifier to `DEMO-ZK-002` and extract `48.2m` |
| `input/demo-borehole-log-missing-depth.pdf` | Validate exception routing | The required depth is absent; do not accept the file as a valid business record and route it to manual review |

The package also contains a ready-to-copy schema, expected JSON for the two valid cases, and acceptance criteria for every case.

The following table lists the example names used in this tutorial. In your environment, use the bucket, volume, and workflow names that already exist or that you actually create, while keeping the input and result volumes clearly separated.

| Object | Example name or location |
| --- | --- |
| S3 source folder | `s3://<bucket>/incoming/borehole-logs/` |
| MinIO Standard S3 connector | `conn_s3` |
| Source Catalog volume | `engineering_automation / documents / incoming` |
| Result Catalog volume | `engineering_automation / documents / extracted` |
| Workflow | `engineering_document_extraction` |

## Steps

### 1. Create and test the MinIO connector

MinIO exposes an S3-compatible object-storage interface, so select **Standard S3** in MOI rather than looking for a separate MinIO connector type. This step stores connection parameters only; it does not import files.

Before opening MOI, confirm the following in MinIO:

- The bucket that will store the engineering files;
- The prefix containing the files to process, such as `incoming/borehole-logs/`;
- The MinIO **S3 API endpoint**, not the browser URL for MinIO Console; and
- An Access Key ID and Access Key Secret with read access to the target bucket and prefix.

Then create the connector in MOI:

1. Sign in to MOI and confirm that the workspace for this tutorial appears in the top-left corner.
2. Go to **Data Connection > Connectors**.
3. Click **New Connector**.
4. Enter `conn_s3` as the connector name.
5. Select **Object Storage > Standard S3**.
6. Select at least **Import** as the purpose. If later export tasks will reuse the connector, you can also select **Export**.
7. Complete the connection form as follows:

| Field | MinIO value | Guidance |
| --- | --- | --- |
| Endpoint | MinIO S3 API address | Include the protocol and port; do not use the Console URL |
| AccessKey ID | Access Key for the MinIO user | Grant only the permissions required for the target bucket and prefix |
| AccessKey Secret | Matching Secret Key | Enter the Secret Key that matches the AccessKey ID |
| File path | `<bucket>/incoming/borehole-logs/` | Combine the bucket name and source prefix without `s3://` |
| S3 address style | **Path Style** | Use the address style configured for your MinIO service |
| Region | Region configured in MinIO | Leave it empty when optional; when required, it must match the server |

8. Click **Test Connection**. After the success message appears in the upper-right corner, click **Confirm**.

![Create a Standard S3 connector for MinIO by selecting Object Storage and Standard S3, entering redacted endpoint and access-key values, a file path, and Path Style, and verifying the success message.](images/minio-standard-s3-connector-config.png)

The screenshot uses the Chinese interface, and all endpoint and access-key values are redacted.

9. Return to the connector list and locate `conn_s3`. Confirm that its type is **Standard S3**, its status is **Connected**, and its purposes include **Import**.

![The connector list shows conn_s3 with type Standard S3 and status Connected.](images/minio-standard-s3-connector-connected.png)

The connection test verifies only that the current parameters can access MinIO. It does not validate the files, filtering rules, target volume, or Catalog write permissions selected by the import task.

**Completion check:** The connector list contains `conn_s3`; its type is **Standard S3**, its status is **Connected**, and its purposes include **Import**.

For other connector types, purposes, editing, and permission rules, see [Connectors](../guides/ai-studio/data-sources/connectors.md).

### 2. Create the source and result volumes

Next, prepare two locations in Catalog: one to receive source files imported from MinIO and one to store information-extraction results.

1. Go to **Resource Center > Catalog**.
2. Create the Catalog `engineering_automation` and database `documents`.
3. Under `documents`, create a volume named `incoming`.
4. In the same database, create a volume named `extracted`.
5. Confirm that both volumes are empty.

Separating inputs and outputs lets you compare the number of imported files with the number of produced results. It also prevents artifacts from a previous batch from becoming inputs to the next workflow run.

**Completion check:** Catalog contains both `engineering_automation / documents / incoming` and `engineering_automation / documents / extracted`, and both volumes are empty.

### 3. Import the S3 folder once

This scenario assumes that the MinIO folder already contains a large collection of engineering files. The import task runs once and writes all currently eligible files in the selected folder to the input data volume as a batch.

To validate the tutorial with synthetic data, first upload the three PDFs from the package's `input/` directory to a test folder in MinIO. For real data, use your own bucket, folder, and file count.

1. Go to **Data Connection > Import Tasks** and click **Create Import Task**.
2. For the import type, select **Unstructured**.
3. Set the data source to **Connector Source** (shown as **Connector Data Source** in some versions), and then select the `conn_s3` connector.
4. Select the target data volume that will receive this batch. If you followed the tutorial example, you can select `engineering_automation / documents / incoming`.
5. Set the import mode to **One-time**.
6. Set the parsing strategy to **Flatten**. If a downstream process needs the MinIO directory hierarchy, select **Preserve Directory Structure** instead.
7. Under the deduplication strategy, select both **File Name** and **MD5**, and set the duplicate-file action to **Skip**.
8. Under the processing scope, retain the PDF, Word, and image types required by this batch. Leave **Path Regex** empty unless you need additional path-based filtering.
9. In the **Import Files** list at the bottom of the page, find and select the MinIO folder that actually contains the source documents. Select the folder row instead of selecting files individually.
10. Click **Create and Start Import**.
11. Return to the **Import Tasks** list, wait until the status changes to **Completed**, and open the details to verify the successful, failed, and skipped file counts.

For a large collection, record the total count and size of S3 objects that match the filters before starting. When using a new file type or parsing configuration, validate a smaller test folder first and then import the full batch.

**Completion check:** The one-time import task is complete; the connector, source folder, and target volume are correct; and the sum of successful, failed, and skipped files reconciles with the number of S3 files in scope for this batch.

For the complete rules for one-time import, deduplication, and retrying failed files, see [Data Loading](../guides/ai-studio/data-sources/load-data.md).

### 4. Create an information-extraction workflow from a template

This step creates a workflow that parses engineering files and extracts information according to a schema. You will configure four nodes in order: read, parse, extract, and save.

1. Go to **Data Processing > Workflows**.
2. Click **Create Workflow**.
3. In the **Create Workflow** window, find **Parsing and Information Extraction** and click **Use Template**.
4. Enter `engineering_document_extraction` as the workflow name.

### 4.1 Configure Read MOI Volume v2

1. In the workflow editor, click **Run Parameters**, and then select **Read MOI Volume v2** on the left.
2. For **Compute Resource**, keep **Follow Workflow Default**.
3. Under **Source File**, select the data volume that actually receives the imported files. If you followed the earlier example in this tutorial, you can select `engineering_automation / documents / incoming`; otherwise, use the volume you created in your environment.
4. Under **File Scope**, select **By File Type**. Select at least **Document** for this scenario. If the MinIO folder also contains standalone scanned images, select the image types you actually use.

![Configure Read MOI Volume v2 by selecting the source data volume and limiting the processing scope by file type.](images/s3-workflow-read-moi-volume.png)

The `catalog_s3 / db_s3 / col_s3` path in the screenshot is an interface example, not a required path.

### 4.2 Configure General Document Parsing

1. Select **General Document Parsing** on the left.
2. Set **Parsing Tier** to **Standard** for a balance of parsing speed, quality, and cost.
3. Keep **Parsing Range** disabled to parse the entire document. Enable it and enter a page range only when you need to process specific pages.

If the input consists mainly of Office documents and speed is the priority, use **Native**. If the input contains scanned pages, complex layouts, tables, or formulas and the Standard tier is insufficient, use **Enhanced**.

![Configure General Document Parsing with the Standard parsing tier and full-document parsing.](images/s3-workflow-document-parsing.png)

### 4.3 Configure Information Extraction

1. Select **Information Extraction** on the left.
2. Set **Configuration Method** to **JSON Configuration**.
3. Paste the following schema into **Extraction Field Definition**, retaining both `description` and `required`:

```json
{
  "type": "object",
  "properties": {
    "depth": {
      "type": "string",
      "description": "Borehole depth. Extract the number and the unit m; it usually appears in the final borehole depth field. Remove the space between the number and unit in the output; for example, output 35.8 m as 35.8m."
    },
    "hole_id": {
      "type": "string",
      "description": "The identifier immediately before '钻孔柱状图' in the title at the top of the borehole log. Remove extra spaces introduced by OCR while preserving hyphens; for example, output D E M O - Z K - 0 0 2 as DEMO-ZK-002."
    }
  },
  "required": ["depth", "hole_id"]
}
```

The `description` text explains each field's meaning, location, and output format, and directly affects extraction quality. This example already places its constraints in each field's `description`, so you can leave **Additional Instructions** empty. Use that field only for rules that apply to every extracted field.

![Configure Information Extraction with a JSON definition containing the depth and hole_id fields.](images/s3-workflow-information-extraction.png)

### 4.4 Configure Save to MOI Volume

1. Select **Save to MOI Volume** on the left.
2. Under **Output Location**, select the data volume that actually stores the extraction results. If you followed the earlier example in this tutorial, you can select `engineering_automation / documents / extracted`; otherwise, use the result volume you created in your environment.
3. Do not use the input data volume as the output location, because a later batch run could treat a result file as a new input.

![Configure Save to MOI Volume by selecting the output data volume for the extraction results.](images/s3-workflow-save-moi-volume.png)

The `catalog_s3 / db_s3 / col_output` path in the screenshot is an interface example. After configuring all four nodes, click **Confirm**, keep the template's default result-naming rule, and save the workflow.

Select **Parsing and Information Extraction**, not another template that writes its result to a data table. This tutorial delivers JSON files, so you do not need to configure an additional embedding model.

**Completion check:** **Read MOI Volume v2** and **Save to MOI Volume** point to the input and result volumes actually used in your environment; **General Document Parsing** uses an appropriate parsing tier; and **Information Extraction** contains the two required fields `depth` and `hole_id`.

For the differences among a workflow definition, a run, and a workflow execution, see [Workflow](../guides/ai-studio/processing/workflow/index.md).

### 5. Run the batch workflow manually

This scenario processes a batch that has already been imported into a data volume, so you run the workflow manually.

1. Confirm that the one-time import task is complete and that the file count in the input data volume matches the import result for this batch.
2. Click **Save and Enable**.
3. On the workflow page, click **Run** to open the parameter confirmation window for this run.
4. Verify the input volume and file-type scope under **Read MOI Volume v2**, and verify the output location under **Save to MOI Volume**.
5. Confirm the parameters, start the run, and view the batch-job status under **Monitoring > Jobs**.

If the input volume also contains files from other batches, use a separate volume or a more precise file scope during import so that unrelated files are not included in this run.

**Completion check:** The workflow is enabled and one manual run has started successfully. The job details show the input volume, file scope, and output volume for this batch.

### 6. Inspect the batch job and extraction results

1. Go to **Monitoring > Jobs** and filter by the workflow name `engineering_document_extraction`.
2. Find the batch job you started manually and wait until its status changes to **Completed**. If it fails, open the first failed node and inspect the error.
3. Open the job details and confirm that the read, parse, information-extraction, and result-saving nodes all ran.
4. Open the result data volume configured in your environment. If you followed the tutorial example, open `engineering_automation / documents / extracted`.
5. Use the source-association information to match every result file to its S3 source file.
6. Open the primary result whose name ends in `_extract.json` and confirm that valid files contain both `depth` and `hole_id`.

The two valid files in the synthetic package should produce these values:

| Input file | `hole_id` | `depth` |
| --- | --- | --- |
| `demo-borehole-log-001.pdf` | `DEMO-ZK-001` | `35.8m` |
| `demo-borehole-log-002.pdf` | `DEMO-ZK-002` | `48.2m` |

The title in `demo-borehole-log-002.pdf` intentionally contains extra spaces, so the result must not retain OCR-style spaces. `demo-borehole-log-missing-depth.pdf` lacks a required business value. Depending on the model and product version, it might produce a schema-validation failure, a failed job, or a result without `depth`. In every case, do not accept it as a valid business record; route it to manual review.

**Completion check:** The input files, job processing records, and output results for this batch can be reconciled. The two valid examples match their expected values, and the file without a depth value does not enter valid business data.

For information about inspecting a failed node, its inputs and outputs, and its logs, see [Run and Debug](../guides/ai-studio/processing/workflow/run-debug.md).

### 7. Retrieve batch results through the API (optional)

If a downstream system needs to consume batch results automatically, it can query the status and download artifacts through the API after the manual job completes. First obtain the workflow ID and execution ID from the MOI job details, and then prepare the result volume ID.

Set variables in a controlled terminal:

```bash
export AI_STUDIO_API_KEY="<YOUR_API_KEY>"
export WORKSPACE_ID="<YOUR_WORKSPACE_ID>"
export WORKFLOW_ID="<YOUR_WORKFLOW_ID>"
export EXECUTION_ID="<YOUR_EXECUTION_ID>"
export RESULT_VOLUME_ID="<YOUR_RESULT_VOLUME_ID>"
```

First query the workflow execution result once to confirm that this manual run has reached a terminal state:

```bash
curl "https://moi.matrixorigin.cn/newmoi/workflow/v2/workflow-apps/$WORKFLOW_ID/executions/$EXECUTION_ID/result" \
  -H "X-API-Key: $AI_STUDIO_API_KEY" \
  -H "X-Workspace-ID: $WORKSPACE_ID" \
  | jq '.data.result | {status, case_result, case_error}'
```

Continue to the download only when `status` indicates that the execution is completed. If `case_result` already contains the primary JSON, the downstream system can consume it directly. When the workflow writes the result as a file, first list the files in the result volume:

```bash
curl -X POST "https://moi.matrixorigin.cn/newmoi/catalog/file/list" \
  -H "X-API-Key: $AI_STUDIO_API_KEY" \
  -H "X-Workspace-ID: $WORKSPACE_ID" \
  -H 'Content-Type: application/json' \
  -d '{
    "filters": [
      {
        "name": "volume_id",
        "values": ["'"$RESULT_VOLUME_ID"'"]
      }
    ],
    "page": 1,
    "page_size": 100
  }' \
  | jq '.data.list[] | {id, name, ref_file_id, ref_workflow_id, workflow_role}'
```

The example queries 100 items per page. If the batch produces more than 100 results, increment `page` and continue until the returned list is empty. Do not treat the first page as the entire batch.

Use `ref_file_id`, `ref_workflow_id`, and the source file name to locate the output for this workflow execution, then record its `id`:

```bash
export RESULT_FILE_ID="<MATCHED_RESULT_FILE_ID>"
```

Then download the corresponding file:

```bash
curl -X POST "https://moi.matrixorigin.cn/newmoi/catalog/file/download" \
  -H "X-API-Key: $AI_STUDIO_API_KEY" \
  -H "X-Workspace-ID: $WORKSPACE_ID" \
  -H 'Content-Type: application/json' \
  -d '{
    "file_id": "'"$RESULT_FILE_ID"'",
    "volume_id": "'"$RESULT_VOLUME_ID"'",
    "as_zip": true
  }' \
  -o engineering-result.zip
```

Repeat the match–record ID–download sequence for the batch, or iterate over the paginated results in a downstream program.

Inspect the archive and read the primary extraction result:

```bash
unzip -l engineering-result.zip
unzip -p engineering-result.zip '*_extract.json' | jq .
```

Do not treat a `2xx` API response as proof that the business operation succeeded. The downstream system must also confirm that the workflow execution is completed and that the result file is readable. Use the association between the source file and workflow execution to avoid downloading the wrong file.

**Completion check:** The API returns a completed status for the manual job. The primary extraction JSON is readable from the downloaded ZIP and matches the result shown in the interface.

For complete fields, see [Query workflow job results](../reference/api/ai-studio/data-processing/workflow-jobs/get-workflow-job-result.md), [Query file list](../reference/api/ai-studio/resource-center/catalog/list-files.md), and [Download file](../reference/api/ai-studio/resource-center/catalog/download-file.md).

### 8. Reconcile batch results and exception files

For a large collection, do not rely on only one page. Before processing, record the number `N` of objects in the S3 source folder that match the filters. After the job completes, reconcile the batch in this order:

| Checkpoint | Passing criterion |
| --- | --- |
| S3 source folder | The number of objects in scope is `N` |
| One-time import task | Successful, failed, and skipped files add up to `N` |
| Input data volume | Its file count matches the successful import count, with no unexpected duplicates |
| Manual workflow job | The job reached a terminal state and its processing scope matches the batch input |
| Result data volume | Valid files have traceable results; exception files are not accepted as valid records |

For the synthetic package, `N` is 3: the two valid files should produce the expected JSON, and the file without a depth should go to manual review. For a real large batch, compare “valid results + business exceptions + technical failures” with the successful import count. Every difference must be explainable.

If a file import fails, select only the failed file in the import-task details and click **Retry**. If the workflow job fails, inspect the first failed node and its error before retrying based on the cause. Rerunning a document with a genuinely missing business field cannot supply a real value, so route it directly to manual review. If the model reports that the input exceeds its context limit, reduce the page range, split the file, or change the model instead of repeatedly submitting the same input.

**Completion check:** The batch reconciles across S3, the import record, the input volume, the workflow job, and the result volume. Valid files produce the expected JSON, and exception files do not enter valid business data.

## Tutorial complete

You have completed S3 engineering-file extraction:

- imported a batch of files from MinIO;
- extracted fixed fields with a workflow; and
- produced structured JSON in the result volume.