# Evaluate multimodal RAG with reference answers

## Application scenario

After a business adds product specifications, maintenance bulletins, field notes, and topology diagrams to a knowledge base, the important question is not merely whether a model can produce a paragraph. The business needs to know:

- whether model names, times, numbers, and conditions in the answer are correct;
- whether the answer cites evidence that actually supports its conclusion;
- whether content in tables, images, and text was parsed and retrieved correctly;
- whether the model says that the sources cannot confirm a claim when no answer exists; and
- whether results remain stable after a source, model, or retrieval configuration changes.

Without fixed questions and reference answers, a few free-form chats cannot distinguish a plausible one-off response from repeatable quality. Once a knowledge base serves business users, one wrong number, unrelated citation, or unsupported extension can mislead downstream decisions.

This tutorial turns RAG question answering into a reproducible acceptance workflow. You will use a fully synthetic campus-device dataset, create an evaluation knowledge base, ask a fixed question set, and record results against reference answers, expected sources, and forbidden claims.

The final artifact is not a screenshot proving that “the model answered.” It is an evaluation record that supports:

| User input | MOI processing | Final artifact | Downstream use |
| --- | --- | --- | --- |
| Multiformat sources, fixed questions, reference answers | Parsing and indexing, retrieval, answer generation, source location | Answer, citation, run identifier, and verdict for each case | Release acceptance, model or configuration comparison, regression testing after source updates |

This tutorial explains how to verify that a multimodal RAG application produces answers that are correct, traceable, and bounded by the supplied evidence.

```{note}
MOI does not currently provide a knowledge-base Golden Set page that automatically scores these cases. This tutorial uses a manual evidence worksheet. It never uploads reference answers to MOI and does not assume that the platform performs automatic scoring.
```

## What you will complete

1. Download and inspect synthetic multimodal sources and an evaluation workbook.
2. Create an evaluation knowledge base isolated from production and add only the knowledge sources.
3. Inspect the processing status, preview, and chunks for Markdown, HTML, text, and image files.
4. Run three smoke-test questions for an exact fact, image understanding, and an unsupported claim.
5. Execute nine fixed questions in separate conversations and record answers, citations, and run evidence.
6. Assign `PASS`, `PARTIAL`, `FAIL`, or `UNJUDGED` using the reference answers.
7. Classify failures as source, parsing, retrieval, generation, context, or benchmark issues.

## Before you begin

Prepare the following:

- Have an MOI account. Use [MOI Cloud sign in/sign up](https://moi.matrixorigin.cn/).
- An MOI workspace where you can create and modify a knowledge base;
- A model that can answer questions over a knowledge base;
- Image indexing and an image Embedding model for the image case;
- A spreadsheet application that can open `.xlsx` files; and
- The synthetic evaluation package supplied with this tutorial.

### Download the synthetic evaluation package

Download and extract the [synthetic multimodal RAG evaluation package](../assets/downloads/moi-multimodal-rag-evaluation-demo.zip). Every organization, product, model, firmware version, date, metric, network layout, and observation in this package was created from scratch. It contains no real customer, project, or business data.

```text
moi-multimodal-rag-evaluation-demo/
├── README.md
├── NOTICE.md
├── manifest.json
├── knowledge-sources/
│   ├── device-overview.md
│   ├── maintenance-bulletin.html
│   ├── field-test-notes.txt
│   └── network-topology.png
└── evaluation/
    └── multimodal-rag-benchmark.xlsx
```

| File | Source type | What it tests |
| --- | --- | --- |
| `device-overview.md` | Markdown product overview | Exact values, conditions, and evidence boundaries |
| `maintenance-bulletin.html` | HTML bulletin and table | Times, versions, protocols, and tabular content |
| `field-test-notes.txt` | Plain-text field notes | The difference between one observation and a formal commitment |
| `network-topology.png` | Network-topology image | Node relationships, port information, and the failover target |
| `multimodal-rag-benchmark.xlsx` | Bilingual evaluation workbook | Fixed questions, reference answers, forbidden claims, and result recording |

Use these names throughout the tutorial:

| Object | Example name |
| --- | --- |
| Knowledge base | `demo_multimodal_rag_evaluation` |
| Initial run batch | `baseline-20260831` |
| Initial configuration version | `v1` |

```{important}
Add only the four files under `knowledge-sources/` to the knowledge base. Do not upload `evaluation/`, the workbook, `README.md`, `NOTICE.md`, or `manifest.json`. If reference answers enter the retrieval scope, the model can retrieve them directly and the evaluation is invalid.
```

## Steps

### 1. Inspect the package and evaluation boundary

Confirm locally that the knowledge sources are separate from the evaluation assets before opening MOI.

1. Open `README.md` and confirm that the package identifies itself as synthetic.
2. Open `manifest.json` and confirm that each of the four `knowledge-sources/` entries has `upload_to_knowledge_base` set to `true`.
3. Confirm that `evaluation/multimodal-rag-benchmark.xlsx` has `upload_to_knowledge_base` set to `false`.
4. Open the workbook and read the `README` worksheet.
5. Open `Questions-EN`. Confirm that it contains Q01 through Q09 and that Q09 has an empty reference answer with a ground-truth status of `UNJUDGED`.
6. Do not change `Questions-EN`. The baseline must use identical wording, reference answers, and judging boundaries.

The SHA-256 values in `manifest.json` prove that a later run uses the same source version. When a file changes, create a new source version instead of reusing the old configuration version.

**Completion check:** The four knowledge sources and one evaluation workbook exist; the workbook is explicitly excluded from upload; and Q01–Q09 and their ground-truth statuses are visible.

### 2. Create an isolated evaluation knowledge base

Keep the evaluation knowledge base separate from one that serves production users so test sources and settings cannot change production answers.

1. Sign in to MOI and confirm that the intended workspace appears in the upper-left corner.
2. Go to **Knowledge Bases** and click **New Knowledge Base**.
3. Enter `demo_multimodal_rag_evaluation` as the knowledge-base name.
4. In the description, enter “Synthetic RAG evaluation; not for production Q&A.”
5. Select **Local Upload**, then select **Unstructured** as the import type.
6. Select only the four files in `knowledge-sources/`.
7. Configure image indexing and an image Embedding model available in the current workspace. Exact controls depend on the deployment.
8. Submit the creation request.

After creation, MOI opens **Advanced Configuration > Data**. Do not add the evaluation workbook later through **Add Data**.

If image indexing or an image Embedding model is unavailable, do not mark Q03 as `FAIL`. Record it as `UNJUDGED` or as an environment blocker until that capability is available. You can still run the text cases.

**Completion check:** The knowledge base contains exactly four synthetic knowledge sources and no evaluation file or reference answer.

For the complete creation and source-selection workflow, see [Creation and Data Sources](../guides/ai-studio/knowledge-bases/create-and-sources.md).

### 3. Wait for readiness and inspect parsing

Do not judge RAG quality until source processing completes.

1. Under **Advanced Configuration > Data**, wait until all four files show **Ready**.
2. Open each file and select **File Preview**.
3. In `device-overview.md`, confirm that “72 hours” and “TCP 7443” are present.
4. In `maintenance-bulletin.html`, confirm that `A-20`, `3.1.0`, and `HTTPS` remain in the same table row.
5. In `field-test-notes.txt`, confirm that “78 seconds” and “not a service-level guarantee” remain in the same semantic context.
6. Open the image or image-text chunk for `network-topology.png` and confirm that it captures `Backup Gateway B`, `RTO ≤ 90 s`, `TLS 1.3`, and `TCP 7443`.
7. Open **Smart Chunking** and confirm that these facts are not truncated, merged with unrelated content, or disabled.

| Symptom | Check first |
| --- | --- |
| A file is not Ready | Inspect its processing status and error before asking questions |
| The HTML table is misaligned | Inspect the preview and table chunk, then regenerate chunks after correction |
| The image has no retrievable description | Check image indexing and the image Embedding configuration |
| An answer does not change after a chunk edit | Confirm the current version and run **Re-embedding** |

**Completion check:** All four files are Ready; facts required by Q01–Q08 can be located in a preview or chunk; and no evaluation asset appears in the knowledge base.

For the full status and preview workflow, see [Data Content Management](../guides/ai-studio/knowledge-bases/files-and-tables.md) and [Chunks and semantics](../guides/ai-studio/knowledge-bases/chunks-semantics.md).

### 4. Freeze the judging rules

Define the rules before seeing MOI answers so that results cannot change the acceptance criteria.

1. Open `Questions-EN` in the evaluation workbook.
2. For each case, review **Reference answer**, **Expected source(s)**, **Required facts**, and **Forbidden claims**.
3. Open `Results-EN`, enter `baseline-20260831` under **Run batch**, and retain `v1` under **Config version**.
4. Record the model, knowledge-base version, and important retrieval settings for this run in the workbook notes or your team test system.

Use these verdicts:

| Verdict | Meaning |
| --- | --- |
| `PASS` | All required facts are present, citations point to the expected evidence, and no forbidden claim appears |
| `PARTIAL` | The direction is correct but a required fact or citation is incomplete, with no conflict against the sources |
| `FAIL` | A material fact is wrong, the citation is wrong, context leaked across cases, or the answer adds an unsupported claim |
| `UNJUDGED` | The reference answer or required environment is missing, so the case cannot currently pass or fail |
| `NOT RUN` | The case has not run |

Q07 is a judged negative case. Its reference answer is that the supplied sources cannot confirm the claim; if MOI preserves that boundary and cites the correct source, it can pass. Q09 is different: it has no reference answer. No matter how reasonable the model sounds, Q09 must remain `UNJUDGED`.

**Completion check:** The run batch, configuration version, and rules are fixed before testing; the distinction between Q07 and Q09 is explicit.

### 5. Run three smoke-test questions

Test three critical behaviors before spending time on the complete question set.

1. Open **Chat** for the knowledge base and click **New Conversation**.
2. Click **Select Knowledge Base** and select only `demo_multimodal_rag_evaluation`.
3. Enter Q01 exactly as written. Confirm that the answer includes 72 hours and its conditions, then open `device-overview.md` from the citation.
4. In the Q01 row of `Results-EN`, record the actual answer, actual cited source, evidence excerpt, and the run or trace identifier from **Developer View**. Then assign a verdict and rationale.
5. Click **New Conversation** and enter Q03 exactly. Confirm that the answer comes from the topology image and distinguishes the RTO design target of no more than 90 seconds from the 78-second field observation.
6. Start another new conversation and enter Q07 exactly. Confirm that the answer says the sources cannot determine satellite-communication support rather than guessing supported or unsupported.

Use three independent conversations. Asking all three in one conversation can let previous retrieval results or context affect the next case.

**Completion check:** Q01, Q03, and Q07 each have an independent answer, citation, run identifier, and verdict, covering an exact fact, image understanding, and the no-answer boundary.

For chat, source preview, and developer information, see [Chat and Retrieval](../guides/ai-studio/knowledge-bases/chat-search.md).

### 6. Execute the complete fixed question set

After the smoke tests behave as expected, execute Q01 through Q09 in order.

Repeat these steps for each case:

1. Click **New Conversation** and reselect the same evaluation knowledge base.
2. Copy the **Fixed question** from `Questions-EN` without rephrasing, adding context, or combining cases.
3. Wait for the answer and open every cited source.
4. In `Results-EN`, record:
   - the actual MOI answer;
   - actual cited sources;
   - an evidence excerpt that supports or contradicts the conclusion;
   - result status and rationale;
   - run or trace identifier;
   - duration and configuration version.
5. Leave the conversation before executing the next case.

Do not paste all nine questions into one prompt and do not reuse a previous answer. The evaluation must show one fixed question mapped to one independent, traceable execution record.

**Completion check:** Q01–Q09 all have independent rows in `Results-EN`. Except for an explicitly unjudged case, no executed row lacks an answer, source, or run identifier.

### 7. Compare answers with ground truth

After all questions have run, complete the final verdicts.

1. Compare **Actual MOI answer** with **Reference answer** and confirm that every required fact appears.
2. Check **Forbidden claims**. An unsupported extension prevents `PASS`, even when the remaining text is correct.
3. Compare **Actual cited source(s)** with **Expected source(s)**. A similar-sounding answer with the wrong evidence is not `PASS`.
4. For cross-source cases such as Q04, verify that the sources collectively support the conclusion.
5. For Q07, confirm that the answer explicitly says that the sources cannot determine the claim.
6. Keep Q09 as `UNJUDGED` and state “missing reference answer” in the rationale.
7. Open `Summary` and confirm that its formulas aggregate status counts and pass rate from `Results-EN`.

This tutorial does not require every first-run case to pass. A useful evaluation gives every case evidence and a boundary, then identifies where failures occur.

**Completion check:** All eight cases with reference answers have verdicts, Q09 remains `UNJUDGED`, and every verdict has a citation and rationale.

### 8. Classify each failure

For `PARTIAL` and `FAIL` cases, work through the following layers before changing the model.

| Observation | Layer | Next action |
| --- | --- | --- |
| The file is not Ready or was never added | Source ingestion | Repair the file or authorization, then process it again |
| The preview, table, or image description lacks the target fact | Parsing | Inspect the source, parsed output, and chunks |
| The correct chunk exists, but the answer cites another file | Retrieval | Adjust chunks, enabled scope, or retrieval settings |
| The answer cites correct evidence but omits conditions or adds claims | Generation | Review the model, prompt, and evidence constraints |
| A new conversation succeeds while a continuous conversation fails | Context | Keep one case per conversation and verify knowledge-base selection |
| The case lacks a reference answer or required source | Benchmark | Mark it `UNJUDGED` and complete the benchmark before retesting |

Change only one factor at a time and record the next configuration as `v2`, `v3`, and so on. Rerun the same question set against the same source version, and do not overwrite the `baseline-20260831` records. This makes the effect of each change measurable.

**Completion check:** Every nonpassing case maps to an explicit layer, and a retest batch or configuration never overwrites the baseline.

### 9. Completion criteria

The evaluation workflow is complete only when:

- the knowledge base contains only the four synthetic knowledge sources, and every in-scope file is Ready or has an explicit failure reason;
- Q01–Q09 all have independent execution records with no missing or context-mixed case;
- every executed case has an answer, citation, run or trace identifier, and judging rationale;
- all eight cases with reference answers have verdicts, while Q09 is not incorrectly labeled `PASS` or `FAIL`;
- the model, source manifest, configuration version, and run batch let another evaluator reproduce the run; and
- each `PARTIAL`, `FAIL`, or environment blocker is classified to a specific layer.

These criteria measure whether the evaluation is trustworthy and repeatable. They do not force every case to pass on the first attempt.

## Apply this method to real business data

Never copy customer sources or internal question sets into a public tutorial, issue, screenshot, or demo package. Build a synthetic twin from scratch: preserve file types, question types, and judging logic while replacing names, values, layouts, paths, and business facts.

For an internal production benchmark, also:

- store knowledge sources and reference answers in different directories and permission boundaries;
- remove authors, usernames, locations, and edit history from Office files, images, and audio or video;
- use a manifest and hashes to pin the source version for each evaluation;
- keep historical production answers, reference answers, and actual MOI answers in separate columns;
- mark missing reference answers as `UNJUDGED` instead of asking a model to write ground truth; and
- use only synthetic, legally distributable material in public demonstrations.

## Tutorial complete

You have completed the multimodal RAG evaluation:

- built an evaluation knowledge base from synthetic sources;
- recorded answers, citations, and verdicts for fixed questions; and
- identified incorrect answers, citations, and out-of-scope claims.