Evaluate multimodal RAG with reference answers

Application scenario

After a business adds product specifications, maintenance bulletins, field notes, and topology diagrams to a knowledge base, the important question is not merely whether a model can produce a paragraph. The business needs to know:

• whether model names, times, numbers, and conditions in the answer are correct;

• whether the answer cites evidence that actually supports its conclusion;

• whether content in tables, images, and text was parsed and retrieved correctly;

• whether the model says that the sources cannot confirm a claim when no answer exists; and

• whether results remain stable after a source, model, or retrieval configuration changes.

Without fixed questions and reference answers, a few free-form chats cannot distinguish a plausible one-off response from repeatable quality. Once a knowledge base serves business users, one wrong number, unrelated citation, or unsupported extension can mislead downstream decisions.

This tutorial turns RAG question answering into a reproducible acceptance workflow. You will use a fully synthetic campus-device dataset, create an evaluation knowledge base, ask a fixed question set, and record results against reference answers, expected sources, and forbidden claims.

The final artifact is not a screenshot proving that “the model answered.” It is an evaluation record that supports:

User input	MOI processing	Final artifact	Downstream use
Multiformat sources, fixed questions, reference answers	Parsing and indexing, retrieval, answer generation, source location	Answer, citation, run identifier, and verdict for each case	Release acceptance, model or configuration comparison, regression testing after source updates

This tutorial explains how to verify that a multimodal RAG application produces answers that are correct, traceable, and bounded by the supplied evidence.

Note

MOI does not currently provide a knowledge-base Golden Set page that automatically scores these cases. This tutorial uses a manual evidence worksheet. It never uploads reference answers to MOI and does not assume that the platform performs automatic scoring.

What you will complete

1. Download and inspect synthetic multimodal sources and an evaluation workbook.

2. Create an evaluation knowledge base isolated from production and add only the knowledge sources.

3. Inspect the processing status, preview, and chunks for Markdown, HTML, text, and image files.

4. Run three smoke-test questions for an exact fact, image understanding, and an unsupported claim.

5. Execute nine fixed questions in separate conversations and record answers, citations, and run evidence.

6. Assign PASS, PARTIAL, FAIL, or UNJUDGED using the reference answers.

7. Classify failures as source, parsing, retrieval, generation, context, or benchmark issues.

Before you begin

Prepare the following:

• Have an MOI account. Use MOI Cloud sign in/sign up.

• An MOI workspace where you can create and modify a knowledge base;

• A model that can answer questions over a knowledge base;

• Image indexing and an image Embedding model for the image case;

• A spreadsheet application that can open .xlsx files; and

• The synthetic evaluation package supplied with this tutorial.

Download the synthetic evaluation package

Download and extract the synthetic multimodal RAG evaluation package. Every organization, product, model, firmware version, date, metric, network layout, and observation in this package was created from scratch. It contains no real customer, project, or business data.

moi-multimodal-rag-evaluation-demo/
├── README.md
├── NOTICE.md
├── manifest.json
├── knowledge-sources/
│   ├── device-overview.md
│   ├── maintenance-bulletin.html
│   ├── field-test-notes.txt
│   └── network-topology.png
└── evaluation/
    └── multimodal-rag-benchmark.xlsx

File	Source type	What it tests
device-overview.md	Markdown product overview	Exact values, conditions, and evidence boundaries
maintenance-bulletin.html	HTML bulletin and table	Times, versions, protocols, and tabular content
field-test-notes.txt	Plain-text field notes	The difference between one observation and a formal commitment
network-topology.png	Network-topology image	Node relationships, port information, and the failover target
multimodal-rag-benchmark.xlsx	Bilingual evaluation workbook	Fixed questions, reference answers, forbidden claims, and result recording

Use these names throughout the tutorial:

Object	Example name
Knowledge base	demo_multimodal_rag_evaluation
Initial run batch	baseline-20260831
Initial configuration version	v1

Important

Add only the four files under knowledge-sources/ to the knowledge base. Do not upload evaluation/, the workbook, README.md, NOTICE.md, or manifest.json. If reference answers enter the retrieval scope, the model can retrieve them directly and the evaluation is invalid.

Steps

1. Inspect the package and evaluation boundary

Confirm locally that the knowledge sources are separate from the evaluation assets before opening MOI.

1. Open README.md and confirm that the package identifies itself as synthetic.

2. Open manifest.json and confirm that each of the four knowledge-sources/ entries has upload_to_knowledge_base set to true.

3. Confirm that evaluation/multimodal-rag-benchmark.xlsx has upload_to_knowledge_base set to false.

4. Open the workbook and read the README worksheet.

5. Open Questions-EN. Confirm that it contains Q01 through Q09 and that Q09 has an empty reference answer with a ground-truth status of UNJUDGED.

6. Do not change Questions-EN. The baseline must use identical wording, reference answers, and judging boundaries.

The SHA-256 values in manifest.json prove that a later run uses the same source version. When a file changes, create a new source version instead of reusing the old configuration version.

Completion check: The four knowledge sources and one evaluation workbook exist; the workbook is explicitly excluded from upload; and Q01–Q09 and their ground-truth statuses are visible.

2. Create an isolated evaluation knowledge base

Keep the evaluation knowledge base separate from one that serves production users so test sources and settings cannot change production answers.

1. Sign in to MOI and confirm that the intended workspace appears in the upper-left corner.

2. Go to Knowledge Bases and click New Knowledge Base.

3. Enter demo_multimodal_rag_evaluation as the knowledge-base name.

4. In the description, enter “Synthetic RAG evaluation; not for production Q&A.”

5. Select Local Upload, then select Unstructured as the import type.

6. Select only the four files in knowledge-sources/.

7. Configure image indexing and an image Embedding model available in the current workspace. Exact controls depend on the deployment.

8. Submit the creation request.

After creation, MOI opens Advanced Configuration > Data. Do not add the evaluation workbook later through Add Data.

If image indexing or an image Embedding model is unavailable, do not mark Q03 as FAIL. Record it as UNJUDGED or as an environment blocker until that capability is available. You can still run the text cases.

Completion check: The knowledge base contains exactly four synthetic knowledge sources and no evaluation file or reference answer.

For the complete creation and source-selection workflow, see Creation and Data Sources.

3. Wait for readiness and inspect parsing

Do not judge RAG quality until source processing completes.

1. Under Advanced Configuration > Data, wait until all four files show Ready.

2. Open each file and select File Preview.

3. In device-overview.md, confirm that “72 hours” and “TCP 7443” are present.

4. In maintenance-bulletin.html, confirm that A-20, 3.1.0, and HTTPS remain in the same table row.

5. In field-test-notes.txt, confirm that “78 seconds” and “not a service-level guarantee” remain in the same semantic context.

6. Open the image or image-text chunk for network-topology.png and confirm that it captures Backup Gateway B, RTO ≤ 90 s, TLS 1.3, and TCP 7443.

7. Open Smart Chunking and confirm that these facts are not truncated, merged with unrelated content, or disabled.

Symptom	Check first
A file is not Ready	Inspect its processing status and error before asking questions
The HTML table is misaligned	Inspect the preview and table chunk, then regenerate chunks after correction
The image has no retrievable description	Check image indexing and the image Embedding configuration
An answer does not change after a chunk edit	Confirm the current version and run Re-embedding

Completion check: All four files are Ready; facts required by Q01–Q08 can be located in a preview or chunk; and no evaluation asset appears in the knowledge base.

For the full status and preview workflow, see Data Content Management and Chunks and semantics.

4. Freeze the judging rules

Define the rules before seeing MOI answers so that results cannot change the acceptance criteria.

1. Open Questions-EN in the evaluation workbook.

2. For each case, review Reference answer, Expected source(s), Required facts, and Forbidden claims.

3. Open Results-EN, enter baseline-20260831 under Run batch, and retain v1 under Config version.

4. Record the model, knowledge-base version, and important retrieval settings for this run in the workbook notes or your team test system.

Use these verdicts:

Verdict	Meaning
PASS	All required facts are present, citations point to the expected evidence, and no forbidden claim appears
PARTIAL	The direction is correct but a required fact or citation is incomplete, with no conflict against the sources
FAIL	A material fact is wrong, the citation is wrong, context leaked across cases, or the answer adds an unsupported claim
UNJUDGED	The reference answer or required environment is missing, so the case cannot currently pass or fail
NOT RUN	The case has not run

Q07 is a judged negative case. Its reference answer is that the supplied sources cannot confirm the claim; if MOI preserves that boundary and cites the correct source, it can pass. Q09 is different: it has no reference answer. No matter how reasonable the model sounds, Q09 must remain UNJUDGED.

Completion check: The run batch, configuration version, and rules are fixed before testing; the distinction between Q07 and Q09 is explicit.

5. Run three smoke-test questions

Test three critical behaviors before spending time on the complete question set.

1. Open Chat for the knowledge base and click New Conversation.

2. Click Select Knowledge Base and select only demo_multimodal_rag_evaluation.

3. Enter Q01 exactly as written. Confirm that the answer includes 72 hours and its conditions, then open device-overview.md from the citation.

4. In the Q01 row of Results-EN, record the actual answer, actual cited source, evidence excerpt, and the run or trace identifier from Developer View. Then assign a verdict and rationale.

5. Click New Conversation and enter Q03 exactly. Confirm that the answer comes from the topology image and distinguishes the RTO design target of no more than 90 seconds from the 78-second field observation.

6. Start another new conversation and enter Q07 exactly. Confirm that the answer says the sources cannot determine satellite-communication support rather than guessing supported or unsupported.

Use three independent conversations. Asking all three in one conversation can let previous retrieval results or context affect the next case.

Completion check: Q01, Q03, and Q07 each have an independent answer, citation, run identifier, and verdict, covering an exact fact, image understanding, and the no-answer boundary.

For chat, source preview, and developer information, see Chat and Retrieval.

6. Execute the complete fixed question set

After the smoke tests behave as expected, execute Q01 through Q09 in order.

Repeat these steps for each case:

1. Click New Conversation and reselect the same evaluation knowledge base.

2. Copy the Fixed question from Questions-EN without rephrasing, adding context, or combining cases.

3. Wait for the answer and open every cited source.

4. In Results-EN, record:

   • the actual MOI answer;

   • actual cited sources;

   • an evidence excerpt that supports or contradicts the conclusion;

   • result status and rationale;

   • run or trace identifier;

   • duration and configuration version.

5. Leave the conversation before executing the next case.

Do not paste all nine questions into one prompt and do not reuse a previous answer. The evaluation must show one fixed question mapped to one independent, traceable execution record.

Completion check: Q01–Q09 all have independent rows in Results-EN. Except for an explicitly unjudged case, no executed row lacks an answer, source, or run identifier.

7. Compare answers with ground truth

After all questions have run, complete the final verdicts.

1. Compare Actual MOI answer with Reference answer and confirm that every required fact appears.

2. Check Forbidden claims. An unsupported extension prevents PASS, even when the remaining text is correct.

3. Compare Actual cited source(s) with Expected source(s). A similar-sounding answer with the wrong evidence is not PASS.

4. For cross-source cases such as Q04, verify that the sources collectively support the conclusion.

5. For Q07, confirm that the answer explicitly says that the sources cannot determine the claim.

6. Keep Q09 as UNJUDGED and state “missing reference answer” in the rationale.

7. Open Summary and confirm that its formulas aggregate status counts and pass rate from Results-EN.

This tutorial does not require every first-run case to pass. A useful evaluation gives every case evidence and a boundary, then identifies where failures occur.

Completion check: All eight cases with reference answers have verdicts, Q09 remains UNJUDGED, and every verdict has a citation and rationale.

8. Classify each failure

For PARTIAL and FAIL cases, work through the following layers before changing the model.

Observation	Layer	Next action
The file is not Ready or was never added	Source ingestion	Repair the file or authorization, then process it again
The preview, table, or image description lacks the target fact	Parsing	Inspect the source, parsed output, and chunks
The correct chunk exists, but the answer cites another file	Retrieval	Adjust chunks, enabled scope, or retrieval settings
The answer cites correct evidence but omits conditions or adds claims	Generation	Review the model, prompt, and evidence constraints
A new conversation succeeds while a continuous conversation fails	Context	Keep one case per conversation and verify knowledge-base selection
The case lacks a reference answer or required source	Benchmark	Mark it UNJUDGED and complete the benchmark before retesting

Change only one factor at a time and record the next configuration as v2, v3, and so on. Rerun the same question set against the same source version, and do not overwrite the baseline-20260831 records. This makes the effect of each change measurable.

Completion check: Every nonpassing case maps to an explicit layer, and a retest batch or configuration never overwrites the baseline.

9. Completion criteria

The evaluation workflow is complete only when:

• the knowledge base contains only the four synthetic knowledge sources, and every in-scope file is Ready or has an explicit failure reason;

• Q01–Q09 all have independent execution records with no missing or context-mixed case;

• every executed case has an answer, citation, run or trace identifier, and judging rationale;

• all eight cases with reference answers have verdicts, while Q09 is not incorrectly labeled PASS or FAIL;

• the model, source manifest, configuration version, and run batch let another evaluator reproduce the run; and

• each PARTIAL, FAIL, or environment blocker is classified to a specific layer.

These criteria measure whether the evaluation is trustworthy and repeatable. They do not force every case to pass on the first attempt.

Apply this method to real business data

Never copy customer sources or internal question sets into a public tutorial, issue, screenshot, or demo package. Build a synthetic twin from scratch: preserve file types, question types, and judging logic while replacing names, values, layouts, paths, and business facts.

For an internal production benchmark, also:

• store knowledge sources and reference answers in different directories and permission boundaries;

• remove authors, usernames, locations, and edit history from Office files, images, and audio or video;

• use a manifest and hashes to pin the source version for each evaluation;

• keep historical production answers, reference answers, and actual MOI answers in separate columns;

• mark missing reference answers as UNJUDGED instead of asking a model to write ground truth; and

• use only synthetic, legally distributable material in public demonstrations.

Tutorial complete

You have completed the multimodal RAG evaluation:

• built an evaluation knowledge base from synthetic sources;

• recorded answers, citations, and verdicts for fixed questions; and

• identified incorrect answers, citations, and out-of-scope claims.

Last updated on Sep 15, 2026