Can You Trust Your AI Assistant’s Sources? A Bilingual RAG Test

A trustworthy knowledge assistant should find the right evidence, preserve its meaning in either language, and make uncertainty visible. A fluent answer with a source link is only the starting point.

For a Saudi or GCC team working across Arabic and English, the useful question is practical: can a colleague open the cited document and confirm the answer without repeating the entire search?

Retrieval-augmented generation, usually shortened to RAG, combines a search over documents with an AI-generated response. To assess it, separate two questions: did the system retrieve useful evidence, and did it use that evidence correctly? Microsoft’s evaluation guidance distinguishes retrieval quality from answer groundedness, relevance and completeness. Source: Microsoft RAG evaluators.

The workflow below is our suggested acceptance test, not an industry certification. Use it after preparing your content with the data checklist for a first AI project.

1. Test all four language combinations

Choose a manageable pilot set, such as 40 real work questions. This is a starting sample, not a statistically sufficient benchmark. Include Arabic questions answered by Arabic documents, English questions answered by English documents, and both cross-language combinations. Keep some questions separate from the examples used to tune the assistant.

This distinction matters: a 2025 study using Arabic–English corporate datasets found weaker retrieval when the question and supporting document used different languages. Its results support testing each direction separately; they do not establish how your own system will perform. Source: The Cross-Lingual Cost.

For each question, record the expected answer, the supporting passage, document version and date, and whether a clarification is required. Add local product names, Arabic spelling variants and mixed-language questions that your team actually uses. A bilingual reviewer should check that paired questions ask the same thing.

2. Inspect the passage, not just the link

For every material claim, ask the reviewer to open the cited source. Does the passage support the exact claim? Is it the approved version? Does the answer preserve exceptions, units, dates and conditions?

Consider this fictional example. A service guide says that standard installation takes five working days after site approval. The assistant replies, “Installation takes five days,” and links to the guide. The link works, but the answer has lost both the working-day qualification and the starting condition. In Arabic, “خمسة أيام” is equally incomplete; the expected meaning is “خمسة أيام عمل بعد اعتماد الموقع”. These are invented test details, not a Masar Next service commitment.

Record separate results for finding the right document, supporting the claim, and retaining the conditions. That makes a failed test actionable: a search problem needs a different fix from an answer that misreads good evidence.

3. Include questions with no supported answer

Add questions whose answers are absent from the approved documents, questions about discontinued offerings, and requests that omit a necessary detail. Define the expected behavior in advance: explain the evidence gap, ask a specific follow-up question, or direct the user to the document owner.

A March 2026 study found that search improved answer accuracy on answerable questions but could reduce appropriate abstention on unanswerable questions in the evaluated systems. More searching should therefore be tested, not assumed to solve uncertainty. Source: Over-Searching in Search-Augmented Large Language Models.

Test a short conversation too: ask about a product, then say “Does that also apply to the older version?” The assistant must resolve “that” and identify the version before answering. July 2026’s MTRAGEval evaluates retrieval, generation and the combined workflow on multi-turn questions, including unclear and underspecified ones. Source: SemEval-2026 MTRAGEval.

4. Check what happens after a document changes

An answer can faithfully reflect an obsolete document. In a test collection, change one approved fact, record the update time, and rerun the relevant questions after your agreed synchronization window. Then withdraw a document and check that it stops supplying answers.

This needs its own test. Azure AI Search documentation, for example, explains that resetting and rerunning an indexer does not remove orphaned search documents by itself. The practical lesson is to verify your connector’s update and deletion behavior explicitly. Source: Run or reset indexers.

Give each document a content owner, approval status and version. Where Arabic and English versions disagree, define which approved version governs the answer and when the assistant should escalate. It should not silently combine conflicting instructions.

5. Use a scorecard your team can explain

Track the following separately for each language combination:

• Evidence found: answerable questions for which the retrieved passages contain the required evidence, divided by answerable questions tested.
• Citation support: reviewed factual claims supported by their cited passages, divided by cited factual claims reviewed. Also count material claims with no citation.
• Answer completeness: answers retaining all required conditions, exceptions and units, divided by answerable questions tested.
• Uncertainty handling: unsupported or ambiguous questions handled as expected, divided by those questions tested.
• Freshness: update and withdrawal scenarios passing within the agreed synchronization window.

Agree acceptance thresholds with the process owner before testing. Treat invented material facts or reliance on withdrawn instructions as blockers for that workflow, then expand the sample after fixes. A small pilot with no observed failures is evidence about that sample, not a guarantee of future reliability. Record response time and cost alongside quality so an improvement remains usable in daily work.

Questions to settle before launch

Does a citation prove that an answer is correct?

No. Check that the source is authoritative and current, that its passage supports the statement, and that the response retains necessary qualifications.

Is translating an English test set enough?

No. Add questions originally written by Arabic-speaking colleagues, cross-language evidence, and realistic follow-ups. Use our Arabic LLM production-readiness guide for broader language considerations.

When should we repeat these tests?

Repeat them after changes to the model, prompt, retrieval settings, document collection or indexing pipeline. Preserve a stable comparison set and add examples from observed failures.

For a first pilot, choose one bounded knowledge task and name the person who approves the evidence. Masar Next’s AI solutions can help you scope that workflow; contact us with the questions your team needs answered and the documents they rely on.

Sources checked on 18 September 2026. Research findings are limited to the systems studied; the test workflow and fictional example above are editorial recommendations.