Question 690 An invoice-extraction agent passes a general fluency judge even when extracted line totals do not equal the declared invoice total. What prevention measure is most targeted to this underlying cause? Maintain a scorer portfolio aligned to business success, correctness and safety requirements. Maintain provenance-aware scorers and negative citation fixtures across retrieval releases. Keep versioned business-rule scorers and fixture-driven unit tests for numeric invariants. Enforce scorer signature-to-dataset column mapping tests and typed evaluation fixtures on each run. ExplanationCorrect Answer C:Generic language-quality judges are not substitutes for exact arithmetic invariants in structured outputs. Other Answers A:This control addresses a different exposure: An evaluation run produces a high Safety score but no measurement of whether the requested invoice total was calculated correctly. It would not prevent this failure, because generic language-quality judges are not substitutes for exact arithmetic invariants in structured outputs. B:That generic language-quality judges are not substitutes for exact arithmetic invariants in structured outputs. D:Generic language-quality judges are not substitutes for exact arithmetic invariants in structured outputs.
Question 691 A citation judge passes responses with valid-looking IDs even when those IDs never appeared in the retrieved evidence. Which implementation decision best restores the intended behavior? Add a custom scorer checking every citation ID against the trace's authorized retrieval-set manifest and relevant supporting text. Inspect nested trace spans and add code-based checks for tool status and tool-output integrity in the evaluation. Implement a deterministic custom scorer that compares parsed line-item sums with the stated total within the business tolerance. Enable approved inference/trace logging with query, index revision, retrieved document IDs and citations while redacting sensitive values. ExplanationCorrect Answer A:The action targets the stated cause: Citation syntax validity is weaker than provenance integrity; a fabricated identifier can still look well formed. Check the repair by: evaluate one fabricated and one genuinely grounded citation and inspect scorer feedback per trace. Other Answers B:Citation syntax validity is weaker than provenance integrity; a fabricated identifier can still look well formed. C:Citation syntax validity is weaker than provenance integrity; a fabricated identifier can still look well formed. D:Citation syntax validity is weaker than provenance integrity; a fabricated identifier can still look well formed.
Question 692 A citation judge passes responses with valid-looking IDs even when those IDs never appeared in the retrieved evidence. Which underlying technical principle explains why this occurs? Without evidence-version provenance, the team cannot separate retrieval mistakes from answer-generation mistakes. Citation syntax validity is weaker than provenance integrity; a fabricated identifier can still look well formed. Generic language-quality judges are not substitutes for exact arithmetic invariants in structured outputs. End-to-end generation failure can originate in tools, retrieval or prompts; one judge score cannot localize every cause. ExplanationCorrect Answer B:Add a custom scorer checking every citation ID against the trace's authorized retrieval-set manifest and relevant supporting text. Other Answers A:That mechanism describes another problem: User reports say a RAG bot cites the wrong policy, but only final answers are stored and engineers cannot see which version was retrieved. It does not explain this case, where citation syntax validity is weaker than provenance integrity; a fabricated identifier can still look well formed. C:Citation syntax validity is weaker than provenance integrity; a fabricated identifier can still look well formed. D:Citation syntax validity is weaker than provenance integrity; a fabricated identifier can still look well formed.
Question 693 A citation judge passes responses with valid-looking IDs even when those IDs never appeared in the retrieved evidence. After correction, what targeted test should demonstrate success? Score one balanced invoice and one mismatched invoice and confirm only the mismatch fails with a clear reason. Evaluate one fabricated and one genuinely grounded citation and inspect scorer feedback per trace. Introduce a mocked failing tool call and verify its span/error is detected independently of the final-answer score. Reproduce a reported answer and find the exact document IDs and index revision in the recorded trace. ExplanationCorrect Answer B:Citation syntax validity is weaker than provenance integrity; a fabricated identifier can still look well formed. Other Answers A:Evaluate one fabricated and one genuinely grounded citation and inspect scorer feedback per trace. C:Evaluate one fabricated and one genuinely grounded citation and inspect scorer feedback per trace. D:Evaluate one fabricated and one genuinely grounded citation and inspect scorer feedback per trace.
Question 694 Two human reviewers score the same hallucinated contract answer as excellent and poor because they use different implicit definitions of correctness. What should the implementation team change first to resolve the described defect? Map the existing expected_answer column explicitly into the scorer invocation and validate its typed inputs before evaluation. Measure factual support and numeric consistency alongside lexical overlap, emphasizing high-severity hallucination frequency. Adopt a shared rubric with supported/unsupported examples, train reviewers and adjudicate disagreements before setting a threshold. Add a task-specific numeric correctness scorer alongside the built-in safety scorer and inspect results per trace. ExplanationCorrect Answer C:The action targets the stated cause: Subjective ratings have limited meaning without calibrated criteria and inter-rater agreement. Check the repair by: double-score a blind set, measure agreement and review large discrepancies against the rubric anchors. Other Answers A:Subjective ratings have limited meaning without calibrated criteria and inter-rater agreement. B:Subjective ratings have limited meaning without calibrated criteria and inter-rater agreement. D:Subjective ratings have limited meaning without calibrated criteria and inter-rater agreement.
Question 695 Two human reviewers score the same hallucinated contract answer as excellent and poor because they use different implicit definitions of correctness. Why does the current approach fail under these conditions? A well-written guess can look complete even when the source cannot support an answer; expert criteria must reward correct uncertainty. A scorer can misjudge valid traces when its required argument mapping omits available reference data even though the dataset itself is complete. N-gram similarity rewards shared wording and cannot prove generated facts are supported by source records. Subjective ratings have limited meaning without calibrated criteria and inter-rater agreement. ExplanationCorrect Answer D:Adopt a shared rubric with supported/unsupported examples, train reviewers and adjudicate disagreements before setting a threshold. Other Answers A:That mechanism describes another problem: An evaluation team mixes policy-faithful answers with intentionally unanswerable cases, but reviewers give high scores to confident guesses. It does not explain this case, where subjective ratings have limited meaning without calibrated criteria and inter-rater agreement. B:Subjective ratings have limited meaning without calibrated criteria and inter-rater agreement. C:Subjective ratings have limited meaning without calibrated criteria and inter-rater agreement.
Question 696 Two human reviewers score the same hallucinated contract answer as excellent and poor because they use different implicit definitions of correctness. Which observation would establish that the requirement is now met? Compare generated numbers against cited source fields for every high-risk evaluation case. Double-score a blind set, measure agreement and review large discrepancies against the rubric anchors. Blind-score labeled answerable and unanswerable examples and measure penalties for fabricated claims. Score one known correct and one known incorrect trace and verify the reference values appear in scorer inputs and recorded feedback. ExplanationCorrect Answer B:Subjective ratings have limited meaning without calibrated criteria and inter-rater agreement. Other Answers A:Double-score a blind set, measure agreement and review large discrepancies against the rubric anchors. C:Double-score a blind set, measure agreement and review large discrepancies against the rubric anchors. D:Double-score a blind set, measure agreement and review large discrepancies against the rubric anchors.
Question 697 Two human reviewers score the same hallucinated contract answer as excellent and poor because they use different implicit definitions of correctness. What lasting control best prevents recurrence of the identified issue? Maintain separate severity-weighted factuality metrics and nonnegotiable high-risk acceptance thresholds. Recalibrate reviewers after rubric or source-domain changes and retain adjudication records. Keep abstention-specific acceptance anchors and audit reviewer bias toward fluent confident text. Enforce scorer signature-to-dataset column mapping tests and typed evaluation fixtures on each run. ExplanationCorrect Answer B:Subjective ratings have limited meaning without calibrated criteria and inter-rater agreement. Other Answers A:That subjective ratings have limited meaning without calibrated criteria and inter-rater agreement. C:Subjective ratings have limited meaning without calibrated criteria and inter-rater agreement. D:This control addresses a different exposure: An MLflow evaluation contains adjudicated expected answers, but a custom correctness scorer receives only the generated response because the answer column was not mapped into its inputs. It would not prevent this failure, because subjective ratings have limited meaning without calibrated criteria and inter-rater agreement.
Question 698 An evaluation team mixes policy-faithful answers with intentionally unanswerable cases, but reviewers give high scores to confident guesses. What is the most targeted corrective step for this observed failure? Enable approved inference/trace logging with query, index revision, retrieved document IDs and citations while redacting sensitive values. Add an explicit abstention criterion and negative examples to the rubric, then test whether reviewers penalize unsupported certainty. Map the existing expected_answer column explicitly into the scorer invocation and validate its typed inputs before evaluation. Measure factual support and numeric consistency alongside lexical overlap, emphasizing high-severity hallucination frequency. ExplanationCorrect Answer B:The action targets the stated cause: A well-written guess can look complete even when the source cannot support an answer; expert criteria must reward correct uncertainty. Check the repair by: blind-score labeled answerable and unanswerable examples and measure penalties for fabricated claims. Other Answers A:A well-written guess can look complete even when the source cannot support an answer; expert criteria must reward correct uncertainty. C:A well-written guess can look complete even when the source cannot support an answer; expert criteria must reward correct uncertainty. D:A well-written guess can look complete even when the source cannot support an answer; expert criteria must reward correct uncertainty.
Question 699 An evaluation team mixes policy-faithful answers with intentionally unanswerable cases, but reviewers give high scores to confident guesses. What is the decisive mechanism behind the observed result? N-gram similarity rewards shared wording and cannot prove generated facts are supported by source records. Without evidence-version provenance, the team cannot separate retrieval mistakes from answer-generation mistakes. A well-written guess can look complete even when the source cannot support an answer; expert criteria must reward correct uncertainty. Subjective ratings have limited meaning without calibrated criteria and inter-rater agreement. ExplanationCorrect Answer C:Add an explicit abstention criterion and negative examples to the rubric, then test whether reviewers penalize unsupported certainty. Other Answers A:A well-written guess can look complete even when the source cannot support an answer; expert criteria must reward correct uncertainty. B:A well-written guess can look complete even when the source cannot support an answer; expert criteria must reward correct uncertainty. D:That mechanism describes another problem: Two human reviewers score the same hallucinated contract answer as excellent and poor because they use different implicit definitions of correctness. It does not explain this case, where a well-written guess can look complete even when the source cannot support an answer; expert criteria must reward correct uncertainty.
Question 700 An evaluation team mixes policy-faithful answers with intentionally unanswerable cases, but reviewers give high scores to confident guesses. Which verification is strongest for this particular failure mode? Reproduce a reported answer and find the exact document IDs and index revision in the recorded trace. Double-score a blind set, measure agreement and review large discrepancies against the rubric anchors. Blind-score labeled answerable and unanswerable examples and measure penalties for fabricated claims. Compare generated numbers against cited source fields for every high-risk evaluation case. ExplanationCorrect Answer C:A well-written guess can look complete even when the source cannot support an answer; expert criteria must reward correct uncertainty. Other Answers A:Blind-score labeled answerable and unanswerable examples and measure penalties for fabricated claims. B:Blind-score labeled answerable and unanswerable examples and measure penalties for fabricated claims. D:Blind-score labeled answerable and unanswerable examples and measure penalties for fabricated claims.
