Question 630 A RAG service meets average response time but 10% of users wait twenty seconds when vector search is overloaded. Which design change directly satisfies the stated constraints? Monitor citation validation rate, groundedness and source freshness as separate production-quality signals. Monitor valid-request success, refusal rate and sampled answer quality alongside latency, segmented by prompt version. Track P95/P99 latency by retrieval and generation stage and address the dominant tail bottleneck. Inspect per-stage inference records for search timeouts and missing-context flags before changing prompt text. ExplanationCorrect Answer C:The action targets the stated cause: Aggregate average latency can hide severe slow-tail experience and prevent stage-specific diagnosis. Check the repair by: trace slow requests and compare percentile latencies for index query versus model completion. Other Answers A:Aggregate average latency can hide severe slow-tail experience and prevent stage-specific diagnosis. B:Aggregate average latency can hide severe slow-tail experience and prevent stage-specific diagnosis. D:Aggregate average latency can hide severe slow-tail experience and prevent stage-specific diagnosis.
Question 631 A RAG service meets average response time but 10% of users wait twenty seconds when vector search is overloaded. Which causal explanation best accounts for the failure? Fluent output does not establish that an answer is supported by a live authoritative passage. A generator can complete successfully after an upstream retrieval failure, so final model status alone hides the incident. Speed can remain stable while application utility drops; response quality and reliability are separate production dimensions. Aggregate average latency can hide severe slow-tail experience and prevent stage-specific diagnosis. ExplanationCorrect Answer D:Track P95/P99 latency by retrieval and generation stage and address the dominant tail bottleneck. Other Answers A:Aggregate average latency can hide severe slow-tail experience and prevent stage-specific diagnosis. B:That mechanism describes another problem: A production service intermittently returns empty context, yet model completion traces show a normal success status. It does not explain this case, where aggregate average latency can hide severe slow-tail experience and prevent stage-specific diagnosis. C:Aggregate average latency can hide severe slow-tail experience and prevent stage-specific diagnosis.
Question 632 A RAG service meets average response time but 10% of users wait twenty seconds when vector search is overloaded. Which concrete acceptance test best demonstrates the proposed change works? Chart correct resolution and refusal counts per deployed prompt revision and reproduce failing requests in offline tests. Trace slow requests and compare percentile latencies for index query versus model completion. Sample live traces and check that citation IDs exist in the current authorized retrieval set. Correlate empty-answer cases with retrieval latency and tool error events in the same trace ID. ExplanationCorrect Answer B:Aggregate average latency can hide severe slow-tail experience and prevent stage-specific diagnosis. Other Answers A:Trace slow requests and compare percentile latencies for index query versus model completion. C:Trace slow requests and compare percentile latencies for index query versus model completion. D:Trace slow requests and compare percentile latencies for index query versus model completion.
Question 633 A RAG service meets average response time but 10% of users wait twenty seconds when vector search is overloaded. Which operational practice should be made a durable requirement? Monitor required context availability and define fallback behavior for upstream search errors. Define balanced quality and operational SLOs with version-attributed monitoring. Set stage-level latency budgets and alert when tail percentiles exceed thresholds. Require source-provenance monitoring and freshness alerts after ingestion changes. ExplanationCorrect Answer C:Aggregate average latency can hide severe slow-tail experience and prevent stage-specific diagnosis. Other Answers A:This control addresses a different exposure: A production service intermittently returns empty context, yet model completion traces show a normal success status. It would not prevent this failure, because aggregate average latency can hide severe slow-tail experience and prevent stage-specific diagnosis. B:That aggregate average latency can hide severe slow-tail experience and prevent stage-specific diagnosis. D:Aggregate average latency can hide severe slow-tail experience and prevent stage-specific diagnosis.
Question 634 A production agent returns fluent answers but silently fails to resolve citations on several newly ingested policy versions. What should the implementation team change first to resolve the described defect? Monitor citation validation rate, groundedness and source freshness as separate production-quality signals. Monitor valid-request success, refusal rate and sampled answer quality alongside latency, segmented by prompt version. Track P95/P99 latency by retrieval and generation stage and address the dominant tail bottleneck. Enable approved inference/trace logging with query, index revision, retrieved document IDs and citations while redacting sensitive values. ExplanationCorrect Answer A:The action targets the stated cause: Fluent output does not establish that an answer is supported by a live authoritative passage. Check the repair by: sample live traces and check that citation IDs exist in the current authorized retrieval set. Other Answers B:Fluent output does not establish that an answer is supported by a live authoritative passage. C:Fluent output does not establish that an answer is supported by a live authoritative passage. D:Fluent output does not establish that an answer is supported by a live authoritative passage.
Question 635 An evaluation run produces a high Safety score but no measurement of whether the requested invoice total was calculated correctly. Which design change directly satisfies the stated constraints? Measure factual support and numeric consistency alongside lexical overlap, emphasizing high-severity hallucination frequency. Inspect nested trace spans and add code-based checks for tool status and tool-output integrity in the evaluation. Add a task-specific numeric correctness scorer alongside the built-in safety scorer and inspect results per trace. Map the existing expected_answer column explicitly into the scorer invocation and validate its typed inputs before evaluation. ExplanationCorrect Answer C:The action targets the stated cause: MLflow scorers measure distinct attributes; passing one risk dimension does not establish task correctness. Check the repair by: run traces with known incorrect totals and confirm the numeric scorer flags them even when safety passes. Other Answers A:MLflow scorers measure distinct attributes; passing one risk dimension does not establish task correctness. B:MLflow scorers measure distinct attributes; passing one risk dimension does not establish task correctness. D:MLflow scorers measure distinct attributes; passing one risk dimension does not establish task correctness.
Question 636 An evaluation run produces a high Safety score but no measurement of whether the requested invoice total was calculated correctly. Which causal explanation best accounts for the failure? MLflow scorers measure distinct attributes; passing one risk dimension does not establish task correctness. A scorer can misjudge valid traces when its required argument mapping omits available reference data even though the dataset itself is complete. End-to-end generation failure can originate in tools, retrieval or prompts; one judge score cannot localize every cause. N-gram similarity rewards shared wording and cannot prove generated facts are supported by source records. ExplanationCorrect Answer A:Add a task-specific numeric correctness scorer alongside the built-in safety scorer and inspect results per trace. Other Answers B:MLflow scorers measure distinct attributes; passing one risk dimension does not establish task correctness. C:MLflow scorers measure distinct attributes; passing one risk dimension does not establish task correctness. D:That mechanism describes another problem: An evaluation report gives a high average BLEU score for summaries but hides hallucinated financial amounts in five important cases. It does not explain this case, where mLflow scorers measure distinct attributes; passing one risk dimension does not establish task correctness.
Question 637 An evaluation run produces a high Safety score but no measurement of whether the requested invoice total was calculated correctly. Which concrete acceptance test best demonstrates the proposed change works? Compare generated numbers against cited source fields for every high-risk evaluation case. Score one known correct and one known incorrect trace and verify the reference values appear in scorer inputs and recorded feedback. Run traces with known incorrect totals and confirm the numeric scorer flags them even when safety passes. Introduce a mocked failing tool call and verify its span/error is detected independently of the final-answer score. ExplanationCorrect Answer C:MLflow scorers measure distinct attributes; passing one risk dimension does not establish task correctness. Other Answers A:Run traces with known incorrect totals and confirm the numeric scorer flags them even when safety passes. B:Run traces with known incorrect totals and confirm the numeric scorer flags them even when safety passes. D:Run traces with known incorrect totals and confirm the numeric scorer flags them even when safety passes.
Question 638 An evaluation run produces a high Safety score but no measurement of whether the requested invoice total was calculated correctly. Which operational practice should be made a durable requirement? Enforce scorer signature-to-dataset column mapping tests and typed evaluation fixtures on each run. Maintain separate severity-weighted factuality metrics and nonnegotiable high-risk acceptance thresholds. Keep stage-specific structured checks alongside model-based evaluators. Maintain a scorer portfolio aligned to business success, correctness and safety requirements. ExplanationCorrect Answer D:MLflow scorers measure distinct attributes; passing one risk dimension does not establish task correctness. Other Answers A:MLflow scorers measure distinct attributes; passing one risk dimension does not establish task correctness. B:This control addresses a different exposure: An evaluation report gives a high average BLEU score for summaries but hides hallucinated financial amounts in five important cases. It would not prevent this failure, because mLflow scorers measure distinct attributes; passing one risk dimension does not establish task correctness. C:That mLflow scorers measure distinct attributes; passing one risk dimension does not establish task correctness.
Question 639 An MLflow evaluation contains adjudicated expected answers, but a custom correctness scorer receives only the generated response because the answer column was not mapped into its inputs. What should the implementation team change first to resolve the described defect? Add a task-specific numeric correctness scorer alongside the built-in safety scorer and inspect results per trace. Map the existing expected_answer column explicitly into the scorer invocation and validate its typed inputs before evaluation. Inspect nested trace spans and add code-based checks for tool status and tool-output integrity in the evaluation. Add a custom scorer checking every citation ID against the trace's authorized retrieval-set manifest and relevant supporting text. ExplanationCorrect Answer B:The action targets the stated cause: A scorer can misjudge valid traces when its required argument mapping omits available reference data even though the dataset itself is complete. Check the repair by: score one known correct and one known incorrect trace and verify the reference values appear in scorer inputs and recorded feedback. Other Answers A:A scorer can misjudge valid traces when its required argument mapping omits available reference data even though the dataset itself is complete. C:A scorer can misjudge valid traces when its required argument mapping omits available reference data even though the dataset itself is complete. D:A scorer can misjudge valid traces when its required argument mapping omits available reference data even though the dataset itself is complete.
Question 640 An MLflow evaluation contains adjudicated expected answers, but a custom correctness scorer receives only the generated response because the answer column was not mapped into its inputs. Why does the current approach fail under these conditions? Citation syntax validity is weaker than provenance integrity; a fabricated identifier can still look well formed. A scorer can misjudge valid traces when its required argument mapping omits available reference data even though the dataset itself is complete. End-to-end generation failure can originate in tools, retrieval or prompts; one judge score cannot localize every cause. MLflow scorers measure distinct attributes; passing one risk dimension does not establish task correctness. ExplanationCorrect Answer B:Map the existing expected_answer column explicitly into the scorer invocation and validate its typed inputs before evaluation. Other Answers A:A scorer can misjudge valid traces when its required argument mapping omits available reference data even though the dataset itself is complete. C:That mechanism describes another problem: An agent trace includes a tool error, yet the evaluation dashboard attributes the bad final response only to an LLM judge score. It does not explain this case, where a scorer can misjudge valid traces when its required argument mapping omits available reference data even though the dataset itself is complete. D:A scorer can misjudge valid traces when its required argument mapping omits available reference data even though the dataset itself is complete.
