Question 350 Two models have similar answer quality, but one exhibits twice the P95 latency on the organization's real workload. Which verification is strongest for this particular failure mode? Use adjudicated summary cases to score citation support and hallucinations separately from n-gram overlap. Run labeled classification and load tests with cost per thousand requests, macro-F1 and P95 latency. Measure quality on unseen intent clusters and report confidence ranges or failure counts alongside point estimates. Run controlled A/B load tests with matched prompts and compare P50, P95, token use and error rate. ExplanationCorrect Answer D:Averages conceal slow-tail behavior; model comparison must use comparable traffic and agreed operational thresholds. Other Answers A:Run controlled A/B load tests with matched prompts and compare P50, P95, token use and error rate. B:Run controlled A/B load tests with matched prompts and compare P50, P95, token use and error rate. C:Run controlled A/B load tests with matched prompts and compare P50, P95, token use and error rate.
Question 351 Two models have similar answer quality, but one exhibits twice the P95 latency on the organization's real workload. What change to engineering governance would reduce this risk over time? Maintain language-segmented evaluation and safe fallback for unsupported locales. Keep evaluation datasets versioned and separated from optimization inputs. Treat high-impact factual errors as hard acceptance criteria in experimental model comparisons. Require reproducible experiments and release gates over both task quality and tail latency. ExplanationCorrect Answer D:Averages conceal slow-tail behavior; model comparison must use comparable traffic and agreed operational thresholds. Other Answers A:This control addresses a different exposure: A multilingual legal assistant's inexpensive model handles English but fails on Japanese clauses that affect customer obligations. It would not prevent this failure, because averages conceal slow-tail behavior; model comparison must use comparable traffic and agreed operational thresholds. B:That averages conceal slow-tail behavior; model comparison must use comparable traffic and agreed operational thresholds. C:Averages conceal slow-tail behavior; model comparison must use comparable traffic and agreed operational thresholds.
Question 352 A prompt change is evaluated on examples copied from the training set and appears perfect, but fails new user intents. Which engineering action most directly addresses this specific requirement? Measure quality, P95 response time and cost on the same stratified workload, then select the model meeting the published SLO. Compare factual groundedness and error severity alongside lexical overlap; reject the model change until unsupported claims are addressed. Recalibrate retrieval thresholds on labeled positives and negatives before promoting the new embedding model. Create a held-out labeled evaluation set that did not drive prompt tuning and compare versions without leakage. ExplanationCorrect Answer D:The action targets the stated cause: Evaluation leakage inflates metrics by testing memorized or overfitted examples rather than generalization. Check the repair by: measure quality on unseen intent clusters and report confidence ranges or failure counts alongside point estimates. Other Answers A:Evaluation leakage inflates metrics by testing memorized or overfitted examples rather than generalization. B:Evaluation leakage inflates metrics by testing memorized or overfitted examples rather than generalization. C:Evaluation leakage inflates metrics by testing memorized or overfitted examples rather than generalization.
Question 353 A prompt change is evaluated on examples copied from the training set and appears perfect, but fails new user intents. What architectural constraint is most directly responsible? Similarity scales and distributions depend on the embedding model and cannot be assumed stable across versions. Evaluation leakage inflates metrics by testing memorized or overfitted examples rather than generalization. Averages conceal slow-tail behavior; model comparison must use comparable traffic and agreed operational thresholds. Surface-form similarity cannot establish factual entailment or safety in free-form generated explanations. ExplanationCorrect Answer B:Create a held-out labeled evaluation set that did not drive prompt tuning and compare versions without leakage. Other Answers A:That mechanism describes another problem: A new embedding model changes similarity scores, and a fixed threshold rejects most valid questions even though the source corpus is unchanged. It does not explain this case, where evaluation leakage inflates metrics by testing memorized or overfitted examples rather than generalization. C:Evaluation leakage inflates metrics by testing memorized or overfitted examples rather than generalization. D:Evaluation leakage inflates metrics by testing memorized or overfitted examples rather than generalization.
Question 354 A prompt change is evaluated on examples copied from the training set and appears perfect, but fails new user intents. What validation evidence would most specifically confirm the resolution? Use adjudicated summary cases to score citation support and hallucinations separately from n-gram overlap. Run controlled A/B load tests with matched prompts and compare P50, P95, token use and error rate. Measure quality on unseen intent clusters and report confidence ranges or failure counts alongside point estimates. Plot positive/negative score distributions, adjust the threshold and verify recall plus false-positive rates. ExplanationCorrect Answer C:Evaluation leakage inflates metrics by testing memorized or overfitted examples rather than generalization. Other Answers A:Measure quality on unseen intent clusters and report confidence ranges or failure counts alongside point estimates. B:Measure quality on unseen intent clusters and report confidence ranges or failure counts alongside point estimates. D:Measure quality on unseen intent clusters and report confidence ranges or failure counts alongside point estimates.
Question 355 A prompt change is evaluated on examples copied from the training set and appears perfect, but fails new user intents. What prevention measure is most targeted to this underlying cause? Treat embedding-model upgrades as re-indexing and evaluation events with threshold regression tests. Require reproducible experiments and release gates over both task quality and tail latency. Treat high-impact factual errors as hard acceptance criteria in experimental model comparisons. Keep evaluation datasets versioned and separated from optimization inputs. ExplanationCorrect Answer D:Evaluation leakage inflates metrics by testing memorized or overfitted examples rather than generalization. Other Answers A:That evaluation leakage inflates metrics by testing memorized or overfitted examples rather than generalization. B:Evaluation leakage inflates metrics by testing memorized or overfitted examples rather than generalization. C:This control addresses a different exposure: A new prompt version improves BLEU-like overlap with expected summaries but increases unsupported legal claims. It would not prevent this failure, because evaluation leakage inflates metrics by testing memorized or overfitted examples rather than generalization.
Question 356 An evaluation dashboard shows a single high win-rate but conceals that the challenger fails every high-risk privacy case. Which implementation decision best restores the intended behavior? Compare factual groundedness and error severity alongside lexical overlap; reject the model change until unsupported claims are addressed. Segment results by risk category and impose an explicit privacy-failure threshold before accepting any aggregate quality gain. Measure quality, P95 response time and cost on the same stratified workload, then select the model meeting the published SLO. Use configurable sampled production monitoring with risk-based selection and targeted offline challenges for known severe cases. ExplanationCorrect Answer B:The action targets the stated cause: A pooled metric can hide concentrated severe failures in a small but important subgroup. Check the repair by: check subgroup confusion and failure lists, including all high-risk privacy cases, before choosing a winner. Other Answers A:A pooled metric can hide concentrated severe failures in a small but important subgroup. C:A pooled metric can hide concentrated severe failures in a small but important subgroup. D:A pooled metric can hide concentrated severe failures in a small but important subgroup.
Question 357 An evaluation dashboard shows a single high win-rate but conceals that the challenger fails every high-risk privacy case. Which underlying technical principle explains why this occurs? Monitoring coverage, inference overhead and tail-risk sensitivity trade off; indiscriminate full-traffic judging may be costly. A pooled metric can hide concentrated severe failures in a small but important subgroup. Averages conceal slow-tail behavior; model comparison must use comparable traffic and agreed operational thresholds. Surface-form similarity cannot establish factual entailment or safety in free-form generated explanations. ExplanationCorrect Answer B:Segment results by risk category and impose an explicit privacy-failure threshold before accepting any aggregate quality gain. Other Answers A:A pooled metric can hide concentrated severe failures in a small but important subgroup. C:A pooled metric can hide concentrated severe failures in a small but important subgroup. D:That mechanism describes another problem: A new prompt version improves BLEU-like overlap with expected summaries but increases unsupported legal claims. It does not explain this case, where a pooled metric can hide concentrated severe failures in a small but important subgroup.
Question 358 An evaluation dashboard shows a single high win-rate but conceals that the challenger fails every high-risk privacy case. After correction, what targeted test should demonstrate success? Use adjudicated summary cases to score citation support and hallucinations separately from n-gram overlap. Run controlled A/B load tests with matched prompts and compare P50, P95, token use and error rate. Verify sampling rates and compare flagged error categories against a separate labeled high-risk test set. Check subgroup confusion and failure lists, including all high-risk privacy cases, before choosing a winner. ExplanationCorrect Answer D:A pooled metric can hide concentrated severe failures in a small but important subgroup. Other Answers A:Check subgroup confusion and failure lists, including all high-risk privacy cases, before choosing a winner. B:Check subgroup confusion and failure lists, including all high-risk privacy cases, before choosing a winner. C:Check subgroup confusion and failure lists, including all high-risk privacy cases, before choosing a winner.
Question 359 An evaluation dashboard shows a single high win-rate but conceals that the challenger fails every high-risk privacy case. Which release safeguard most directly addresses this failure class? Require risk-stratified dashboards and non-negotiable safety thresholds in model experiments. Treat high-impact factual errors as hard acceptance criteria in experimental model comparisons. Require reproducible experiments and release gates over both task quality and tail latency. Establish sampling budgets plus escalation rules for incident-triggered high-risk audits. ExplanationCorrect Answer A:A pooled metric can hide concentrated severe failures in a small but important subgroup. Other Answers B:A pooled metric can hide concentrated severe failures in a small but important subgroup. C:This control addresses a different exposure: Two models have similar answer quality, but one exhibits twice the P95 latency on the organization's real workload. It would not prevent this failure, because a pooled metric can hide concentrated severe failures in a small but important subgroup. D:That a pooled metric can hide concentrated severe failures in a small but important subgroup.
Question 360 An agent produces a wrong invoice answer, and its current logs show only the final chat message, not the SQL tool call used. Which design change directly satisfies the stated constraints? Reconcile the March exam's Agent Framework terms with the October 2026 Agent Bricks CLI deployment guidance before migrating code. Log tool result identifiers and prompt context spans in the trace, then use recorded traces to review the failing branches. Instrument the framework stages with MLflow traces carrying prompts, retrieval events, tool results and output identifiers under appropriate redaction. Enable MLflow tracing of the authorized tool invocation, input parameters and result metadata with sensitive values redacted. ExplanationCorrect Answer D:The action targets the stated cause: Agent behavior spans tool calls and intermediate messages; missing spans block root-cause attribution. Check the repair by: replay the invoice and inspect an end-to-end trace linking the request, SQL query span and returned amount. Other Answers A:Agent behavior spans tool calls and intermediate messages; missing spans block root-cause attribution. B:Agent behavior spans tool calls and intermediate messages; missing spans block root-cause attribution. C:Agent behavior spans tool calls and intermediate messages; missing spans block root-cause attribution.
