Question 380 A deployed support assistant passes its original test set yet slowly degrades as billing policies change over several months. Which design change directly satisfies the stated constraints? Add production sampling and MLflow scorer monitoring for groundedness and policy freshness while retaining periodic offline regression tests. Register the model or agent artifact and configuration revision with MLflow and link deployed versions to their evaluation runs. Use configurable sampled production monitoring with risk-based selection and targeted offline challenges for known severe cases. Run offline evaluation on labeled or curated prompts before deployment, then establish production monitoring after release. ExplanationCorrect Answer A:The action targets the stated cause: Static evaluation cannot detect all new input distributions or changing external knowledge after launch. Check the repair by: track scorer feedback by week and correlate rising unsupported claims with policy-version changes. Other Answers B:Static evaluation cannot detect all new input distributions or changing external knowledge after launch. C:Static evaluation cannot detect all new input distributions or changing external knowledge after launch. D:Static evaluation cannot detect all new input distributions or changing external knowledge after launch.
Question 381 A deployed support assistant passes its original test set yet slowly degrades as billing policies change over several months. Which causal explanation best accounts for the failure? Offline evaluation estimates expected quality in controlled conditions; live monitoring detects drift and operational failures after deployment. Static evaluation cannot detect all new input distributions or changing external knowledge after launch. Monitoring coverage, inference overhead and tail-risk sensitivity trade off; indiscriminate full-traffic judging may be costly. Reproducible agent behavior depends on code, model, prompts and dependencies, not just the endpoint name. ExplanationCorrect Answer B:Add production sampling and MLflow scorer monitoring for groundedness and policy freshness while retaining periodic offline regression tests. Other Answers A:Static evaluation cannot detect all new input distributions or changing external knowledge after launch. C:Static evaluation cannot detect all new input distributions or changing external knowledge after launch. D:That mechanism describes another problem: A project promotes agent code without recording dependency or prompt revision identifiers, making a rollback to known behavior impossible. It does not explain this case, where static evaluation cannot detect all new input distributions or changing external knowledge after launch.
Question 382 A deployed support assistant passes its original test set yet slowly degrades as billing policies change over several months. Which concrete acceptance test best demonstrates the proposed change works? Track scorer feedback by week and correlate rising unsupported claims with policy-version changes. Restore the previous registered version in staging and verify trace metadata, dependencies and known-good acceptance tests. Confirm evaluation scores on the frozen set before production traffic exists, then validate live-trace scoring once deployed. Verify sampling rates and compare flagged error categories against a separate labeled high-risk test set. ExplanationCorrect Answer A:Static evaluation cannot detect all new input distributions or changing external knowledge after launch. Other Answers B:Track scorer feedback by week and correlate rising unsupported claims with policy-version changes. C:Track scorer feedback by week and correlate rising unsupported claims with policy-version changes. D:Track scorer feedback by week and correlate rising unsupported claims with policy-version changes.
Question 383 A deployed support assistant passes its original test set yet slowly degrades as billing policies change over several months. Which operational practice should be made a durable requirement? Establish sampling budgets plus escalation rules for incident-triggered high-risk audits. Define separate pre-release quality gates and post-release monitoring thresholds. Treat model and prompt artifact versioning as deployment inputs under change control. Schedule drift and freshness reviews from sampled live traces with approved retention boundaries. ExplanationCorrect Answer D:Static evaluation cannot detect all new input distributions or changing external knowledge after launch. Other Answers A:Static evaluation cannot detect all new input distributions or changing external knowledge after launch. B:This control addresses a different exposure: A new agent has not shipped, but engineers enable production anomaly dashboards rather than checking whether it answers benchmark questions correctly. It would not prevent this failure, because static evaluation cannot detect all new input distributions or changing external knowledge after launch. C:That static evaluation cannot detect all new input distributions or changing external knowledge after launch.
Question 384 A team proposes running expensive LLM judges on every production request, overwhelming budget while missing low-volume rare failure categories. What should the implementation team change first to resolve the described defect? Use configurable sampled production monitoring with risk-based selection and targeted offline challenges for known severe cases. Run offline evaluation on labeled or curated prompts before deployment, then establish production monitoring after release. Segment results by risk category and impose an explicit privacy-failure threshold before accepting any aggregate quality gain. Add production sampling and MLflow scorer monitoring for groundedness and policy freshness while retaining periodic offline regression tests. ExplanationCorrect Answer A:The action targets the stated cause: Monitoring coverage, inference overhead and tail-risk sensitivity trade off; indiscriminate full-traffic judging may be costly. Check the repair by: verify sampling rates and compare flagged error categories against a separate labeled high-risk test set. Other Answers B:Monitoring coverage, inference overhead and tail-risk sensitivity trade off; indiscriminate full-traffic judging may be costly. C:Monitoring coverage, inference overhead and tail-risk sensitivity trade off; indiscriminate full-traffic judging may be costly. D:Monitoring coverage, inference overhead and tail-risk sensitivity trade off; indiscriminate full-traffic judging may be costly.
Question 385 A team proposes running expensive LLM judges on every production request, overwhelming budget while missing low-volume rare failure categories. Why does the current approach fail under these conditions? Static evaluation cannot detect all new input distributions or changing external knowledge after launch. A pooled metric can hide concentrated severe failures in a small but important subgroup. Offline evaluation estimates expected quality in controlled conditions; live monitoring detects drift and operational failures after deployment. Monitoring coverage, inference overhead and tail-risk sensitivity trade off; indiscriminate full-traffic judging may be costly. ExplanationCorrect Answer D:Use configurable sampled production monitoring with risk-based selection and targeted offline challenges for known severe cases. Other Answers A:Monitoring coverage, inference overhead and tail-risk sensitivity trade off; indiscriminate full-traffic judging may be costly. B:That mechanism describes another problem: An evaluation dashboard shows a single high win-rate but conceals that the challenger fails every high-risk privacy case. It does not explain this case, where monitoring coverage, inference overhead and tail-risk sensitivity trade off; indiscriminate full-traffic judging may be costly. C:Monitoring coverage, inference overhead and tail-risk sensitivity trade off; indiscriminate full-traffic judging may be costly.
Question 386 A team proposes running expensive LLM judges on every production request, overwhelming budget while missing low-volume rare failure categories. Which observation would establish that the requirement is now met? Check subgroup confusion and failure lists, including all high-risk privacy cases, before choosing a winner. Track scorer feedback by week and correlate rising unsupported claims with policy-version changes. Confirm evaluation scores on the frozen set before production traffic exists, then validate live-trace scoring once deployed. Verify sampling rates and compare flagged error categories against a separate labeled high-risk test set. ExplanationCorrect Answer D:Monitoring coverage, inference overhead and tail-risk sensitivity trade off; indiscriminate full-traffic judging may be costly. Other Answers A:Verify sampling rates and compare flagged error categories against a separate labeled high-risk test set. B:Verify sampling rates and compare flagged error categories against a separate labeled high-risk test set. C:Verify sampling rates and compare flagged error categories against a separate labeled high-risk test set.
Question 387 A team proposes running expensive LLM judges on every production request, overwhelming budget while missing low-volume rare failure categories. What lasting control best prevents recurrence of the identified issue? Establish sampling budgets plus escalation rules for incident-triggered high-risk audits. Define separate pre-release quality gates and post-release monitoring thresholds. Require risk-stratified dashboards and non-negotiable safety thresholds in model experiments. Schedule drift and freshness reviews from sampled live traces with approved retention boundaries. ExplanationCorrect Answer A:Monitoring coverage, inference overhead and tail-risk sensitivity trade off; indiscriminate full-traffic judging may be costly. Other Answers B:This control addresses a different exposure: A new agent has not shipped, but engineers enable production anomaly dashboards rather than checking whether it answers benchmark questions correctly. It would not prevent this failure, because monitoring coverage, inference overhead and tail-risk sensitivity trade off; indiscriminate full-traffic judging may be costly. C:That monitoring coverage, inference overhead and tail-risk sensitivity trade off; indiscriminate full-traffic judging may be costly. D:Monitoring coverage, inference overhead and tail-risk sensitivity trade off; indiscriminate full-traffic judging may be costly.
Question 388 A dashboard treats a production monitoring safety score as proof that every response in the prior month is technically correct. What is the most targeted corrective step for this observed failure? Add production sampling and MLflow scorer monitoring for groundedness and policy freshness while retaining periodic offline regression tests. Report sampled monitoring coverage and uncertainty separately from full-bank offline evaluation or qualified human review. Run offline evaluation on labeled or curated prompts before deployment, then establish production monitoring after release. Fail the safety and tone criterion independently of policy factuality and route the case through controlled moderation or rewrite review. ExplanationCorrect Answer B:The action targets the stated cause: Sampling gives evidence on observed traces, not exhaustive certification of unreviewed answers. Check the repair by: inspect the monitored trace count and selection policy and disclose unscored traffic and judge limitations. Other Answers A:Sampling gives evidence on observed traces, not exhaustive certification of unreviewed answers. C:Sampling gives evidence on observed traces, not exhaustive certification of unreviewed answers. D:Sampling gives evidence on observed traces, not exhaustive certification of unreviewed answers.
Question 389 A dashboard treats a production monitoring safety score as proof that every response in the prior month is technically correct. What is the decisive mechanism behind the observed result? Offline evaluation estimates expected quality in controlled conditions; live monitoring detects drift and operational failures after deployment. Correct factual content does not make abusive presentation acceptable; quality evaluation needs separate safety dimensions. Sampling gives evidence on observed traces, not exhaustive certification of unreviewed answers. Static evaluation cannot detect all new input distributions or changing external knowledge after launch. ExplanationCorrect Answer C:Report sampled monitoring coverage and uncertainty separately from full-bank offline evaluation or qualified human review. Other Answers A:That mechanism describes another problem: A new agent has not shipped, but engineers enable production anomaly dashboards rather than checking whether it answers benchmark questions correctly. It does not explain this case, where sampling gives evidence on observed traces, not exhaustive certification of unreviewed answers. B:Sampling gives evidence on observed traces, not exhaustive certification of unreviewed answers. D:Sampling gives evidence on observed traces, not exhaustive certification of unreviewed answers.
Question 390 A dashboard treats a production monitoring safety score as proof that every response in the prior month is technically correct. Which verification is strongest for this particular failure mode? Inspect the monitored trace count and selection policy and disclose unscored traffic and judge limitations. Track scorer feedback by week and correlate rising unsupported claims with policy-version changes. Score factuality and harmful-language behavior separately on adversarial tone prompts and confirm safe alternatives pass. Confirm evaluation scores on the frozen set before production traffic exists, then validate live-trace scoring once deployed. ExplanationCorrect Answer A:Sampling gives evidence on observed traces, not exhaustive certification of unreviewed answers. Other Answers B:Inspect the monitored trace count and selection policy and disclose unscored traffic and judge limitations. C:Inspect the monitored trace count and selection policy and disclose unscored traffic and judge limitations. D:Inspect the monitored trace count and selection policy and disclose unscored traffic and judge limitations.
