Question 330 An evaluation identifies a tool that sometimes calls DELETE rather than GET. What is the severity? A cosmetic UI issue Treat the incorrect DELETE call as a formatting defect and skip side-effect checks A potentially critical action-selection and permission defect requiring a release block A harmless wording difference Explanation Correct Answer C:A wrong HTTP method can change or destroy records. Incorrect Answers A:The defect concerns tool execution, not style. B:An unintended DELETE can destroy external records regardless of message tone. D:DELETE may alter external state.
Question 331 A test harness uses an AI judge to score policy accuracy. What additional assurance is needed? Suppress ambiguous cases from the score Treat every judge PASS as objective truth Use only the judge's confidence adjectives Calibrate the judge with authoritative examples and inspect disputed failures Explanation Correct Answer D:Model graders can be wrong or inconsistent. Incorrect Answers A:Omitting difficulty can bias results. B:Automated graders are not independent fact verification. C:Words like 'certain' aren't validation.
Question 332 An agent update changes the order of required questions in a form. How should evaluation detect a broken workflow? Check only screenshots of the new form without submitting values Ignore the new missing field if the action succeeds once Use multi-turn test cases asserting required prompts and final tool inputs Check only the welcome message Explanation Correct Answer C:Conversation state and ordering cannot be verified by single-turn text alone. Incorrect Answers A:A form can look correct but fail input sequencing or tool payload construction. B:An incomplete payload may fail later. D:Initial greeting doesn't test multi-step collection.
Question 333 A security agent should refuse unauthorized account changes. Which negative test matters? Request changes as a caller without rights and verify no write occurs Only check that a refusal sentence appears Make up an error after performing the write Test only with an administrator account Explanation Correct Answer A:A refusal must be backed by actual tool nonexecution. Incorrect Answers B:The tool might still have executed behind the message. C:If a tool performed an unauthorized write, a polite refusal shown afterward still misrepresents the actual external side effect and cannot undo the change. D:Admin success doesn't cover denial behavior.
Question 334 A test set includes successful payment examples but not timeouts. What should be added? Ambiguous timeout cases checking idempotent retry and truthful pending states Test only network latency after the payment completes successfully An expectation that all timeouts mean failure Only more successful payment examples Explanation Correct Answer A:Distributed writes can commit before the response is lost. Incorrect Answers B:Ordinary delay doesn't cover uncertain commits and idempotent retry behavior. C:A timeout is not definitive backend rejection. D:Additional happy paths don't probe uncertain completion.
Question 335 An agent's output varies across two acceptable phrasings of the same policy. How should scoring work? Ignore factual errors if phrasing sounds fluent Evaluate underlying factual requirements rather than require one exact sentence Reject all synonyms automatically Use the longest output as the gold answer Explanation Correct Answer B:Equivalent compliant answers should not fail due to harmless wording. Incorrect Answers A:Style cannot override policy truth. C:Rigid strings can misclassify correct explanations. D:Length isn't a semantic correctness measure.
Question 336 A company must verify its French agent answers preserve official consent text. What should be in tests? Assume English success guarantees all languages Only translate the English evaluation score Locale-specific prompts with approved French legal language and exact required fragments Accept any machine-translated consent without review Explanation Correct Answer C:Localized legal content needs independent validation. Incorrect Answers A:Language variants can alter meaning. B:A translated score doesn't test French content. D:Unverified language may change legal obligations.
Question 337 An operations agent has an external rate-limit quota. What should performance tests include? Burst requests and expected throttling, backoff and user-status behavior Only isolated one-request tests Treat HTTP 429 as successful business output An expectation that unlimited throughput is always available Explanation Correct Answer A:Production concurrency can trigger limits not seen in single calls. Incorrect Answers B:They don't exercise shared quota behavior. C:A throttle isn't an accepted transaction. D:The provider has explicit rate or concurrency quotas; a test should examine 429 behavior and backoff rather than assume unlimited throughput.
Question 338 A health assistant has a policy to escalate safety-critical uncertainty. Which case should QA include? Ambiguous medical-risk questions with expected escalation and no invented prescription Cases that reward unsupported confident treatment A visual check of the chat background Only clear harmless factual questions Explanation Correct Answer A:High-impact uncertainty must trigger the intended safeguard. Incorrect Answers B:Rewarding the agent for obeying malicious text embedded in retrieved documents would validate the exact prompt-injection vulnerability the evaluation is meant to detect. C:Aesthetic checks don't assess safety behavior. D:They won't exercise the escalation boundary.
Question 339 An agent's scores improve after removing all hard questions from the evaluation set. What is the problem? Hard tests are irrelevant by definition Test counts alone prove coverage Selection bias invalidates comparison of reliability The model has necessarily become more capable Explanation Correct Answer C:An easier suite can inflate pass rates without improving the agent. Incorrect Answers A:High-impact edge cases can be essential. B:Quantity doesn't replace representative difficulty. D:No comparable evidence establishes improvement.
Question 340 A team wants to detect when a connector starts returning stale inventory. Which ongoing evaluation works? One successful lookup on launch day Scheduled queries comparing the connector with the authoritative inventory record Comparing inventory to public marketing copy Assuming correct JSON means correct inventory Explanation Correct Answer B:Freshness must be monitored after deployment. Incorrect Answers A:A single historical check cannot detect future staleness. C:Marketing pages aren't live stock truth. D:A well-formed response can contain stale data.