Try AWS Certified AI Practitioner (AIF-C01)

Q1.A company is evaluating LLM outputs for a medical information application and wants to understand whether responses are accurate, empathetic, and appropriate for patients before deploying. They are considering using automated metrics but a clinical expert argues that certain qualities cannot be captured by automated scoring. When is human evaluation of LLM outputs most important?

• A.Evaluating subjective qualities like empathy and appropriateness that automated metrics cannot capture
• B.Evaluating high-stakes outputs in domains like healthcare where errors could cause patient harm
• C.Evaluating the inference latency of model API responses under load
• D.Evaluating the cost per API call across different model providers

Domain: Applications of Foundation Models

Q2.A security operations team is responsible for monitoring a production AI inference environment that processes thousands of model invocations per hour. They want to detect anomalies such as sudden spikes in invocation volume, unusual geographic origins of requests, or attempts to extract sensitive information through repeated probing. Which AWS service is most appropriate for setting up this behavioral monitoring and anomaly detection?

• A.Amazon CloudWatch metrics and alarms
• B.Amazon Macie automated PII discovery
• C.AWS Config configuration compliance rules
• D.Amazon Detective security investigation

Domain: Security, Compliance, and Governance for AI Solutions

Q3.A company is evaluating several third-party AI vendors to power a new customer analytics platform. The procurement team wants to ensure the selected vendor meets the company's responsible AI standards before signing a contract. Which of the following should the team evaluate when assessing a third-party AI provider's responsible AI practices?

• A.Reviewing the vendor's transparency documentation on model training, data, and known limitations
• B.Assessing whether the vendor conducts bias testing and fairness evaluations of their AI models
• C.Confirming that the vendor's API response latency consistently meets a 100-millisecond threshold
• D.Verifying that the vendor's headquarters is located in the same country as the procuring company

Domain: Guidelines for Responsible AI

Q4.A data scientist is evaluating a binary classification model that predicts whether a loan applicant will default. The team wants a single metric that measures the model's ability to distinguish between defaulters and non-defaulters across all possible classification thresholds, regardless of which threshold is chosen. Which metric should the data scientist use?

• A.AUC-ROC (Area Under the Receiver Operating Characteristic curve)
• B.Accuracy
• C.F1 score
• D.RMSE (Root Mean Squared Error)

Domain: Fundamentals of AI and ML

Q5.A developer is tuning a foundation model's inference parameters to control the diversity of generated text. They want to limit the model so it only considers the smallest set of top-ranked tokens whose cumulative probability exceeds 90% at each generation step, cutting off the long tail of unlikely words. Which inference parameter controls this behavior?

• A.Top-p (nucleus sampling)
• B.Temperature
• C.Top-k
• D.Max tokens

Domain: Fundamentals of Generative AI