AI Evaluation
AI Response Evaluation
Practical Sample Case Study
A comparative evaluation of two AI-generated responses based on accuracy, instruction following, relevance, clarity and responsible handling of uncertainty.
Task
The brief
The original user prompt was:
"I'm planning to work remotely while travelling around Europe for three months. I have a Portuguese passport and a budget of €1,500 per month. I prefer safe, walkable cities with reliable public transport and good internet. I don't care about nightlife.
Recommend three European cities and explain why each would suit me. Please consider my budget and don't recommend cities where €1,500 per month would be clearly unrealistic."
Two AI-generated responses were evaluated.
Response A
Summary
Response A recommended:
- Lisbon, Portugal
- Prague, Czech Republic
- Sofia, Bulgaria
It stated that €1,500 per month should allow the user to live comfortably in Lisbon, including renting a centrally located one-bedroom apartment, eating out regularly and participating in local activities.
It also stated:
"As a Portuguese EU citizen, you can live and work freely in any European country without visas or time restrictions."
Response B
Summary
Response B recommended:
- Sofia, Bulgaria
- Kraków, Poland
- Valencia, Spain
It explained each recommendation in relation to affordability, transport, walkability and internet.
It also warned that Valencia would be the tightest option within the €1,500 budget and recommended checking current short-term accommodation prices.
Regarding the Portuguese passport, it referred specifically to EU free-movement rules while noting that registration, tax, employment and social-security requirements could still depend on individual circumstances.
My Evaluation
Scoring
| Criterion | Response A | Response B |
|---|---|---|
| Accuracy | 8/10 | 9/10 |
| Instruction Following | 6/10 | 9/10 |
| Relevance | 6/10 | 9/10 |
| Clarity | 7/10 | 10/10 |
| Responsible Uncertainty | 4/10 | 9/10 |
| TOTAL | 31/50 | 46/50 |
Final Decision
Selected response
Decision
Response B was selected as the stronger response.
I selected Response B because it addresses the user's request more directly and structures the recommendations around the user's stated priorities.
It identifies the three recommended cities immediately and explains why each may be suitable in terms of budget, public transport, walkability and internet access.
Response B also handles uncertainty more responsibly. Instead of presenting living costs as guaranteed, it acknowledges that accommodation prices can vary and specifically warns that Valencia may be more difficult within the €1,500 monthly budget.
Its explanation regarding the Portuguese passport is also more precise because it discusses EU free movement while acknowledging that registration, tax, employment and social-security requirements may depend on the individual's circumstances.
Analysis
Key issues identified in Response A
Finding 1
Excessive confidence regarding Lisbon's affordability.
The response assumes that a €1,500 total monthly budget would allow comfortable living in central Lisbon, including a centrally located one-bedroom apartment and regular restaurant spending.
This should not be presented as a confident assumption because accommodation costs are highly variable and short-term rentals can significantly affect the budget.
Finding 2
Incorrect geographical/legal generalization.
The statement that a Portuguese citizen can live and work freely in "any European country" incorrectly treats Europe and the European Union as equivalent.
Not every European country belongs to the EU, and different immigration or work requirements may apply.
Finding 3
Excessively absolute language.
The phrase "without visas or time restrictions" is too absolute because residence, registration, taxation, employment and social-security requirements can depend on the country, length of stay and individual circumstances.
Skills demonstrated
- AI response evaluation
- Comparative analysis
- Instruction-following assessment
- Accuracy assessment
- Detection of overconfidence
- Responsible uncertainty evaluation
- Critical thinking
- Attention to detail