AI Heuristic Evaluation: Building a UX Claude skill
Edit: In a follow-up post, I tested the skill against five different websites, uncovered critical issues that improved prompts can’t fully address, and proposed next steps for diving in into how to incorporate AI in research workflows.
I built a Claude skill for conducting usability heuristic evaluations because I wanted to see what happens when you try to encode parts of expert UXR into a system that can scale beyond one person’s attention span.
More honestly, I wanted to find out where it breaks.
The goal
Can I create an automated heuristic evaluation that feels defensible, expert-led, and grounded in real UXR reasoning rather than generic AI critique?
You have probably seen AI-generated UXR audits floating around LinkedIn. I wanted something that was closer to how skilled researchers actually think through an interface and something that was more accessible to anyone.
The process
I started by talking my husband’s ear off then brain-dumping ideas into Mural. Mostly my thoughts swirled around defensibility, severity scoring, and where the AI evaluation could fail.
The original concept centred around a few ideas:
Ground evaluations in established heuristic frameworks like Nielsen’s heuristics and Shneiderman’s Eight Golden Rules.
Use dual severity ratings (Nielsen’s severity rankings + FMEA) because a single scoring system can miss the true impact of an issue without human judgment and contextual interpretation.
Simulate different expert perspectives during evaluation.
Force explicit rationale for why an issue violates a heuristic for better traceability.
Require reproducible steps rather than vague observations.
Distinguish between verified issues (i.e., actually reproducible) and assumptions.
My core concern underneath all of it was validity.
UXR findings are not just observations. They are judgments made within context. Skilled researchers constantly calibrate confidence, question assumptions, and contextualize what they are seeing. A lot of that work is invisible in final deliverables because traditionally, the researcher is in the room to explain it. Once you try to automate parts of that process, you realize how much hidden judgment needs to be surfaced explicitly.
I shared the brainstorming notes with Claude and expanded on the intent behind the project. We discussed the failures I was worried about and the pieces that would make the evaluation actually useful. Claude drafted initial instructions. Then we iterated. A lot.
I pushed on places where the reasoning felt weak, overconfident, or methodologically shaky. Sometimes the fixes were small wording changes. Other times the structure itself needed to change because the workflow encouraged evaluations that would be unsound or potentially riddled with hallucinations.
The test
Once the instructions were stable enough, I asked Claude to evaluate my own website, danielanapoli.com.
This was critical because I needed a site where I could compare the findings against a website where I already understood the design rationale, constraints, tradeoffs, and implementation realities. Otherwise, it would have been difficult to pin point actual UX insight from potential hallucinations.
Some findings were genuinely useful. Especially around providing sufficient information to visitors, properly setting expectations, and improving flows between pages. Others were overinflated or completely wrong.
Specifically, Claude confidently flagged problems that did not exist, especially around dynamic content, client-rendered components, or navigation paths the model could not fully verify. Static fetches can return outdated pages without making that obvious, which means the model can generate entirely valid reasoning against a version of the interface that no longer exists. Every downstream artifact inherits the problem. Findings. Severity ratings. Recommendations. Everything.
Once I saw that happen, “freshness” verification became a required step in the workflow rather than a nice-to-have.
From there, I asked Claude to reflect on its own evaluation process and identify where the workflow itself introduced problems. The model got surprisingly good at identifying its own failure patterns once explicitly prompted to audit the process rather than just the interface.
The guardrails
A lot of the final workflow is less about generating UX insight and more about constraining bad AI behaviour. Things like separating candidate generation from severity scoring, requiring reproducible steps, explicitly documenting assumptions, verifying rendered behaviour, and tracking disagreements between evaluation systems.
Like my own research work, I cared most about whether the process was defensible. The skill’s workflow forces Claude to check:
Are all claims verified?
Are simulated quotes clearly labelled?
Are reproduction steps specific enough for a third party?
Were freshness checks completed?
Are disagreements between scoring systems explained?
Are out-of-scope areas documented?
At first glance this can seem like overkill, but without these sorts of checks the evaluation can quickly drift into confident-sounding pattern matching instead of something that resembles careful research reasoning.
The iterations
I ran the workflow against five other websites. After each iteration, the report was shared with the owners (my friends 😊). The intention was to recreate the same kind of feedback loop you would normally build into any UXR process; are these findings meaningful, actionable, and contextually accurate to the people closest to the product?
Sometimes the evaluations surfaced things owners already suspected but had trouble articulating. Sometimes the findings missed the mark entirely. The skill evolved every time, not through “prompt engineering hacks,” but by improving methodology and forcing clearer standards.
My next post goes into what happened when people actually received these reports.
The final step
I refactored the instructions into a reusable Claude skill. This way, things are easier for humans to parse and edit as they experiment building their own versions. You can download the skill from my Github and drag it into Claude’s skills folder. I’ll push more refinements as I keep working through things.
If you want to build your own skills in the meantime, these were the two resources I referenced most: