Five Websites, One Claude Skill, and the Thing Prompts Can't Fix
In my last post, I argued that the validity layer (with a skilled researcher reviewing findings before they go anywhere) is the thing you can’t automate out of AI-assisted UXR.
Then, I built a skill that runs a heuristic evaluation for Claude to run through a website, flag issues by severity, synthesize the findings, and generate a report with actionable recommendations.
Did it hold up?
I tried to answer that honestly by running the skill against five personal websites and getting candid reactions to the outputs. Here’s what I found.
The setup
Five websites. Five participants including me. The sites were all career-focused portfolio pieces: places to communicate skill and invite engagement. They covered a range of expertise including UX, software development, digital art and game design, and book editing.
For each site, Claude generated a full evaluation report and an executive summary with prioritized, actionable recommendations. I shared both with the other site owners and gave them one instruction: “Be candid. This isn’t my baby. I genuinely want to know what you think of the evaluation.” They reviewed the reports on their own time and sent back their thoughts. I reviewed the feedback, looked for patterns, and noted anything that surprised me.
Disclaimer: This study is informal. The feedback happened asynchronously, mostly over email and Discord, so there wasn’t much room for probing. I write this shortly after the reports were provided, so long-term impact isn’t something I can speak to. There’s probably some friend bias in the feedback provided. And the sites themselves were simple, mostly static pages, which means the skill didn’t get stress-tested against anything complex. It’s a first look, not a final verdict.
What happened, site by site
Site 1: My own website
Claude’s findings were detailed and useful in a way that’s hard to achieve when looking at your own work. The skill caught things I wouldn’t have caught unless I walked away for a while and came back with fresh eyes and a systematic checklist.
Issues were ranked by severity. Most of mine were quick fixes, which was genuinely reassuring.
But here’s where it got interesting. I had recently updated my navigation, and Claude kept flagging nav inconsistencies across pages as a critical issue. I asked it to walk me through the recreation steps. I followed them exactly but could not reproduce the problems. When I pushed back, Claude explained that it was likely facing caching issues and was pulling from a version of the page that didn’t reflect my current website. We discussed how this probably wasn’t a real issue for an actual visitor.
Before sharing the skill with others, we worked on adding a validation step. Before flagging something as critical, Claude must check its confidence that the issue actually exists. Issues it isn’t sure about get marked for manual review and hedged appropriately in the report.
It helped. But as you’ll read, it didn’t fully solve it. Claude still surfaced noise.
Site 2: A software engineer’s portfolio
The site owner’s reaction was something like surprised appreciation. They weren’t skeptical of AI — they use it regularly at work — but they hadn’t thought about this specific application. What got their attention was that the evaluation surfaced things they genuinely hadn’t considered when building their website.
The severity labels were a sticking point. “Critical” sounds alarming. When the owner read the actual issues underneath that label and found things like minor layout inconsistencies, that mismatch between the label and the lived experience of the issue discouraged them from taking the report seriously.
The more interesting finding was a JavaScript typewriter component on the site. Claude flagged it as the highest-priority issue, arguing that the intermediate frames created a broken-looking experience for users. But anyone watching that component for more than a second understands it’s text being typed out. It’s a deliberate design choice. It reads correctly as a human.
The site owner put it well: “I understand the logic behind the issues, but I’m also not confident it’s seeing things as a human sees.”
Site 3: A digital artist’s portfolio
A running theme so far: the site owner’s first reaction was surprise at the scope of what got caught. The owner hadn’t expected Claude to test links or read discoverability settings. Those were intentionally turned off because the owner sends the site to specific people during hiring conversations, with no intention for the website to be indexed for general traffic.
The skill flagged disabled search indexing as the single highest-impact defect on the site. For most portfolios, that finding would be critical. For this one, it was a deliberate choice.
What stood out beyond that was less about what the skill flagged and more about what it didn’t. The site owner was expecting criticism of the creative work itself, the posing, the angles, the portfolio curation choices. The skill didn’t go there. It stayed in its lane, surfacing structural and editorial issues: Wix template scaffolding leaking onto live project pages, a project named three different things across the nav, the URL, and the page heading. While unexpected, these are the kind of issues a heuristic evaluation is designed to detect.
Site 4: A software developer’s portfolio
This site was intentionally minimal. One page, a bio, three social links, and one featured project. The skill found five issues, two critical. The scope felt appropriate.
The most significant finding was a light-mode contrast failure. The site was built dark-mode-first, and the amber link colour passes contrast in dark mode but doesn’t pass in light mode. The fix is a single CSS override. Claude also flagged that the email address is hidden behind an icon with no visible text, meaning anyone on a device where mailto links don’t resolve hits a dead end trying to make contact.
The report also recommended that the site owner add a case study or skills section to better communicate their technical background. A recruiter landing on a one-page bio with no visible tech stack has to pivot immediately to LinkedIn to evaluate qualifications.
Site 5: A book editor’s portfolio
The findings were generally sound. The skill caught things that matter for a professional services site: a missing call-to-action on the homepage, vague pricing with no worked examples, a contact form that asks one open-ended question when it needs five structured ones, a portfolio last updated three years ago. For a book editor whose entire business runs on trust and credibility, those are real problems.
One moment stood out. When describing a cluster of WordPress defaults bleeding through the site, the report noted that the issues, “[…] make a professional editing service look like a hobby blog. None of these add value. All are removable.” The first sentence did the work. The second and third didn’t need to be there.
What the skill does well
It’s fast.
A comprehensive review, synthesized findings, and a prioritized report took under ten minutes. A skilled researcher doing this work thoroughly could spend twenty to thirty hours on an evaluation this deep. That’s no small potatoes.
The reports also landed better than I expected with people who weren’t researchers. Site owners consistently noted that it wasn’t just flagging problems, it was also explaining why something was a problem, which usability principle it violated, and what the downstream impact might be for the their goals (attracting clients, getting hired). That context is what makes a finding actionable versus just feedback.
One detail I hadn’t anticipated, since reports are markdown that made them easy to paste directly into a coding agent to support implementation. Only I had time to do this and that made my evaluation a starting point for a fixing workflow, not just a document.
These outcomes are a direct result of how the skill was designed. The UXR expertise went into defining the evaluation approach, writing careful instructions, and building in safeguards. The speed is real, but it comes from deliberate design choices, not from the model figuring it out on its own as I shall elaborate.
Where it breaks down
Fragile trust
The caching problem from Site 1 kept showing up. The skill works from what it can see, and that’s not always the current version of the site. When it’s unsure, it can flag something for manual review, but it still surfaced issues that don’t exist. That noise degraded trust.
The severity calibration made it worse. When “critical” got applied to issues that, in context, weren’t actually urgent, the label lost its weight. The value of a prioritized report is to give people a shortcut for focusing their remediation efforts. That falls apart if people can’t trust the labels.
The typewriter component from Site 2 is where it really fell apart. It flagged something that looked broken but wasn’t. A skilled researcher might have noticed it too, but they’d take a beat, watch the component complete its loop, and immediately understand what it was doing. The skill observed a state. Figuring out whether something is actually a problem requires interacting with the interface, not just inspecting it.
Lackluster recommendations
The skill asks clarifying questions before starting, follows a set of instructions, and runs a final checklist. It still managed to be inconsistent. It recommended that the software developer add a case study to better communicate their technical background. It didn’t make the same suggestion for the digital artist, where the same gap existed. The skill inferred what each site needed from the artifact alone, without asking what conventions apply in each domain or what the site owner actually needed the evaluation to do for them. A skilled researcher would have asked those questions before starting either evaluation. When those inferences are right, the report lands well. When they’re off, the evaluation is technically sound but not useful to the people it was meant to help.
The skill also never recommended next research steps. A skilled researcher knows when findings are pointing at a question a heuristic evaluation can’t answer, and says so. They’d recommend a follow-up study, flag that the method has real limitations in this context, or identify that what the site actually needs is a different kind of research entirely. The skill delivers what it was designed to deliver. It doesn’t know what it doesn’t know, and it won’t tell you when to go further.
Being kind of a jerk
Site 5 surfaced a different kind of failure. One that I didn’t expect.
The skill flagged a cluster of WordPress defaults and noted that they make a professional editing service look like a hobby blog, and kept going: none of them add value, all are removable.
A skilled researcher knows when a finding has landed and stops. Knowing when enough has been said requires reading the person on the other side of the report. That’s social judgment, not a prompt.
What comes next
Running this evaluation didn’t just surface findings about the websites. It surfaced problems with the skill itself. The skill could clearly benefit from tweaks like forcing hard stops to prevent assumptions that lead to inconsistency. But there’s much more to do.
Token optimization
After testing all the sites, I worked through an optimization pass with Claude. I give full details on what changed and why in the notes.md file in the skill’s GitHub repository.
The optimization process mirrored the dynamic I’ve been writing about: Claude did the heavy lifting on identifying what could be cut and proposing the changes but before anything was touched, I asked what each change would actually do, what was being lost, and what the implications were for the integrity of the evaluation.
That back-and-forth is the job of a researcher. Not reviewing a finished output, but probing the reasoning before giving the green light. The researcher needs to apply pressure throughout the work, not just be a quality check at the end.
We got the token footprint down by about 25% per full invocation. On Pro with Opus, this skill can evaluate one site, maybe two, every five hours. But there’s only so much optimization that can be done.
A thorough heuristic evaluation has a high token cost because it needs to be thorough. Trimming any further runs the risk of degrading quality and that’s not a worthwhile trade off.
Pressure-testing
The portfolios in this study were a reasonable starting point, but simple and mostly static sites don’t stress the skill in other ways that matter in industry.
The next test should be something closer to what’s actually at stake in industry. It should evaluate a SaaS tool with multi-step flows, an e-commerce experience where the goal is a completed purchase, a complex informational site with tens of pages and competing user needs. These are the sorts of sites where the evaluation isn’t just a conversation starter, but potentially an entire web team’s roadmap.
Whether the skill holds up under volume (more findings, more pages, more competing priorities to reconcile) and comes up with something a team could actually build against is a worthwhile investigation. A prioritized, actionable roadmap is a different deliverable than a list of flagged issues. The skill would have to handle the pure weight of findings without losing the thread of what matters most, in a context where getting that wrong has real (financial) consequences.
Human versus AI
An honest test of the rigour of this skill would be a 1-to-1 comparison to researcher-led heuristic evaluations on the same sites. It would assess what got caught, what got missed, how severity rankings hold up, and where interpretations diverge.
What would make this comparison interesting is comparing Claude’s output against a range of UXR expertise. Instead of using a single benchmark, this approach could tell us something more about where the skill actually sits on the spectrum of skilled research. We hear a lot that AI performs like a competent junior researcher, solid on the brunt work but still needing guidance on judgment calls. A multi-researcher comparison would let us actually investigate that claim.
The number of findings wouldn’t be the point. The meaningful difference, if there is one, will show up in the depth of insight and the quality of the recommendations. That’s where UXR earns its place on a team. Our methods have to be sound, but truly influential work shows up downstream in a product’s trajectory shifts and vision sharpening. Findings that accurately identify usability issues are helpful. Recommendations that change how a team thinks about their users are something else entirely.
The Takeaway
AI heuristic evaluation does not replace researchers.
What it does is shed light on something we already knew but don’t usually need to defend: the humanistic qualities we bring to our work can outweigh the rigour of the processes we follow. It’s true, AI has a real advantage in speed and systematic coverage. But speed and coverage aren’t always enough to produce something meaningful for the team on the other side of the report.
What’s missing isn’t more instructions or better guardrails. It’s judgment that lives outside the context window.
The skill knew when to flag an issue. It didn’t know when it was wrong. It knew how to apply a framework. It didn’t know which conventions mattered in a domain it hadn’t been asked about. It couldn’t read the room and sense when a finding had already landed.
No amount of prompt engineering fully accounts for these factors because they are genuinely human. We can design for it, constrain toward it, and reduce the surface area where it fails. But there’s no guarantee it follows through consistently, and even less that it follows through humanely.
This is the case for why skilled researchers are still the most important part of AI-assisted UXR.
Every guardrail exists because a researcher understood what could go wrong and built toward it. Every finding that landed well did so because someone who understood the work, the context, and the person receiving it was in the loop. The system didn’t produce that.
The faster and more capable these tools get, the more researcher judgment becomes the thing worth protecting. Not because AI can’t do the work, but because without you, the work isn’t worth doing.