Usability testing of AI services in the UK Government: Testing beyond usability

In our previous article, How GenAI Transforms User Research in UK Government Services, we explored how generative AI challenges assumptions that underpin traditional digital services. We discussed how AI introduces greater variability into user journeys, changes how people seek information, and makes traditional definitions of success, trust and user behaviour more difficult to apply.

One theme we touched on was uncertainty. Unlike traditional government services, people can interact with AI in highly variable ways, often pursuing the same goal through different approaches, receiving different responses, and interpreting them differently.

Over the past year, our User-Centred Design team has been supporting a major government agency in exploring how staff can use AI agents to understand and apply regulatory rules and guidance based on authoritative guidance texts. What became increasingly clear was that the usability testing challenge was not simply understanding how users interacted with AI. Rather, it raised a more fundamental question: what exactly are we testing when we conduct usability testing with generative AI?

Usability is only part of what we are testing

Traditional usability testing often operates within relatively stable boundaries. Researchers define realistic scenarios, agree on what successful completion broadly looks like, and observe participants attempting tasks. While individual experiences may vary, researchers usually understand the intended journey, expected system behaviour, and likely destination.

Generative AI introduces a different dynamic.

Users may approach the same goal with different levels of existing knowledge, different questions, and different expectations of the AI. These differences shape their prompts, the responses they receive, and the follow-up paths they pursue. Consequently, being able to use the AI agent is only one part of task success. We also need to understand whether the interaction helps users bridge the gap between what they already know and what they need to know to progress towards their goal.

As a user researcher, I quickly realised that evaluating AI services was less about assessing a predefined user journey and more about making sense of the many possible interaction paths and the factors shaping them. Evaluating task success therefore requires us to consider the user’s starting point and intentions, how they interact with the AI, and whether that interaction helps them move forward.

The hardest work happens before the testing begins

Scenario design matters in any usability study. However, with AI, particularly in complex and high-stakes environments such as government regulatory decision-making, small differences in context can propagate throughout the interaction. They influence how a participant interprets the task, what they ask the AI, the response they receive, and what they do next.

Creating meaningful research activities therefore required close collaboration with designers, product teams, and client-side subject matter experts who understood both the regulatory context and the realities of day-to-day decision-making. We needed to provide enough shared context for participants to respond to broadly the same underlying situation, without prescribing how they should interact with the AI.

This is perhaps one of the less discussed aspects of AI usability testing (Nielsen, 2026). Much of the complexity lies in designing the conditions that make those conversations interpretable in the first place.

A poor outcome does not always mean the same thing

AI interactions can create greater ambiguity when researchers try to diagnose what happened.

When users struggle with a traditional service, the source of the issue is often relatively straightforward to identify. A form may be unclear, navigation may be difficult to find, or the information architecture may not align with user expectations.

With AI, a disappointing outcome may originate from several points in the interaction. The user’s question may lack context. The system may misunderstand their intent. Relevant information may not have been retrieved. The response may be technically incorrect or operationally unhelpful. Alternatively, the user may expect the AI to perform a role the service was never designed to fulfil.

Researchers therefore need to understand what the user intended to achieve, how they engaged with the AI, and whether the response helped them progress towards that goal.

For example, one user may be looking to validate a decision they have already made, while another may use the same service to explore possible options. The same technically accurate response may therefore provide very different levels of value.

This makes task success harder to define. Rather than simply asking whether users reached a predefined endpoint, researchers need to consider whether the interaction meaningfully supported their intended goal. Understanding user intention therefore becomes central to interpreting usability evidence.

User behaviour is becoming a moving target

A further challenge emerging across multiple rounds of research is that the people we are testing with are changing.

As AI becomes increasingly embedded in the workplace, users are developing new habits, expectations and behaviours. They learn from each interaction, share approaches with colleagues, and develop their own strategies for working with AI.

This raises an interesting question: are we evaluating the usability of the system, or are we observing a snapshot of users’ AI literacy at a particular point in time?

In practice, the answer is both. This means findings should rarely be treated as permanent truths. What users struggle with today may not be what they struggle with six months from now. Their experience of working with AI increasingly forms part of the context within which usability evidence needs to be interpreted.

Applying user-centred design principles to an AI-enabled future

Reflecting on this programme of AI usability testing, I found that some of the most valuable insights came not from measuring the AI’s capabilities, but from understanding how people adapt their behaviours, expectations and decision-making when AI becomes part of their work.

One reassuring conclusion is that AI does not invalidate the principles that have underpinned successful public sector service design for years. If anything, the uncertainty introduced by AI makes those principles even more important. The UK Government Digital Service (GDS) Standards encourage teams to understand users and improve services through continuous testing and learning. The Artificial Intelligence Playbook for the UK Government builds on this by reminding us that, even in AI-enabled services, people remain responsible for decisions and outcomes.

Our experience suggests that these principles become even more important when introducing generative AI. While interactions may be less predictable, the fundamentals remain the same: engage with users early, understand how technology fits into real-world workflows, and continuously learn from users’ experiences.

Ultimately, successful AI adoption in government will depend not only on what the technology can do, but on how effectively people can work with it.

Reference: Nielsen, J. (2026, August 5). 76 open research questions in AI usability. UX Tigers. https://www.uxtigers.com/post/ai-ux-research