The “AI-Free” Test
5 minute read
Originally posted on Substack.
If you say “we need AI to [fix something]” you haven’t defined the problem well enough yet.
Recently I was told by a product leader at a major tech company that their team had been tasked with: “Use Generative AI to make the user experience better.”
“What should I do?” they asked me.
That’s not a problem statement. That’s a technology mandate in search of a problem.
Why This Is Backwards
When you start with “we need AI,” you’ve already limited your thinking to what AI can do, rather than what your users need. It’s like telling a chef, “Use a blender to improve dinner,” before considering what dinner is going to be.
If the user experience has real pain points, the first question should be: What exactly is wrong, and how will we know when it’s better? Is onboarding too complex? Is navigation unclear? Are key tasks buried under too many clicks?
Without that clarity, AI is just a shiny layer over an unchanged core experience.
The AI-Free Test
Here’s how I’d reframe that mandate:
- State the user problem in plain English.
- Don’t mention AI, machine learning, models, or anything about the tech solution.
If you can’t do that, you’re not ready.
Example:
- “Use Gen AI to make the user experience better.”
Why? Why is the user experience bad?
“Because its confusing.”
How do you know it is confusing? What parts are confusing?
“Users are not finishing their first deployment.”
Aha! That seems problematic. Where do they give up?
“A lot of the time it’s during the deployment configuration step.”
Define ‘a lot of the time’. How often?
- “New users fail to complete their first deployment 35% of the time, where they give up before completing the configuration step.”
We started with an expensive hope. Now we have something we can measure against. You can continue this exercise by asking about the deployment steps or asking about “why new users only.”
Would an AI assistant help here? Maybe… But we managed to get to a better defined issue by removing the word “AI” from our vocabulary.
I’d wager you can solve a lot of UI problems without using AI at all by applying good user-flow principles and crispy navigation. This particular issue may be cleaned up with a better flow wizard, or perhaps less configuration pieces. Maybe some documentation/guide to help out when someone is stuck.
And yes, maybe you could fix this with a helpful AI assistant…maybe…but I think you can explore half a dozen other solves before you end up there.
How to Get There
- Talk to real users: Watch them click. Watch them rage-click. Watch them vanish into the void. Then ask why.
- Use Tools to Monitor Experience: Tracking your users through the platform can be as effective as looking over their shoulder. Find a tool you like and use it!
- Quantify pain points: Error rates, abandonment rates, time-on-task. Numbers that matter to this specific experience. Avoid the red herrings.
- Start tech-agnostic: Decide on the right fix after the problem is clear.
Examples of Tools I Like
I’ve seen a lot of success in tracking realtime-user monitoring events (RUM) using Datadog’s Digital Experience module. I visit my dashboard nearly daily to see what my users are doing. I often watch a lot of replay video to see how they navigate. The best way to learn is to relive their experience.
Survey tools are boundless. I prefer MSFT Forms because I work at a MSFT-partner company. But you can’t really go wrong with any of them.
Figma’s Figjam is my favorite tool for brainstorming and mapping problem spaces. Collaborating with my fellow PMs is a must.
If you need AI systems, I like the Databricks agent tool stack. See? We got to AI eventually, and it was for a good reason. That’s partially because I really like Databricks, and also because I’m a stan for the lakehouse paradigm for data and analytics. I’m planning to come out with a post on how we’ve used it for some neat use cases, so stay tuned.
Final Thought
Generative AI might very well improve this product team’s user experience. But starting with “make it better with AI” is like picking the paint color before you’ve designed the house.
Before your next project kickoff, try this: Write the problem statement without using the word “AI” once. If you can’t, you’re not solving a problem. You’re shopping for a hammer and hoping a nail shows up. Figure out the problem first. Then choose the right tool for the job.
