Discovery: When We Replaced Users With Dashboards

The sixty-year journey from clipboard surveys to AI-assisted interviews, and what got lost along the way.

May 29, 2026

Brought to you by ExecReps.ai — AI-powered executive presence coaching for teams.

If you are not already part of our community, subscribe below for more content like this, delivered to your inbox every week.

There is a question I keep turning over in my head. When did product managers stop talking to people and start talking to dashboards instead?

I do not mean this as an accusation. I mean it as a genuine puzzle. Because somewhere along the line, something shifted. The industry went from "go talk to your customers" to "check your Amplitude dashboard," and nobody quite noticed the switch happening.

This issue draws on the work of thirteen practitioners who wrote about product discovery in Product Coalition. Not consultants selling frameworks. Not influencers optimising for reach. Working product people, from Tel Aviv to Madrid to Melbourne, sharing what they learned in the messy middle of building products. That is the kind of writing that changes how you think.

The story of how discovery evolved is also the story of how product management grew up. And like most growing-up stories, it involves a few wrong turns.

Era 1: The Requirements Document (1990s-2000s)

If you were building software in the late 1990s, "discovery" meant something very specific. It meant writing requirements. Lots of them. In enormous documents that nobody fully read.

Noa Ganot captured this era perfectly in Let's Say "Discovery": Why "Validation" Is the Wrong Word. She recalled her days as a system architect at the Israeli Air Force, where the prevailing belief was that if you planned well enough, nothing would need to change. "The idea that if only you planned well enough your code wouldn't need to change seems so naive today," she wrote. But back then, it was simply how things worked.

The assumption was simple. Gather every requirement upfront. Write them all down. Hand the document to engineering. Wait. Get exactly what you asked for. Ship.

Except it never worked that way. John Utz, writing in Start Without the End in Mind, described the fundamental flaw: "I wasn't solving their problem; I was solving my own." He had built a feature he was sure users would love, only to watch engagement flatline. The problem was not the feature. The problem was that he never asked anyone if they wanted it.

This was the era of the 200-page specification. Of waterfall timelines that stretched across years. Of "requirements gathering" as a distinct phase that happened once and was never revisited.

And the thing I keep coming back to is how confident everyone was. Nobody thought they were guessing. They thought they were planning.

Era 2: Get Out of the Building (2008-2015)

Then Steve Blank said four words that changed everything: "Get out of the building."

The Lean Startup movement landed like a bomb in product teams that had spent years arguing over requirements documents. Customer Development said something radical for the time, that you could not know what customers wanted by sitting in a conference room debating it. You had to go ask them.

Scott Middleton, in his Product Discovery Playbook, described the core of this shift: "Product Discovery is an exercise in working out whether there are customers that want the product (or feature) you're working on and that you can deliver a solution to them." It sounds obvious now. In 2010, it was a revelation.

This era introduced ideas that are now so embedded in product practice that we forget they were once controversial. Minimum viable products. Hypothesis testing. Pivots. The idea that building the wrong thing was worse than building nothing at all.

Scott Middleton also wrote about an underappreciated side effect in Discovery for Non-Product People. As product teams embraced discovery, a gap opened up with the rest of the organisation. He shared an actual quote from a General Manager: "We gave an outstanding idea to the Lab, but they just told us our idea was no good and we had to go do more work." Discovery created power. It also created friction.

Meanwhile, Fitra Akbar was documenting the craft of the customer interview in How to Run a Customer Interview Correctly. The rules were simple but counter-intuitive: don't talk about your solution, don't share your own opinions, create a comfortable environment. The interview was supposed to be about them, not you.

The Lean Startup era gave product managers permission to be wrong. That was its gift. Its limitation was that discovery still felt like a project with a beginning and an end. You "did discovery," then you "did building." The two tracks ran separately.

Era 3: Continuous Discovery (2015-2022)

Teresa Torres changed the game again. Her book, Continuous Discovery Habits, introduced the idea that discovery was not a phase. It was a practice. Something you did every single week, forever.

Austin Nichols, in Five Lessons on Continuous Discovery, quoted her definition directly: "Weekly touchpoints with customers, by the team building the product, where they conduct small research activities, in pursuit of a desired outcome." That was it. No six-week research sprint. No discovery "phase." Just a steady habit of staying close to the people you are building for.

Nichols also flagged something important: "The antithesis of modern Product is mindlessly building a list of features. It's an anti-pattern to rarely talk to your customers." The bar was no longer "have you done any research?" The bar was "are you doing research this week?"

Anthony Rousounelos built on this in A Continuous Product Discovery Framework for Agile Teams, pointing out the irony that most product teams lived in: "product teams are agile in their implementation efforts while lacking an agile approach in the whole feature prioritization process." Teams were using Scrum to build things fast. They were not using anything resembling agility to decide what to build.

Ant Murphy, in How to Kickoff Product Discovery Like a Pro, addressed the practical question that came up most often when he coached PMs: "I know what I want to do discovery on, but how do I get started?" His answer was Assumptions Mapping and Experiment Boards. Simple tools. Low ceremony. The kind of thing you could do in an afternoon, not a quarter.

And Jerel Lee, in Designers, Forget Lengthy Discovery Activities, Start With Jobs-To-Be-Done Instead, pushed back on discovery becoming its own bureaucracy. His argument was practical: "context matters, in the private sector, business needs dictate quicker turnarounds and an expedient go-to-market approach." Jobs-To-Be-Done, he argued, could compress weeks of discovery into something far tighter.

The continuous discovery era was a genuine breakthrough. But it also created a new failure mode.

Era 4: The Dashboard Trap (2020-Present)

Here is where things get uncomfortable.

Somewhere around 2020, product analytics tools got very, very good. Amplitude. Mixpanel. Pendo. FullStory. Hotjar. Suddenly you could watch exactly what users did without ever talking to them. You could run A/B tests, track funnels, measure retention cohorts, all from the comfort of your laptop.

And slowly, quietly, discovery started to become a dashboard exercise.

I wonder if this is the biggest risk product management faces right now. Not that teams skip discovery entirely. That would be too obvious. The risk is that teams believe they are doing discovery because they look at data. They check metrics. They review dashboards. But they have stopped having conversations.

Michael Goitein captured the extreme version of this in Let's Rethink How We Build Products. He described a leader who dismissed his team's user research with: "I don't care what some random customers told you. You're going to build what I tell you to build." The irony, as Goitein pointed out, was devastating: "Without continuous product discovery, we never would have come to this leader's attention." The very practice that made the team successful was the first thing to get squashed.

Keren Koshman sounded a different alarm in The Balance Between Product Discovery and Delivery. Her concern was that "in the zeal to innovate and strategize, the pendulum has swung too far toward the discovery end of the spectrum." Some teams were doing so much research that they had forgotten to ship anything.

And Afonso Franco wrote what might be the most honest diagnosis of all in How to Escape the "Analysis Paralysis" Trap in Product Discovery: "Endless refinement of your idea. Completely lost in customer insights. Drawn in data. And not much progress apart from more and more user interviews. In essence, you're stuck." His list of causes reads like a mirror: perfectionism, fear, misunderstanding what good discovery looks like, and something he called "discovery fanaticism," where a PM "sees her/his job as to 'do discovery and design', usually following a certain framework religiously."

Maret Kruve brought mathematical rigour to this tension in The Discovery Dilemma: When to Just Ship It. Her framing was brilliant: "Think of discovery as buying risk insurance: you pay a premium to reduce the probability of a larger potential loss." And then the punchline: "Insurance is worth the price only when the potential loss is high compared to the premium. It makes little sense when the potential loss is trivial. You insure your house, not your stapler."

That last line should be tattooed on every product team's wall.

Era 5: The AI Discovery Frontier (2024-Present)

And now we are here. In the era of synthetic users and AI-assisted research.

Tools are emerging that promise to simulate customer interviews. AI can summarise transcripts, cluster feedback, generate personas, even suggest opportunity spaces. The question is no longer whether AI can help with discovery. It can. The question is whether it can replace the part that matters most.

And the part that matters most has never been the data. It has been the surprise.

John Utz put it best: "Great product strategists fall in love with the user's problem, not the solution." You cannot fall in love with a problem you have never seen up close. You cannot be surprised by a dashboard. You can only be surprised by a person saying something you did not expect.

Noa Ganot's argument about language matters here too. She argued that "validation" was the wrong word because it implied you already had the answer and were just checking it. "Discovery" was better because it implied you were genuinely looking for something you did not already know. As AI makes it easier than ever to "validate" at scale, the danger is that we lose the genuine looking-for-something-new part entirely.

The thing I keep coming back to is this: every era of discovery has been a reaction to the previous era's failure mode. Requirements documents failed because they assumed you could know everything upfront. Customer Development failed because it was a one-off phase. Continuous Discovery risks failing because it can become performative. And AI-assisted discovery risks failing because it might give us answers to questions we should not have been asking in the first place.

So here is my open question for you. If you replaced every customer conversation with an AI simulation, and the simulation gave you the same insights, would you have learned the same thing? Or would you have missed the thing that only shows up in the awkward pause, the offhand comment, the thing the customer says after you have already stopped recording?

What I Am Hearing on the Podcast

Three recent podcast conversations keep circling back in my thinking around this topic.

Dave West, CEO of Scrum.org, came on EP 74: Embracing the Product Mindset and described a problem that sits at the root of the dashboard trap. He talked about teams whose "connection to the users, and to the stakeholders, and to the problem domain is tenuous at best." These teams work on something, then work on something else, then something else again. They never stay close enough to any one set of users to develop real understanding.

Dave also painted the picture of what good looks like, drawing from startups: "continuously innovating, everybody's motivated, everybody connects to the customer, everybody's flexible because you have to be." That is the environment where discovery happens on its own, not because someone scheduled a research sprint, but because everyone is close enough to the problem that they cannot avoid learning.

And when asked about what matters most in complex product work, Dave cut right to the core: "if it's a complex problem, are you incrementally delivering, learning against that problem, that's defining your understanding of it and delivering more value?" That is discovery in one sentence. Not a phase. Not a dashboard. Learning against the problem.

Then Dan Balcauski, founder of Product Tranquility and a program leader at Northwestern's Kellogg School of Management, came on EP 83: Profit by Design and said something that landed hard. Talking about how companies understand the value they create, Dan was blunt: "the value is not in the feature. The value is in the mind of the customer."

Dan went further, pushing back on the idea that you can understand customers from data alone: "understanding customer value, you can't just do in a room with a couple of spreadsheets. You have to go talk to customers." He described running qualitative interviews before any pricing work, asking not about numbers but about context, what customers were trying before, what caused them to look for a new solution, why alternatives fell short. Discovery, it turns out, is not just a product management practice. It is how any function that touches customers should be thinking.

And Ana Bello-Elliott, founder of Bello Insights Lab, came on EP 94: Do We Really Need Research for This? and brought something I had not heard anyone frame so clearly before. Ana started her career in cybersecurity, and she drew a direct line between risk documentation and product discovery: "In the cybersecurity world, risks have to be documented. They have to be acknowledged... I think we need something similar in our discipline, where we are at least having the discussion of what is the risk that we're taking on by not taking an action."

That reframe hit hard. Discovery is not a nice-to-have. It is risk management. And Ana had the four questions to back it up: Do we know what customer needs we are solving for? How do we know, what is the evidence? Is this impacting more than one customer? And what is the risk if we move forward without research?

Ana also gave the most honest assessment I have heard of AI-assisted research. She described using an AI moderator for a recent study: "There were things that the tool was not very good at, like my user was sharing their screen. It could not interpret what was happening on the screen." A participant thought they were sharing their screen but were not, and the AI had no way to notice. A human moderator would have caught it immediately. But then the flip side: "If we need 100 interviews overnight, you can get 100 interviews overnight." The power is real. The gaps are real too.

And then the line that ties it all together: "Research is, at the end of the day, strategic. And something that we often forget because UX and UX research lives within the little box of UX design. But at the end of the day, research is strategy. It feeds into your strategy." That is the argument for discovery that cuts through every era of this newsletter. Not a phase. Not a checkbox. Strategy.

What these thirteen writers and three podcast guests are doing is the work that Product Coalition exists for. They are not waiting for someone to hand them a framework. They are building in the open, sharing what they learn, and being honest about what does not work.

Ant Murphy is building Assumptions Maps with his teams. Maret Kruve is doing the maths on when discovery is and is not worth the investment. Afonso Franco is naming the failure modes that nobody else wants to talk about. Michael Goitein is pushing back against leaders who would rather dictate than listen. Austin Nichols is running small experiments every week. Noa Ganot is questioning the very words we use to describe what we do. And Ana Bello-Elliott is treating discovery as the risk management discipline it always should have been.

These are the people doing the work. Not from stages. From desks and calls and sprint reviews and awkward customer interviews. This is what open practice looks like.

Discovery has been many things over the decades. A requirements document nobody read. A post-it wall in a WeWork. A Dovetail dashboard with 400 tags and zero conviction. Each era tried to solve the same problem: how do you build something people actually want when you are not the person who will use it?

The answer has never changed. You talk to them. You watch them. You sit in the discomfort of being wrong. The tools will keep evolving. AI will get better at summarising transcripts and worse at noticing that someone is not actually sharing their screen. But the discipline underneath, the willingness to be surprised, that part is not automatable. It never was.

The best discovery teams I see right now are not choosing between human and machine. They are using AI to move faster and humans to move deeper. That is not a compromise. That is the frontier.

👋 Jay

If you are not already part of our community, subscribe below for more content like this, delivered to your inbox every week.

Sources

Discussion about this post

Ready for more?