A few weeks ago I met a friend at a cafeteria. He is a product manager, and within ten minutes he was not talking about his roadmap. He was venting.
Sprint review was the next day. Jira said the ticket was "in progress". Slack said the engineer was waiting on a design decision. The dashboard on his screen was from yesterday. And his manager had just asked, on a call, "so where are we on this?"
So he does what every PM does. He chases. He opens five tabs, pings three people, and stitches together an answer that is already stale by the time he says it out loud. He is the human glue between his own tools, and he is tired of it.
I had some free evenings and a lot of curiosity, so instead of just sympathizing, I offered to help him fix it properly. This is the story of what we built, including the parts that broke.
Step 1: We mapped habits before touching any tool
The mistake most people make with AI is starting with the tool. We started with a question: how does a status question actually get answered today?
So we wrote it down, step by step, for each of his teams. Where does engineering status live? Jira, mostly, plus whatever people actually said in Slack. Where do product metrics live? In the data warehouse, in dashboards, and in someone's memory. Where do decisions live? In meetings, and then nowhere.
That map turned out to be the most valuable thing we made. His problem was never "we lack data". The problem was that the answer to any real question was scattered across tools, formats and people's heads, and assembling it was manual labor.
Only after that did we pick up the tools.
Step 2: The system, layer by layer
The core idea: split data by freshness, and never let AI touch production systems directly.
Delayed data (a few hours old). The data lake, queried through Databricks. This is where trends, cohorts, funnels and retention analysis live. Perfect for "how did last week's release land", useless for "is the site on fire right now".
Real-time data. Grafana, for live operational data and health checks. This answers "what is happening right now".
Dashboard layer. QuickSight, for the metrics the business already trusts. If a number is going to be quoted upward, it should match what leadership already sees.
The safety layer. This is the part we were most careful about. The AI never connects to his company's production databases directly. It goes through Redash, which sits in front of the databases with approved queries, with personal data masked out. One line that matters: the AI only sees what a Redash query is allowed to return. Access is read-only.
On top of this we built skills: reusable instruction files that teach the AI each workflow. "Pull this week's activation numbers" is a skill. "Draft the blockers report" is a skill. For each one, we asked the AI to write the connection code to the data source, then generate the reports and graphs on top. He works in Cursor, using MCP servers to connect to tools like Jira and Slack, so the same setup that reads a database can also read where the team actually talks.
Step 3: What we measure
A dashboard is only useful if a product team can read it in ten seconds. These are the metric groups we landed on:
- Acquisition and activation: signups, activation rate (share of new users reaching the first key action), time to first value.
- Engagement: DAU, WAU, MAU, stickiness (DAU/MAU), sessions per user, feature adoption rate.
- Retention: day 1 / 7 / 30 retention by cohort, churn, resurrection rate.
- Funnels: step-by-step conversion and drop-off, with time between steps.
- Quality and reliability: error rate, latency, crash rate, MTTR, support tickets per 1,000 users.
- Business: conversion to paid, revenue per user, and LTV vs CAC where they apply.
- Delivery health: cycle time, blocked tickets and how long they have been blocked, sprint carry-over. This is the group that answers the original "what's the status" question.
The delayed vs real-time split does real work here: retention curves come from the lake, incident signals come from Grafana, and we never confuse the two.
Step 4: The memory layer
Dashboards tell you what happened. They do not tell you what was decided. For that we needed meetings.
So he keeps transcripts of his meetings locally, and only with consent. Everyone in the meeting knows it is being transcribed. That line is non-negotiable, and honestly it made adoption easier, not harder.
The retrieval is almost embarrassingly simple: Claude Code searches the local transcript files with grep. No vector database, no embeddings, no cloud service holding his company's conversations. When he needs "what did we decide about the pricing experiment three weeks ago", grep finds it in seconds.
Step 5: The part we did not expect: other people started using it
Here is the thing about workflows: processes are mostly the same across teams. Once the skills and boilerplates worked for him, they worked for others too.
Other PMs around him started copying the base setup and adapting it to their own teams. Each person tweaks it for their own use case, which is exactly the point. We were not trying to build a product. We were trying to remove one person's friction. But a reusable base that spreads on its own is the clearest signal that the friction was never just his.
Step 6: Where it broke (the honest part)
This is the section most AI blog posts skip, and it is the one that matters.
Connection issues. AI could write the code, but it could not fix connectivity to internal systems. Network access, connection strings, internal addresses: these are real-world problems that no amount of prompt engineering solves. The fix was boring. He went to the people who own the infrastructure, got the right access path, and the problem went away.
Package and dependency errors. Internal packages failed in ways AI could not see, and we would watch it loop, retrying the same fix until we stopped it and went to find a human.
Secrets. You cannot paste credentials into a prompt, and you should not. He went to a developer on his team and asked: how do we do this properly? They walked him through the right process for getting and storing secrets, and that is how we did it.
What actually ate the time was not the big pieces. It was the edge cases, at two levels. At the technical level: the package failures and connection issues above, each one its own small investigation. At his level, the PM level: working out which metrics the team actually needed, then tweaking the code, the queries and the processes until the numbers matched how the team really works. None of it was hard in the heroic sense. It was a lot of attempts, a lot of trials, and choosing the best option each time. Anyone who builds something like this goes through it. What carried us through was time and patience.
The lesson we keep coming back to: AI gets you about 80% of the way. The last 20% (access, secrets, internal infrastructure) still needs a person who knows the system. And that is fine. The win is not replacing people. The win is that he now spends his time with people on the 20% that actually needs judgment, instead of burning their time on questions a script can answer.
What I learned
The bottleneck moved. It used to be "who can write the code". Now AI writes most of the code. The scarce skill is knowing which question to ask, and where the answer lives. That is, not coincidentally, a very product-manager skill.
Leaders run on metrics, so meet them there. His job is to understand the vision his leadership is driving toward and the KPIs they are measured on. If he can show them the right number faster, with context, he earns trust and time. The system is not about dashboards for their own sake. It is about shrinking the distance between a question and a confident answer.
Remove friction first, then add AI. Every gain in this project came from understanding how each team already works before automating anything. AI layered on top of a broken process just produces broken output faster. And budget for the edge cases. They, not the happy path, are where the time goes.
Making an LLM behave is its own discipline. Halfway through, I realized what we were actually doing had a name: harness engineering. An LLM left on its own drifts. It answers a slightly different question than the one you asked, formats things its own way, and slowly wanders off the brief. You stop the drift with a harness. Skill files give it a fixed set of rules and instructions for each workflow. And the guardrails that matter live at the code level, inside the connection code itself, so the important rules do not depend on the model remembering them. The skill files steer it. The code holds it.
What actually improved
- Less friction. The routine "where are we on this?" questions mostly answer themselves now.
- Faster answers. What used to mean chasing across tools and people is now one prompt.
- Fresher data. He always knows how old a number is before he quotes it.
- Better prepared calls. Conversations go straight to blockers and decisions, not status.
- Self-serve metrics. Routine pulls and graphs no longer wait on someone else.
- A memory for decisions. Old meeting details are findable in seconds.
- No trade-off on safety. Access stays read-only and masked, so speed did not cost security.
- A base others can reuse. The setup is not tied to one person or one team.
And one line on what it does not replace: people. Judgment, access approvals, secrets and real decisions still belong to humans. The system just stops wasting their time on questions a script can answer.
What's next: a PM who can ship small fixes
There is one more thing I want to try with the same stack: clearing the small product issues that sit in every team's backlog. The P2s and P3s. Not heavy engineering work. Product-level changes: a copy fix, a filter that should default the other way, a rough edge in a form, the small polish that never becomes anyone's priority.
I have seen how these go today. My friend spots one, writes it up, and asks a developer to pick it up. The developer is busy with bigger things, so the small fix waits. Three days, sometimes four, for a change that itself takes a few hours. He waited because they had the expertise, not because the work was hard.
Now the same setup can connect to GitHub, make the code change, and raise a pull request, with the PM driving it. One thing stays exactly as it is: a developer still reviews and merges the PR. The point is not to cut them out. The point is that they get a finished change to review instead of a ticket to interpret, and a review takes minutes where the whole loop used to take days. Today the human gate is non-negotiable, and I would keep it. Over time, as these loops earn trust, I expect that barrier to shrink too. Because the real goal was never to remove people. It is to change the winning metric: how fast a team can go from spotting an issue to shipping the fix.
If this works even for the smallest category of issues, the "not important enough for this sprint" backlog starts to shrink. That is the next thing I want to test, and I will write about what happens.
The roles are converging. PMs who can build, builders who can think in metrics. But the through-line is not the technology. It is the willingness to go past your job description, sit with the friction another team lives with, and fix it properly.
My friend started out chasing status updates. He ended up with a system. And I ended up learning more about data infrastructure, security processes and team dynamics than any product book ever taught me.
Names and some details changed. Nothing here names a company, a team or an internal system.