Every product team has this meeting. The funnel is leaking. Signups come in, and somewhere between the home page and the first real action, people quietly leave. The room fills with opinions. "The button should be higher." "Users want more personalisation." "Maybe the paywall is the problem." Everyone is smart. Everyone has a reason. And nobody actually knows.
The way out is not a louder opinion. It is a fair test.
This post walks through a simple experimentation framework: what to test, who gets into the test, how the data gets there, and what to do when the result arrives.
1. The problem: a funnel leaks, and opinions do not fix it
A funnel is just the path a user walks: open the app, browse, pick something, pay or play or order. Every step loses people. That loss is the leak.
The instinct in the meeting is to fix the leak with taste. Someone senior likes a layout, someone read a case study, someone has a feeling. Taste is not useless, but it has a structural problem: it cannot tell you what your users will do. It can only tell you what the people in the room would do. The people in the room may use the product very differently from the people they are building for.
So the first shift is cultural. The question stops being "what do we think is better?" and becomes "what would users actually do, and how would we see it?"
An entertainment example: the next episode
For a streaming or episodic-story product, the journey might be: choose a title, start an episode, finish it, then decide whether to continue. Each step has its own small funnel. A team could test whether a clearer next-episode preview improves continuation, while watching errors, viewing quality and longer-term retention.
Some episodic products charge to unlock the next episode. In that model, an unlock screen is another step to test, with clear pricing and guardrails such as refunds and complaints. This is an illustrative model, not a claim about how Netflix charges or runs experiments.
2. What an experiment actually is
Strip the jargon away and an experiment is a fair comparison.
You take a small slice of users and show them a new version (the variant). Another slice keeps the old version (the control). Both run at the same time, users are randomly assigned, and you compare one primary metric you chose in advance: click to purchase, play-start rate, order completion, whatever maps to the leak you are fixing.
Every real experiment starts with a hypothesis, and the hypothesis has a shape:
For [this slice of users], changing [X] will improve [metric], because [something you believe about the user].
That last part is the aha most people miss. The "because" is the whole point. Anyone can move a button. The reason you think it will work is a belief about your user, and the experiment adds evidence for or against that belief. Both outcomes teach you something. That is why teams say the goal is not to win experiments. The goal is to understand users.
Two disciplines keep the comparison useful. First, agree on the sample size, duration and analysis method before launch. You can monitor safety throughout, but repeatedly checking a conventional significance test and stopping the moment it looks good increases the chance of a false positive. A valid sequential method is different: it is designed for repeated checks.
Second, allow for weekly habits and novelty. A short-lived rise in clicks may reflect curiosity, not lasting value. Keep assignment stable so the same user does not bounce between versions. Before reading a result, check that exposure events, metric events and the control/variant split look right. A broken test cannot give you a reliable product decision.
An inconclusive result is still inconclusive. It does not prove the two versions are the same.
3. How a user gets into an experiment
Before any comparison can start, the product has to answer one question for every user: are you in this experiment or not?
There is more than one way that happens. The most common: the moment the app launches, it calls a backend API. That API runs a set of checks, looks at the experiments currently running, and decides whether this user is eligible for any of them. The app gets back an answer - experiment A, variant B - and shows the right version from the first screen.
The other common way is lazier, in a good sense: nothing is decided at launch. Only when the user reaches a certain module or a certain page does the experiment trigger. A payments experiment comes alive when the user opens the payments screen, not before. Users who never go there simply never enter the test.
There are multiple ways to trigger an experiment, but underneath all of them sits the same machinery: an experiment framework. It holds the list of running experiments, the audience rules for each, and the assignment logic that keeps a user in the same group every time. For a PM, the mental model is three boxes: trigger, eligibility check, assignment.
In practice that means any screen in the app can attach to the framework - home, search, payments, profile. The dashed block is the part of the screen an experiment owns. Each screen asks the framework before it renders: which version should this user see?
Getting started does not mean building this framework yourself. If you want to release to the market fast, use a third-party experiment tool: Statsig, PostHog and GrowthBook handle flagging, targeting and analysis out of the box, and a team can run its first experiment within days. If you want full control, build it in-house. That control comes at a cost: real development time to set it up, build it and stabilize it before the first experiment ever runs. Most teams start with a tool and build in-house only when they outgrow it. What no tool gives you is the discipline in the rest of this post. That part is still on the PM.
4. The two kinds of parameters
Eligibility checks run on parameters, and there are two kinds worth knowing. The difference is when they are computed.
Real-time parameters come straight from the user's mobile or frontline device, at the moment of the check. Device ID. Location. Which app, when the company runs more than one. Which outlet or store. App version, OS, network, time of day. These answer: what is true about this user right now?
Asynchronous parameters are computed later, from history. User activity is collected in the data lake, and jobs run over it to put labels on users. Spending activity: how much they spend and how often. Listening activity: what they listen to, including hook points - what the user is hooked to, which genres, which artists. Scrolling activity: how far and how fast they move through a feed. These answer: what kind of user is this?
Here is the aha: real-time parameters catch the moment, asynchronous parameters catch the person. "Show this variant to users on the checkout screen in this city" is a moment. "Show it to users whose spending label says price-sensitive" is a person. Most real eligibility rules mix both. The boundary is not absolute - a device property may be cached, a label may refresh often - but the model holds.
Two habits keep the slice honest. Define the audience before the treatment starts, not after you have seen the data. And do not let behavior caused by the variant change who enters the comparison, or you are no longer comparing like with like. A broad experiment with no parameters at all can also be the right choice.
5. Rolling out: the variant earns its traffic
You almost never ship a variant to everyone at once. An illustrative rollout might start at one percent of eligible users, then move to five and twenty-five percent before wider release. The right steps depend on risk and traffic. A safety ramp is not the same as declaring a statistical win at every step. The variant earns the right to more traffic.
6. Guardrail metrics: a win that hurts something else is not a win
Say the variant wins. The primary metric is up. Before anyone celebrates, you check the guardrails.
Guardrail metrics exist because a win in one place can quietly be a loss somewhere else. More clicks, but the page got slower. More plays started, but more refunds a week later. More orders, but more cancellations and support tickets. The classic guardrails are latency and error rate, retention, uninstalls, complaints, and revenue per user.
The distinction is simple: A primary metric tells you whether the variant did the thing. Guardrails tell you whether the thing was worth doing. A team that only watches the primary metric will happily ship its way into a worse product, one "win" at a time.
7. Where the data goes
This part sounds technical, but the shape of it is simple.
Every event the app emits - a screen view, a play, an order - flows in real time into a real-time database, something like ScyllaDB or Postgres. That is what anything fresh reads from: live counters, current sessions, the experiment dashboards that cannot wait.
Behind the scenes, a cron job picks up the same events and lands them in Databricks, typically with about a one-hour delay. That is it. Real-time databases for now, Databricks for depth.
On top of both sit the tools a product team actually touches: interactive internal dashboards over Databricks for deep dives into behavior, and dashboards over the real-time databases for what is happening this minute. This is how a PM figures out user behavior and activity without filing a data request and waiting a week.
8. Hooks: nudging users toward the funnel
Sometimes an experiment needs help getting users to the step you are testing. That is what hooks are for.
A hook is a nudge attached to the experiment: an email, a pop-up inside the app when the phone opens, or a push notification at a chosen time. Testing a new checkout? A hook can bring users who abandoned a cart back to it. Testing a new listening feature? A hook can notify users at the hour they usually listen. Hooks get the right users to the right funnel step, and they let you observe behavior you would otherwise wait weeks to see.
The same power makes hooks easy to abuse. Every hook interrupts someone. Too many pop-ups and notifications and you are no longer running an experiment - you are troubling the user. So hooks come with manners: cap how often a user sees them, make dismissing them easy and permanent, and only nudge when the nudge fits what the user was already doing. A hook the user hates also ruins the data: you end up measuring irritation, not the change.
9. Reading the result: ship, kill or iterate
Once the test is valid and the evidence is sufficient, there are three useful product decisions. If the test is broken or underpowered, fix it or gather more evidence first.
Ship. The evidence supports a useful improvement, the guardrails are acceptable, and a staged rollout continues. Statistical significance alone is not enough: the improvement must be worth the cost and risk.
Kill. The variant is clearly worse, and you roll it back. Teams new to experimentation treat a kill as a failure. It is the opposite: a kill is a cheap lesson. You limited the exposure while testing a belief, rather than sending the change to everyone.
Iterate. The result suggests a better next question. Perhaps the change was too small, or a preplanned segment behaved differently. An unexpected subgroup result is a lead to test, not a win to announce. Now the "because" in your hypothesis earns its keep: you ask why, adjust the variant or the targeting, and run it again.
Whatever the call, write down the learning. Not the metric movement - the belief that was confirmed or killed. "In this test, the eligible repeat buyers responded better to a delivery-time promise than to the discount message" is a useful, bounded learning. "Variant B won by a bit" is trivia.
10. What this actually teaches you about users
Here is the aha that ties the whole framework together: an experiment shows what people do under a controlled change. It does not, by itself, explain why. Interviews, usability tests and support conversations help fill that gap.
Run this loop for a year and something compounds. The decision log becomes a map of your users. You learn which moments they act in and which they ignore, which promises they believe and which they scroll past, what annoys them enough to leave and what delights them enough to stay. That map is the real output of an experimentation culture. Features ship and get replaced. The map stays.
The goal was never the metric. The goal is to understand your users better, with the data you need and one fair comparison at a time.
A template for the next test
If you take one thing from this post, take this. Before your next experiment, fill in every line:
- Hypothesis: for [slice], changing [X] improves [metric], because [belief about the user].
- Trigger and allocation: where the experiment comes alive - at app launch, or on a module or page - and how users get assigned.
- Primary metric: one number, chosen in advance.
- Guardrails: the metrics that would catch a win that hurts something else.
- Slice: who sees it, defined by real-time parameters (the moment) and asynchronous parameters (the person).
- Hooks, if any: which nudges will bring users to the step, and how you will keep them from becoming spam.
- Ramp plan: start small, earn more traffic.
- Duration and decision rule: set before launch, no early peeking.
- Learning note: the belief confirmed or killed, written down whether you ship, kill or iterate.
Fill it in before launch, then return to it when the result arrives. The tooling is the easy part. The discipline is the product.