Personalization metrics and a cohort experiment

Context & problem

The Wall Street Journal and Barron’s are part of Dow Jones’s consumer subscription products, which reach more than 10 million monthly users. During my time at Dow Jones, an ML personalization system chose which articles appeared in their home and app feeds, and personalized the newsletters, push notifications and subscription offers readers received. My part was its success metrics and its A/B testing.

The product question was what good meant for a subscriber and for the business, and how to measure it so the answer would hold up.

I was a Senior Product Manager. I worked in a 25-person cross-functional group and managed about two PMs. I owned the success metrics and the A/B testing. The model inputs, the ranking and the models themselves were owned by other teams. This page is about the measurement side, not the modelling side.

The four surfaces matter because they are different promises. A feed is something you scroll; a newsletter is something you open; a push is something that interrupts you; an offer is something you pay for. One personalization system served all four, and a change that helps one can cost another.

That is the problem measurement solves. A personalization system makes a choice every time a reader opens a feed or an email. Without success metrics agreed in advance and tests that separate a real change from noise, it is easy to ship something because it looks better, or to call a win on a number nobody agreed to beforehand.

Discovery & insight

Alongside the program, a cohort analysis on acquired subscribers led to a targeted experience for them, and I led the experiment that tested it.

The habit behind it is one I keep: an average across every subscriber hides the groups that behave differently, and a subscription business can’t afford to miss one of those.

HypothesisBy changing the existing experience to a targeted experience for acquired subscribers, we expect retention to move, because a cohort analysis on acquired subscribers led to a targeted experience for them.

Strategy & bets

My part was the success metrics and the A/B testing, so the bets I can speak to are about measurement, not modelling.

Two numbers, not one score. Retention and engagement are the two results on record for the program. I keep them as separate results rather than blending them into one score, because a personalization system can raise reading without changing whether someone renews, and the business needs to know which of the two moved.

Decide what counts before the test runs. A/B testing is how a model change is told apart from noise. I want the success metric and the comparison agreed before a test starts, so a result can’t be argued into a win afterwards.

Read segments, not only the average. The cohort experiment is this bet in practice: a segment read on its own, a change built for it, and one metric that decides whether it worked.

Know the cost of optimizing too well. Alongside the headline metrics, what I watch for in news personalization is a feed that narrows a reader into one topic, or one that stops carrying the stories the newsroom considers essential. A lift that comes from either is not one I would want to ship.

Solution

There are no screenshots for this work by design. The output was success metrics and A/B tests, not a screen.

The program was an ML system choosing articles for the WSJ and Barron’s home and app feeds and personalizing newsletters, push notifications and subscription offers. My part was its success metrics and its A/B testing.

Separately, the cohort experiment: the segment was acquired subscribers, the change was a targeted experience, the metric was retention.

Execution

I owned the success metrics and the A/B testing, and I led the cohort experiment on acquired subscribers.

I worked in a 25-person cross-functional group and managed about two PMs. Model inputs, ranking and the models were owned by other teams, so my leverage was in the definitions and the tests: what counts as a win, and how we would know.

Validation & results

Retention +28% and engagement +38% are the results on record for the personalization program. They are program results, so I don’t attribute them to a single surface or a single model change.

Retention +40% is the result in the acquired-subscriber cohorts, from the experiment I led. The two results answer different questions, one about the personalization program and one about a single segment, and I keep them apart for that reason.

Reflection

Owning the metrics for a system other teams built taught me where a PM’s leverage sits in ML work: in the definitions, the tests and the segments, not in the model. The habit I kept is to read the important segments on their own before trusting an average. If I owned the metrics for a program like this again, I would write each metric’s definition, window and comparison into the same document as its result, so the number and its method always travel together.