Machine Learning
Netflix Recommendation Engine
How a catalogue of thousands becomes a homepage of dozens, personalised for every viewer without anyone browsing a menu.
How a catalogue of thousands becomes a homepage of dozens, personalised for every viewer without anyone browsing a menu.
A streaming catalogue holds far more titles than anyone could browse. The homepage you see is not a catalogue at all — it is a constructed page, assembled for you, where even the order of the rows is a prediction.
The system cannot score every title for every user on every visit; that is far too much work for the hundred milliseconds it has. So it splits the problem in two: cheaply narrow thousands of candidates down to a few hundred, then spend real computation ranking only those.
Almost everything it learns from is implicit. Very few people rate anything, but everybody reveals preference by what they start, what they finish, what they abandon after four minutes, and what they scroll straight past.
Think of it like this: it is a shop that rebuilds its window display for each customer as they walk in — using what people like them bought, what they lingered over last time, and what they pointedly ignored.
Personalisation at this scale runs into several problems simultaneously:
Scoring every title for every user on every page load is computationally impossible within the latency budget.
Any individual user has interacted with a tiny fraction of the catalogue, leaving mostly blanks to reason about.
New users have no history and new titles have no audience, yet both need sensible treatment immediately.
Recommending something makes it more likely to be watched, which then reinforces the recommendation regardless of whether it was good.
Taste shifts, and what someone watched an hour ago should influence what they are shown now.
The pipeline separates cheap breadth from expensive precision: candidate generation casts a wide net quickly, ranking applies heavy models to a small set, and assembly turns ranked items into an actual page.
Every play, pause, abandon, search and scroll is recorded as an event, forming the raw material for every model downstream.
In shortEverything you do is written down, especially the things you did not finish.
Users and titles are each represented as vectors in a shared space, learned so that a user sits near the titles they are likely to enjoy.
In shortPeople and shows are placed on the same giant map, and closeness means likely interest.
Several cheap retrieval strategies each nominate a few hundred plausible titles — nearest neighbours in embedding space, trending items, continue-watching, and similar-to-recent.
In shortA few quick shortlists are drawn up by different methods, no deep thinking yet.
A heavier model scores each candidate using rich features about the user, the title, the context and their interaction, producing a predicted engagement score.
In shortNow the shortlist gets properly assessed, one by one, with everything the system knows.
Ranked items are grouped into themed rows, deduplicated, diversified and ordered, since a page is more than a flat list.
In shortThe winners are arranged into rows with titles, so it reads as a page rather than a leaderboard.
What you actually do with the page becomes training data, closing the loop and continuously updating the models.
In shortHow you react to the page teaches it what to do next time.
The system gathers the viewer's profile, recent activity, device, time of day and anything left partly watched.
In shortIt picks up who you are, what you were doing, and where you left off.
Multiple retrieval sources each propose a few hundred titles, which are merged into one pool of a few thousand at most.
In shortSeveral shortlists are drawn up and pooled together.
Already-watched titles, regionally unavailable content and anything explicitly dismissed are removed before any scoring happens.
In shortAnything you cannot watch or already have is dropped straight away.
The ranking model scores every remaining candidate for predicted engagement, given this user in this context.
In shortEach survivor gets a score for how likely you are to actually watch it now.
Near-duplicates are suppressed and titles are organised into coherent rows, balancing confident picks with some variety.
In shortIt avoids showing you ten near-identical things and sorts the rest into themed rows.
Impressions, clicks, plays and abandons are recorded against exactly what was shown, enabling later evaluation and retraining.
In shortIt writes down what it offered and what you did about it.
Explicit ratings are rare, unreliable and often aspirational. The useful signal is almost entirely behavioural.
Finishing a title is strong positive evidence. Abandoning after a few minutes is strong negative evidence, and far more common than any rating.
In shortWatching it all the way through says yes. Bailing out after five minutes says no, loudly.
Repeatedly showing someone a title they never select is meaningful negative feedback, even though nothing was clicked.
In shortIgnoring the same thumbnail ten times tells the system plenty.
Time of day, device and session length matter — a thirty-minute comedy on a phone at lunchtime is a different proposition from a film on a television at night.
In shortWhat you want on a phone at lunch is not what you want on the sofa at ten.
Recent behaviour is weighted more heavily than old behaviour, so the model tracks shifting taste rather than averaging a lifetime.
In shortWhat you watched last week counts for more than what you watched two years ago.
The obvious design is one model that scores every title for every user. It is conceptually clean and completely impractical:
In short: the trick is refusing to think hard about most of the catalogue. A cheap method throws away ninety-nine percent of the options in milliseconds, which buys the budget to think properly about the remaining one percent.
Collaborative approaches need history, and both new users and new titles have none. This is the single most common failure mode of a naive recommender.
With no history, the system falls back to popularity within the user's region and any onboarding preferences, then adapts rapidly over the first few sessions.
In shortIt starts with what is broadly popular near you, then learns fast from your first few choices.
A title nobody has watched has no collaborative signal, so content-based features — genre, cast, description, visual similarity — carry it until behavioural data accumulates.
In shortA brand-new show is recommended based on what it is like, not who has watched it.
A deliberate fraction of slots is given to uncertain items, because a system that only shows confident picks never learns anything new.
In shortIt gambles on a few unknowns on purpose, otherwise it would never discover anything.
Content-based and collaborative scores are blended with weights that shift toward collaborative as evidence accumulates.
In shortIt leans on descriptions early and on real behaviour later.
The serving path has a hard latency budget, so almost everything expensive is moved offline or precomputed:
| Layer | Common Choices |
|---|---|
| Event Collection | Streaming pipelines capturing impressions, plays and abandons |
| Feature Store | Shared online and offline feature storage to keep training and serving consistent |
| Training | Distributed batch training for embeddings and ranking models |
| Retrieval Index | Approximate nearest neighbour indexes such as FAISS or ScaNN |
| Ranking Service | Low-latency model serving with strict timeout budgets |
| Page Assembly | Row construction, deduplication and diversity rules |
| Experimentation | A/B testing infrastructure with guardrail metrics |
A recommender is really a two-stage funnel with a feedback loop attached — cheap breadth, then expensive precision, then learning from the outcome. The same shape applies to any system that must select a small number of things to put in front of someone from a pool far too large to evaluate exhaustively.
A quick, no-nonsense translation of the technical terms used above.
Recommending based on what similar users liked, rather than on properties of the item itself.
Recommending based on attributes of the item — genre, cast, description — rather than on other users' behaviour.
A list of numbers representing a user or item, arranged so that similar things end up close together.
The cheap first stage that narrows a huge catalogue down to a manageable shortlist.
The expensive second stage that carefully scores and orders the shortlist.
The problem of recommending for a new user or a new item with no interaction history.
Preference inferred from behaviour — watching, abandoning, ignoring — rather than from an explicit rating.
A method for finding items close to a point in embedding space quickly, trading a little accuracy for a lot of speed.