Can one model handle both memorization and generalization?

A recommender has two opposed needs. Memorization captures frequent co-occurrences that historical data demonstrates — if users who installed Netflix tend to install Pandora, that conjunction should fire. Cross-product transformations on sparse features memorize beautifully but don't generalize to unseen feature pairs. Generalization, on the other hand, comes from dense embeddings that smooth across feature combinations, but those over-smooth on sparse high-rank query-item matrices, predicting positive interactions where there should be none.

Wide & Deep's claim is that you don't have to choose. The wide tower carries cross-product features for memorization; the deep tower carries embedding-based MLPs for generalization. The mechanically important detail is that they're trained jointly, not as an ensemble. In an ensemble, each model has to be reasonably accurate alone, so each must be larger. In joint training, the gradients flow through both at once: the wide part only needs to add cross-product features that complement what the deep part missed. This means the wide part is a small set of carefully chosen feature crosses rather than a full linear model.

The architecture implicitly trades off two distinct failure modes — overgeneralization on sparse data (deep alone) and inability to extend to new combinations (wide alone) — by letting each fix the other's failure mode under shared supervision.

Source: Recommenders Architectures