Can one model memorize and generalize better than two?

The recommender ranking problem demands two opposing capabilities. Memorization — learning frequent feature co-occurrences — fits cross-product transformations on sparse features (e.g. "user installed Netflix AND impression is Pandora"). Generalization — predicting on never-seen feature combinations — fits dense embeddings that map sparse features into continuous space. Cross-products fail to generalize across unseen pairs; dense embeddings over-generalize and produce nonzero predictions for niche items the user shouldn't see.

The Wide & Deep insight is that you can put both halves into one model and train them jointly rather than ensembling separately-trained models. The technical distinction matters: in an ensemble, each component is optimized standalone and combined at inference, which means each must be full-size to perform reasonably alone. In joint training, the wide part is optimized knowing the deep part exists, so it only needs to complement the deep model's weaknesses with a small number of cross-product features. Gradients from the output flow into both branches simultaneously via mini-batch SGD.

The result is a model where memorization and generalization are not two competing strategies forced into a weighted average but two specialized roles in one network. The wide part becomes small and surgical because the deep part handles the common cases; the deep part doesn't have to learn rare exceptions because the wide part captures them.

Source: Recommenders Architectures