Many claim that Transformers are ill-suited for recommendation systems because of the heterogeneous nature of features like user IDs, item IDs, and categories. However, this post argues that such features can be converted into embeddings of the same dimension, making them compatible with Transformer input. The real challenge, it suggests, lies in other architectural aspects, such as handling sparse interactions and long-term user behavior sequences. This perspective is crucial for ML engineers who are exploring Transformer-based recommenders. It shifts the focus from feature engineering to model design, encouraging a reevaluation of how attention mechanisms can be adapted for sequential recommendation tasks. Understanding this can lead to more effective and scalable recommendation models.
A common belief is that Transformers fail in recommendations due to heterogeneous features. This post argues otherwise, sparking a deeper architectural discussion.