Data Structure
Networking
RDBMS
Operating System
Java
MS Excel
iOS
HTML
CSS
Android
Python
C Programming
C++
C#
MongoDB
MySQL
Javascript
PHP
Physics
Chemistry
Biology
Mathematics
English
Economics
Psychology
Social Studies
Fashion Studies
Legal Studies
- Selected Reading
- UPSC IAS Exams Notes
- Developer's Best Practices
- Questions and Answers
- Effective Resume Writing
- HR Interview Questions
- Computer Glossary
- Who is Who
Handling sparsity issues in recommendation system
Introduction
In Recommendation Systems, Collaborative filtering is one of the approaches to building a model and finding seminaries between users. This concept is highly used in Ecommerce sites and OTT and video-sharing platforms. One of the highly talked about issues that such systems face while in the initial modeling phase is that of data sparsity, which occurs when only a few users give ratings or reviews on the platform and are in any way involved in the interaction.
In this article let us understand the problem of data sparsity in the Recommendation System and know about ways to handle it.
Data Sparsity
The main objective of Collaborative Filtering is to aggregate users that have similar thinking and common choices. This is done by gathering user-level information based on their rating or reviews on products/movies, etc. Thus a matrix of users and item ratings is generated. However, most of the time this matrix is highly sparse and that may be as high as 99%. Another problem occurs when new users come, for which very less information regarding rating are available.
This is also visible in the cold start problem.
How to handle data sparsity?
There are a few approaches to handling data sparsity.
Dimensional Reduction − A dimensional reduction algorithm is employed that reduces the user and item interaction matrix into a denser form while keeping one of the most relevant users who have interacted and provided their ratings. All the predictions are based on this denser matrix which is in reduced form. This method can improve the performance of many recommendation systems however there is one drawback that is it leads to valuable information loss.
Infer trust between users − In this method, we attempt to find the trust factor between two users that may not be directly associated with each other. However, they can be related through an intermediate user for ex. P.
For example, If user S and N have related I1 and user N and T have rated I2, to find relations between S and T, we may use a trust path that goes through user N as N is a common link between S and T.
The trust paths so defined can vary in length (k) and possibly become infinite if the source and target user has no common relations or trust users.
Social network in systems − Inference and relations are derived by users that have items that are co-rated by both. There are other kinds of interactions as well like feedback, transactions, etc. This is achieved by building a social network in recommendation systems. There are two processes involved in this approach – membership and evolution. In membership, any new or existing user has to at least rate one item to join such a network.
In the evolution phase as more and more users interact with the networks the interactions and links grow and strengthen and more associations are built.
Conclusion
Sparsity issue is a very common issue in Recommendation systems primarily with the Collaborative Filtering approach. Sparsity occurs when the data is sparse and neighbors cannot be identified from it. This can limit the quality of recommendations by the recommendation algorithm/system however there are a few methods like Dimensionality reduction, and inferring trust between users and social networks that have proven useful in solving this issue.
72 Views
