Search

8. Datasets, Evaluation Metrics and Applications

Person
비고

8.1. Datasets

GNN 기반의 recommendation systems에 사용된 dataset들만 소개. (일반적인 RecSys에서 사용되는 dataset은 여기서 소개 X)
Search
Name
Data info
Task
Miscellaneous
MovieLens
Open
User-item rating pair: timestamp, movie의 attribute, tag, user demographic feature와 함께 제공됨.
User-Item CF
KG-based Rec
총 3가지 scale: MovieLens-100K, MovieLens-1M, MovieLens-20M
Amazon
Open
Reviews (ratings, text, helpfullness votes), product metadata (description, category, price, brand, image feature), link (also viewed / also bought graph)
User-Item CF
Sequential Rec
Category에 따라 sub-dataset으로 구분될 수 있음. (Amazon-Books, Amazon-Electronics 등)
Yelp
Open
User check-in 정보
User-Item CF
POI
Gowalla
Open
User check-in 정보, User간 social relationship 정보.
User-Item CF
POI
Sequential Rec
Yoochoose
Open
Stream of user clicks on e-commerce website within 6 months
Diginetica
Open
시간 순서대로 정리된 transactional data
Session-based Rec
RetailRocket
Open
6개월 간의 user browsing activity
LastFM
Open
Musician listening 정보 (2,000명의 user와 artist attribute)
Sequential Rec
Social Rec
KG-based Rec
Epinions & Ciao
Open
User의 rating, item review User 간 directed trust relationship.
Social Rec
Book-Crossing
Open
1M rating of books, attributes of books (title, author, ..)
KG-based Rec

8.2. Evaluation Metrics

•
HR: 추천된 item 중에 1개 이상 click 한 user의 비율
HR@K=1∣U∣∑u∈UI(∣RK(u)∩T(u)∣>0)\text{HR}@K = \frac{1}{|\mathcal{U}|}\sum_{u \in \mathcal{U}}I(|R^K(u) \cap T(u)| > 0)
◦
RK(u)R^K(u): top-KK 개의 추천된 item set
◦
T(u)T(u): groud truth
◦
II: indicator function
•
Precision, Recall, F1
◦
Precision@K(u)=∣RK(u)∩T(u)∣K\text{Precision}@K(u) = \frac{|R^K(u) \cap T(u)|}{K}
◦
Recall@K(u)=∣RK(u)∩T(u)∣∣T(u)∣\text{Recall}@K(u) = \frac{|R^K(u) \cap T(u)|}{|T(u)|}
◦
F1@K(u)=2×Precision@K(u)×Recall@K(u)Precision@K(u)+Recall@K(u)\text{F1}@K(u) = \frac{2 \times \text{Precision}@K(u) \times \text{Recall}@K(u)}{\text{Precision}@K(u) + \text{Recall}@K(u)}
•
NDCG: ranking position에 따라 맞게 추천된 item의 contribution을 구분한다.
◦
NDCG@K=1∣U∣∑u∈U∑k=1KI(RkK(u)∈T(u))log⁡(k+1)∑k=1K1log⁡(k+1)\text{NDCG}@K = \frac{1}{|\mathcal{U}|}\sum_{u \in \mathcal{U}} \frac{\sum^K_{k=1}\frac{I(R^K_k(u) \in T(u))}{\log (k+1)}}{\sum^K_{k=1}\frac{1}{\log(k+1)}}
•
MAP: average precision over users
◦
MAP@K=1∣U∣∑u∈U∑k=1KI(RkK(u)∈T(u))Precision@k(u)K\text{MAP}@K = \frac{1}{|\mathcal{U}|}\sum_{u \in \mathcal{U}} \sum^K_{k=1}\frac{I(R^K_k(u) \in T(u)) \text{Precision}@k(u)}{K}
•
AUC: Click된 item을 click 되지 않은 item 보다 model이 더 높게 rank할 확률.
◦
AUC(u)=∑i∈T(u)∑j∈I\T(u)I(r^i>r^j)∣T(u)∣∣I\T(u)∣\text{AUC}(u) = \frac{\sum_{i \in T(u)} \sum_{j \in \mathcal{I}\backslash T(u)} I(\hat{r}_i > \hat{r}_j)}{|T(u)||\mathcal{I}\backslash T(u)|}

8.3. Applications

GNN 기반의 추천 모델이 산업에서 사용될 수 있는 분야
•
E-commerce platform에서 상품 추천 또는 광고
◦
IntentGC: user-item interaction 정보와 knowledge graph 정보를 GCN으로 모두 사용하고, Alibaba platform에 deploy 되었음.
•
Content 추천: 뉴스, 글 추천
◦
DANSER: user-article interaction과 social relationship 정보를 활용해 실제 article 추천 시스템인 WeChat Top Story에 적용됨.
•
App 추천
◦
GraphSAIL: App Store에 deploy 됨.
•
Image 추천
◦
PinSage: Pinterest에 deploy 됨.