Enabling Explainable Recommendation in E-commerce with LLM-powered Product Knowledge Graph

How to leverage large language model’s superior capability in e-commerce recommendation has been a hot topic. In this paper, we propose LLM-PKG, an efficient approach that distills the knowledge of LLMs into product knowledge graph (PKG) and then applies PKG to provide explainable recommendations. Specifically, we first build PKG by feeding curated prompts to LLM, and then map LLM response to real enterprise products. To mitigate the risks associated with LLM hallucination, we employ rigorous evaluation and pruning methods to ensure the reliability and availability of the KG. Through an A/B test conducted on an ecommerce website, we demonstrate the effectiveness of LLM-PKG in driving user engagements and transactions significantly.

Introduction. The recent progress in Large Language Models (LLMs) has offered excellent capabilities in understanding, generating, and reasoning with human-like text. It is a promising tool to augment recommendation systems by articulating the rationale behind their recommendations in a manner that is comprehensible to users. This not only enhances users’ trust and confidence in the recommendations but also fosters a deeper engagement with the recommender system. Nowadays modern recommender systems on e-commerce websites are complicated and extremely sensitive to the response time, making calling LLM in real-time an unacceptable solution. So a natural idea is to distill useful knowledge from LLM into a knowledge graph (KG) and then apply the KG into recommendation for E-commerce. Trained on vast amounts of textual data and external knowledge sources, LLM is assumed to own world knowledge. So it is a relatively mild condition that LLM understands most usecases of products and user intention behind a product purchase.

Discussion / Conclusion. The results of the experiment are listed in Table 3. Notable improvements across various metrics were observed in the Treatment group compared to the Control, including Clicks, Transactions, and GMB (Gross Merchandise Bought). Specifically, clicks experienced a significant increase of 5.19% in the Treatment group, indicating enhanced user engagement and interaction with recommended products. Moreover, Transactions and GMB exhibited a substantial positive uplift (7.59% and 8.56% respectively), suggesting that users interacting with LLM-PKG-powered recommendations are more likely to initiate transactions compared to those in the Control group. We argue that the benefits of LLM-PKG bring to recommendations are two folds: 1) LLM- PKG can understand the user needs based on their historical behavior, thus providing more accurate recommended items to customers. 2) LLM-PKG can provide explanations of its recommendations to attract users. Unlike other KG based explanation for recommendations, this implementation is very simple in LLM-PKG. We only need to display edge information of KG without any additional operations or modifications to the LLM-PKG, making this approach more efficient and scalable.