{"title":"Impact of Similarity Ratings on Human Judgement","authors":"Ian A. McCulloh, Madelaine Zinser, Jesse Patsolic, Michael Ramos","volume":205,"journal":"International Journal of Computer and Information Engineering","pagesStart":28,"pagesEnd":33,"ISSN":"1307-6892","URL":"https:\/\/publications.waset.org\/pdf\/10013446","abstract":"<p>Recommender systems are a common artificial intelligence (AI) application. For any given input, a search system will return a rank-ordered list of similar items. As users review returned items, they must decide when to halt the search and either revise search terms or conclude their requirement is novel with no similar items in the database. We present a statistically designed experiment that investigates the impact of similarity ratings on human judgement to conclude a search item is novel and halt the search. In the study, 450 participants were recruited from Amazon Mechanical Turk to render judgement across 12 decision tasks. We find the inclusion of ratings increases the human perception that items are novel. Percent similarity increases novelty discernment when compared with star-rated similarity or the absence of a rating. Ratings reduce the time to decide and improve decision confidence. This suggests that the inclusion of similarity ratings can aid human decision-makers in knowledge search tasks.<\/p>","references":"[1]\tMelville, P., Sindhwani, V., Sammut, C. and Webb, G.I., 2010. Recommender Systems. In Encyclopedia of machine learning.\r\n[2]\tCena, F., Gena, C., Grillo, P., Kuflik, T., Vernero, F. and Wecker, A.J., 2017. How scales influence user rating behaviour in recommender systems. Behaviour & Information Technology, 36(10), pp.985-1004.\r\n[3]\tGarland, R. 1991. \u201cThe Mid-Point on a Rating Scale: Is it Desirable.\u201d Marketing Bulletin 2: 66\u201370.\r\n[4]\tFriedman, H. H., and T. Amoo. 1999. \u201cRating the Rating Scales.\u201d Journal of Marketing Management 9 (3): 114\u2013123.\r\n[5]\tAmoo, T., and H. H. Friedman. 2001. \u201cDo Numeric Values Influence Subjects Responses to Rating Scales?\u201d Journal of International Marketing and Marketing Research 26: 41\u201346.\r\n[6]\tCummins, R., and E. Gullone. 2000. \u201cWhy We Should Not Use 5-Point Likert Scales: The Case for Subjective Quality of Life Measurement.\u201d In Proceedings of the Second International Conference on Quality of Life in Cities, 74\u201393. Singapore: The School.\r\n[7]\tCosley, D., S. K. Lam, I. Albert, J. A. Konstan, and J. Riedl. 2003. \u201cIs Seeing Believing?: How Recommender System Interfaces Affect Users\u201d Opinions.\u201d In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI \u201803, 585\u2013592. New York: ACM.\r\n[8]\tVaz, P. C., R. Ribeiro, and D. M. de Matos. 2013. \u201cUnderstanding the Temporal Dynamics of Recommendations Across Different Rating Scales.\u201d UMAP Workshops 2013, CEUR-WS.org.\r\n[9]\tCena, F., F. Vernero, and C. Gena. 2010. \u201cTowards a Customization of Rating Scales in Adaptive Systems.\u201d In User Modeling, Adaptation, and Personalization \u2013 18th International Conference, UMAP 2010, edited by P. De Bra, A. Kobsa, and D. N. Chin, Big Island, HI, USA, June 20\u201324. Proceedings, Volume 6075 of Lecture Notes in Computer Science, 369\u2013374.\r\n[10]\tGena, C., R. Brogi, F. Cena, and F. Vernero. 2011. \u201cThe Impact of Rating Scales on User\u2019s Rating Behavior.\u201d In User Modeling, Adaption and Personalization \u2013 19th International Conference, UMAP 2011, Girona, Spain, July 11\u201315. Proceedings, Lecture Notes in Computer Science 6787, 123\u2013134.\r\n[11]\tPreston, C., and A. Colman. 2000. \u201cOptimal Number of Response Categories in Rating Scales: Reliability, Validity, Discriminating Power, and Respondent Preferences.\u201d Acta Psychologica 104 (1): 1\u201315.\r\n[12]\tWeng, L. J. 2004. \u201cImpact of the Number of Response Categories and Anchor Labels on Coefficient Alpha and Test-Retest Reliability.\u201d Educational and Psychological Measurement 64 (6): 956\u2013972\r\n[13]\tBest Movies, https:\/\/bestsimilar.com\/movies, retrieved on October 31, 2022.","publisher":"World Academy of Science, Engineering and Technology","index":"Open Science Index 205, 2024"}