Tag Archives: computer vision

Learning Faces to Predict Matching Probability in an Online Dating Market (ICIS 2022)

Kwon, Soonjae, Gene Moo Lee, Dongwon Lee, Sung-Hyuk Park (2024) “Digital Cupid: Empowering Generative AI for Fair and Efficient Matchmaking,” Working Paper.

  • Previous title: Learning Faces to Predict Matching Probability in an Online Dating Market
  • Presentations: DS (2021), AIMLBA (2021), WITS (2021), ICIS (2022)
  • Preliminary version in ICIS 2022 Proceedings
  • Based on an industry collaboration

With the increasing prevalence of online transactions, enhancing matching efficiency has emerged as a critical objective for most matching platforms. However, these efforts often lead to decreased fairness, making it challenging to balance these two elements. This study presents a novel generative AI-based approach to increase the platform’s efficiency and fairness simultaneously in the context of online dating. By developing a model that utilizes users’ multimodal features to predict individual preferences, we assess the impact of various matching algorithms on platform efficiency and fairness. Extensive simulations show that our fairness-aware algorithm significantly enhances both metrics, addressing conventional methods’ severe efficiency-fairness tradeoff issue. We also introduce a novel generative AI-based personalization technique that modifies users’ profile images in different directions according to their counterparts, further boosting efficiency without sacrificing fairness. Our matching framework can be applied to platforms with various objectives, contributing to all stakeholders in digital platforms.

IS / Marketing Papers on Multimodal Data Analytics (Image, Video, Audio)

Last update: Sep 7, 2023

With the advent of social media and mobile platforms, visual and multimodal data are becoming the first citizen in big data analytics research. Compared to textual data that require significant cognitive efforts to comprehend, visual data (such as images and videos) can easily convey the message from the content creator to the general audience. To conduct large-scale studies on such data types, researchers need to use machine learning and computer vision approaches. In this post, I am trying to organize studies in Information Systems, Marketing, and other management disciplines that leverage large-scale analysis of image and video datasets. The papers are ordered randomly:

  1. Yang, Yi, Yu Qin, Yangyang Fan, Zhongju Zhang (2023). Unlocking the Power of Voice for Financial Risk Prediction: A Theory-Driven Deep Learning Design Approach. MIS Quarterly 47(1): 63-96.
  2. Ceylan, G., Diehl, K., & Proserpio, D. (2023). EXPRESS: Words Meet Photos: When and Why Visual Content Increases Review Helpfulness. Journal of Marketing Research, forthcoming.
  3. Alex Burnap, John R. Hauser, Artem Timoshenko (2023) Product Aesthetic Design: A Machine Learning Augmentation. Marketing Science, forthcoming.
  4. Gao, Jia, Ying Rong, Xin Tian, Yuliang Yao (2023) Improving Convenience or Saving Face? An Empirical Analysis of the Use of Facial Recognition Payment Technology in Retail. Information Systems Research, forthcoming.
  5. Guan, Yue, Yong Tan, Qiang Wei, Guoqing Chen (2023) When Images Backfire: The Effect of Customer-Generated Images on Product Rating Dynamics. Information Systems Research, Forthcoming.
  6. Son, Y., Oh, W., Im, I. (2022) The Voice of Commerce: How Smart Speakers Reshape Digital Content Consumption and Preference. MIS Quarterly, forthcoming.
  7. Hou, J., Zhang, J., & Zhang, K. (2022). Pictures that are Worth a Thousand Donations: How Emotions in Project Images Drive the Success of Crowdfunding Campaigns? An Image Design Perspective. MIS Quarterly, Forthcoming.
  8. Lysyakov, Mikhail, Siva Viswanathan (2022) Threatened by AI: Analyzing Users’ Responses to the Introduction of AI in a Crowd-Sourcing Platform. Information Systems Research, Forthcoming.
  9. Hanwei Li, David Simchi-Levi, Michelle Xiao Wu, Weiming Zhu (2022) Estimating and Exploiting the Impact of Photo Layout: A Structural Approach. Management Science, Forthcoming.
  10. Bharadwaj, N., Ballings, M., Naik, P. A., Moore, M, Arat, M. M. (2022) “A New Livestream Retail Analytics Framework to Assess the Sales Impact of Emotional Displays,” Journal of Marketing, 86(1): 24-47.
  11. Chen, Z., Liu, Y.-J., Meng, J., Wang, Z. (2022) “What’s in a Face? An Experiment on Facial Information and Loan-Approval Decision“, Management Science, forthcoming.
  12. Lu, T., Wang, A., Yuan, X., Zhang, X. (2020) “Visual Distortion Bias in Consumer Choices,” Management Science, forthcoming.
  13. Zhou, M., Chen, G. H., Ferreira, P., Smith, M. D. (2021) “Consumer Behavior in the Online Classroom: Using Video Analytics and Machine Learning to Understand the Consumption of Video Courseware,” Journal of Marketing Research 58(6): 1079-1100.
  14. Zhang, Shunyuan, Dokyun Lee, Param Vir Singh, Kannan Srinivasan (2021) What Makes a Good Image? Airbnb Demand Analytics Leveraging Interpretable Image Features. Management Science 68(8):5644-5666.
  15. Gunarathne, P., Rui, H., Seidmann, A. (2021) “Racial Bias in Customer Service: Evidence from Twitter,” Information Systems Research 33(1): 43-54.
  16. Shin, D., He, S., Lee, G. M., Whinston, A. B., Cetintas, S., Lee, K.-C. (2020) “Enhancing Social Media Analysis with Visual Data Analytics: A Deep Learning Approach,” MIS Quarterly 44(4): 1459-1492. [Details]
  17. Li, Y., Xie, Y. (2020) “Is a Picture Worth a Thousand Words? An Empirical Study of Image Content and Social Media Engagement,” Journal of Marketing Research 57(2): 1-19.
  18. Zhang, Q., Wang, W., Chen, Y. (2020) “Frontiers: In-Consumption Social Listening with Moment-to-Moment Unstructured Data: The Case of Movie Appreciation and Live comments,” Marketing Science 39(2).
  19. Liu, L., Dzyabura, D., Mizik, N. (2020) “Visual Listening In: Extracting Brand Image Portrayed on Social Media,” Marketing Science 39(4): 669-686.
  20. Peng, L., Cui, G., Chung, Y., Zheng, W. (2020) “The Faces of Success: Beauty and Ugliness Premiums in E-Commerce Platforms,” Journal of Marketing 84(4): 67-85.
  21. Liu, X., Zhang, B., Susarla, A., Padman, R. (2020) “Go to YouTube and Call Me in the Morning: Use of Social Media for Chronic Conditions,” MIS Quarterly 44(1b): 257-283.
  22. Zhao, K., Hu, Y., Hong, Y., Westland, J. C. (2020) “Understanding Characteristics of Popular Streamers in Live Streaming Platforms: Evidence from Twitch.tv,” Journal of the Association for Information Systems, Forthcoming.
  23. Ordenes, F. V., Zhang, S. (2019) “From words to pixels: Text and image mining methods for service research,” Journal of Service Management 30(5): 593-620.
  24. Wang, Q., Li, B., Singh, P. V. (2018) “Copycats vs. Original Mobile Apps: A Machine Learning Copycat-Detection Method and Empirical Analysis,” Information Systems Research 29(2): 273-291.
  25. Lu, S., Xiao, L., Ding, M. (2016) “A Video-Based Automated Recommender (VAR) System for Garments,” Marketing Science 35(3): 484-510.
  26. Xiao, L., Ding, M. (2014) “Just the Faces: Exploring the Effects of Facial Features in Print Advertising,” Marketing Science 33(3), 315-461.
  27. Suh, K.-S., Kim, H., Suh, E. K. (2011) “What If Your Avatar Looks Like You? Dual-Congruity Perspectives for Avatar Use,” MIS Quarterly 35(3), 711-729.
  28. Todorov, A., Porter, J. M. (2014) “Misleading First Impressions: Different for Different Facial Images of the Same Person“, Psychological Science 25(7): 1404-1417.
  29. Todorov, A., Madnisodza, A. N., Goren, A., Hall, C. C. (2005) “Inferences of Competence from Faces Predict Election Outcomes“, Science 308(5728): 1623-1626.
  30. Mueller. U., Mazur, A. (1996) “Facial Dominance of West Point Cadets as a Predictor of Later Military Rank“, Social Forces 74(3): 823-850.
  31. Lee, H, Nam, K. “When Machine Vision Meets Human Fashion: Effects of Human Intervention on the Efficiency of CNN-Driven Recommender Systems in Online Fashion Retail”, Working Paper.
  32. Lysyhakov M, Viswanathan S (2021) “Threatened by AI: Analyzing users’ responses to the introduction of AI in a crowd-sourcing,” Working Paper.
  33. Park, S., Lee, G. M., Shin, D., Han, S.-P. (2020) “Targeting Pre-Roll Ads using Video Analytics,” Working Paper.
  34. Choi, A., Ramaprasad, J., So, H. (2021) Does Authenticity of Influencers Matter? Examining the Impact on Purchase Decisions, Working Paper.
  35. Park, J., Kim, J., Cho, D., Lee, B. Pitching in Character: The Role of Video Pitch’s Personality Style in Online Crowdsourcing, Working Paper.
  36. Yang, J., Zhang, J., Zhang Y. (2021) First Law of Motion: Influencer Video Advertising on TikTok, Working Paper.
  37. Davila, A., Guasch (2021) Manager’s Body Expansiveness, Investor Perceptions, and Firm Forecast Errors and Valuation, Working Paper.
  38. Peng, L., Teoh, S. H., Wang, U., Yan, J. (2021) Face Value: Trait Inference, Performance Characteristics, and Market Outcomes for Financial Analysts, Working Paper.
  39. Zhang, S., Friedman, E., Zhang, X., Srinivasan, K., Dhar, R. (2020) Serving with a Smile on Airbnb: Analyzing the Economic Returns and Behavioral Underpinnings of the Host’s Smile,” Working Paper.
  40. Park, K., Lee, S., Tan, Y. (2020) “What Makes Online Review Videos Helpful? Evidence from Product Review Videos on YouTube,” UW Working Paper.
  41. Doosti, S., Lee, S., Tan, Y. (2020) “Social Media Sponsorship: Metrics for Finding the Right Content Creator-Sponsor Matches,” UW Working Paper.
  42. Koh, B., Cui, F. (2020) “Give a Gist: The Impact of Thumbnails on the View-Through of Videos,” KU Working Paper.
  43. Hou J.R., Zhang J., Zhang K. (2018) Can title images predict the emotions and the performance of crowdfunding projects? Workshop on e-Business.

Trustworthy Face? The Effect and Drivers of Comprehensive Trust in Online Job Market Platform

Kwon, Jun Bum, Donghyuk Shin, Gene Moo Lee, Jake An, Sam Hwang (2020) “Trustworthy Face? The Effect and Drivers of Comprehensive Trust in Online Job Market Platform”. Work-in-progress.

The abstract will appear here.

Enhancing Social Media Analysis with Visual Data Analytics: A Deep Learning Approach (MISQ 2020)

Shin, Donghyuk, Shu He, Gene Moo Lee, Andrew B. Whinston, Suleyman Cetintas, Kuang-Chih Lee (2020) Enhancing Social Media Analysis with Visual Data Analytics: A Deep Learning Approach, MIS Quarterly, 44(4), pp. 1459-1492. [SSRN]

  • Based on an industry collaboration with Yahoo! Research
  • The first MISQ methods article based on machine learning
  • Presented in WeB (Fort Worth, TX 2015), WITS (Dallas, TX 2015), UT Arlington (2016), Texas FreshAIR (San Antonio, TX 2016), SKKU (2016), Korea Univ. (2016), Hanyang (2016), Kyung Hee (2016), Chung-Ang (2016), Yonsei (2016), Seoul National Univ. (2016), Kyungpook National Univ. (2016), UKC (Dallas, TX 2016), UBC (2016), INFORMS CIST (Nashville, TN 2016), DSI (Austin, TX 2016), Univ. of North Texas (2017), Arizona State (2018), Simon Fraser (2019), Saarland (2021), Kyung Hee (2021), Tennessee Chattanooga (2021), Rochester (2021), KAIST (2021), Yonsei (2021), UBC (2022), Temple (2023)

This research methods article proposes a visual data analytics framework to enhance social media research using deep learning models. Drawing on the literature of information systems and marketing, complemented with data-driven methods, we propose a number of visual and textual content features including complexity, similarity, and consistency measures that can play important roles in the persuasiveness of social media content. We then employ state-of-the-art machine learning approaches such as deep learning and text mining to operationalize these new content features in a scalable and systematic manner. For the newly developed features, we validate them against human coders on Amazon Mechanical Turk. Furthermore, we conduct two case studies with a large social media dataset from Tumblr to show the effectiveness of the proposed content features. The first case study demonstrates that both theoretically motivated and data-driven features significantly improve the model’s power to predict the popularity of a post, and the second one highlights the relationships between content features and consumer evaluations of the corresponding posts. The proposed research framework illustrates how deep learning methods can enhance the analysis of unstructured visual and textual data for social media research.