How would you feel if you were rejected from a job because you didn't look competent enough? Or if you were apprehended at a public place by the police because you looked like a criminal? Although these scenes sound dystopic and generate a sense of fear and anxiety, technology that claims that people's traits can be inferred from their faces already exists and is being used by businesses and governments worldwide.

An Israeli start-up markets a machine-learning algorithm that uses photographs of people's faces to classify them into categories, such as terrorists, academic researchers, gamblers, and pedophiles, to name a few. A smartphone app generates users' personality profiles and tells them how compatible they will be with others based on just a selfie that users upload to the app.

Although personality quizzes and matchmaking may seem like harmless fun, a company claims it can predict how much risk people are willing to take for high-stakes decisions just based on their face. Apart from these examples, academic researchers have also developed machine learning algorithms that claim to predict people's personality and demographic traits from their faces, including sexual orientation, criminality, and openness, agreeableness, and neuroticism.

Not surprisingly, such technologies provoke controversy. People who support these facial profiling technologies argue that these technologies should be welcomed because they can help make our lives:

  • Safer—allowing police to apprehend people with criminal intent,
  • Better—providing personalized treatment guidance,
  • Richer—providing tailored learning opportunities,
  • More fun—providing  personalized entertainment recommendations, and
  • Convenient—allowing businesses to offer targeted products and services.

On the other hand, opponents claim that facial profiling technologies are unreliable and sometimes grossly inaccurate, are based on stereotypes that discriminate against marginalized groups, and often violate people's privacy. For example, Google's face recognition algorithm once classified people of African origin as gorillas.

Despite these concerns, governments and businesses across the world are deploying facial profiling technologies to infer people's traits from their faces. In our recent research, we sought to understand the psychological basis for why people support these technologies. We argued that for people to support the use of facial profiling technologies, they must first believe that a reliable association exists between individuals' faces and their character. In other words, only if people believe that someone's face communicates something hidden about them would they support the use of facial profiling.

Why Do People Think Individuals' Appearance Reveals Their Character?

One's social environment plays a role in whether they believe that people's appearance reveals their character or not. For example, in TV series and movies, heroes look attractive whereas villains are often portrayed as hideous and unseemly. In the movie Wizard of Oz, when Dorothy asks the good witch why she's so beautiful, the good witch replies, "Why, only bad witches are ugly." On the other hand, folklore includes sayings such as "a wolf in sheep's clothing" or "not all that glitters is gold," which suggest that appearances do not reveal character.

Across nine studies conducted with a total of nearly 3,000 participants, we found that the more people believed that individuals' appearance reveals their character, the more they supported adopting such facial profiling technologies. We included extreme uses of facial profiling in our studies, such as allowing the police to apprehend people because the facial profiling software predicts they may be criminals, or putting a student in the remedial track because the facial profiling software ranks them as low in intelligence, or allowing financial institutions to charge higher interest rates to certain customers because the facial profiling software rates them as less trustworthy. We assessed people's support for such severe uses of facial profiling because many people would probably be OK with using a smartphone app to rate their personality using a selfie just for fun. However, in industry and government, facial profiling is moving far beyond these innocuous uses—it is being used to make consequential decisions. The fact that those who believed that people's appearance reveals their character supported such extreme yet realistic uses of facial profiling demonstrate the strong role this belief plays in shaping people's responses on consequential issues. On a positive note, the relationship between the appearance reveals character belief and support for facial profiling disappeared when participants were told that evidence is unclear whether people's appearance can reliably reveal their character traits.

Intuition says that people may be hesitant to allow the use of facial profiling on themselves. However, contrary to this expectation, we found that the target of facial profiling did not matter—if people believed that individuals' appearance reveals their character, they supported facial profiling even if they themselves were the target of such profiling. We also examined the underlying reason for this relationship. We found that the more people believed that individuals' appearance reveals their character, the more confident they felt in their ability to infer others' character from their appearance, and thus, the more they supported facial profiling.

Based on the finding, we developed a personality-prediction game that either made participants feel that they, and people in general, were adept at predicting people's personalities from their faces (to bolster their confidence) or made them feel that they (and people in general) were quite bad at guessing people's personality from their face (to reduce their confidence). As expected, high confidence increased people's support for facial profiling.

Artificial intelligence seems to be an essential part of humanity's future. One of its more far-reaching uses is facial profiling—predicting what people are like just based on their faces, and using this information for critical decisions such as whom to arrest and whom to hire. Among the myriad practical reasons why people may support or oppose facial profiling, their beliefs about whether or not people's appearance reveals their character plays an important role.

Next time you encounter a claim to infer people's personality from their picture, think about the story your face can, or cannot, tell. Research shows that people can judge some traits such as extraversion from others' appearance at above-chance levels (i.e., greater than just guessing) but not other traits such as intelligence or trustworthiness. Importantly, although this research has focused on whether character traits can be predicted at above-chance accuracy, research at the heart of facial profiling has claimed that machine learning models (which are often proprietary trade secrets) can assess character traits with near-perfect accuracy. It is an open question whether these claims would hold up to scrutiny.

For Further Reading

Madan, S., Savani, K., & Johar, G. V. (2022). How you look is who you are: The appearance reveals character lay theory increases support for facial profiling. Journal of Personality and Social Psychology.

Shilpa Madan is an Assistant Professor of Marketing at Virginia Tech. Her research focuses on people's lay beliefs and cultural differences to enhance individual and societal well-being.

Krishna Savani is a Professor of Management at the Hong Kong Polytechnic University. He conducts research on culture, decision-making, diversity, morality, and machine learning.

Gita V. Johar is the Meyer Feldberg Professor of Business at the Graduate School of Business, Columbia University. Her research focuses on consumer identity, beliefs, and persuasion.