The next time you are waiting to pay for your groceries or your morning coffee, take a look at the people around you and ask yourself: Whom would I trust? Who looks like they would have my back in a fight? You will probably find that it is relatively easy to form impressions of others based on their appearance. Research has shown that, in spite of the old adage not to judge a book by its cover, we intuitively judge how trustworthy or dominant others are based on their facial appearance. But where do these first impressions come from?

Dozens, if not hundreds, of studies have been published on this topic. To name just a few, previous studies found that people are perceived as more trustworthy if look attractive, if they have a relatively narrow face, if they have baby-like facial features, and if they appear to be smiling ever so slightly (even when they are completely neutral).

So do we rely on all these features when judging others’ trustworthiness? It’s difficult to say. Most studies only looked at one facial feature at a time, which makes it difficult to understand if people are actually relying on that facial feature, or on another related one. Imagine you are trying to understand what kind of muffin Charlie likes best. You note down the colors of the muffins he chooses and observe that Charlie is always picking light over dark muffins. You might conclude that Charlie likes light-colored muffins. However, you could be easily mistaken. Charlie might only pick the light ones because they are the ones that have blueberries in them. Based on our observations we could easily draw the wrong conclusion that it’s one thing (light color) even though it is actually another (blueberries).

Is It The Muffin’s Color or the Blueberries?

The same logic applies to understanding which facial features people rely on when forming first impressions. Many facial features go together. People who appear happy are also seen as more attractive, people with wider faces are also seen as more baby-like, and so on. To address this problem, Alex Jones and I measured emotionally neutral faces on many different dimensions and we let all facial characteristics compete against each other to test how much each one influences observers’ ratings of trustworthiness and dominance.

We analyzed almost 600 face images from the Chicago Face Database. All of the target people were U.S. Americans, but they varied on gender, age, race, and many other characteristics. The targets were photographed with a neutral facial expression in front of a uniform background. Then the images were rated by more than 1,000 U.S. American raters on many different dimensions: trustworthiness, dominance, attractiveness, babyfacedness, and so on. Other characteristics like facial width-to-height ratio, were measured with computer software. In the end, we had a large data set that allowed us to predict impressions of trustworthiness and dominance from many different facial characteristics (28 in total)—some of which have been the focus of many previous studies (e.g., facial width-to-height ratio, attractiveness, babyfacedness), and some which have received relatively less attention (such as gender, age, and race).

When we compared the unique predictive power of all facial characteristics, some findings from earlier studies didn’t come out: in fact, some of those facial features did surprisingly poorly. For example, how narrow or wide a face is was not very informative for predicting how trustworthy or dominant it was perceived to be. Characteristics like attractiveness (attractive people are seen as more trustworthy) and gender (men are seen as more dominant than women) were more important. But the strongest influence on impressions was how much the facial features resembled emotional expressions. People who appear to be smiling (even when they are neutral) are seen as trustworthy and people who appear to be angry are perceived as dominant. This process is called “emotion overgeneralization.”

Emotion overgeneralization is a two-step process. First, there is our oversensitive emotion detection system. We are constantly on the look-out for emotional signals. This makes a lot of sense because emotional expressions, like a smile or a frown, tell us important things about a person. But we even see emotions when they are not there. Even when a person is not sending any emotional signals, we might detect a smile, just because the corners of their mouth are naturally tilted upwards a little bit.

Then, there is our tendency to overgeneralize. When we think we detect a smile, we not only think that this person is happy right now, but also that this person is happy, sociable, and nice in general. We really want to know what kind of person we are dealing with. That’s why even a fleeting smile, or a facial appearance that resembles a smile, can make us jump to the conclusion that we are dealing with a nice person.

So when we look at someone and immediately think that they cannot be trusted, this is probably not because we are actually able to judge others’ character based on their facial appearance. It may just be our oversensitive emotion detection system.

For Further Reading

Jaeger, B., & Jones, A. L. (2021). Which facial features are central in impression formation? Social Psychological and Personality Science.

Zebrowitz, L. A. (2012). Ecological and social approaches to face perception. In G. Rhodes, A. Calder, M. Johnson, & J. V. Haxby (Eds.), Oxford handbook of face perception. Oxford University Press.

Bastian Jaeger is an Assistant Professor at the Vrije Universiteit Amsterdam. He studies how first impressions are formed, how accurate they are, and how they guide our behavior.