How you see others

Written by Eloise Ballou
We do not see things as they are. We see them as we are. Proverb

You would be surprised at how much we bring to what we see.

This is particularly true about faces. What you may assume to be objective, factual information is in fact profoundly coloured by you, the viewer. 

We are particularly vulnerable to making mistakes when reading ambiguous information on people’s faces. Unfortunately, many nonverbal cues are ambiguous.

Researchers have made several interesting findings in this area.

Seeing what you expect rather than what is there

Context is very important when interpreting facial expressions. Researchers have demonstrated this with the following experiment. Why don’t you try it out for yourself?

First, read the following story:

“This is a story of a woman who wanted to treat her sister to the most expensive, exclusive restaurant in their city. Months ahead, she made a reservation. When she and her sister arrived, they were told by the maitre d’ that their table would be ready in 45 minutes. Still, an hour passed, and no table. Other groups arrived and were seated after a short wait. The woman went to the maitre d’ and reminded him of her reservation. He said that he’d do his best. Ten minutes later, a local celebrity and his date arrived and were immediately shown to a table. Another couple arrived and were seated immediately. The woman went to the maitre d’ who said that all the tables were now full, and that it might be another hour before anything was available.”

Now, imagining that she is the woman in the story, look at this woman’s face.

What emotion is this woman feeling? Happiness, surprise, fear, anger, disgust, or sadness?

After reading the narrative, the large majority of participants identified her expression as anger.

However, take a look at her face again, separate from the story. What emotion is she feeling?

Screenshot 2016-07-24 19.35.54

When shown her face on its own, without being primed by the story, you can see that she is actually displaying fear (and possibly surprise). Her face is not angry.

Participants consistently read the emotion that went along with the narrative, rather than the true emotion. This speaks to the power of context when interpreting someone’s face.

Have you ever been treated as if you were displaying a certain emotion when in fact you weren’t feeling the emotion that was attributed to you?

In women, this phenomenon is captured in the concept of “resting bitch face” or RBF. 

Kristen Stewart, poster child of RBF

How disorienting and frustrating is it to be misread by others?

How emotions colour what you see

Your emotional state plays a large role in how you interpret facial expressions.

This was tested by inducing happiness in subjects, then showing them a series of images of faces that are changing from happiness to sadness (this is known as the face morph paradigm).

Face morph from happy to neutral (above) and sad to neutral (below)

Subjects were asked to identify when the original expression had dissipated from the face. 

People who were happier saw happiness last longer on the face than those who had not been primed to feel happier. Those primed to feel sad saw sadness for a greater time than those who hadn’t been placed in a sad emotional state. This revealed that subjects’ temporary emotional state changed the way they saw emotions on the faces of others in a consistent, measurable way.

Given an ambiguous expression, your emotional state exerts a top-down influence that powerfully colours what you see. If you feel happy, you will see more happiness in every face you encounter. If you feel sad, you will see even less happiness than a neutral observer would.

Think of how this could have the opposite effect: If you’re feeling angry, you will see more anger in the face of someone with a neutral expression. In fact, you might see anger where there is none. This will likely fuel and perpetuate your own anger, creating a self-fulfilling prophecy.

“When you’re holding a hammer, everything looks like a nail”.

On the other hand: “When you smile, the whole world smiles with you.”

Wouldn’t you rather choose the latter?

JFK and Jackie's wedding, TIME magazine, 1953.


Seeing what you hope to see

Your perception is also affected by your expectations and goals.

In another study, researchers showed men either a clip from a romantic movie or a neutral movie. They were then shown an image of a woman’s face with a neutral expression. The men who first saw the romantic clip interpreted the ambiguous expression on the woman’s face as displaying sexual interest, while those who weren’t shown the romantic clip didn’t identify as much sexual interest. This effect was even more prominent with an attractive woman’s face compared to a less attractive woman’s face.

Ruth Orkin, American Girl in Italy, 1951.
Ruth Orkin, American Girl in Italy, 1951.

In other words, men primed to think of women as potential romantic partners saw sexual interest in the faces of women who were not displaying it. Their goals and expectations unconsciously coloured their perceptions.

How do your own expectations colour what you see in others? How are you perceived through the expectations of others?

The power of groups

Another powerful determinant of how accurately you perceive others’ emotions is caused by in-group vs. out-group differences.

Hazel Bryan and Elizabeth Eckford, Little Rock, Arkansas, September 1957.


But it goes further than that. Even being arbitrarily assigned to a group affects how well you perceive the emotions of others.

This was demonstrated by giving participants a bogus personality test and then dividing them into “red” and “green” personality types (though they were in fact randomly assigned to either group).  These two groups were shown images of unambiguous facial expressions that clearly showed a single emotion (e.g. happiness). But the faces were shown with either a red or green background, and the subjects were told that the colour corresponded to the face’s personality type.

The results revealed that subjects were more accurate at identifying emotions on faces they believed to be part of their in-group (i.e. “red” or “green” personality type). This was true despite the fact that both groups were randomly assigned, and there were no actual differences between them.

Why did this happen?

One explanation is that participants were more motivated to interpret the emotions of members in their own group. When subjects were given less time to view the images (preventing them from spending more time and effort to identify the emotion), the in-group advantage was reduced, which supports this idea.

Reading faces is a motivated action. We are more motivated to interpret the emotions of individuals who are powerful or dominant. With someone who is subservient or whom we don’t respect as much, we will not make this extra effort. And we may make mistakes as a result.

Anticipated stigma or the self-fulfilling prophecy

What if you expect to be treated differently? Might you perceive something that isn’t there?

Maud Wagner, the first well-known female tattoo artist in the United States, 1907.

Maud Wagner, the first well-known female tattoo artist in the United States, 1907.

A very revealing study demonstrated this effect.

Subjects were shown images of faces morphing from contempt to a neutral expression. They were asked to indicate when the expression changed. Female participants were also asked to complete a validated questionnaire measuring their level of stigma consciousness, that is, the extent to which they felt they were subject to sexism.

Women who scored high on this questionnaire perceived greater contempt in a neutral/ambiguous male face than in a female face. Women who did not have high stigma consciousness did not show this difference, and did not perceive as much contempt in the male face.

What do you see in this face?

Let’s try it out. How much contempt do you see in the face above? 

Now compare it to the range of expressions below.

Face morph from contempt to happiness.
Face morph from contempt to happiness.

Face C is the same as the first image shown above. That means that, on the morph, it’s actually closer to happiness than to contempt. But if you expected to see contempt, this face might appear contemptuous to you.

This can become a self-fulfilling prophecy: if you expect to be stigmatized, you will see evidence of it in other peoples’ faces. This is important because what you perceive in others (especially in ambiguous situations) powerfully changes how you will respond to them. And what was at first ambiguous (or nonexistent) might become true if you respond in a particular way to elicit that response.

If a woman expects men to show contempt towards her, she might incorrectly assume a man is showing contempt when he isn’t. Or, she may perceive a small amount of contempt as more intense than it actually is. As a result of seeing him as openly contemptuous, her reaction may be more angry or curt than it would otherwise be. The man might then begin to feel anger or true contempt when he sees the woman “overreact”, from his perspective. The woman has elicited in him the very emotions she wrongly believed were there in the first place. And she can then feel justified by his response, though it reflected her own actions rather than his original intentions.

And it works both ways.

Men who expect women to be “too emotional” may over-read an ambiguous expression on a woman’s face as displaying inappropriate emotionality (for example, as being more sad or angry), even when this isn’t the case. The woman may then become more emotionally expressive as a result of his reactions, confirming his initial assumptions.

This type of exchange will confirm both biases– that men are contemptuous towards women, and that women are “irrational” and “overemotional”. Neither was true to begin with. It’s another case of self-fulfilling prophecy and confirmation of bias.

The key here is not to blame people for their beliefs, or state that perceptions are always incorrect. Stigma is a very real thing, and women who are stigma conscious often have good reasons to feel that way, based on their actual experience. Ambiguous faces sometimes do show the emotion perceived, irrelevant of expectations.

The problem is that these biases lead to over-generalizations and reaffirming of incorrect beliefs. It makes life harder than it needs to be, and prevents true communication.

Alfred Eisenstadt - Student nurses in the amphitheater of Roosevelt Hospital, New York City, 1938.
Alfred Eisenstadt – Student nurses in the amphitheater of Roosevelt Hospital, New York City, 1938.
What you bring to the faces you see

We can misread obvious facial cues given the right context. Our mood colours the emotions we see in others, and our expectations cause us to see what fits with our ideas, rather than what is actually there. Whether we perceive others as being part of our group or not changes how accurately we read their emotions, and how hard we try to make sense of ambiguous emotions rather than jumping to conclusions.

Faces are often ambiguous. If you expect to see contempt, you’ll see it everywhere. You may then respond in a way to perpetuate this belief. The closed loop will strengthen your conviction that you know what others are feeling, when this may not be true.

The goal is to realize that you might be jumping to conclusions, and to check in with the other person when things don’t seem to fit. Search for evidence for and against your beliefs, before deciding that your gut feeling was right. Even the knowledge of these biases may help you question your perceptions from this day on.

How might keeping an open mind (and trying to recognize when other factors are influencing your perceptions) transform the way you interact with others? Striving to maintain an open and questioning attitude is difficult, but has the potential to radically change how you see others.