Time Travel
2/2/22
Welcome back to the Odyssey! Today I am going to deviate a bit from the regular program to talk about one of my favorite and probably most interesting journal articles I ever read.
When I was younger I was obsessed with ancient civilizations — the Sumerians, Ancient Greeks, Ancient Egyptians, the Chinese dynasties, you name it. I was enthralled by their magical cultures and legendary stories. If I had one wish back then, it would have been to travel back in time to talk to the ancient Greeks or Egyptians and learn what their life was like. In a way though, we can do this with AI. If you were to train a language model on the texts of an ancient civilization, it would learn the values, beliefs, and norms that define its society, allowing us to ‘talk’ with the civilization.
The journal article that this post is about does something similar. The authors process historical texts and use the word embeddings generated to study historical trends and social change regarding ethnic and gender stereotypes in the last 100 years.
The Paper
Word Embeddings
I have discussed word embeddings in a previous post. To summarize, a word embedding is a learned representation of the text in your data set where each word is represented by a unique vector in some vector space of predefined size. This representation encodes the meaning of each word such that the words that are closer in the vector space are expected to be similar in meaning.
This context based representation means that embeddings can be used to measure the relative strength of association between words by comparing the Euclidian distances between their corresponding embedded vectors. For example, you would expect the distance between the vector for ‘ocean’ and ‘water’ to be smaller than the vector for ‘desert’ and ‘water’, meaning that the word ‘water’ is more associated with the word ‘ocean’ than it is with ‘desert’.
Methods
The paper studies the changes in ethnic and gender stereotypes over time by measuring the strength of association between occupations and adjectives. To be more exact:
“As an example, we overview the steps we use to quantify the occupational embedding bias for women. We first compute the average embedding distance between words that represent women—e.g., she, female—and words for occupations—e.g. teacher, lawyer. For comparison, we also compute the average embedding distance between words that represent men and the same occupation words. A natural metric for the embedding bias is the average distance for women minus the average distance for men. If this value is negative, then the embedding more closely associates the occupations with men. More generally, we compute the representative group vector by taking the average of the vectors for each word in the given gender/ethnicity group. Then we compute the average Euclidean distance between each representative group vector and each vector in the neutral word list of interest, which could be occupations or adjectives. The difference of the average distances is our metric for bias—we call this the relative norm difference or simply embedding bias.”
Results - Gender Stereotypes
Through their use of word embeddings, the authors of this paper found that language today is even more biased than traditional methods like occupational data analysis show. Further, the embedding bias captures stereotypes in a far more nuanced and accurate fashion than occupational statistics.
Their results also show that bias, as seen through adjectives associated with men and women, has decreased over time and that the women’s movement in the 1960s and 1970s especially had a systemic and drastic effect in women’s portrayals in literature and culture.
Using word embeddings to analyze biases in adjectives is especially important because while the effects of the women’s movement on inclusive language is well documented, the literature currently is lacking systematic and quantitative metrics for adjective biases.
The change in the biases, as seen through adjectives associated with men and women, is shown quantitatively and qualitatively below:
Results - Ethnic Stereotypes
The paper illustrates the effectiveness of word embeddings to study ethnic biases over time.
Word embeddings allowed the researchers to better understand how broad trends in the 20th-century influenced the view of Asians in the United States. The embedding bias showed that prior to 1950 strongly negative words, especially those often used to describe outsiders, are among the words most associated with Asians: barbaric, hateful, monstrous, bizarre, and cruel.
However, a rising Asian population in the United States after 1950, led to these words being largely replaced by words often considered stereotypic of Asian Americans today: sensitive, passive, complacent, active, and hearty, for example. Word embeddings allowed the researchers to quantify this change, illustrating a remarkable change in the attitudes towards Asian Americans as words related to outsiders steadily decrease in Asian association over time (the exception being WWII).
The researchers also found that word embeddings serve as an effective tool to analyze finer-grained trends, using the stereotypes towards Islam, Russians, and Hispanics as their foci. Word embeddings were able to capture how global events, such as 9/11 and the Cold War, lead to a sharp change in the stereotypes towards ethnicities while more frequent but less salient events have a more gradual impact on these stereotypes.
Results - Validation
To validate the effectiveness of this approach, the researchers compared their results to the occupational differences between the groups in question using US census data, which supported their findings.
Why Should You Care?
Accurately and quantitatively measuring changes in a societal characteristic as subtle as bias is a very difficult, but vitally important, task. Typically, metrics like poverty, GDP, and inequality are used to measure the improvement or progression of society. If a country is richer and more equal, it is better off. And while that is generally true, these metrics miss the intricate details of human behavior that have a major impact on our lives. If we are not able to quantify and accurately measure the change in social biases, we won’t know what direction to head in. Hence, AI’s ability to pick up these subtleties allows us to map a path forward, make better decisions, and evaluate the effectiveness of initiatives that try to tackle these issues.
