Is This a Weapon of Math Destruction?: An Inquiry Activity
CoT Announcements
Next Monthly Tech Talk on Tuesday, 11/07/23. Join our monthly tech talks to discuss current events, articles, books, podcast, or whatever we choose related to technology and education. There is no agenda or schedule. Our next Tech Talk will be on Tuesday, November 7th, 2023 at 8-9pm EST/7-8pm CST/6-7pm MST/5-6pm PST. Learn more on our Events page and register to participate.
Next Book Club on Tuesday, 12/17/23: We are discu\ssing Blood in the Machine The Origins of the Rebellion Against Big Tech by Brian Merchant led by Dan Krutka. Register for this book club event if you’d like to participate.
Critical Tech Study Participation: If you self-identify as critical of technology, please consider participating in our study. We are seeking participants who self-identify as holding critical views toward technology to share their stories by answering the following questions: To you, what does it mean to take a critical perspective toward technology? How have you come to take on that critical perspective? Please consider participating in our study via OUR SURVEY. You are welcome to share with others. Thank you!
by Jacob Pleasants
In a recent Civics of Tech book club, we dug into Cathy O’Neil’s Weapons of Math Destruction: How Big Data Increases Inequality and Threatens Democracy. In the book, O’Neil describes a series of technological systems built on mathematical models that, she argues, have become uniquely influential and harmful in our society. Examples include predictive policing, online advertising, and systems for rating job applications. What makes these examples of Weapons of Math Destruction (WMDs) is that they show the following characteristics:
They are opaque in that the ways they work are not visible to those they affect.
They have entered widespread use and thus reach into the lives of many, if not all, members of society.
They cause substantial and irreversible damage on those that they harm.
One of the neat ideas that emerged from our book club conversation was how we might use her definition of a WMD in a classroom setting to spur inquiry. If students were introduced to the core characteristics of a WMD and shown an example or two, how might they then use the idea to examine some of the technologies in their own lives?
This seemed to me like a great idea, and so I decided to get the ball rolling by drafting up a basic outline of such a lesson that you can find below. In my mind, this lesson is just a starting point, and I would love to hear your ideas for how it could be further developed, expanded, and modified. How might you play with this basic idea for your educational context?
What I’d love to see is a small collection of options, versions, and renditions of this lesson, so please contribute your creative ideas and we will add them on to the basic version that I have created!
The Lesson
Part 1: What is a WMD?
In this initial part of the inquiry, the teacher will introduce students to the foundational ideas and language that O’Neil uses. O’Neil’s playful phrase draws attention to the destructive power of mathematics. But how could math be destructive? Her concern lies in the creation and use of mathematical models, which are mathematical representations of the world. The teacher begins the inquiry by helping students understand what we mean by mathematical model.
Introduce the Book and Mathematical Models
Share the book cover and the provocative title with students, and raise the question: “How could mathematics be destructive?”
Share with students that O’Neil isn’t concerned with math in general but about something more specific: mathematical models, which are ways of representing the world using math. Share some examples of mathematical models that we use often, highlighting that they aren’t always destructive. For instance:
We use mathematics to represent the weather. We measure things like wind, pressure, temperature, and the like. And then we use complex equations and computers to predict the future. It’s a representation because those measurements and equations aren’t the actual weather, but the model is still useful to us.
Your grades and GPA are also a sort of mathematical model. Here, they are representing your achievements in school. All the things that you do in school are pretty complex, and those mathematical representations offer a simpler to communicate all that complexity. You might have some mixed feelings about this. Is this representation too simple? Does it miss some important things? Might boiling all of your achievements down to a set of letter grades or even a single number end up causing some unnecessary anxiety?
The uneasiness around grades is what O’Neil’s ideas build upon. Let’s look at a specific example that she brings up in her book.
O’Neil’s Example: Predictive Policing
Share this example with students as a case of a mathematical model that O’Neil believes is harmful. Students will need an overview of how this works, as well as the problems that arise:
In this example, police want to figure out where crimes are most likely to occur so that they can deploy their resources to those places. And so, they turn to a mathematical model, in this case to represent where and when crimes are committed. To build a model, you need measurements (like the ones used in weather models). For predictive policing, those measurements are typically crime statistics about past crimes.
One predictive policing model looks at where crimes have been committed in the past. Those data will show that crimes are not evenly spread out across a city. Cases of theft, for instance, are more likely to be reported in some places than others. Convictions for illegal drugs are similarly concentrated in certain places.
Many police departments have used those kinds of data to decide where to send their police officers. It makes sense to send those officers to neighborhoods where there have been lots of crimes in the past. Right?
But wait, let’s think that through just a little bit. There are many crimes that are not enforced because no police officer was around to see it. Traffic violations are a pretty classic example of that. If you put a bunch of police offers into a neighborhood, they will probably catch more traffic violations. Which will make it look like that neighborhood is a real hot spot for, say, speeding. But is it really? Or is that just because you’ve put a lot of police officers there?
In New York City, there was a notorious example of this called “Stop and Frisk.” Police officers were placed in specific “high crime” locations and empowered to stop and search whoever they chose. If you search a lot of people, you’re destined to find things from time to time – usually illegal drugs. But you also end up searching a lot of people who are doing nothing wrong. That’s bad enough. But the worse part is where those police officers were sent. Which neighborhoods do you suppose were subjected to “Stop and Frisk?” It wasn’t the wealthy ones, even though there is pretty good evidence that those communities actually use illegal drugs at least as often as others. No, it was predominantly non-white and low-income neighborhoods. Stop and Frisk was harmful, but it was more harmful to some people than others.
What Makes This a WMD?
After surveying what Predictive Policing is and the ways in which it is problematic, introduce the label of a “Weapon of Math Destruction.” O’Neil offers up three essential features of a WMD, and Predictive Policing fits:
Opaque: The details of how the mathematical works are not visible to the people who are affected by it. The police know what the model is doing, but the people who are harmed by it do not. In fact, those people might not even know that the model exists!
Widespread: Mathematical models can be applied to a very specific and small context, but they can also be applied to a whole city, a state, a country, even the whole world. The more widely they are used, the more potentially dangerous they become. If a single police officer used a predictive policing model, that’s one thing. If police departments all across the country use it, that’s another.
Damaging: This feature might seem somewhat obvious given the WMD name (you can’t get destruction without the ability to cause damage). It’s worth highlighting all the same. WMDs are models that can be used to cause actual and irreversible damage. Predictive policing has this ability because placing someone in jail or prison is an inherently damaging act. Putting many members of a community in jail or prison causes damage to that whole community.
Ask students to consider one of the previous examples in light of this definition: would Weather Models be considered WMDs?
Are they opaque? Only in the sense that they are complex and technical, and so you would need some specific knowledge and training to fully understand them. But if you wanted to know more about them, you could pretty easily find out.
Are they widespread? Definitely.
Do they cause irreversible damage? You could argue that when they get the weather wrong, there is harm. But on balance, being able to predict the weather generally results in far less damage being done. Most of us appreciate knowing when, say, a hurricane or tornado or hailstorm is on its way.
Key Point: Having one or even two of these features does not necessarily make a mathematical model a “bad thing.” It’s the combination of all three that is especially dangerous.
Part 2: Is This a WMD?
In this part of the inquiry, the teacher will lead students in an examination of a possible WMD that O’Neil does not address in the book. There are many possibilities, but a familiar and school-oriented one is: using standardized testing to rate and rank different public schools.
The teacher should begin by clarifying a few things about this example:
What is this model representing? People want to represent something about the quality of different schools. Some schools, we suppose, are better than others.
What makes it a “mathematical” model? We are representing “quality” using a mathematical measurement of some kind along with a set of calculations.
What is being measured? The “data” used in this model comes from students’ scores on specific standardized tests.
What decisions are made based on the model? This is very important! We wouldn’t bother talking about this if nobody paid attention to the test scores. But people pay a lot of attention to them. Parents will use test scores to decide where to send their kids to school. Low-performing schools have, at various points in time, faced consequences including funding cuts, firing of staff, even closing the school.
Students’ Task [In Small Groups]: Based on the definition of WMDs that we’ve seen, does this qualify? Work through each of the three features and develop an argument for whether that feature is present. Seek out evidence for your claims. Be prepared to present your arguments to your classmates.
Part 3: What Other WMDs Might Exist?
In this part of the inquiry, students explore a mathematical model that affects their lives and that might be a WMD. They will conduct an inquiry into that model that is similar to what they just did with standardized test scores. At the end of the inquiry, they will need to create an argument for whether that model really is a WMD.
When introducing this task, the teacher should emphasize that students might decide that it is not a WMD, and that’s fine – as long as they can defend your ideas! Below is an outline of what students will need to address.
Define Your Model
What is this model representing?
What makes it a “mathematical” model?
What is being measured?
What decisions are made based on the model?
Compare it to the WMD Definition
Is it opaque?
Is it widespread?
Is it damaging?
Imagine Alternatives
*Defusing the WMD: If you decide that it is a WMD, what changes could be made (to the model or our society) so that one (or more) of the three features was removed?
*Imagine a Dystopian Future: If you decide that it is not a WMD, what changes would have to occur (to the model or our society) so that it became one?
You can find a link to this lesson on our Curriculum page.