New research funded by the Pentagon suggests that artificial intelligence can scan and analyze blocks of text to discern whether the humans who wrote it have done something wrong or not.
David Byrne on New Tech and AI | Gizmodo Interview
The paper, written by two researchers at Ben-Gurion University, leverages predictive models that can analyze messages for what they call “social norm violations.” To do this, researchers used GPT-3 (a programmable large language model created by OpenAI that can automate content creation and analysis), along with a method of data parsing known as zero-shot text classification, to identify broad categories of “norm violations” in text messages. The researchers break down the purpose of their project like this:
While social norms and their violations have been intensively studied in psychology and the social sciences the automatic identification of social norms and their violation is an open challenge that may be highly important for several projects…It is an open challenge because we first have to identify the features/signals/variables indicating that a social norm has been violated…For example, arriving at your office drunk and dirty is a violation of a social norm among the majority of working people. However, “teaching” the machine/computer that such behavior is a norm violation is far from trivial.
Of course, the difficulty with this premise is that norms are different depending on who you are and where you’re from. Researchers claim, however, that while various cultures’ values and customs may differ, human responses to breaking with them may be fairly consistent. The report notes:
While social norms may be culturally specific and cover numerous informal “rules”, how people respond to norm violation through evolutionary-grounded social emotions may be much more general and provide us with cues for the automatic identification of norm violation…the results [of the project] support the important role of social emotions in signaling norm violation and point to their future analysis and use in understanding and detecting norm violation.
Researchers ultimately concluded that “a constructive strategy for identifying the violation of social norms is to focus on a limited set of social emotions signaling the violation,” namely guilt and shame. In other words, the scientists wanted to use AI to understand when a mobile user might be feeling bad about something they’ve done. To do this, they generated their own “synthetic data” via GPT-3, then leveraged zero-shot text classification to train predictive models that could “automatically identify social emotions” in that data. The hope, they say, is that this model of analysis can be pivoted to automatically scan text histories for signs of misbehavior.
Somewhat unsettlingly, this research was funded by the Pentagon’s Defense Advanced Research Projects Agency (DARPA). Created in 1958, DARPA has been at the forefront of U.S. military research and development for the better part of a century, frequently helping to create some of the most important technological innovations of our time (see: drones, vaccines, and the internet, among many others). The agency funds a broad diversity of research areas, always in the hopes of finding the next big thing for the American war machine.
Ben-Gurion researchers say their project was supported by DARPA’s computational cultural understanding program—an initiative with the vague mandate of developing “cross-cultural language understanding technologies to improve a DoD operator’s situational awareness and interactional effectiveness.” I’m not 100 percent sure what that’s supposed to mean, though it sounds (basically) like the Pentagon wants to create software that can analyze foreign populations for them so that, when the U.S. inevitably goes to war with said populations, we’ll understand how they’re feeling about it. That said, why DARPA would specifically want to study the topic of “social norm violation” is a bit unclear, so Gizmodo reached out to the agency for additional context and will update this story if it responds.
In essence, the research seems to be yet another form of sentiment analysis—an already fairly well-traversed area of the surveillance industrial complex. It’s also yet another sign that AI will inexorably be used to broaden the U.S. defense community’s powers, with decidedly alarming results.
