Countering Violent Extremism: The FBI and the Department of Homeland Security Need to Develop Strategies and Objectives for Sharing Threat Information with Social Media and Gaming Companies

Key Chapter Title List

1. Background: Definition of Domestic Violent Extremism and Threat Categories
2. Roles and Responsibilities of Federal Agencies in Addressing Domestic Violent Extremism
3. Social Media and Gaming Companies and Their Legal Considerations
4. Utilization of Online Platforms by Domestic Violent Extremists for Radicalization, Recruitment, and Mobilization
5. Selected Company Reports on Various Tools Used to Mitigate Content Promoting Domestic Violent Extremism
6. Impact of Company Leadership, Financial Considerations, and Evasion Efforts on Content Mitigation
7. Mechanisms for the FBI and DHS to Share and Receive Information with Social Media and Gaming Companies
8. The FBI and DHS Have Not Established Objectives or Strategies for Sharing Information on Domestic Violent Extremists with Social Media and Gaming Companies
9. Conclusion
10. Recommendations for the Executive Branch
11. Agency Comments
12. Appendix I: Experts and Their Affiliations Interviewed by the Government Accountability Office
13. Appendix II: Perspectives of the FBI and DHS on Addressing Online Content Promoting Domestic Violent Extremism
14. Appendix III: Efforts by Online Platforms to Prevent and Combat Violent Extremism

Document Introduction

This report is a special audit report issued by the U.S. Government Accountability Office in January 2024, in response to a congressional request, focusing on the issue of domestic violent extremists utilizing social media and gaming platforms. The report aims to examine the current state of domestic violent extremism's exploitation of online spaces, evaluate the content governance practices of major platform companies, and review the information-sharing mechanisms between two core counterterrorism agencies—the Federal Bureau of Investigation (FBI) and the Department of Homeland Security (DHS)—and the private sector. The research background is based on the fact that several high-profile attacks in recent years have been influenced by online content promoting domestic violent extremism, and that the FBI and DHS have identified primarily online-radicalized lone offenders and small groups as one of the most significant terrorism threats currently facing the United States.

The report first defines the operational definition of domestic violent extremism and lists five categories of domestic terrorism threats jointly determined by the FBI and DHS, including racially or ethnically motivated, anti-government or anti-authority motivated, animal rights or environmental motivated, abortion-related motivated, and other undefined threats. The report clarifies the FBI's role as the lead agency for federal terrorism investigations and domestic intelligence work, and the DHS's (specifically its Office of Intelligence and Analysis) core role in collecting, analyzing, and sharing information on emerging terrorism threats. Simultaneously, the report notes that private social media and gaming companies are not directly bound by the First Amendment and can autonomously manage platform content according to their terms of service, typically protected from liability under Section 230 of the Communications Decency Act.

Through interviews with 16 experts from research institutions, advocacy organizations, and academia, as well as in-depth research on five selected social media and gaming companies (including Discord, Reddit, Roblox, and two anonymous companies), the report reveals the multiple purposes for which domestic violent extremists utilize online platforms: widespread dissemination of information, integration of extremist ideologies into the mainstream, and radicalization, recruitment, and mobilization of others. Their strategies are highly targeted; for example, using diluted rhetoric on large public platforms to attract attention, then moving recruitment targets to more private or encrypted platforms for in-depth communication and action planning. Online games and their related platforms are used to build trust and social bonds, creating social spaces for the spread of extremist ideologies.

The report details a series of content moderation tools adopted by selected companies to identify and remove content promoting domestic violent extremism, including terms of service explicitly prohibiting such content, machine learning tools for scanning violative content, user and trusted reporter reporting mechanisms, human trust and safety team reviews, and design features intended to suppress violative content (such as like/dislike systems). However, efforts at the company level are constrained by various factors, including leadership preferences, financial considerations (such as advertising revenue incentives to maintain user engagement), uneven resource allocation, and the evolving evasion strategies of violent extremists, such as using coded language, modifying audiovisual content, and cross-platform migration, making content moderation a whack-a-mole-like challenge.

Although the FBI and DHS have established various mechanisms for sharing and receiving domestic violent extremism threat information with social media and gaming companies—including participation in non-governmental organizations like the Global Internet Forum to Counter Terrorism, appointing industry liaison program managers, hosting annual private sector partnership meetings, establishing local contacts through field offices, accepting and reviewing reports, and issuing joint intelligence products—the report's core finding indicates that neither agency has developed an overarching information-sharing strategy that clearly articulates how to identify and select partner companies, nor has it set specific objectives and expected outcomes. The lack of strategy and objectives prevents the agencies from comprehensively assessing the effectiveness of their communications with companies and makes it difficult to ensure that resource investments align with the overall counterterrorism mission.

Based on the above findings, the U.S. Government Accountability Office concludes that establishing program objectives and strategies is crucial to ensuring that information-sharing mechanisms serve the agencies' core missions. Therefore, the report makes one recommendation each to the Director of the FBI and the Under Secretary for Intelligence and Analysis at DHS, requesting them to develop strategies and objectives for sharing domestic violent extremism-related information with social media and gaming companies. Both DHS and the Department of Justice, after reviewing the draft report, agreed with the recommendations. Among them, the DHS Office of Intelligence and Analysis plans to complete the development of the relevant strategy by June 2024. This report provides policymakers, law enforcement agencies, the intelligence community, platform companies, and researchers in related fields with an authoritative assessment and action guide regarding the ecosystem of online violent extremism threats and the current state and challenges of public-private collaboration.