Sujata Mukherjee heads the Trust & Safety research team at Google. Jan Eissfeldt serves as Director and Global Head of Trust & Safety at the Wikimedia Foundation, the non-profit hosting Wikidata, Wikipedia, and other free knowledge projects.

Shutterstock

Trust & Safety policymaking is becoming a multi-dimensional balancing act: between sweeping “universalization” of global policies and excessive “balkanization” of and within individual countries/locales; between scalable, federated moderation processes and distributed, community-reliant systems; between business interests and societal impact. Compounding this complexity is the issue of measurement: the impact of Trust & Safety policies is often indirect, can take time to materialize and unfolds across complex, entangled systems. In this commentary, we review the forces shaping the industry’s balancing act and the challenge of building agile, resilient systems that are future-ready.

A new field emerges

Trust & Safety (T&S) is an emerging field in the tech industry that has grown rapidly in recent years. T&S is used to refer to the teams and corporate function(s) at internet companies and service providers dedicated to ensuring that users are protected from unwanted or harmful experiences. Key functions include Policy, which is the team developing the company’s rules to govern its products, platforms and services, and Enforcement or Operations, the teams that work on ensuring that said rules are followed. Larger platform providers usually have whole ecosystems of support functions for these two foundational elements, including engineering and machine learning teams that build tools and models used for enforcement; data science professionals helping with metrics and analytical research; threat intelligence and enterprise management professionals; attorneys, public policy, and communications folks dedicated to the topic area in-house; and often outsourcing providers and software vendors complementing existing capabilities required to run scaled, global operations.

This growth in T&S over the last decade is due in part to the increasing importance of online platforms in people’s lives. People rely on digital platforms for everything from communication to commerce, and expect them to be safe and trustworthy. Most internet platforms are now aware of product safety principles and social benefits as revenue enablers, not least because there is accelerating pressure from regulators and consumers to do more to protect users from harm. Most platforms realize that investing in Trust & Safety can lead to long-term benefits, such as improved user engagement and loyalty. 

However, there are many headwinds facing the T&S industry. Based on our combined two decades of experience working on T&S, we seek to provide a comprehensive mapping of the issues influencing the industry at this moment in time. Our hope is that a situational summary of this type will help to ensure that proposed solutions are grounded in a solid understanding of the sociotechnical context in which this field operates. Finally, this article highlights the speed of evolution in the field, particularly the role of technology drivers. This is important because solutions that were effective in the past may not continue to be so in the future, highlighting the need for future-proof practices, processes, and infrastructure that form the foundation of the field. 

Market Forces

Role in Society

Many platforms were created with the promise of the free and open web, to host and extend participation, expression, and connection across the world. But as these platforms grew, the challenges of free interaction and content creation became apparent: illegality, nonconsensual pornography, violence, abuse, and hate. Platforms found themselves needing to moderate to prevent criminal activity, and increasingly, to protect users from each other, prevent the marginalization of already vulnerable groups, and safeguard the integrity of their products and services. 

The conversation about the impact of technology has expanded to include not just the impact to individuals, but also impacts on society and geopolitical relationships. Political and social movements are increasingly influenced and amplified by online conversations, and a series of world events including the US elections, UK Brexit Referendum, India Demonetization– all in 2016– made this phenomenon painfully clear. This has also caused a paradigm shift in the perception of a platform’s “responsibi