Content Moderation At Scale Is Impossible: Recent Examples Of Misunderstanding Context

I've said over and over and over again that content moderation at scale is impossible to do well, and one of the biggest reasons for this is that it's difficult to understand context. Indeed, I've heard that some content moderation policies explicitly suggest that moderators don't try to take context into account. This is not because companies don't think it's important, but the recognition that understanding the context behind every bit of content, would make the process so slow as to be absolutely useless. Yes, it would be great if every content moderator had the time and resources to understand the context of every tweet or Facebook post, but the reality is that we'd then need to employ basically every human being alive to be researching context. Low level content moderators tend to only have a few seconds to make decisions on content, or the entire process slows to a crawl, and then the media will slam those companies for leaving "dangerous" content up too long. So tradeoffs are made, and often that means that understanding context is a casualty of the process. And, of course, that leads to some ridiculous (and hilarious) results. Here are three recent ones that came across my radar. First, someone marketing a tabletop roleplaying game discovered that when they tried to purchase ads on Facebook and Instagram marketing their TTRPG, they had their entire account shut down because they used the word "supplement."

The confusion here is not hard to figure out. First off, lots of roleplaying games have "supplements," or variations/adjustments/add-ons. However... "supplements" also refer to dietary supplements, a market filled with often highly questionable things that people put into their bodies with promises of... well... all sorts of things. And, making matters even worse (as I just discovered!) there's actually a dietary supplement called "RPG" so the Google searches are, well... complex.

And, to make matters even more complex, you may recall that a decade ago, the Justice Department got Google to hand over $500 million for displaying ads for non-approved drugs and supplements. So, I'm sure that both Facebook and Google are extra sensitive to any advertisement that might contain sketchy drugs and supplements -- and thus the rules are designed to be overly aggressive. To them, it's worst case, you shut down an account advertising a role playing game... which is better than having the DOJ show up and confiscate $500 million.

That's not to say this is a good result -- but to explain what likely happened on the back end.

Next up, we have Kate Bevan, who wrote about another content moderation fail on Facebook:

Again, the context here seems fairly obvious. Commenting about a picture of a cat and saying "beautiful puss" is... um... referring to a cat. Not anything else. But, again, in these days when companies are getting sued for all kinds of "bad" things online, you can see why a content moderator having to make a decision in seconds might get this one wrong.

Finally, we've got one that hits a little closer to home. Many of you may be familiar with one of our prolific commenters, That Anonymous Coward (or TAC, for short) who also is a prolific Twitter user. Or was until about a week ago when his account got suspended. Why did his account get suspended? Because of a reply he made to me! Chris Messina had tweeted following the news that