TikTok empowered these plus-sized women, then took down some of their posts. They still don’t know why

Scrolling through videos on TikTok, the short-form video app especially popular among teens and 20-somethings, there’s no shortage of scantily clad women and sexually suggestive content. So when curvier influencers like Bader and Touma post similar videos that are then removed, they can’t help but question what happened: Was it a moderator’s error, an algorithm’s error or something else? Adding to their confusion is the fact that even after appealing to the company, the videos don’t always get reinstated.

Content-moderation issues aren’t limited to TikTok, of course, but it’s a relative newcomer compared to Facebook, Twitter, and others that have faced blowback for similar missteps for years. Periodically, groups and individuals raise concerns that the platforms are inappropriately and perhaps deliberately censoring or limiting the reach of their posts when the truth is far less clear. In the case of the plus-sized influencers, it’s not evident whether they’re being impacted more than anyone else by content takedowns, but their cases nonetheless offer a window to understand the messy and sometimes inconsistent content moderation process.

The murkiness of what actually happened to these influencers highlights both the mystery of how algorithms and content moderation work and also the power that these algorithms and human moderators — often working in concert — have over how we communicate, and even, potentially, over whose bodies have a right to be viewed on the internet. Those in the industry say likely explanations range from artificial intelligence bias to cultural blindspots from moderators. But those outside the industry feel left in the dark. As Bader and Adore Me found, posts can disappear even if you believe you’re following the rules. And the results can be confounding and hurtful, even if they’re unintentional.

“It’s frustrating for me. I have seen thousands of TikTok videos of smaller people in a bathing suit or in the same type of outfit that I would be wearing, and they’re not flagged for nudity,” Touma said. “Yet me as a plus sized person, I am flagged.”

A sense of not knowing is pervasive

For years, tech platforms have relied on algorithms to determine much of what you see online, whether it’s the songs Spotify plays for you, the tweets Twitter surfaces on your timeline, or the tools that spot and remove hate speech on Facebook. Yet, while many of the big social media companies use AI to complement the experience their users have, it’s even more central to how you use TikTok.

TikTok’s “For You” page, which relies on AI systems to serve up content it thinks individual users will like, is the default and predominant way people use the app. The prominence of the “For You” page has created a pathway to viral fame for many TikTok users, and is one of the app’s defining features: Because it uses AI to highlight certain videos, it occasionally enables someone with no followers to garner millions of views overnight.

But TikTok’s choice to double down on algorithms comes at a time of widespread concerns about filter bubbles and algorithmic bias. And like many other social networks, TikTok also uses AI to help humans sift through large numbers of posts and remove objectionable content. As a result, people like Bader, Kondratink and Touma who have had their content removed can be left trying to parse the black box that is AI.

TikTok told CNN Business that it doesn’t take action on content based on body shape or other characteristics, as Adore Me alleges, and the company said it has made a point of working on recommendation technology that reflects more diversity and inclusion. Furthermore, the company said US-based posts may be flagged by an algorithmic system but a human ultimately decides whether to take them down; outside the United States, content may be removed automatically.

AI experts acknowledge that the processes can seem opaque in part because the technology itself isn’t always well understood, even by those who are building and using it. Content moderation systems at social networks typically use machine learning, which is an AI technique where a computer teaches itself to do one thing — flag nudity in photographs, for instance — by poring over a mountain of data and learning to spot patterns. Yet while it may work well for certain tasks, it’s not always clear exactly how it works.

“We don’t have a ton of insight a lot of times into these machine learning algorithms and the insights they’re deriving and how they’re making their decisions,” said Haroon Choudery, cofounder of AI for Anyone, a nonprofit aimed at improving AI literacy.

But TikTok wants to be the poster child for changing that.

A look inside the black box of content moderation

Eventually, the company said, guests at actual offices in Los Angeles and Washington, D.C. “will be able to sit in the seat of a content moderator, use our moderation platform, review and label sample content, and experiment with various detection models.”

“TikTok’s brand is to be transparent,” said Mutale Nkonde, a member of the TikTok advisory council and fellow at the Digital Civil Society Lab at Stanford.

Even so, it’s impossible to know precisely what goes into each decision to remove a video from TikTok. The AI systems that large social media companies rely on to help moderate what you can and can’t post do have one major thing in common: They’re using technology that’s still best suited to fixing narrow problems to address a problem that is widespread, ever changing, and so nuanced it can even be tricky for a human to understand.

Because of that, Miriam Vogel, president and CEO of nonprofit EqualAI, which helps companies decrease bias in their AI systems, thinks platforms are trying to get AI to do too much when it comes to moderating content. The technology is also prone to bias: As Vogel points out, machine learning is based on pattern recognition, which means making snap decisions based on past experience. That alone is implicit bias; the data that a system is trained on and a number of other factors can present more biases related to gender, race, or many other factors, as well.

“AI is certainly a useful tool. It can create tremendous efficiencies and benefits,” Vogel said. “But only if we’re conscious of its limitations.”

For instance, as Nkonde pointed out, an AI system that looks at text that users post may have been trained to spot certain words as insults — “big”, “fat”, or “thick”, perhaps. Such terms have been reclaimed as positive among those in the body positivity community, but AI doesn’t know social context; it just knows to spot patterns in data.

Furthermore, TikTok employs thousands of moderators, including full-time employees and contractors. The majority are located in the United States, but it also employs moderators in Southeast Asia. That could result in a situation where a moderator in the Philippines, for instance, may not know what body positivity is, she said. So if that sort of video is flagged by AI, and is not part of the moderator’s cultural context, they may take it down.

Moderators work in the shadows

It remains unclear exactly how TikTok’s systems misfired for Bader, Touma and others, but AI experts said there are ways to improve how the company and others moderate content. Rather than focusing on better algorithms, however, they say it’s important to pay attention to the work that must be done by humans.

Liz O’Sullivan, vice president of responsible AI at algorithm auditing company Arthur, thinks part of the solution to improving content-moderation generally lies in elevating the work done by these workers. Often, she noted, moderators work in the shadows of the tech industry: the work is outsourced to call centers around the world as low-paid contract work, despite the often unsavory (or worse) images, text, and videos they’re tasked with sorting through.

To fight unwanted biases, O’Sullivan said a company also has to look at every step of building their AI system, including curating the data that’s used to train the AI. For TikTok, which already has a system in place, this may also mean keeping a closer watch on how the software does its job.

Vogel agreed, saying companies need to have a clear process not just for checking AI systems for biases, but also for determining what biases they’re looking for, who’s responsible for looking for them, and what kinds of outcomes are okay and not okay.

“You can’t take humans outside of the system,” she said.

If changes aren’t made, the consequences may not just be felt by social media users, but also by the tech companies themselves.

“It lessened my enthusiasm for the platform,” Kondratink said. “I’ve contemplated just deleting my TikTok altogether.”