Many major social media safety lawsuits come down to two things: what platforms knew, and what they decided to do about it. A confidential TikTok document brought this exact issue into sharp focus. In late 2021, TikTok built an algorithmic safety patch to stop users from seeing endless videos about self-harm, suicide and depression. It then deliberately withheld the fix from approximately 15 million people, or around 10% of its US user base, just to see how safety impacted engagement numbers.
Among those users was Chase Nasca, a 16-year-old whose account the document describes as receiving an “onslaught” of suicide and self-harm videos. The document states: “TikTok’s filter bubble prevention strategies did not take effect on this user by design.” He died in February 2022. His family’s case is now among more than 2,600 pending lawsuits TikTok faces, and the document has become central evidence in the litigation.
The document focuses on TikTok’s US deployment, but the platform operates in over 150 countries. Algorithmic testing affecting users outside the US is a question the litigation hasn’t yet addressed.
What “By Design” Actually Means
The phrase carries weight here. The focus shifts away from basic claims that TikTok’s algorithm accidentally served up harmful material, a well-documented issue already facing heavy litigation. The documents show the company created a safer system, chose to hold it back from millions of users and clearly framed that move as a calculated testing method. The internal report describes the exclusion as necessary to “strike a delicate balance between safety and the ability to measure impact on DAU and core metrics”, where DAU means daily active users.
To run a valid A/B test on whether the safety feature affected engagement, TikTok needed a control group that continued receiving recommendations from the older system. That is a standard experimental design choice. It’s also, in this context, a choice to leave 15 million people on an algorithm the company had already judged higher-risk. That remained the case for as long as the test continued.
The document notes that after the death, TikTok recommended shrinking the control group and shortening test durations. It also notes that even a 1% pool would leave at least 1.8 million US users alone in the unprotected group at any given time. By early 2022, the safeguarded algorithm had reached around 90% of US users. The remaining 10% remained in the control group.
More from News
- The UK Government Wants Your Encrypted Data And Apple Is Fighting Back
- Seriös Group Launches New Research Exposing ‘Chief Information Overload’ Facing UK Data Leaders
- From Driverless Cars To Parentless Parenting: Is Sam Altman Running Out Of Things To Outsource?
- How Does Google Intend On Producing 15 Million AI Chips Over The Next Two Years?
- TikTok Parent ByteDance Has Officially Passed The $4 Billion Mark In AI Revenue
- Russia’s Latest Move Against Pavel Durov Shows That Telegram Is No Longer Just A Messaging App
- Experts React To The UK’s Decision To Make Tech Subjects Compulsory In Schools
- Microsoft Has Confirmed Copilot’s Super App Will Launch Soon – But What Is It For?
Why This Changes The Legal Picture
TikTok is defending more than 2,600 consolidated cases in US courts covering claims from wrongful death to product liability and design defect. The legal theories in most of these cases depend on establishing that TikTok knew its product could cause serious harm to users, particularly young users, and failed to act adequately. The leaked document addresses both sides of that test.
On foreseeability: the document shows TikTok not only understood that its recommendation algorithm could trap users in harmful content loops, but had developed a specific technical intervention to address it. Knowing the risk existed is harder to dispute when the company’s internal team built a tool to mitigate it.
On reasonable action: the document shows the mitigation was deployed to 90% of users but withheld from 10% by design choice, not accident, not technical limitation and not resource constraint. The reason given is the need to preserve measurement capability for engagement metrics. Courts will decide how to weigh that trade-off. For the plaintiffs’ lawyers across 2,600 cases, the document provides something rarely available in platform litigation. That’s the company’s own internal characterisation of a conscious decision to limit a safety feature.
How TikTok Has Responded
TikTok’s official statement paints the test as a routine comparative evaluation of safety controls, confirming that internal procedures have improved since. Crucially, the company hasn’t challenged the authenticity of the report.
The wider dynamic shown here, balancing platform engagement against safety controls, is a challenge shared across the entire industry. What’s unusual about this document is that it records the trade-off explicitly, in the company’s own language, in a case where a named user in the unprotected group died.
This has been described as among the most damaging internal leaks in TikTok’s litigation history. Based on what the document contains, that assessment is difficult to argue with.
