Meta Just Paid Nearly $17 Billion To Make Sure It Gets To Write The Kid Safety Rules For Every Other Social Media Platform

from the regulate-me-daddy dept

By now you’ve almost certainly heard the news that Meta has settled with 52 state and local Attorneys General who had sued the company in some form or another over child safety on Meta’s platforms. The headlines are all covering the basics: the years-long case these states filed against Meta ends, and Meta pays somewhere between $12.7 billion and $18 billion, depending on which document you read (the consent judgment itself caps the total at $16,680,647,753.21; Meta’s press release rounds it up to “approximately $18 billion”). Also Meta will implement a bunch of changes to its platforms with the aim of improving child safety on those platforms. It will also “encourage” YouTube and TikTok to enable the same safety features even though (bizarrely), if YouTube and TikTok follow suit, then Meta will have to pay more.

You can read the details of the proposed settlement here.

Notably, the whole point of doing this as a “settlement” is that everyone involved knows full well that no government could mandate these feature changes without violating the First Amendment. But now that it’s in a “settlement” the courts may need to explore if these choices — which Meta could make freely on its own — suddenly have become a “state action,” implicating the First Amendment.

As with the various rulings against Meta over the last few months, people are cheering this on, without realizing the damage it will do. We’ll explore why this is problematic in a moment, but just to highlight that I’m not alone in thinking so, both EFF and Fight for the Future are warning how bad this settlement is. Here’s EFF:

Under this settlement, young users will now have less access to Meta products, and a lesser ability to exercise their rights to speak, access information and art and culture, associate and form communities, and play. The settlement also embeds age assurance into every product, mandating the collection of even more personal information from users of all ages; this enshrines Meta’s harmful surveillance into law, and it will compromise users’ privacy and anonymity while increasing their exposure to data breaches and government data requests. And the data minimization and security measures don’t keep states from using data collected under the agreement for other law enforcement purposes – which could include things like criminal investigations of abortions or gender-affirming care. 

And here’s Fight’s emailed statement:

Big Tech does pose harm to our kids through its business practices and exploitation, but pushing for more censorship, age-gating, and surveillance of young people at the hands of the same Big Tech companies that have already harmed young people is not the answer. Online ID checks when implemented put vital information behind age-gates, stamp down teenagers’ right to speak, and expose all of us to even more of our data being collected, hacked, and leaked. Meta knows that managing this amount of personal information and enforcing these agegates will be messy and that’s why they are seeking to offload the burden to anyone but themselves, while being seen to comply by the public and lawmakers. Instead of actually damaging their exploitative business model, this result allows Meta to bring everyone else down with them, from app stores to other social media companies. We feared that these lawsuits would manufacture consent for invasive age verification and content controls and our fears have been proven correct. We will continue to oppose online ID checks everywhere and be on the watch for more censorship creeping into Meta’s platform.

We’ll get into the specifics of why this settlement is so bad, but first some important background. For a few decades now, when basically all Attorneys General would get together to threaten and/or sue tech companies, it was almost always over bullshit headline grabbing claims where the AGs either had no jurisdiction or ability to legally do anything. Sixteen years ago, we wrote one story about an account written by a CEO of a company who faced down dozens of state AGs who were way more concerned about the headlines they generated than actually making platforms safe.

It was similar to other stories that we’d heard, where no matter what companies did to explain to the AGs what steps they were taking to keep a platform safe the AGs would simply turn around and misrepresent what they were told, out of context, to make the platforms look worse and worse until they agreed to some sort of settlement. It happened with Craigslist. It happened with ISPs being forced to kick their users off at the behest of the recording industry. Even John Oliver has covered how grandstanding state Attorneys General will target just about anyone they want to shake down in some form or another.

That’s not to say that there aren’t righteous cases brought by Attorneys General, but there are so many examples of them being much more about getting headlines than actually making people safer. And the simple fact is that these efforts are so resource intensive, so expensive, and so draining that it’s no surprise that most companies end up “settling” by agreeing to do things that the government simply cannot force a company to do. But because it’s a “settlement” people act like it’s not the government doing it.

In this case, given some of the recent court decisions, it’s no surprise that Meta would strike some sort of settlement. As these cases continued, the headlines would only get worse for the company. And Meta deserves some bad headlines, but as I’ve discussed, many of the bad headlines in these cases involved lawyers and the media taking things way out of context. The classic case with Meta is that many of its efforts to study how to make its platforms safer were used against the company as proof that “they knew!” their platforms were unsafe!

The lesson for the rest of the tech industry is grim and unambiguous: never study whether your own platform is causing harm. The mere existence of the research will be turned into Exhibit A that “they knew,” both in the court of public opinion and in actual courts.

The other bit of background worth understanding here is that Meta has been desperately seeking a path to regulatory capture for quite some time now. It’s been practically begging for Congress to pass child safety legislation that only the largest companies (like itself) could comply with. Indeed, Meta has done this before. It went against the rest of the internet industry in embracing FOSTA, again to try to create a regulatory moat. So this shouldn’t be surprising.

Meta’s failed forays into the “metaverse” and AI have shown that it’s been pretty consistently losing the innovation race, and the government granting it a regulatory moat that smaller competitors can’t cross would be a godsend.

And it’s even better when it can be done in a way that looks like Meta “losing” a lawsuit.

So that’s what Meta gets here. They “settle” the lawsuit so the AGs and Meta haters can all claim that they’ve “protected the children.” Meta pays out over a decade — enough that it’s taking a $10 billion legal charge in Q3, which stings for a bit but will mostly be forgotten by next year. Meta can easily eat the cost. And then Meta agrees to implement a bunch of kid safety features, most of which we have no idea whether they actually protect any kids. Notably, a legislature could not have mandated most of these features without running straight into the First Amendment — but coming out of a settlement, they carry the imprimatur of law anyway (more on that in a moment), and the structure of the agreement makes it so that Meta has to actively encourage Google and TikTok to take identical steps, thereby setting in concrete what steps any platform will have to take to be considered following “best practices” and therefore acceptable to most of the country’s Attorneys General.

The specific features don’t even matter that much, but for the record:

• Time Limit: A default two-hour daily time limit that teens can only turn off with a parent’s permission. This limit is cumulative across Facebook and Instagram, and time spent scrolling on both apps counts toward the total, including if we detect that someone has multiple accounts.
• Night Mode: A default block from our apps between midnight and 6am. This means teens will not be able to post or view their Feed, Stories, Explore, or Reels, for example.
• School Mode: Notifications will be muted by default between 8 AM and 3 PM. During those hours, teens will no longer receive push notifications, except for direct messages and alerts about their account security or safety.
• Regular Prompts: Teens will receive prompts after every 15 minutes of continuous screen time on Facebook or Instagram. They’ll also receive prompts when their total daily usage hits 60 minutes and 90 minutes. These prompts are designed to encourage intentional use.

• Algorithmic Feed Control: Teens will be able to choose a non-algorithmic feed — one that isn’t personalized by our recommendation systems — as their default. We will periodically remind them of this option, and parents can choose to adjust their teen’s default experience to require this setting.
• Autoplay Control: Teens will be able to turn off autoplay, so that content no longer automatically plays. Instead, they’ll need to take a deliberate action, like a tap or swipe, to see more. Parents can choose to adjust their teen’s default experience to require this setting.
• Hidden Likes: Teens won’t see the number of likes and reactions on posts — both their own and those from others — by default.
• Disabling cosmetic surgery and extreme makeup filters: In addition to our existing policy to block teens from using cosmetic surgery filters, we’ll now block teens from using extreme makeup filters.
• Age Assurance: We work hard to find and remove underage accounts from our apps and, as part of our agreement, we’re investing in even stronger technology to proactively catch accounts that may belong to under-13s. We’re also strengthening the technology we use to identify accounts that may be between the ages of 13 and 17, so we can ensure those accounts are placed in experiences designed for teens, even if they give us an adult birthday. However, to ensure teens are consistently protected across the many apps they use, app stores must provide developers with verified age information. This will allow platforms to put age-appropriate protections in place for as many teens as possible. That’s why we’ll continue to advocate for legislation that empowers parents by requiring app stores to verify age and obtain parental approval before a teen downloads an app.
• Age-appropriate content restrictions: We will maintain our current content standards so that, by default, teens are placed into 13+ content settings, inspired by movie ratings criteria and parent feedback. We will also continue to prevent teens from following or interacting with accounts we consider age-inappropriate. We will work to continually improve these systems to ensure age-appropriate content experiences for teens.
• Unwanted contact from strangers: We will maintain our current practices of defaulting teens into private accounts on Instagram and private default settings on Facebook, and we’ll continue to restrict potentially suspicious adults from contacting them. We will also strengthen our efforts to make it harder for those adults to find, follow, or interact with teens.
• Reporting and ongoing protection from harmful content: We will continue to give teens easy ways to report content that concerns them, and we’ll work to improve our response times. We will also continue our work to protect teens from potentially harmful experiences by regularly evaluating how often teens are exposed to them. We’ll draw on research and expert input to improve our work.
• Strengthening our parental controls: We will encourage parents to set up our supervision tools and give them new controls and insights. This includes notifying parents when a teen links a secondary account, alerting them to interactions with potentially suspicious accounts, and providing periodic updates on their teen’s usage and any changes their teen attempts to make to their protective settings.

Some of those might be good features. Some of them might not be. Some of them might be good for some kids, but very bad for other kids.

Part of the problem is we really don’t know.

There is something of an accountability structure here too. Meta and the states will appoint an “independent” auditor for five years, and the age assurance system gets tested annually to meet certain thresholds. But it’s important to look at what’s actually being audited here. It’s whether or not Meta is implementing the things it’s promised to do, not whether any of those things actually work.

But now these are, effectively, mandated by law. Even though if Congress or the states had passed a law requiring these, it would almost certainly be thrown out as unconstitutional under the First Amendment.

The weirdest part of the agreement is that Meta has to try to convince Google (YouTube) and TikTok to implement some (but not all?) of these same features. Indeed, Meta has already put up a settlement-mandated open letter to those two companies asking them to implement those features.

What’s so weird is that if YouTube and TikTok agree to do this and to voluntarily throw billions of dollars at the states, then Meta also needs to pay more. The breakdown of the money Meta owes is partially dependent on them arm-twisting those two companies to do the same things:

The agreement includes a payment of approximately $18 billion, which can be used to fund youth online safety initiatives, among other state priorities. The payment will be distributed in annual installments over a 10-year period. Participating states will receive approximately 70% (approximately $12.7 billion) of the allocated payment over the decade. The remaining 30% (approximately $5.3 billion) will be released only after two specific conditions are met.

1. YouTube and TikTok implement a one-hour Daily Limit, Night Mode, and age assurance measures.
2. YouTube and TikTok each pay an amount matching the 30% figure, with half of the remaining funds tied to YouTube’s payment and half tied to TikTok’s.

You can argue that Meta might not actually want YouTube and TikTok to do this, so they won’t have to pay that extra $5.3 billion, but from a competitive standpoint, you have to think that Meta absolutely needs to have YouTube and TikTok implement these features or its already somewhat dwindling market share will dwindle faster.

It’s quite possible that YouTube and TikTok will go along with this, rather than get bogged down in a similarly costly legal fight. But, again, that would create many problems. First, we still don’t know if those feature changes are actually helpful or effective. But now they’re effectively government mandated.

In theory, this could open up room for other platforms to come in and sweep up the youth market by not implementing these same features. But the nature of this agreement is that if the state AGs suddenly feel like any platform is becoming too popular with the kids, it can point to this agreement and call it “industry standard” or “industry best practices” to insinuate that other companies not doing the same are deliberately choosing to keep kids unsafe.

Indeed, within the agreement there’s a bit of weirdness, in which Meta has to push for “industry wide adoption” which is currently defined as YouTube and TikTok, but which the agreement makes clear could include any new social media platform if such a new platform meets the thresholds. In other words, Meta is basically being forced into guaranteeing this settlement creates an industry-wide standard.

And that’s a real problem when we still don’t know how to actually help keep kids safer online. So if a web service comes up with a unique or innovative or different idea that works differently than what Meta has agreed to do, then that may be too risky to even try. Better to just follow what the AGs have “blessed” in this settlement.

As noted, we already know that some of these things are directly harmful. Age assurance is a privacy nightmare. Enshrining it as the industry standard means the end of meaningful online anonymity, and it “forces” Meta to collect more data about all of us — including adults — while handing the states a pipeline to that data for whatever else they decide it’s useful for.

That’s bad.

Also, there are some oddly specific requirements:

Meta SMPs will disable Teen Users from applying Cosmetic Procedure Filters to their content.

The agreement clarifies that this means:

… any digital filter or augmented reality (AR) effect that distorts, sculpts, redefines, or idealizes a user’s face in a way that cannot be achieved without cosmetic surgery or extreme makeup techniques.

And, sure, I can understand why such content might be unhealthy for teens. But it is, in fact, Constitutionally-protected speech. Meta could decide internally to block that speech specifically on its own platform (that’s its own editorial right). But now that it’s being done at the behest of government pressure, it almost certainly violates the First Amendment.

Also, somewhat oddly, some of the rules appear to only apply to content in English or Spanish:

With respect to Potentially Harmful Reported Content submitted in English or Spanish, Meta SMPs shall maintain processes designed to permit Teen Users to receive a response indicating Meta’s decision on the report within 6 hours in at least 90% of cases.

The implication is obvious: those are the languages most reports come in, and Meta is expected to staff up enough to clear them fast. But it also means the government has just negotiated a moderation service level that varies by the language you happen to speak — English and Spanish speakers get a six-hour guarantee, Tagalog and Mandarin speakers get whatever Meta feels like. That’s a strange thing for a state to be dictating at all.

This is also a perfect example of the kind of standard that only a giant can meet. A six-hour turnaround on 90% of reports is achievable when you have thousands of trust & safety staff and a decade of tooling. For a startup with four employees and a Discord server, it’s a fantasy — and now it’s the benchmark against which every AG will measure them.

So what happens now? The judge will need to review the settlement, but I’m actually wondering if some teenage users would have standing to challenge this. Meta is clearly restricting First Amendment protected speech under this agreement. It is free to do so on its own if it chooses to do so, but this is different. Here it’s doing so because it’s being forced to by various state AGs, making it a state action.

Under the Supreme Court’s recent (unanimous) Vullo decision, that seems pretty clearly unconstitutional. In that case,the justices said, quite clearly:

[A] government official cannot do indirectly what she is barred from doing directly: A government official cannot coerce a private party to punish or suppress disfavored speech on her behalf.

That seems like it should be the whole ballgame, because that’s what’s happening here.

One other point on all of this. Here’s the list of 52 Attorneys General that have agreed to this settlement:

Alabama, Alaska, American Samoa, Arizona, Arkansas, California, Colorado, Connecticut, Delaware, District of Columbia, Georgia, Hawaii, Idaho, Illinois, Indiana, Iowa, Kansas, Kentucky, Louisiana, Maine, Maryland, Massachusetts, Michigan, Minnesota, Mississippi, Missouri, Montana, N. Mariana Islands, Nebraska, Nevada, New Hampshire, New Jersey, New York, North Carolina, North Dakota, Ohio, Oklahoma, Oregon, Pennsylvania, Puerto Rico, Rhode Island, South Carolina, South Dakota, Tennessee, Texas, Utah, Vermont, Virginia, Washington, West Virginia, Wisconsin, and Wyoming.

Notice anyone missing? Yup. There’s no New Mexico. Remember, New Mexico won its initial case against Meta recently, enabling the judge to force a different set of feature changes on the company. So… now Meta may have certain features for New Mexico, and different features for everywhere else?

None of this is to say that Meta shouldn’t do a better job trying to protect kids on its platform. Obviously, it can certainly do more. But this settlement seems much more like Meta using this case as a way to force the industry into a set of required steps (which might not help much, and may do real harm in some cases), making it difficult for smaller competitors to enter the market, and giving them a bit of regulatory capture through mass lawsuit settlement.

As law professor Jess Miers wrote, this settlement is about Meta selling out the entire social media industry, forcing them to embrace impractical and unhelpful features that serve only to lock in giants and lock out upstarts:

But for all the people “celebrating” this as a win “against” Meta, you’ve been fooled. Meta just cut a deal to put itself in charge of how social media works going forward. As Justin Maurer wrote on Bluesky, this is Meta taking a “please regulate me Daddy” approach to the government, and getting exactly what it really has been asking for.

We still don’t have any actual evidence that this will help anyone, let alone every kid. The state AGs didn’t have to prove how this would help kids. Meta didn’t have to prove it. The judge won’t be asked to. It’s just taken on faith. Meta offered this up, the AGs okayed it… and it all becomes a grand experiment on kids.

You can argue that these feature changes sound like they should help kids. Limiting access to two hours a day (unless parents grant more, which many will), lights out at midnight, disappearing like counts — these all sound like they’ll help some kids. But if it turns out that locking kids out of these systems actually pushes the most vulnerable ones to darker places with no trust & safety team at all, you won’t hear about that from Meta or the AGs.

We just spent three years teaching the entire industry that if you do research on child safety, you’ll have it held against you. Do we really think that all of this is going to actually enable anyone to figure out what works to help actual kids?

Meta bought itself a moat. The AGs bought themselves headlines that will be useful next election season. And every teenager in the country was just automatically enrolled in an untested experiment. There’s a five year independent auditor requirement to confirm that Meta follows the rules. But not to see if the rules work.

Filed Under: child safety, kid safety, settlements, state ags, states

Companies: meta, tiktok, youtube