AI Companion Moderation Explained: Who Reviews Reports and What Gets Removed

When you hit "report" in an AI companion app, or when a companion suddenly refuses to continue, you're brushing up against a moderation system most users never see. Moderation is the machinery that decides what content is allowed to exist on the platform, what gets removed, and what happens to accounts that break the rules. Knowing roughly how it works makes the app's behavior far less mysterious — and helps you tell a well-run platform from a careless one.

The two layers: automated and human

Almost all companion moderation is a mix of two layers working together, because neither alone is enough at scale.

Automated moderation runs first and handles the overwhelming majority of content. Classifiers scan messages and generated media in real time and act instantly — blocking a reply, refusing an image request, or flagging an exchange for later review. It's fast and tireless but blunt: it misreads context, produces false positives, and can be inconsistent, which is why refusals sometimes feel arbitrary.

Human moderation handles what automation can't: appeals, edge cases, reports that need judgment, and patterns that only make sense with context. Human reviewers are slower and see far less volume, but they're where nuanced decisions actually get made. On responsible platforms, humans also audit the automated system to catch where it's being too strict or too lax.

Who actually reviews your report

When you file a report, it doesn't usually go straight to a person. A typical flow looks like this:

• Your report enters a queue and is often triaged automatically by severity and category.
• Low-risk or clearly automatable cases may be resolved by the system without a human ever looking.
• Higher-severity reports — anything involving safety, minors, real people, or credible harm — get escalated to a human moderation or trust-and-safety team.
• The most serious categories can trigger separate, urgent processes, sometimes involving specialized teams or outside authorities.

Who those humans are varies: an in-house trust-and-safety team, an outsourced moderation vendor, or a mix. Larger or more responsible platforms tend to have dedicated teams and published processes; smaller or shadier apps may have almost no human review at all, which is a meaningful quality and safety signal.

What actually gets removed

Removal decisions cluster into a few tiers, roughly by how firm the line is:

• Always removed, no exceptions — sexual content involving minors or anything resembling it, credible threats, and content built around real, identifiable private individuals without consent. These are hard lines on any legitimate platform.
• Removed by policy — content that violates the app's specific terms, which differ widely: some apps permit adult content for verified users, others prohibit it entirely.
• Blocked in the moment — a lot of "moderation" isn't removal at all but real-time refusal, where the companion simply won't generate something. This overlaps heavily with how content filters and sensitive modes work.

It's worth understanding that "removed" can mean several different things: a message blocked before it's sent, content deleted after the fact, a feature disabled, or an entire account suspended.

What happens to the account

Moderation doesn't only act on content — it acts on accounts. Depending on severity and history, consequences range from a warning, to loss of specific features, to temporary suspension, to a permanent ban. Serious, illegal categories can go beyond the platform entirely. Reputable apps tend to publish these consequences and offer an appeal path; the absence of any stated process is itself informative.

Why moderation feels inconsistent

Users often experience moderation as unpredictable, and there are real reasons for that. Automated classifiers misjudge context. Policies get updated, sometimes silently, so the same content is allowed one week and gone the next. Human reviewers apply judgment that varies. And payment processors or app stores can force policy changes from the outside. This is the same inconsistency that shows up in filtering, and it's usually a sign of many systems interacting rather than one clear rulebook.

What this means for you

A few practical takeaways when you're evaluating or using an app:

• Reporting works better on serious things. High-severity reports are the ones most likely to reach a human and produce action; expect less from reports on minor annoyances.
• Published moderation and appeal processes are a green flag. They signal a platform that takes trust and safety seriously — part of the broader due diligence in our review template.
• No visible moderation is a red flag. Apps that advertise "anything goes" and show no signs of real oversight often overlap with the shady operators in our guide to AI romance scams and fake companions.

The bottom line

Moderation is the invisible layer that makes an AI companion app livable: automated systems handling volume, humans handling judgment, and a set of removal tiers from hard legal lines down to in-the-moment refusals. You can't see most of it, but you can judge it — by whether the platform publishes its rules, offers appeals, and shows evidence that real people are minding the store.