What is AI-Generated Content (AIGC)?

AI-generated content is text, image, audio, or video produced by a machine learning model rather than written, drawn, filmed, or recorded by a human. The term became mainstream in late 2022 with the public release of ChatGPT and Stable Diffusion, but the underlying category is broader: anything a generative model produces, from a three-line tweet to a full feature film sequence, falls inside it.

What counts as AI-generated

In practice, the line between AI-generated and human-written is already blurry.

A journalist who runs their draft through a grammar model. A designer who uses Midjourney to create a placeholder that ships. A developer whose IDE autocompletes entire functions. All of these produce content that is partly synthetic. Regulators and platforms have landed on a few working definitions: the EU AI Act treats content as AI-generated if a model "meaningfully" produced it, and requires labeling of synthetic media that could be mistaken for real. Meta, TikTok, and YouTube have all introduced disclosure requirements for synthetic audio and video of real people.

Why it matters for moderation

Generative models change the economics of abuse. Writing convincing spam used to take effort. Producing a deepfake video used to require skill and hours of compute. Both are now a few API calls away, and this shifts the defender's problem from filtering individual bad posts to handling abuse at a volume no human review queue can absorb.

The specific categories where AI-generated content causes the most trouble:

• Non-consensual intimate imagery, including AI-generated sexual images of real people. NCMEC reported a sharp increase in AI-generated CSAM cases in 2023 and 2024, and several US states have passed laws criminalizing synthetic intimate imagery.
• Election and political misinformation, where cloned voices and deepfake video of candidates are used to spread fabricated statements.
• Fraud and impersonation, including voice cloning used in CEO scams and investment fraud.
• Low-quality spam and SEO content, which is not illegal but degrades search results and floods user-generated content platforms.

Detection, in brief

Reliable detection of AI-generated text is an open problem.

The best classifiers get high accuracy on known model families and degrade quickly on new ones, and they produce false positives on non-native English writing, which has already caused real harm in academic settings. Image and video detection is easier because synthesis artifacts are more consistent, but it is still an arms race. The working consensus is that detection alone cannot be the answer. Platforms are moving toward provenance instead: cryptographic content credentials (C2PA), watermarking at the model output stage, and metadata that travels with a file through editing and re-upload. These approaches don't answer whether a given piece of content is AI-generated, they answer where it came from, which is usually the question that actually matters.

What platforms are doing

Most large platforms now require users to disclose when they upload realistic synthetic media. Some apply automated labels on top, based on classifier signals or embedded watermarks. Detection is paired with policy: a synthetic video of a real public figure saying something they didn't say is handled very differently from an obviously stylized AI illustration.

The goal is not to ban generative content, which would be both impossible and undesirable, but to give viewers enough context to judge what they are looking at.