Community Management

Abuse detection is the process of identifying harmful or abusive user-generated content — such as harassment, threats, or hate speech — using automated tools, human moderators, or a combination of both.

An appeal process is a mechanism that lets a user contest a moderation decision and request that the removal, restriction, or account action be reviewed again. Under the EU Digital Services Act, providing an accessible internal complaint-handling system is a mandatory user right.

Astroturfing is a coordinated campaign disguised as a spontaneous grassroots movement, where paid or organized actors pose as ordinary users to push a political, commercial, or ideological message. It typically relies on networks of sock puppet accounts to manufacture the appearance of broad public support.

Banning is the act of permanently revoking a user's access to a platform or service after they have violated the community guidelines or terms of service. Bans may be enforced by account, IP address, device fingerprint, or payment method.

Catfishing is the practice of creating a fake online persona to deceive another person into a romantic or emotional relationship, often as a prelude to financial fraud, blackmail, or manipulation. Detection typically combines image reverse-search, behavioral signals, and conversation analysis.

Chat moderation is the practice of monitoring and managing real-time conversations — in messaging apps, game lobbies, or live streams — to enforce community guidelines, prevent harassment, and stop spam as messages are sent.

Community guidelines are a set of rules and standards published by a platform that define what behavior and content are acceptable, giving users and moderators a shared framework for keeping discussions safe and respectful.

Content flagging is a feature that lets users report posts, comments, or media they find inappropriate or harmful, sending the item to a moderation queue where moderators can review it and take action.

Content moderation is the practice of monitoring and managing user-generated content on a platform to ensure it adheres to community guidelines and legal standards, using a mix of automated tools and human reviewers.

Content review is the process of examining flagged or reported content to determine whether it violates a platform's community guidelines or policies, performed by human moderators, AI classifiers, or a hybrid review queue.

Coordinated inauthentic behavior is a term for networks of fake or compromised accounts that work together to mislead users about their identity and purpose, often to manipulate political discourse, amplify narratives, or run influence operations. Detection focuses on cross-account signals such as shared infrastructure, posting patterns, and content overlap.

Cyberbullying is the use of digital communication tools — social media, messaging apps, comments, or forums — to repeatedly harass, threaten, intimidate, or humiliate another person, often requiring moderator intervention or reporting to authorities.

Disinformation is false information that is deliberately created and spread to deceive, manipulate, or cause harm. It is the raw material of influence operations and coordinated inauthentic behavior, and is distinguished from misinformation by the intent of whoever originates it.

Doxxing is the act of publicly sharing someone's private personal information — such as home address, phone number, workplace, or identity documents — without their consent, typically to enable harassment, intimidation, or real-world harm. Most major platforms treat doxxing as an immediate takedown offense.

Flagging is the act of marking a piece of content for moderator review, typically by a user who has encountered something they find inappropriate, harmful, or in violation of the platform's community guidelines.

Hate speech is content that promotes violence, discrimination, or hostility toward individuals or groups based on protected attributes such as race, religion, ethnicity, national origin, gender, sexual orientation, or disability.

Human in the loop is a moderation approach where AI handles bulk decisions but escalates low-confidence or borderline cases to human reviewers, combining the speed of automation with the judgment of trained moderators.

Human moderation is the practice of having trained people — rather than algorithms — review user-generated content for policy violations. It is essential for nuanced cases involving context, sarcasm, or cultural specifics that AI struggles with.

Manual review is the process of a human moderator examining a piece of content directly — rather than relying on an automated system — to decide whether it violates community guidelines or policies.

Misinformation is false or misleading information that is shared without the intent to deceive — the person spreading it believes it is true. It is distinct from disinformation, which is deliberately fabricated, though the two can blur as content travels through a network.

A moderation queue is the prioritized list of flagged or reported content waiting for moderator review. Modern queues sort items by severity, AI confidence score, and virality so the most urgent cases are handled first.

Offensive content is user-generated material likely to upset or alienate readers — including hate speech, harassment, slurs, graphic violence, and sexually explicit imagery — even when it does not necessarily break the law.

Proactive moderation is the practice of detecting and acting on policy violations before users report them, typically using AI classifiers that scan every new post the moment it is submitted.

Reactive moderation is the practice of waiting for users to report violations and only reviewing content after it has been flagged. It is cheaper than proactive moderation but lets harmful content reach users first.

Self-harm detection is the identification of user-generated content that expresses suicidal ideation, self-injury, or eating disorder behaviors. Best practice is to route such content to crisis support resources and trained human reviewers rather than issuing blunt removals, which can compound harm for vulnerable users.

Shadow banning is the practice of silently reducing the visibility of a user's posts — hiding them from feeds, search, or replies — without notifying the user that any action has been taken. Platforms use it to contain spammers and trolls without giving them a signal they can adapt to.

A sock puppet is a fake online identity created to deceive other users, usually operated by someone who already has a primary account. Sock puppets are the basic building block of coordinated inauthentic behavior, astroturfing, and review manipulation.

Toxicity in content moderation describes language that is harmful, abusive, or disruptive to a community — typically including hate speech, harassment, threats, and personal attacks. Toxicity classifiers score text on this dimension.

Trust & Safety is the discipline within an online platform responsible for protecting users from harm — covering content moderation, fraud prevention, policy enforcement, and incident response. Content moderation is one pillar of a broader Trust & Safety function that also handles abuse, scams, and regulatory compliance.

User-generated content (UGC) is any text, image, video, audio, or comment created and published by the users of a platform rather than by the platform itself. UGC is the primary subject of content moderation.