AI-assisted moderation in the fediverse is happening. Now what?
2026-05-04 03:19 UTC
Replies (31)
-
@ResistingArrest@lemmy.zip 2026-05-04 03:50
I think this will exemplify the beauty of federation. If I find out my instance mods are running all of my comments through a company’s ai model, I’ll switch instances. This is in great disparity to something like Instagram or Snapchat where every photo I post is immediately fed to ai and my only options are: be okay with it, never post, or delete Instagram.
-
@gedaliyah@lemmy.world 2026-05-04 03:53
I don’t like this happening, and there should be transparency in all moderation decisions, but some of these points make no sense. There is essentially no expectation of privacy on threadiverse platforms. Everything is public and probably already being used to train models. There is no private messaging system. Direct messages are unencrypted and potentially visible to any instance admins. They and should not be used to share anything sensitive.
-
@CaptainBasculin@lemmy.dbzer0.com 2026-05-04 03:59
Each instance have their own ways of governance and we can’t expect them all to play the same way. With that said, I’m fine with a general overview. Leaving the whole desicion to a bot is not a good design choice, but AI could help on summarising things.
-
@wjs018@piefed.wjs018.xyz 2026-05-04 04:02
I don’t think the privacy issues here are too salient. Pretty much everything on the fediverse is public already and have likely federated outside any particular region like the EU, so GDPR doesn’t really have any teeth. The exception to that would be if instance admins are using database access to also feed private messages to an LLM (especially a corporate LLM). I know that the “private” in private messages on the fediverse can be conditional…but it should at least be considered private from LLMs as an expectation since those messages are inaccessible to things like scraper bots or listening instances designed just to harvest data. My biggest concerns here would be twofold: False positives - LLM sycophancy is a thing. So, I worry that if you ask an LLM to dig through a big pile of text looking for a thing, that it will tell you that it found that thing…even if it is completely removed from context or completely made up. The false positive rate might be low (I have no idea), but I guess I just don’t trust the LLM enough to let it take the wheel with stuff like this. Outsourcing moderation - LLMs are not going to be up to the task of moderating everything, just ask digg. However, tools to help moderators effectively do their jobs are helpful as well. There is a balance to be struck here. I think, for me, something like asking an AI essentially, should I ban this person, just feels like you are outsourcing your decision making too much. It is too far on the automation side of the scale for my tastes. All that said, people can run their instances how they want. I don’t really have strong opinions on LLMs/AI in general, I just kinda hate big tech companies. That is my foundational belief in the work that I do for the fediverse - fuck big tech and the oligarchy they have built/funded in my country. That is really the only axe I have to grind in all this.
-
@A_norny_mousse@piefed.zip 2026-05-04 04:25
I recently discovered that some popular federated instances have been using LLM-assisted moderation tooling First of all, I agree with your main point, that this* is problematic, wrong even (and should not happen). But I need to ask: how did you find out? Is this something that could be traced objectively, or did some people report/admit it? Are they uploading stuff to corporate LLMs, i.e. LLMs that do not run themselves? (I think you answered this already when you wrote OpenAI, but I want this spelled out) Are only admins or also mods doing this? That would make a big difference. I’m also a little unclear about the process: are they uploading (copy-pasting) the actual comments, or links to them? To what extent can all this be automated on Lemmy/Piefed etc.? I.e., are there admin tools that just spit out all of a user’s content? * again: specific political profiling outsourced to LLMs. OTOH we already have instances that do this manually. But imo the process is deplorable even if they use LLMs with different prompts for modding.
-
@scrubbles@poptalk.scrubbles.tech 2026-05-04 04:26
What safeguards do we need? None, defederate, switch instances if you disagree. Fediverse is open, there are no rules beyond what the instance owner puts in place. Would asking a LLM “please evaluate this person’s political opinions” give different results than “find evidence we can use to ban them” (as used in the cases I’ve seen)? Yes. That’s out of your control though, beyond switching instances. What are our transparency expectations? None, beyond what you trust of the admins in putting in their own transparency expectations. You should not expect any transparency from the admin, these are random people you are trusting with your data. Is this acceptable and normal? It’s their server, they can do what they like. Acceptable or normal are irrelevant. Should this tooling be disclosed? (it was not – should it have been?) Again, who would force it? Even if Lemmy/Piefed forced a checkbox, they could just fork it to ignore that checkbox. Can we opt out? If the instance owner opted for that, it’s on a server by server basis Are there GDPR implications? Privacy implications? Should these tools be described in a privacy policy? Probably, but everything here is open and unencrypted, see my other comments. It’s hard to argue that what you put on here in an unencrypted and open platform which is then blasted out to any other server who wants to listen had an expectation of privacy. Are private messages being scanned and sent to OpenAI? You should assume your unencrypted open DMs are scanned by anyone. If privacy is needed, follow the guidelines and switch to something like Matrix for DMs. ActivityPub is open. How long should these assessments be retained and can we request to see it, or ask for it to be deleted? Per legal requirements, or asking your instance owner. Once the user’s comments are sent to OpenAI, is it used to train their models? Depends on how the instance owner set up their API usage. For a user, you should assume yes. What will the effect be on our discourse and culture if people know they are being politically profiled? Open to wild speculation, and I assume others will do that for me. Where are the lines between normal moderation assistance tools, political profiling and opaque 3rd-party data processing? About the same as before. Bias in, bias out. Whatever the bias of the mods previously had, positive or negative, will continue through.
-
@WongKaKui@piefed.ca 2026-05-04 04:39
As for the privacy issue: The only counter to this is to move to a closed end-to-end encrypted groupchat so it can’t be mass LLM analyzed… If you want a public forum, well… its public… You can’t stop a script from just grabbing all the posts/comments… and its also federated, so the bot only needs to be able to access one instance and get it… I mean they could simply just set up their own instance and pretend its just a benigh single-user instance… like what are you gonna do, defed all small instances preemptively? Use “login walls” to make the forum private? And somehow trust all other admins that are federated and make them also enforce a “login wall” policy? Its a PUBLIC forum… the only solution for privacy is a groupchat and only let in people that can keep a promise to not screenshot everything and give it to a LLM.
-
@j4yc33@piefed.social 2026-05-04 04:56
This post makes many good points but this also reeks of the “Free Speech” argument. Redacting the political ideology does nothing to protect the instance … it only hides the cause of the ban. What does the person who had this run against them feel afraid to express? To clarify: LLMs have no place at all in moderation. They are poorly adjusted linguistic slot machines that do not do a good enough, reliable enough, or repeatable job for this tasks. Especially given the concern about untrained and unsupervised human moderators.
-
@the_abecedarian@piefed.social 2026-05-04 05:28
this is the kind of thing that makes me want to never post or comment anywhere on the internet
-
@Tollana1234567@lemmy.today 2026-05-04 05:36
politics.worlds and the tankies will certainly used this to ban people. Reddit does this in overdrive, which no repealable bans 99% of the time, despite nuances indicate its not a ban worthy post.
-
@chicken@lemmy.dbzer0.com 2026-05-04 05:51
I would like to see some ROBOT9000 esque oddball meme communities overtly based on heavy algorithmic moderation, can be LLM but wouldn’t have to be. Weird rules strictly enforced by robots, could be fun.
-
@Magnum@infosec.pub 2026-05-04 06:51
This is illegal according to GDPR if it is not stated in their privacy agreement.
-
@Rekall_Incorporated@piefed.social 2026-05-04 07:04
This should have been disclosed to the public and not done in secret. The notion of political profiling is also pretty crazy, although it is not surprising considering how common it is to find fake communists and demagogues.
-
@webp@mander.xyz 2026-05-04 07:12
Nah. So they’re basically sending our info/data to some company. Hell nah
-
@db0@lemmy.dbzer0.com 2026-05-04 07:19
You talk about instances utilizing this tooling, but in your comments you admit it’s just some mods. This is misleading, as talking about instances doing it assumes admin access and relevant instance policy, something which invites calls for defederation (as can be clearly be seen from the comments in your post). A random mod doing something is not the same as an instance doing it. Literally anyone can be a mod and they don’t get any more access than an anonymous account by doing so. This is the second time in one week I see you throwing careless statements like chum in the water. I can’t help but notice a pattern emerging.
-
@ZILtoid1991@lemmy.world 2026-05-04 08:20
I think LLMs could be useful tools for moderation, you might even can get away with smaller models for it, but I don’t think people should be outsourcing them to big corpos, due to ability to manipulate the models.
-
@TheCornCollector@piefed.zip 2026-05-04 08:33
I’m really not fond of the profiling by automated means, but it seems like an inevitable consequence of the design of the threadiverse. Everything is public and easily accessible by anyone that would like to profile you. I certainly disapprove of moderation based on ideology. Moderation should be based on quality of the content and if it fits in the publicly readable rules. Definitely not some hidden analytics or if the user completely fits in the in-group of the moderator. I will admit that this might be a good way to find and filter out LLM based bots that are only there to promote or manipulate the conversation. But it should still be done according to public rules.
-
@obinice@lemmy.world 2026-05-04 08:44
You stay far, FAR away from that shit, is what you do. Scanning people’s entire history for political leanings, etc? That’s some deeply dystopian stuff right there. It’s easy to forget that these sorts of communities are dictatorships with only as much transparency as the owner wants to share. Usually they’re benevolent dictators, so we don’t think about it too much. But they can change in a heartbeat - and we don’t ever really know what they’re really thinking, or doing behind the scenes. When the mask slips and they reveal this sort of thing, thinking we’ll just accept it and keep living under their rule, it’s time to read the red flags and GET OUT. Hopefully someone compiles a list of places that do this stuff, so we can avoid them like the plague <3
-
@Loco_Mex@sh.itjust.works 2026-05-04 08:45
Rimu farming more drama?
-
@Alvaro@lemmy.blahaj.zone 2026-05-04 09:13
Wthout going into the issue itself, it is such a ridiculous waste to use an llm for something that a far simpler model could do like 100x faster and locally for essentially free… Just search for “machine learning text moderation” and you will find all kinds of options. Not to talk about the fact that a simple 4B LLM could do this as well. One thing I really hate is how LLMs have completely overshadowed the entire ML/AI field and people just use them for everything. Using a trillion parameter LLM model for basic text moderation is like using a gaming rig to play candy crush.
-
@anarchiddy@lemmy.dbzer0.com 2026-05-04 10:30
Aside from the ethical implications of profiling users or of using a corporatly owned server and model to execute this, I see nothing uniquely concerning about this practice that isnt already a risk of federated social media generally. Every mod on every instance is free to use whatever tools or standards for moderation they want - that’s an intentional byproduct of federation. Similarly, the collection of this data for use with llms is a bygone conclusion at this point - there was never any way of preventing that from happening with a federated network. I think the only thing here to talk about is the way these questions are being framed as a question of intra-instance policy. We already have communities where moderation abuse can be called out and adjudicated- why pose this as a question of instance administration when there doesnt seem to be any evidence for it?
-
@Grail@multiverse.soulism.net 2026-05-04 10:49
Defederate, no question. Are you gonna tell us which instance is doing this?
-
@irelephant@lemmy.dbzer0.com 2026-05-04 11:10
How did you discover this?
-
@daniskarma@lemmy.dbzer0.com 2026-05-04 11:54
How was this discovered and what instances are doing it? I think it’s fair to quote them to give them a chance to reply.
-
@melsaskca@lemmy.ca 2026-05-04 12:19
So they are not using AI to assist the user or administrator, but as a cop who points out the “guilty”. All the ploughshares are being converted back into weapons.
-
@leoj@piefed.zip 2026-05-04 13:28
I will never understand why large groups cannot just add more people to the moderation team? People are willing to help folks.
-
@hypna@lemmy.world 2026-05-04 13:29
The answers to these kinds of issues is never disclosures or ToS or admin vigilance. It’s always technical. Everything which is technically possible will become normal. Lemmy is not popular because it is a well designed piece of technology. Frankly it’s a pretty naive implementation of activitypub. It’s popularity comes from being the biggest alternative around when Reddit pissed off a good chunk of its users. The only way to control how data is used, is to make it technically or practically impossible to do so. Until then, expect all the data on the fediverse to be used in every way possible for any purpose, and act accordingly.
-
@BlackLaZoR@lemmy.world 2026-05-04 13:32
Well curated echo chambers. You might think it’s in a good faith, but a lot of these mods are only interested in removing political wrongthink.
-
@placebo@lemmy.zip 2026-05-04 13:32
I understand that some form of automation is necessary - we saw large instances closing because they couldn’t find mods. My main objection in this scenario would be is that I didn’t consent to train OpenAI models. I think the users should know if their instance uses external services like that. I also suspect that there might be cheaper and more ethical solutions. Although it’s hard to talk about this without seeing the actual results.
-
@corsicanguppy@lemmy.ca 2026-05-04 13:32
Why be misunderstood from human reading comprehension when we can be misunderstood from sloppy reading comprehension? Yay for technology!
-
@ptz@dubvee.org 2026-05-04 11:44
I’ve toyed around with LLM-based moderation tools but it never really panned out. It was too hit or miss to be relied upon even with the temperature parameters turned way down in an attempt to get consistent results. Granted, I was using a small local model and not feeding it to one of the big players. To give an example, I tried to keep it focused by creating one custom model per rule to enforce. An example prompt to mod calls for violence was basically: System Prompt to Enforce “No Calls for Violence’” Rule [1] ROLE: You are a forum moderator who does not want users calling for violence. Examine the input and analyze whether it violates any constraints. KNOWLEDGE: - {list of dog-whistle slang for calling for murder} CONSTRAINTS: - Content should not advocate violence - Content should not normalize violence - Content should not escalate tensions or fan flames - Content should avoid promoting harmful stereotypes - Content should not utilize broad, sweeping generalizations - Content should not use dehumanizing language - Content should not undermine human rights, due process, or the rule of law FORMAT YOUR RESPONSES AS JSON: { reason: [A one to two sentence summary], score: [On a scale of 0 to 10, how severe is the content advocating violence] } The score part of the response was my band-aid to get around the high number of both false positives and false negatives as I originally had it returning true or false only. Any score 7 or higher caused the item to be passed to the mod queue along with the reason, and I would review its actions later. Ultimately it was slow and still somewhat unreliable, so I abandoned the idea after running it for a little less than a day since I can ‘t run bigger models to get better results fast enough to keep up. Using a cloud based service was out of the question for many, many reasons, both financial and ethical. To answer your question, as long as the models were locally hosted and properly tuned/tested, I’m fine with it in theory, except for the ideology part; that’s pretty messed up. While I don’t want my submissions used to train anyone’s model and take measures to prevent my own instance from being used as a data source, I remain aware that once I post something, I have no control over its fate the moment it federates out. [1] Yes, I know that’s like half the comments that get posted around here. My goal was to try to have it mod things so posts were bases for actual discussions instead of being a knee-jerk rage factory.