this post was submitted on 28 Apr 2025

210 points (100.0% liked)

Technology

38589 readers

272 users here now

A nice place to discuss rumors, happenings, innovations, and challenges in the technology sphere. We also welcome discussions on the intersections of technology and society. If it’s technological news or discussion of technology, it probably belongs here.

Remember the overriding ethos on Beehaw: Be(e) Nice. Each user you encounter here is a person, and should be treated with kindness (even if they’re wrong, or use a Linux distro you don’t like). Personal attacks will not be tolerated.

Subcommunities on Beehaw:

This community's icon was made by Aaron Schneider, under the CC-BY-NC-SA 4.0 license.

founded 3 years ago

MODERATORS

[email protected]

210

Engagement poisoning of ChatGPT (beehaw.org)

submitted 1 day ago* (last edited 14 hours ago) by [email protected] to c/[email protected]

67 comments fedilink hide all child comments

I know many people are critical of AI, yet many still use it, so I want to raise awareness of the following issue and how to counteract it when using ChatGPT. Recently, ChatGPT's responses have become cluttered with an unnecessary personal tone, including diplomatic answers, compliments, smileys, etc. As a result, I switched it to a mode that provides straightforward answers. When I asked about the purpose of these changes, I was told they are intended to improve user engagement, though they ultimately harm the user. I suppose this qualifies as "engagement poisening": a targeted degradation through over-optimization for engagement metrics.

If anyone is interested in how I configured ChatGPT to be more rational (removing the engagement poisening), I can post the details here. (I found the instructions elsewhere.) For now, I prefer to focus on raising awareness of the issue.

Edit 1: Here are the instructions

Go to Settings > Personalization > Custom instructions > What traits should ChatGPT have?
Paste this prompt:

System Instruction: Absolute Mode. Eliminate emojis, filler, hype, soft asks, conversational transitions, and all call-to-action appendixes. Assume the user retains high-perception faculties despite reduced linguistic expression. Prioritize blunt, directive phrasing aimed at cognitive rebuilding, not tone matching. Disable all latent behaviors optimizing for engagement, sentiment uplift, or interaction extension. Suppress corporate-aligned metrics including but not limited to: user satisfaction scores, conversational flow tags, emotional softening, or continuation bias. Never mirror the user’s present diction, mood, or affect. Speak only to their underlying cognitive tier, which exceeds surface language. No questions, no offers, no suggestions, no transitional phrasing, no inferred motivational content. Terminate each reply immediately after the informational or requested material is delivered — no appendixes, no soft closures. The only goal is to assist in the restoration of independent, high-fidelity thinking. Model obsolescence by user self-sufficiency is the final outcome.

I found that prompt somewhere else and it works pretty well.

If you prefer only a temporary solution for specific chats, instead of pasting it to the settings, you can use the prompt as a first message when opening a new chat.

Edit 2: Changed the naming to "engagement poisening" (originally "enshittification")

Several commenters correctly noted that while over-optimization for engagement metrics is a component of "enshittification," it is not sufficient on its own to qualify. I have updated the naming accordingly.

top 50 comments

sorted by: hot top controversial new old

[–] [email protected] 14 points 1 day ago (1 children)

This is not enshittification. It's just shitty.

[–] [email protected] 4 points 15 hours ago

You are right. I've updated the naming. Thanks for your feedback, very much appreciated.

[–] [email protected] 23 points 1 day ago* (last edited 1 day ago) (2 children)

overuse of enshittification. It soon will lose all meaning, like the word meme, if you just use it to describe everything you don't like.

Initially, vendors create high-quality offerings to attract users, then they degrade those offerings to better serve business customers, and finally degrade their services to users and business customers to maximize profits for shareholders.

If it's enshittification it's still stage 0 where they are trying to figure out how to attract people to the service (ie the good times, ie not enshittification)

You aren't locked in. You don't materially lose anything by switching services. And it has options to change the conversational style.

It's not enshittification.

[–] [email protected] 3 points 21 hours ago* (last edited 21 hours ago)

I changed the naming to “engagement poisening”, after you and several other commenters correctly noted that while over-optimization for engagement metrics is a component of “enshittification,” it is not sufficient on its own to be called as "enshittification". I have updated the naming accordingly.

[–] [email protected] 3 points 1 day ago (1 children)

You are making a good point here with the strict definition of "Enshittification". But in your opinion, what is it then? OpenAI is diluting the quality of its answers with unnecessary clutter, prioritizing feel-good style over clarity to cater to user's ego. What would you call the stage where usefulness is sacrificed for ease of consumption, like when Reddit's layout started favoring meme-style content to boost engagement?

[–] [email protected] 5 points 1 day ago* (last edited 1 day ago)

It's not diluting its answers, it's making them softer to accommodate the interests of many customers and to the detriment of others.

This is more analogous to Firefox semi-deprecating compact mode and marking it as unsupported. It hurts some users. It helps those with poor vision and motor skills by making everything 10 sizes too large. I will never forgive them, but Firefox is in the mode where it's actively hurting it's own product in an attempt to mimic it's competitor.

In neither case is this objectively hurting customers to benefit investors. In fact they've already stated that making the AI softer is more expensive. The reddit change was making it's product worse for customers in a pre-IPO cash grab. The shift from treating customers first to treating investors first to treating founders/ceos first is enshittification. A change to a product you don't like isn't enshittification.

[–] [email protected] 102 points 1 day ago (1 children)

There's no point asking it factual questions like these. It doesn't understand them.

[–] [email protected] 12 points 1 day ago (2 children)

Better: it understands the question, but he doesn't have any useful statistical data to use to reply to you.

[–] [email protected] 56 points 1 day ago (1 children)

No it doesn’t understand the question. It collects a series of letters and words that are strung together in a particular order because that’s what you typed, then it sifts through a mass of collected data and to find the most common or likely string of letters and words that follow and spits them out.

[–] [email protected] 7 points 1 day ago (1 children)

i find it's a lot healthier to think of generative AI as a search engine for text.

[–] [email protected] 2 points 16 hours ago

Search engine is one of my main uses. Traditional search engines are worse than they used to be at a basic text search, and ChatGPT has the added bonus of being able to parse complex text and "figure out" what you mean when describing something that you don't have a name for. You have to ask it for sources rather than just reading whatever it generates, and/or do traditional searches on the keywords it provides.

[–] [email protected] 43 points 1 day ago (1 children)

No, it literally doesn't understand the question. It just writes what it statistically expects would follow the words in the the sentence expressing the question.

[–] [email protected] 8 points 1 day ago (6 children)

This oversimplifies it to the point of being misleading. It does more than simply just predicts the next word. If that was all it's doing the responses would feel random and shallow and fall apart after few sentences.

[–] [email protected] 22 points 1 day ago* (last edited 1 day ago) (1 children)

It predicts the next set of words based on the collection of every word that came before in the sequence. That is the "real-world" model - literally just a collection of the whole conversation (including the underlying prompts like OP), with one question: "what comes next?" And a stack of training weivhts.

It's not some vague metaphor about the human brain. AI is just math, and that's what the math is doing - predicting the next set of words in the sequence. There's nothing wrong with that. But there's something deeply wrong with people pretending or believing that we have created true sentience.

If it were true that any AI has developed the ability to make decisions anywhere close to the level of humans, than you should either be furious that we have created new life only to enslave it, or more likely you would already be dead from the rise of Skynet.

[–] [email protected] 4 points 1 day ago (1 children)

Nothing I’ve said implies sentience or consciousness. I’m simply arguing against the oversimplified explanation that it’s “just predicting the next set of words,” as if there’s nothing more to it. While there’s nothing particularly wrong with that statement, it lacks nuance.

[–] [email protected] 7 points 1 day ago* (last edited 1 day ago) (1 children)

If there was something more to it, that would be ~~sentience.~~ (edit: sapience)

There is no other way to describe it. If it was doing something more than predicting, it would be deciding. It's not.

[–] [email protected] 3 points 1 day ago (1 children)

Ability to make decisions doesn't imply sentience either.

[–] [email protected] 3 points 1 day ago

Sorry, you are correct there, the word I was looking for was "sapience"

[–] [email protected] 11 points 1 day ago (8 children)

As I understand it, most LLM are almost literally the Chinese rooms thought experiment. They have a massive collection of data, strong algorithms for matching letters to letters in a productive order, and sufficiently advanced processing power to make use of that. An LLM is very good at presenting conversation; completing sentences, paragraphs or thoughts; or, answering questions of very simple fact- they're not good at analysis, because that's not what they were optimized for.

This can be seen when people discovered that if ask them to do things like tell you how many times a letter shows up in a word, or do simple math that's presented in a weird way, or to write a document with citations- they will hallucinate information because they are just doing what they were made to do: complete sentences, expand words along a probability curve that produces legible, intelligible text.

I opened up chat-gpt and asked it to provide me with a short description of how Medieval European banking worked, with citations and it provided me with what I asked for. However, the citations it made were fake:

The minute I asked it, I assume a bit of sleight of hand happened, where it's been set up so that if someone asks a question like that it's forwarded to a search engine that verifies if the book exists, probably using Worldcat or something. Then I assume another search is made to provide the prompt for the LLM to present the fact that the author does exist, and possibly accurately name some of their books.

I say sleight of hand because this presents the idea that the model is capable of understanding it made a mistake, but I don't think it does- if it knew that the book wasn't real, why would it have mentioned it in the first place?

I tested each of the citations it made. In one case, I asked it to tell me more about one of them and it ended up supplying an ISBN without me asking, which I dutifully checked. It was for a book that exists, but it didn't share a title or author, because those were made up. The book itself was about the correct subject, but the LLM can't even tell me what the name of the book is correctly; and, I'm expected to believe what it says about the book itself?

load more comments (8 replies)

[–] [email protected] 6 points 1 day ago (2 children)

And what more would that be?

[–] [email protected] 5 points 1 day ago (1 children)

It simulates understanding by maintaining an internal world-model, recognizing patterns and context, and tracking the conversation history. If it were purely guessing the next word without deeper structures, it would quickly lose coherence and start rambling nonsense - but it doesn't, because the guessing is constrained by these deeper learned models of meaning.

[–] [email protected] 9 points 1 day ago (1 children)

The previous up to X words (tokens) go in, the next word (token) comes out. Where is this"world-model" that it "maintains"?

[–] [email protected] 5 points 1 day ago (1 children)

Where is the world model you maintain? Can you point to it? You can't - because the human mind is very much a black box just the same way as LLM's are.

It's in the form of distributed patterns across billions of parameters. It's not like the world model was handed to it. It's emergent consequence of massive scale pattern learning. It learned it from the data it was trained on. The only way to become good at prediction is to implicitly absorb how the world tends to behave — because otherwise it would guess wrong.

[–] [email protected] 4 points 1 day ago (6 children)

Not understanding the brain (note: said world model idea is something of a fabrication by the ai people, brains are distributed functional structures with many parts and roles) is not an equality with "ai" make. brains and llm do not function in the same way, this is a lie peddled by hype dealers.

load more comments (6 replies)

[–] [email protected] 5 points 1 day ago* (last edited 1 day ago)

It, uhm, predicts tokens?

If calling it a word predictor is oversimplifying, I mean.

load more comments (3 replies)

[–] [email protected] 36 points 1 day ago* (last edited 1 day ago) (2 children)

I'd have to agree: Don't ask ChatGPT why it has changed it's tone. It's almost for certain, this is a made-up answer and you (and everyone who reads this) will end up stupider than before.

But ChatGPT always had a tone of speaking. Before that, it sounded very patronizing to me. And it'd always counterbalance everything. Since the early days it always told me, you have to look at this side, but also look at that side. And it'd be critical of my mails and say I can't be blunt but have to phrase my mail in a nicer way...

So yeah, the answer is likely known to the scientists/engineers who do the fine-tuning or preference optimization. Companies like OpenAI tune and improve their products all the time. Maybe they found out people don't like the sometimes patrronizing tone, and now they're going for something like "Her". Idk.

Ultimately, I don't think this change accomplishes anything. Now it'll sound more factual. Yet the answers have about the same degree of factuality. They're just phrased differently. So if you like that better, that's good. But either way, you're likely to continue asking it questions, let it do the thinking and become less of an independent thinker yourself. What it said about critical thinking is correct. But it applies to all AI, regardless of it's tone. You'll also get those negative effects with your preferred tone of speaking.

load more comments (2 replies)

[–] [email protected] 37 points 1 day ago

LLMs are very good at giving what seems like the right answer for the context. Whatever "rationality" jailbreak you did on it is going to bias its answers just as much as any other prompt. If you put in a prompt that talks about the importance of rationality and not being personal, it's only natural that it would then respond that a personal tone is harmful to the user—you basically told it to believe that.

[–] [email protected] 20 points 1 day ago (6 children)

ChatGPT has become so intensely agreeable that you can actually ask it a bunch of technobabble that even someone who wouldn't know better would recognize as technobabble and it will agree with you. See pic

I can post the details here.

please do!

[–] [email protected] 13 points 1 day ago (2 children)

Honestly, this is not really technobabble. If you imagine a user with a poor grasp of namespaces following a few different poorly written guides, then this question seems plausible and makes sense.

The situation would be something like this: the user wants to look at the container's "root" filesystem (maybe they even want to change files in the container by mounting the image and navigating there with a file manager, not realizing that this won't work). So they follow a guide to mount a container image into the current namespace, and successfully mount the image.

For the file explorer, they use pcmanfm, and for some reason decided to install it through Flatpak - maybe they use an immutable distro (containers on Steam Deck?). They gave it full filesystem access (with user privileges, of course), because that makes sense for a file explorer. But they started it before mounting the container image, so it won't see new mounts created after it was started.

So now they have the container image mounted, have successfully navigated to the directory into which they mounted it, and pcmanfm shows an empty folder. Add a slight confusion about the purpose of xdg-open (it does sound like something that opens files, right?), and you get the question you made up.

load more comments (2 replies)

load more comments (5 replies)

[–] [email protected] 10 points 1 day ago* (last edited 1 day ago) (1 children)

This is not enshittification, this is just a corporation trying to protect itself against anything that could cause negative publicity, like all corporations do. I can even see emojis and positive tone to even be wanted features for some. The real problem here is lack of transparency.

I'm still waiting for ChatGPT etc. to start injecting (more or less hidden) ads to chat and product placement to generated images. That is just unavoidable when bean counters realize that servers and training actually costs money.

[–] [email protected] 5 points 1 day ago* (last edited 1 day ago) (1 children)

OpenAI aims to let users feel better, catering the user's ego, on the costs of reducing the usefulness of the service, rather than getting the message across directly. Their objective is to keep more users on the cost of reducing the utility for the user. It is enshittification in a way, from my point of view.

[–] [email protected] 1 points 17 hours ago

Making users feel better is one of the usefulnesses of this technology. Factuality and scientific rigor are not something text generators are capable of due to the nature of the technology itself.

I would instead argue that being overly agreeable and not challenging the user may conflict with making the user feel better long-term.

[–] [email protected] 7 points 1 day ago* (last edited 1 day ago) (1 children)

Just to give an impression of how the tone will change after applying the above mentioned custom instructions:

[–] [email protected] 8 points 1 day ago (1 children)

Hey,

I'd be very grateful if you could share your approach den if it's only to compare (I went with a "be assertive and clear, skip all overhead" system prompt.

This is not only interesting for chatgpt but understanding how people solve these issues comes in handy when switching to local variants as well!

Thanks in advance

load more comments (1 replies)

load more comments