One of my big worries with the way people are using LLMs is that they're being trained to trust whatever they spit out. Hey Google, what's the nutritional content of peanuts? And people are learning not to ask where the information came from or to check sources.
One of the many reasons this worries me is that very soon these businesses are going to need to recoup the billions they're spending, and I wonder how long until these systems start feeding paid promotions to a population that's been trained to accept whatever they're told. imagine what some businesses, or governments, would pay to have exactly their choice of words produced on demand in response to knowledge queries.