this post was submitted on 25 Feb 2024
319 points (87.2% liked)

Technology

59217 readers
3414 users here now

This is a most excellent place for technology news and articles.


Our Rules


  1. Follow the lemmy.world rules.
  2. Only tech related content.
  3. Be excellent to each another!
  4. Mod approved content bots can post up to 10 articles per day.
  5. Threads asking for personal tech support may be deleted.
  6. Politics threads may be removed.
  7. No memes allowed as posts, OK to post as comments.
  8. Only approved bots from the list below, to ask if your bot can be added please contact us.
  9. Check for duplicates before posting, duplicates may be removed

Approved Bots


founded 1 year ago
MODERATORS
 

A new report from plagiarism detector Copyleaks found that 60% of OpenAI's GPT-3.5 outputs contained some form of plagiarism.

Why it matters: Content creators from authors and songwriters to The New York Times are arguing in court that generative AI trained on copyrighted material ends up spitting out exact copies.

you are viewing a single comment's thread
view the rest of the comments
[–] [email protected] 86 points 8 months ago (7 children)

The individual GPT-3.5 output with the highest similarity score was in computer science (100%), followed by physics (92%), and psychology (88%).

And that’s why this claim is mostly bullshit. These use cases are all sciences, where the correct solution is usually the same or highly similar no matter who writes it. Small snippets of computer code cannot be copyrighted anyway.

Not surprisingly, softer subjects like “English” and “Theatre” rank extremely low on this scale.

[–] [email protected] 33 points 8 months ago (2 children)

Not to mention that a response "containing" plagiarism is a pretty poorly defined criterion. The system being used here is proprietary so we don't even know how it works.

I went and looked at how low theater and such were and it's dramatic:

The lowest similarity scores appeared in theater (0.9%), humanities (2.8%) and English language (5.4%).

[–] [email protected] 9 points 8 months ago

Yeah, anyone who has written a thesis knows those tools are bullshit. My handwritten 140 page master's thesis had a similarity index of 11%.

[–] [email protected] 9 points 8 months ago
[–] [email protected] 26 points 8 months ago

So, if the Ai gives you a correct answer to a science question, it’s “infringing copyright” and if it spits out a bullshit answer, it’s giving you wrong, and unsupported claims.

[–] [email protected] 13 points 8 months ago (1 children)

Right? Nod doubt that output can be similar to training data, and I would believe that some of it is plagiarism, but plagiarism detectors are infamous among uni students for being completely unreliable and flagging pronouns, dates and citations. Until someone can go "here's an example of actual plagiarism" (which is obvious when pointed out), these claims make no sense.

[–] [email protected] 2 points 8 months ago

If it's plagiarizing, so are Google search results summaries.

It's not like it doesn't cite where it found the data.

[–] [email protected] 11 points 8 months ago (1 children)

Eh, kinda. It’s not like a science paper is just going to be an equation and nothing else. An author’s synthesis of the results is always going to have unique language. And that is even more true for a social science paper.

[–] [email protected] 4 points 8 months ago (1 children)

Are those "best matches" paper-sized, or snippet-sized?

[–] [email protected] 5 points 8 months ago

Article mentioned 400-word chunks, so much less than paper-sized.

[–] [email protected] 10 points 8 months ago (2 children)

But also, there is far less training data to mix and match responses from, so naively I would expect a higher plagiarism rate, by its very nature.

Less than 2% of the world's population has a doctorate. According to the US Census Bureau, only 1.2% of the US population has a PhD.

source

[–] [email protected] 7 points 8 months ago* (last edited 8 months ago) (2 children)

"Only" 1 in a hundred Americans are PhDs? Thats far higher than I would have expected.

[–] [email protected] 3 points 8 months ago

Surely many who have them received them from elsewhere before immigration to America, and likewise the proportion of immigrants who have them I would expect to be oversized. Americans tend to be more greedy than anything else and don't put in the effort required for such small (financial) rewards.

Also, those with PhDs tend to congregate into certain areas that support those jobs, i.e. cities but not even a goodly number of those so much; plus smaller college towns too ofc. As such, many in the general populace might rarely if ever run into one for the largest majority of their lives, unless traveling specifically to those areas for some reason?

And ofc rural areas are far larger, geographically speaking, than places where a person with a PhD would (likely) go. So you could randomly pick a spot on a map 100 times and never manage to find someone with a PhD anywhere within tens of miles, I would expect - although that line of thinking reveals my own biases: do most educated farmers stop at like an MS and just follow up with their own (possibly even extensive) self studies, or go all the way to PhDs while working their actual farms? (I doubt it bc it does not sound practical, and that is a hallmark of farmers afaik, but I could be wrong...) Anyway, I expect the unequal distribution is a contributing / exasperating factor to the general rarity.

[–] [email protected] 4 points 8 months ago* (last edited 8 months ago)

Ironically, in the article, the link to the original Census source of the 1.2% datum is now dead.

Also, it’s 2.1% now (for people over 25), according to the Wikipedia article’s source: https://www.census.gov/data/tables/2018/demo/education-attainment/cps-detailed-tables.html

Edit: the Wikipedia citation is from 2018 data. The 2023 tables are here: https://www.census.gov/data/tables/2022/demo/educational-attainment/cps-detailed-tables.html

Citation party!

[–] [email protected] 2 points 8 months ago

I think the issue is more about HOW they wrote it, rather than WHO wrote it.

[–] [email protected] 1 points 8 months ago* (last edited 8 months ago) (1 children)

You can’t write a paper covering scientific topics without plagiarism. A human would be required to. Generative AI should be held to at least as high of a standard.

[–] [email protected] 4 points 8 months ago (1 children)

Turns out ChatGPT isn’t writing a scientific paper though, it’s conversing with the user.

[–] [email protected] 0 points 8 months ago

If it’s regurgitating other people’s work then it needs citations.