Wikipedia talk:Large language models

Chatbot to help editors improve articles

After selecting text, the control panel on the right is used to give instructions. The responses by the AI model are presented in the chat panel on the left.

I wrote a user script called WikiChatbot. It works by selecting text in an article and then clicking one of the buttons on the right to enquire about the selected text. It includes many functions. For example, it can summarize and copyedit the selected text, explain it, and provide examples. The chat panel can also be used to ask specific questions about the selected text or the topic in general. The script uses the AI model GPT 3.5. It requires an API key from OpenAI. New OpenAI accounts can use it freely for the first 3 months with certain limitations. For a more detailed description of all these issues and examples of how the script can be used, see the documentation at User:Phlsph7/WikiChatbot.

I was hoping to get some feedback on the script in general and how it may be improved. I tried to follow WP:LLM in writing the documentation of the chatbot. It would be helpful if someone could take a look to ensure that it is understandable and that the limitations and dangers are properly presented. I also added some examples of how to use edit summaries to declare LLM usage. These suggestions should be checked. Feel free to edit the documentation page directly for any minor issues. I'm also not sure how difficult it is to follow the instructions so it would be great if someone could try to set up the script, use it, and explain which steps were confusing. My OpenAI account is already older than 3 months so I was not able to verify the claims about the free period and how severe the limitations are. If someone has a younger account or is willing to open a new account to try it, that would be helpful.

Other feedback on the idea in general, on its problems, or on new features to implement is also welcome. Phlsph7 (talk) 12:45, 12 July 2023 (UTC)

I meant to reply to this sooner. This is awesome and I'm interested in this (and related ideas) related to writing / reading with ML. I'll try to have a play and give you some feedback soon. Talpedia 10:18, 17 July 2023 (UTC)

Related: see also m:ChatGPT plugin. Mathglot (talk) 07:22, 18 July 2023 (UTC)

Whilst I rather like the ability of this nifty little script to do certain things, I do have some criticism. These functions strike me as extremely risky, to the point that they should probably be disabled:

• "is it true?" - ChatGPT likely uses Wikipedia as a source, and in any case, we want verifiability, not truth. I feel quite strongly, based on several other reasons too, that this function should be disabled and never see the light of day again.
• "is it biased?" - ChatGPT lacks the ability to truly identify anything more than glaring "the brutal savages attacked the defenceless colonist family" level bias (i.e. something that any reasonably aware human should spot very quickly indeed). Best left to humans.
• "is this source reliable?" - Same as the first one, this has so much potential to go wrong that it just shouldn't exist. Sure it might tell you that Breitbart or a self-published source isn't reliable, but it may also suggest that a bad source is reliable, or at least not unreliable.

I don't think that any amount of warnings would prevent misuse or abuse of these functions, since there will always be irresponsible and incompetent people who ignore all the warnings and carry on anyway. By not giving them access to these functions, it will limit the damage that these people would cause. Doing so should not be a loss to someone who is using the tool responsibly, as the output generated by these functions would have to be checked so completely that you might as well just do it without asking the bot.

The doc page also needs a big, obvious warning bar at the top, before anything else, making it clear that use of the tool should be with considerable caution.

The doc page also doesn't comment much on the specific suitability of the bot for various tasks, as it is much more likely to stuff up when using certain functions. It should mention this, and also how it may produce incorrect responses for the different tasks. It also doesn't mention that ChatGPT doesn't give wikified responses, so wikilinks and any other formatting (bolt, italics, etc) must be added manually. The "Write new article outline" function also seems to suggest unencyclopaedic styles, with a formal "conclusion", which Wikipedia articles do not have.

Also, you will need to address the issue of WP:ENGVAR, as ChatGPT uses American English, even if the input is in a different variety of English. Mako001 (C)  (T)  🇺🇦 01:14, 23 July 2023 (UTC)

You can ask it return wikified responses and it will do it with reasonable good success rate. -- Zache (talk) 03:03, 23 July 2023 (UTC)

@Mako001 and Zache: Thanks for all the helpful ideas. I removed the buttons. I gave a short explanation at Wikipedia:Village_pump_(miscellaneous)#Feedback_on_user_script_chatbot and I'll focus here on the issues with the documentation. I implemented the warning banner and add a paragraph on the limitations of the different functions. That's a good point about the English variant being American so I mentioned that as well. I also explained that the response text needs to be wikified before it can be used in the article.

Adding a function to wikify the text directly is an interesting idea. I'll experiment a little with that. The problem is just that the script is not aware of the existing wikitext. So if asked to wikify a paragraph that already contains wikilinks then it would ignore those links. This could be confusing to editors who only want to add more links. Phlsph7 (talk) 09:12, 23 July 2023 (UTC)

I made summaries/translations/etc it so that I gave wikitext as input to chatgpt instead of plaintext. However, the problem here is how to get the wikitext from page in first place. -- Zache (talk) 09:48, 23 July 2023 (UTC)

In principle, you can already do that with the current script. To do so, go to the edit page, select the wikitext in the text area, and click one of the buttons or enter your command in chat panel of the script. I got it to add wikilinks to an existing wikitext and a translation was also possible. However, it seems to have problems with reference tags and kept removing them, even when I told it explicitly not to. I tried it for the sections Harry_Frankfurt#Personhood and Extended_modal_realism#Background, both with the same issue. Maybe this can be avoided with the right prompt. Phlsph7 (talk) 12:09, 23 July 2023 (UTC)

Thanks for setting this up. I've recently had success drafting new Wikipedia articles by feeding the text of up to 5 RS into GPT4-32k through openrouter.com/playground and simply asking it to draft the article. It does a decent job with the right prompt. You can see an example at Harrison Floyd. I'll leave more details on the talk page of User:Phlsph7/WikiChatbot, but I wanted to post here for other interested parties to join the discussion. Nowa (talk) 00:02, 20 September 2023 (UTC)

Thanks for the information. I've responded to you at Talk:Harrison_Floyd#Initial_content_summarized_from_references_using_GPT4 so that we don't have several separate discussion about the same issue. Phlsph7 (talk) 07:44, 20 September 2023 (UTC)

Ran into a brick wall I thought might be helpful to know about. I've been working on the bios of people associated with Spiritual_warfare#Spiritual_Mapping_&_the_Charismatic_movement. GPT 4 and LLama refused to read the RS claiming that it was "abusive". I can see from their point of view why that is, but nonetheless, RS is RS, so I just read it manually. Between that and the challenges of avoiding copyvios I'm a bit sour on the utility of LLMs for assisting in writing new articles. It's just easier to do it manually. Having said that, the Bing chatbot does have some utility in finding RS relative to Google. Much less crap. Nowa (talk) 00:35, 9 October 2023 (UTC)

If we're going to allow LLM editing, this is a great tool to guide editors to the specific use cases that have community approval (even if those use cases are few to none at this point). I found it to be straightforward and easy to use. –dlthewave ☎ 16:06, 23 July 2023 (UTC)

There is no policy or guideline disallowing the use of LLM or other machine learning tools. No need for any approval unless that changes. MarioGom (talk) 17:29, 11 February 2024 (UTC)

Should vs. must?

I had assumed that LLMDISCLOSE was mandatory, but I now realize it's optional. That seems totally broken, and so I'm wondering if anyone here could help enlighten me as to why it was written that way. I ask this with an eye to making this conversation a pre-RfC to make LLM disclosure mandatory. CaptainEek ^{Edits Ho Cap'n!}⚓ 19:37, 2 November 2025 (UTC)

Probably to offer some flexibility as is the norm. Even COI disclosure is only a should, must only comes into play with PAID and that is mandated by the TOU. Maybe not as important to be precise when wording an essay as opposed to PAGs, but still usually best to try. 184.152.65.118 (talk) 22:01, 2 November 2025 (UTC)

Previous discussion about making it mandatory is at Wikipedia:Village pump (policy)/Archive 205#Alternative approach: make transparency policy. --Tryptofish (talk) 00:52, 3 November 2025 (UTC)

@Tryptofish that seems like a rather decent pre-RfC. I see the point about a technical solution, but I think that a policy solution would probably be necessary first to both be a stopgap and to spur on the technical solution. It seems like the path forward is to propose that LLMDISCLOSE be made mandatory and elevated to policy? You obviously have way more experience in this arena so I'll happily defer to you, just don't want this to wither on the vine :) CaptainEek ^{Edits Ho Cap'n!}⚓ 17:27, 3 November 2025 (UTC)

Thanks. I'm going to give a more detailed reply below, but it applies to both what you are saying and what isaacl is saying. --Tryptofish (talk) 21:20, 3 November 2025 (UTC)

I submitted a closure request for that village pump discussion a week and a half ago. — Newslinger talk 20:41, 3 November 2025 (UTC)

Thanks for that, too. Pity it got archived without a consensus being found. --Tryptofish (talk) 21:20, 3 November 2025 (UTC)

I'm mildly hopeful the LLMDISCLOSE may make the cut, the rest was my being unaware of the prior RFC and basically re-asking for something the community had already supported. But even if LLMDISCLOSE doesn't make policy, I think tackling this one piece at a time will prove more useful than trying to push one grand proposal (unless it's easily broken into pieces and put up as a Watchlist notice, etc for a month). —Locke Cole • t • c • b 23:32, 3 November 2025 (UTC)

Generally speaking, the community doesn't object to all uses of programs to assist with writing. The concerns are about using programs that generate original content, with greater detail than any human input used to trigger the generation. Thus in my view, any guidance should focus on how the program is being used, rather than the underlying technology (which in any case might not be readily apparent to the end user).

I think from a holistic perspective, a key question is what happens next if disclosure is required? If it's just a pre-cursor to removing the text, then maybe we should be banning generated text from mainspace instead (as with a disclosure requirement, of course this only works for those who read and follow guidance). If it's to queue up edits for examination, then what's the best format for disclosure that assists with this (template used on talk page, perhaps?) and do we need to organize more volunteers to manage and process the queue? Can we build more tools to help with analyzing edits (including presumably the vast majority of problematic edits from editors who won't comply with any relevant guidance)? isaacl (talk) 17:30, 3 November 2025 (UTC)

I've been paying very close attention to as many community discussions about these issues as I can find, and yes, we definitely need to find a carefully calibrated middle ground between no regulation and too much regulation. It's very clear to me that many members of the editing community don't want a complete ban on LLM use, and want to be careful that it doesn't become something that editors might weaponize against one another, and from my own observation, I've come to agree that this is a risk that we need to avoid. At the same time, by now, most of us have seen use of LLM-generated content that is obviously disruptive.

I've been working (I promise I have!) on trying to pull together a draft proposal for a potential policy (a full policy page, including, but going beyond, disclosure), that others could then workshop further before putting it to the community for an RfC. It's not easy, and I've been swamped with other stuff, both on and off site. So I want to promise CaptainEek that I won't let it wither on the vine – but I expect that it will take a while before it ripens on the vine. --Tryptofish (talk) 21:20, 3 November 2025 (UTC)

User:CaptainEek - Why disclosure was written that way -- Archives shows the discussion in Oct 2023 about a RFC showing this page was clearly not going to be promoted to either policy or guideline, so it was relabeled as essay and to make the phrasing more essay-like replaced "must"s with "should"s to "better reflect its status as an essay". See also discussion about LLMDISCLOSE lacking clarity on how to disclose and being incentivised to hide it completely instead of disclosing. Cheers Markbassett (talk) 05:30, 4 November 2025 (UTC)

Maybe mentioning the new guideline higher on the page

Heard of the new LLM guideline but didn't know its name, landed on this essay page, expected to see it in the disambiguation, and it was not in the disambiguation. I know the guideline is in fact linked lower in the page, but point is, it should probably be mentioned at the top of the page. -- Lampyscales (🐍 | C) 16:25, 25 November 2025 (UTC)

added it. User:Bluethricecreamman (Talk·Contribs) 05:53, 1 December 2025 (UTC)

I made this page an information page

See title. I did it in Special:Diff/1325613952. Feedback is welcome. I think this change is for the better. SuperPianoMan9167 (talk) 03:19, 4 December 2025 (UTC)

@Cambalachero: Why did you revert this page back to an essay? It's not really an essay; it describes the community's general procedures and expectations for LLM use and mainly provides information that supports other guidelines like WP:NEWLLM and WP:AITALK. You said in your edit summary too many "you must" and "you should" to be a page just providing information, but pages like WP:MINOR and WP:MERGE have similar directions. If this page isn't an information page, then what should it be? Should it be a "supplemental essay" instead? SuperPianoMan9167 (talk) 14:29, 4 December 2025 (UTC)

Does it? Not according to Wikipedia talk:Large language models/Archive 6#RfC: Is this proposal ready to be promoted?. One example: although there seems to be consensus against including unreviewed LLM into articles, many users are fine with reviewed LLM content: that is, using it as a starting point in a sandbox and then fixing and referencing it where needed before going live. "Writing articles" forbids it, and with no clear explanation to boot. Drafts are not checked for notability or sanity, and "is not one of the purposes of draft space or user space" does not really provide any actual reason (and yes, it remarks that the problem would be "unaltered" LLM output, but where else would you alter it if not in a draft?) Cambalachero (talk) 14:52, 4 December 2025 (UTC)

That's fair, I did indicate that anyone was free to revert me when I made the change. You might want to revert my contributions to templates (like this one) though, as I made several edits acknowledging the change after I did it. SuperPianoMan9167 (talk) 16:49, 4 December 2025 (UTC)

I have reverted myself wherever necessary to acknowledge WP:LLM is still an essay. SuperPianoMan9167 (talk) 16:52, 4 December 2025 (UTC)

Discussion at Wikipedia:Village pump (policy) § RfC: Replace text of Wikipedia:Writing articles with large language models

 You are invited to join the discussion at Wikipedia:Village pump (policy) § RfC: Replace text of Wikipedia:Writing articles with large language models. –Novem Linguae (talk) 23:41, 5 December 2025 (UTC)