Draft SPEC on options for use of AI in project contributions

[matthew-brett]

This follows from a discussion on a policy on generative AI assisted contributions.

See that discussion for links to other discussions.

The use of generative AI brings two things into tension:

1. A desire to honor the copyright of code from which the AI training data has been derived and
2. The hope that generative AI will lead to more contributions and / or contributions of greater quality.

It would be useful and timely to set out the issues in some detail in preparation for:

1. A SPEC offering guidance on the use of generative AI in contributions and
2. Collecting further specialist advice on copyright law and the nature of generated AI code, relative to its training set.

At a first pass, the SPEC should cover the issues that Ralf laid out in his discussion post, but it would also be useful to cover:

• The distinction between legal liability and our desire, as project authors, to respect the copyright of our fellow authors.
• How and whether a policy should be shaped by the difficulty of enforcement. For example, one might argue that it would be practically impossible to be sure that a contributor had not used generative AI - while also accepting that setting the policy to forbid such use would likely reduce AI generated code submissions.
• Whether there are uses of AI that cannot plausibly leak substantial copyrighted code - for example, perhaps this is true of automated code review.

Different projects will put different weights on different factors. For example, some may feel that copyright has effectively become unenforceable, and should therefore no longer be a factor in accepting submissions, while others will put substantial weight on preserving copyright, and be prepared to accept some loss of volume or quality as a result. Hence the SPEC should lay out these issues carefully and clearly, in order to help projects choose their own position on the factors involved.

[drammock]

I'd also advocate for including a discussion of energy cost (which I think @ilayn alluded to in that thread). In particular, folks I've talked to are generally aware that training LLMs incur a very large one-time energy cost, but are often unaware of the ongoing inference-time energy cost (https://www.rwdigital.ca/blog/how-much-energy-do-google-search-and-chatgpt-use/ estimates it at 10x a traditional search query).

[matthew-brett]

I'd also advocate for including a discussion of energy cost

Certainly - but I'd like to distinguish carefully between general concerns about, or enthusiasm for AI, and the specifics of copyright and code quality.

[drammock]

I'd like to distinguish carefully between general concerns about, or enthusiasm for AI, and the specifics of copyright and code quality.

agreed --- the license issue is the most important concern to me, and also I think the one which is least subjective, so it deserves pride of place / emphasis.

[ivanov]

Any project that accepts code that was generated by tools violating license terms dilute and invalidate their own license.

These include approximately all of the automated tools in use today, as even training set that exclude GPL nevertheless violate the terms of BSD and MIT licenses which require attribution.

I think a way to address this would be to have attestations in the form of checkboxes and fill-in-the-blank, akin to those in Contributor License Agreements (CLAs), that states either that "I did not use code-generation tools in the production of the code submitted here", or that "The code generation tools I used was trained only on public domain, no attribution such as zero-clause BSD, zero-clause MIT, or code within this project." with a fill-in-the-blank requirement to specify the models that were used. In particular, in no uncertain terms it should reject contribution that utilized any of the popular tools in use today, which includes GitHub CoPilot, ChatGPT, Claude, Llama, Mistral, Replit Code, StarCoder, CodeLLama.

Projects without such a direct approach are open to drifting and sliding into a post-legal world. It doesn't matter what license they use, as they lack the means to assert their own license due to a lack of provenance of their incoming contributions.

[sanketverma1704]

Photos of the whiteboard from the discussion we had today. Notes from the meeting to follow.

[drammock]

high-level summary of the discussion on an AI SPEC. This is (I think) approximately where we landed at the end of 90 minutes (though I'm not sure everyone in the room would agree to all of these points):

1. license laundering is not acceptable. therefore:
2. full, unfettered use of AI in generating contributions is not acceptible. therefore:
3. projects should decide upon a policy about use of GenAI in contributions to their project. Some possible example policies:
   • no AI use allowed in generation of the code you contribute (though asking an LLM to summarize or explain how code is organized or works, is ok)
   • use of AI to generate example code is ok, as long as you read it, internalize it, and then write the code yourself (but no copy-pasting what it generates)
4. projects should request / insist on some kind of attestation that their policy is being followed when contributions are made (similar to what is done with Code-of-Conduct)
5. projects should think about what enforcement actions they might take if it is discovered that a contributor misrepresented their use of AI in a contribution (i.e., reverting PRs, suspending/banning users, etc)
6. projects should revisit their codes of conduct to ensure that enforcements of GenAI policy violations are discussed there / unambiguously constitute a CoC violation.

many thanks to @matthew-brett for facilitating. Also present at the meeting: @drammock @munkm @sanketverma1704 @dschult @stefanv @j-bowhay @mihaimaruseac @matthewfeickert @virchan @KirstieJane @nabobalis

[j-bowhay]

Thanks for summarizing what was quite a long conversation @drammock!

I thought I'd share the following as I found it quite interesting and because we mostly talked about beginners using AI. Terrance Tao (fields medalist for those not in maths circles) posted the following video two days ago in which he uses copilot to help him write a formalization of a proof in lean.

https://www.youtube.com/watch?v=cyyR7j2ChCI

[mdhaber]

I have a case that I thought would be an acceptable use of AI, but doesn't look like it would be under these criteria: translation of (identically-licensed) code from one language to another. I used ChatGPT to translate this BSD-3 licensed code from Matlab to Python, resulting in this gist. The functionality of the code can be verified with extensive property-based tests, and I did not intend to review the code line-by-line or rewrite it, treating it as if it were vendored code from an old Fortran library.

I did not intend to submit a PR with it until the question of AI generated code was resolved, so we started discussing it in this forum.

One reason I can see that this might not be OK is that ChatGPT may have trained on incompatibly-licensed code and reproduce it (in some form) in the translation. If this were the concern, I am surprised to see that this is acceptable:

use of AI to generate example code is ok, as long as you read it, internalize it, and then write the code yourself (but no copy-pasting what it generates)

This seems to be at odds with @matthew-brett's most recent comment in the SciPy discussion:

I’m afraid the problem of the license remains. AI generated some
code where it is effectively impossible for you to know what
license(s) should correctly apply. Now you’ve seen the code with
the unknowable license, you are at grave risk of reproducing aspects
of that code, thereby pulling in the unknowable license. Your
resulting code, even if you have rewritten it - cannot be asserted
free of the original (unknowable) license). And therefore you cannot
assert your own copyright, or freedom from other people’s copyright.

Anyway, I respect the consensus of this group and do not have the expertise to weigh in; I just think it might be helpful if translation were explicitly addressed in the SPEC.

[ilayn]

I don't know if this would come out as dismissive but I don't intend to make it so. However, there is a case to be made about making things enforcable. We can't know if contributions are coming from suitable sources even the contributor is all organic matter and no AI.

So as I mentioned in the discourse thread, having a idealistic stance is one thing, but making things unnecessarily restrictive as if we have a way to check the validity of the claim or the originality of the code is another. What we would do is exactly the same thing when we find the said code is incompatibly licensed, whether we have a legal warning or not, we will take it out and apologize. Hence I don't think the license problem is solved with any legal sounding policy text if we don't have the accompanying imposition power. This is a trust based operation anyways so that's the perimeter of our enforcement.

Code assistance has quite valid use cases such as boilerplate code generation or code completion, or as Matt mentioned, snippet translation work, (how I wish it was good at Fortran77 translation but alas). Moreover, I have fundamental disagreements with everything these AI companies providing. It is way way worse in the creative domain (say, graphics or music) than code. So I am on the anti-generative AI team. I try to turn it off everywhere I can. But this does not mean that I am oblivious to the fact that the times are changing and many people have been using these tools extensively. So in my opinion it is better to set reasonable boundaries instead of outright rejecting.

[drammock]

The summary I wrote above omitted an important caveat on the second bullet point under item 3:

use of AI to generate example code is ok, as long as you read it, internalize it, and then write the code yourself (but no copy-pasting what it generates)

This was meant to include a caveat such as "If you choose a policy like this for your project, you expose your project to the same kind of license laundering that comes from reading/copying/translating GPL-licensed code".

Regarding @mdhaber's comment, we discussed this in person and I agree that something like his translation example probably ought to be included in the SPEC, as a case that maintainers are likely to be confronted with eventually (if they haven't already) --- again with the caveat that it raises the same license laundering concerns as small copy-pasted snippets that were LLM-generated.

there is a case to be made about making things enforcable. We can't know if contributions are coming from suitable sources even the contributor is all organic matter and no AI.

This is, I think, why the SPEC needs a section discussing enforcement (points 5 and 6 in my summary above), encouraging projects to have clear written procedures to follow if/when it is discovered that a contributor flouted the project's stated GenAI policy. As @ilayn says, it's the same action that many (most? all?) of us are already doing when someone opens a PR saying "I copied/translated this GPL code from some other library" : "sorry, this violates our license/policy. <close as not planned>"

[matthew-brett]

@ivanov and I drafted a proposed summary - here : https://github.com/matthew-brett/sp-ai-post/blob/main/notes.md - PR's and comments welcome.

[tupui]

I really like your proposal @matthew-brett 👏

[betatim]

I support @ilayn's point that rules should be enforceable. If you want to make a rule it should be possible to check it was followed. This doesn't mean that it isn't interesting to have something that communicates intent: please don't use X, Y, Z to create your contributions.

Given that there is no way for a maintainer to check how a contributor created their contribution I think the important thing to have is a way to deal with the case where someone files a complaint. Many project already have a way to deal with license infringement. From my naive understanding nothing bad happens to you as long as you make a credible effort at preventing license infringements and act swiftly when one is reported. In that sense it doesn't matter how people create their contributions, a project already needs a way to deal with license infringement and that same machinery can be use to deal with license infringement as the result of reading another libraries code or using a tool that generated infringing code.

LLM powered tools create problems we already have: license infringement and large volume of work to review - and for both (and others I forgot) we need a way to deal with them, but they are not unique to the use of LLM powered tools. Automated vulnerability scanning, applying spell checkers to the whole code base, enabling "all the linter and code formatter rules", to name a few are all types of contribution that are in this category. They can be super useful and move the project forward or they can be a large time sink. I'd rather have a recommendation that deal with this whole class of problems than something specific to the use of LLMs.

[matthew-brett]

Automated vulnerability scanning, applying spell checkers to the whole code base, enabling "all the linter and code formatter rules", to name a few are all types of contribution that are in this category. They can be super useful and move the project forward or they can be a large time sink.

I could well imagine a SPEC or similar on those topics - but I don't think it would make sense to restrict discussion of AI and licenses to such a SPEC.