AI/ML Model and Pre-Trained Weight Packaging in Fedora
by Tim Flink
These questions came up in a FESCo ticket [1] recently and the primary purpose of this thread is to have some public record of the conversation around the handling of pre-trained weights for AI/ML models as packaged for Fedora.
[1] https://pagure.io/fesco/issue/3175
Intro and Definitions
=====================
Previous conversations have involved a decent amount of confusion around terminology and I want to be clear about what I'm asking so I'm starting with a few definitions in the context of my questions.
Artificial Neural Network (ANN) - effectively structured data consisting of neurons (nodes containing some value) organized into layers with various connections between the neurons. There are connections between neurons which control the flow of data through the entire network. The exact value of how the connections affect flow through the network is found through the training process and these values are generally referred to as weights.
Model - A model by itself is a description of a specific ANN - how layers are configured, how they interact with each other, how model training is done, how data needs to be structured for using a trained model and so on. A model by itself is rarely, if ever useful. Models generally need to be trained on data before they can be used but many models offer a mechanism through which weights can be loaded from a model which has already been trained. An untrained model without pre-trained weights or training is pretty much code.
Pre-Trined Weights - Pre-trained weights are essentially the data contained in a model after training the model on some input data. Training modern ANN models is a very expensive and time consuming process; pre-trained weights allow people to use models without having to train the model locally or even have access to data needed to train the model.
Questions
=========
1. Are pre-trained weights considered to be normal non-code content/data or do they require special handling?
2. If an upstream offers pre-trained weights and indicates that those weights are available under a license which is acceptable for non-code content in Fedora, can those pre-trained weights be included in Fedora packages?
3. Extending question 2, is it considered sufficient for an upstream to have a license on pre-trained weights or would a packager/reviewer need to verify that the data used to train those weights is acceptable?
4. Is it acceptable to package code which downloads pre-trained weights from a non-Fedora source upon first use post-installation by a user if that model and its associated weights are
a. For a specific model?
b. For a user-defined model which may or may not exist at the time of packaging?
I can provide examples of any of these situations if that would be helpful.
Thanks,
Tim
3 weeks, 3 days
Re: Redis will no longer be OSS... now what?
by Neal Gompa
On Wed, Mar 20, 2024 at 6:26 PM Jonathan Wright via devel
<devel(a)lists.fedoraproject.org> wrote:
>
> We can potentially look to https://github.com/Snapchat/KeyDB which I've been loosely working on packaging anyway.
>
I'll want to test this for Pagure at least, since we're going to have
to switch our recommendations around soon because of this.
--
真実はいつも一つ！/ Always, there's only one truth!
1 month, 1 week
Re: Trivy for licenses
by Maxwell G
On Tue Mar 5, 2024 at 04:06 +0000, Maxwell G wrote:
> On Mon Mar 4, 2024 at 22:35 +0100, Sandro wrote:
> > On 04-03-2024 07:59, Miroslav Suchý wrote:
> > > It would welcome if anyone can help Robert here:
> > > https://bugzilla.redhat.com/show_bug.cgi?id=2235055
> >
> > I had a look and it seems the package is currently stuck on broken
> > python-pymaven-patch, which requires python-lxml < 5~~. In rawhide and
> > f40 python-lxml was updated to 5.1.0 two months ago.
> >
> > For about as long there has been a PR open for python-pymaven-patch
> > removing that version constraint. Notably, the maintainer of
> > python-pymaven-patch is the same person, who submitted the
> > scancode-toolkit review request.
> >
> > There may be more trouble down the road. But for the moment, I don't see
> > how others can help driving this forward. A proven packager could merge
> > the PR. But I don't know how eclipseo, who's a proven packager, would
> > feel about that.
>
> Fixing FTI bugs that are unaddressed by a package's maintainer
> definitely falls under the purview of a provenpackager.
> I rebased the PR [1] and will merge it once CI passes.
>
> [1] https://src.fedoraproject.org/rpms/python-pymaven-patch/pull-request/1
It also looks like a bunch of the tests are failing and have been
skipped. That's not super ideal. It looks like [1] has been open
upstream for some time. I have not yet looked closely at the failures,
but Philippe, if you have any pointers to give, that would certainly be
helpful.
[1] https://github.com/nexB/scancode-toolkit/issues/3496
1 month, 3 weeks
Re: Trivy for licenses
by Maxwell G
On Mon Mar 4, 2024 at 07:59 +0100, Miroslav Suchý wrote:
> Dne 03. 03. 24 v 20:22 Philippe Ombredanne napsal(a):
>
> > If you want robust license detection, consider using ScanCode [2] and
> > Scancode.io [3] for more complex pipelines. Both are tools that I
> > co-maintain and are considered as better tools for this. Do not
> > hesitate to reach out for help!
>
> *nod*
>
> It would welcome if anyone can help Robert here:
> https://bugzilla.redhat.com/show_bug.cgi?id=2235055
Robert has not been very responsive as of late. It might be a good idea
for someone else to pick it up and start a new review ticket.
1 month, 3 weeks
Trivy for licenses
by Maxwell G
Hi,
Has anyone every used trivy [1] to scan for licenses? It appears more
robust and better maintained than askalono-cli and can detect files with
multiple licenses and licenses embedded in file headers. I have been
running it with "trivy fs --scanners license --license-full ."
[1] https://github.com/aquasecurity/trivy
--
Maxwell G (@gotmax23)
Pronouns: He/They
1 month, 3 weeks
License check for pyuca
by Sandro
Hi,
A while ago I opened issue #39[1] on GitLab. There have been some recent
developments regarding the license. Upstream is welcoming a PR for
fixing/clarifying the license situation.
I have prepared a draft PR, but before I submit that for upstream to
review and hopefully merge, I'd like someone from legal to go over my
proposal and tell me if that PR would allow me to proceed with the
package review of python-pyuca.
More information can be found in the ticket, in particular my latest
comment.
[1] https://gitlab.com/fedora/legal/fedora-license-data/-/issues/379
Cheers,
--
Sandro
FAS: gui1ty
Matrix: Penguinpee
Elsewhere: [Pp]enguinpee
1 month, 3 weeks
Fwd: SPDX Statistics - Beginning of the year edition
by Miroslav Suchý
-------- Přeposlaná zpráva --------
Předmět: SPDX Statistics - Beginning of the year edition
Datum: Fri, 1 Mar 2024 09:27:48 +0100
Od: Miroslav Suchý <msuchy(a)redhat.com>
Společnost: Red Hat Czech, s.r.o.
Komu: Development discussions related to Fedora <devel(a)lists.fedoraproject.org>
Hot news:
fedora-license-data has Copr project https://copr.fedorainfracloud.org/coprs/g/osci/fedora-license-data where new
package is built whenever new PR is merged
The last phase is ready for wrangler https://fedoraproject.org/wiki/Changes/SPDX_Licenses_Phase_4 and we will
proceed when approved with FESCO.
I corrected lots of SPDX formula where you used lowercase "and", "or". The specification allows only "AND", "OR".
This will likely change in specification version 3, but now the operator has to be upper case.
Two weeks ago we had:
> * 23737spec files in Fedora
>
> * 30335license tags in all spec files
>
> * 11314 tags have not been converted to SPDX yet
>
> * 5105 tags can be trivially converted using `license-fedora2spdx`
>
> * Progress: 62,70% ░░░░░░████ 100%
>
> ELN subset:
>
> 128 out of 2412 packages are not converted yet (progress 94.69%)
>
Today we have:
* 23786spec files in Fedora
* 30396license tags in all spec files
* 11182 tags have not been converted to SPDX yet
* 5044 tags can be trivially converted using `license-fedora2spdx`
* Progress: 63,21% ░░░░░░████ 100%
ELN subset:
112 out of 2411 packages are not converted yet (progress 95.35%)
Graph of these data with the burndown chart:
https://docs.google.com/spreadsheets/d/1QVMEzXWML-6_Mrlln02axFAaRKCQ8zE80...
The list of packages needed to be converted is here:
https://pagure.io/copr/license-validate/blob/main/f/packages-without-spdx...
List by package maintainers is here
https://pagure.io/copr/license-validate/blob/main/f/packages-without-spdx...
List of packages from ELN subset that needs to be converted:
https://pagure.io/copr/license-validate/blob/main/f/eln-not-migrated.txt
New version of fedora-license-data has been released. With:
7 new licenses (plus some public domain declarations).
17 licenses are waiting to be review by SPDX.org (and then to be added to fedora-license-data)
https://gitlab.com/fedora/legal/fedora-license-data/-/issues/?label_name%...
Legal docs and especially
https://docs.fedoraproject.org/en-US/legal/allowed-licenses/
was updated too.
License analysis of remaining packages: http://miroslav.suchy.cz/fedora/spdx-reports/
New projection when we will be finished is 2025-02-01 (+17 days from last report). Pure linear approximation.
If your package does not have neither git-log entry nor spec-changelog entry mentioning SPDX and you know your license
tag matches SPDX formula, you can put your package on ignore list
https://pagure.io/copr/license-validate/blob/main/f/ignore-packages.txt
Either pull-request or direct email to me is fine.
Why Beggining of the year? That should be January 1st, right? Before the advent of the Gregorian calendar, March 1st was
considered the beginning of the year. Hence Septemeber as the "seventh month" despite the fact it is 9th now. But in the
Republic of Venice, for example, March was considered the beginning of the year until 1797. So in Venice, March 1790 <
January 1790. For more interesting facts about time (and time zones) see this legendary video
https://www.youtube.com/watch?v=-5wpm-gesOY
Miroslav
1 month, 3 weeks
