Hi Guys,
AFAIK, the Yocto Project is an open source collaboration project that provides tools, templates, and methods to help developers create custom Linux-based systems for embedded devices.
My confusion is that:
1. what`s the tool to make our Fedora Linux 38 released like Yocto? 2. And Centos ? 3. And Ubuntu ?
Warm regards Jun Miao
Miao, Jun wrote:
Hi Guys,
First of all, we are not all guys. (I happen to be one though.)
AFAIK, the Yocto Project is an open source collaboration project that provides tools, templates, and methods to help developers create custom Linux-based systems for embedded devices.
My confusion is that:
- what`s the tool to make our Fedora Linux 38 released like Yocto?
There is more than one tool in use. The main build tool is Koji, but it depends on other underlying tools such as Mock, DNF, RPM, etc. Also keep in mind that in Fedora, all binaries are natively compiled, not cross-compiled as in Yocto. I.e., the rpmbuild process needs to run on a (real or emulated) CPU of the architecture for which you want to build the package. RPMs do not typically support cross-compilation.
- And Centos ?
The only CentOS that is left is CentOS Stream, which (as far as I know) also uses Koji.
The CentOS successors such as Rocky Linux and AlmaLinux are free to either also use Koji or use their own tools, or a combination. E.g., Rocky Linux has developed Peridot.
- And Ubuntu ?
Ubuntu has its own completely different tooling built around Debian's tooling and Canonical's Launchpad platform.
Kevin Kofler
On Sun, Jul 16, 2023 at 7:30 AM Kevin Kofler via devel devel@lists.fedoraproject.org wrote:
Miao, Jun wrote:
Hi Guys,
First of all, we are not all guys. (I happen to be one though.)
I would advise not to get too picky about this. The term "guys" isn't necessarily gender-specific. In its plural form, it has basically replaced "folks" in many English dialects, and holds the same non-gendered meaning. The antonyms, "guy" and "gal" have largely fallen out of favor in English, with the latter being quite rare, and the former's plural form evolving into a non-gendered usage, similar to "dude(s.)" and "dudes(pl.)" being used in non-gendered ways. While I think it's a good idea to use gender-neutral language when gendered language isn't needed, this one actually is pretty gender-neutral already, at least in the way many people use it. So, I recommend not reading too much into casual greetings like this. Getting too picky runs the risk of polarizing people and making it harder to make changes for the better where it matters. Assuming good intentions, it seems like the author intended this in the common, modern, non-gendered plural form to me.
AFAIK, the Yocto Project is an open source collaboration project that provides tools, templates, and methods to help developers create custom Linux-based systems for embedded devices.
My confusion is that:
- what`s the tool to make our Fedora Linux 38 released like Yocto?
There is more than one tool in use. The main build tool is Koji, but it depends on other underlying tools such as Mock, DNF, RPM, etc. Also keep in mind that in Fedora, all binaries are natively compiled, not cross-compiled as in Yocto. I.e., the rpmbuild process needs to run on a (real or emulated) CPU of the architecture for which you want to build the package. RPMs do not typically support cross-compilation.
When I saw the original post, I was interested to see what responses it would trigger. I must admit that this one is a bit disappointing. While I've been building RPMs for Fedora for awhile now, one of the things that has eluded me is that the Fedora OS composes seem to be very opaque to me. I know enough to understand that Koji tags rpm builds from individual buildroots, but from there, the process to build the repos and the installation media, or how the buildroots are created in the first place for Koji, or any other weirdness involved in the construction of the OS as a whole from the individual RPM builds, all of that seems opaque to me. I think it'd be great if there was an up-to-date and detailed step-by-step guide for how to build a Fedora release. Such documentation should be detailed enough that one could stand up their own fork of the Fedora OS... not because we want to encourage that... but because that's the level of detail that I think is needed to allow volunteers to step in and get involved, as the current folks move on, due to retirement, death, boredom, or whatever. If such documentation already exists, I don't know where it could be found.
Do you know of any such detailed documentation, step-by-step instructions, or maybe slides/presentations on the compose process or overall Fedora OS build systems?
- And Centos ?
The only CentOS that is left is CentOS Stream, which (as far as I know) also uses Koji.
The CentOS successors such as Rocky Linux and AlmaLinux are free to either also use Koji or use their own tools, or a combination. E.g., Rocky Linux has developed Peridot.
- And Ubuntu ?
Ubuntu has its own completely different tooling built around Debian's tooling and Canonical's Launchpad platform.
Kevin Kofler
devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Am 16.07.23 um 21:24 schrieb Christopher:
On Sun, Jul 16, 2023 at 7:30 AM Kevin Kofler via devel devel@lists.fedoraproject.org wrote:
Miao, Jun wrote:
Hi Guys,
First of all, we are not all guys. (I happen to be one though.)
I would advise not to get too picky about this. The term "guys" isn't necessarily gender-specific. In its plural form, it has basically replaced "folks" in many English dialects, and holds the same non-gendered meaning. The antonyms, "guy" and "gal" have largely fallen out of favor in English, with the latter being quite rare, and the former's plural form evolving into a non-gendered usage, similar to "dude(s.)" and "dudes(pl.)" being used in non-gendered ways. While I think it's a good idea to use gender-neutral language when gendered language isn't needed, this one actually is pretty gender-neutral already, at least in the way many people use it. So, I recommend not reading too much into casual greetings like this. Getting too picky runs the risk of polarizing people and making it harder to make changes for the better where it matters. Assuming good intentions, it seems like the author intended this in the common, modern, non-gendered plural form to me.
AFAIK, the Yocto Project is an open source collaboration project that provides tools, templates, and methods to help developers create custom Linux-based systems for embedded devices.
My confusion is that:
- what`s the tool to make our Fedora Linux 38 released like Yocto?
There is more than one tool in use. The main build tool is Koji, but it depends on other underlying tools such as Mock, DNF, RPM, etc. Also keep in mind that in Fedora, all binaries are natively compiled, not cross-compiled as in Yocto. I.e., the rpmbuild process needs to run on a (real or emulated) CPU of the architecture for which you want to build the package. RPMs do not typically support cross-compilation.
When I saw the original post, I was interested to see what responses it would trigger. I must admit that this one is a bit disappointing. While I've been building RPMs for Fedora for awhile now, one of the things that has eluded me is that the Fedora OS composes seem to be very opaque to me. I know enough to understand that Koji tags rpm builds from individual buildroots, but from there, the process to build the repos and the installation media, or how the buildroots are created in the first place for Koji, or any other weirdness involved in the construction of the OS as a whole from the individual RPM builds, all of that seems opaque to me. I think it'd be great if there was an up-to-date and detailed step-by-step guide for how to build a Fedora release. Such documentation should be detailed enough that one could stand up their own fork of the Fedora OS... not because we want to encourage that... but because that's the level of detail that I think is needed to allow volunteers to step in and get involved, as the current folks move on, due to retirement, death, boredom, or whatever. If such documentation already exists, I don't know where it could be found.
Do you know of any such detailed documentation, step-by-step instructions, or maybe slides/presentations on the compose process or overall Fedora OS build systems?
https://docs.pagure.org/releng/
On Sun, 2023-07-16 at 15:24 -0400, Christopher wrote:
There is more than one tool in use. The main build tool is Koji, but it depends on other underlying tools such as Mock, DNF, RPM, etc. Also keep in mind that in Fedora, all binaries are natively compiled, not cross-compiled as in Yocto. I.e., the rpmbuild process needs to run on a (real or emulated) CPU of the architecture for which you want to build the package. RPMs do not typically support cross-compilation.
When I saw the original post, I was interested to see what responses it would trigger. I must admit that this one is a bit disappointing. While I've been building RPMs for Fedora for awhile now, one of the things that has eluded me is that the Fedora OS composes seem to be very opaque to me. I know enough to understand that Koji tags rpm builds from individual buildroots, but from there, the process to build the repos and the installation media, or how the buildroots are created in the first place for Koji, or any other weirdness involved in the construction of the OS as a whole from the individual RPM builds, all of that seems opaque to me. I think it'd be great if there was an up-to-date and detailed step-by-step guide for how to build a Fedora release. Such documentation should be detailed enough that one could stand up their own fork of the Fedora OS... not because we want to encourage that... but because that's the level of detail that I think is needed to allow volunteers to step in and get involved, as the current folks move on, due to retirement, death, boredom, or whatever. If such documentation already exists, I don't know where it could be found.
Do you know of any such detailed documentation, step-by-step instructions, or maybe slides/presentations on the compose process or overall Fedora OS build systems?
This is the wrong question, kinda. There is no detailed step-by-step process. The process for creating a compose is, more or less, "push the magic COMPOSE NOW" button. (Okay, there's a *bit* more to it than that, but not a lot). The SOP for it is https://docs.pagure.org/releng/sop_composing_fedora.html .
All the complexity, these days, is in what happens when you push the button. Which is so complex I just couldn't stand the thought of sitting down and writing it all out.
*basically*...more or less...what happens when you hit the button is that pungi - https://pagure.io/pungi - following the Fedora pungi config - https://pagure.io/pungi-fedora - creates a whole bunch of Koji tasks. Each of those Koji tasks does...something, there are a lot of somethings, often using different tools.
The whole process is logged in overwhelming detail - https://kojipkgs.fedoraproject.org/compose/rawhide/Fedora-Rawhide-20230716.n... . Often the pungi logs (those ones) wind up just pointing you to a Koji task, where you will find the actual logs, e.g. the pungi 'log' for the KDE live media creation is https://kojipkgs.fedoraproject.org/compose/rawhide/Fedora-Rawhide-20230716.n... , which points you to https://koji.fedoraproject.org/koji/taskinfo?taskID=103428940 where you can find the 'real' logs.
Adam Williamson wrote:
This is the wrong question, kinda. There is no detailed step-by-step process. The process for creating a compose is, more or less, "push the magic COMPOSE NOW" button. (Okay, there's a *bit* more to it than that, but not a lot). The SOP for it is https://docs.pagure.org/releng/sop_composing_fedora.html .
All the complexity, these days, is in what happens when you push the button. Which is so complex I just couldn't stand the thought of sitting down and writing it all out.
*basically*...more or less...what happens when you hit the button is that pungi - https://pagure.io/pungi - following the Fedora pungi config - https://pagure.io/pungi-fedora - creates a whole bunch of Koji tasks. Each of those Koji tasks does...something, there are a lot of somethings, often using different tools.
Note though (and I am writing this for other readers, I know *you* know this) that the compose process you describe here does not actually compile any packages. It just collects the packages that have previously been built in a given Koji tag and builds repositories and/or images from those.
Kevin Kofler
On Mon, 2023-07-17 at 02:58 +0200, Kevin Kofler via devel wrote:
Adam Williamson wrote:
This is the wrong question, kinda. There is no detailed step-by-step process. The process for creating a compose is, more or less, "push the magic COMPOSE NOW" button. (Okay, there's a *bit* more to it than that, but not a lot). The SOP for it is https://docs.pagure.org/releng/sop_composing_fedora.html .
All the complexity, these days, is in what happens when you push the button. Which is so complex I just couldn't stand the thought of sitting down and writing it all out.
*basically*...more or less...what happens when you hit the button is that pungi - https://pagure.io/pungi - following the Fedora pungi config - https://pagure.io/pungi-fedora - creates a whole bunch of Koji tasks. Each of those Koji tasks does...something, there are a lot of somethings, often using different tools.
Note though (and I am writing this for other readers, I know *you* know this) that the compose process you describe here does not actually compile any packages. It just collects the packages that have previously been built in a given Koji tag and builds repositories and/or images from those.
Right, of course - this part of the overall process starts from whatever set of packages is tagged 'f39' (or whatever) at the time it starts. Thanks for noting that!
On Thu, 13 Jul 2023 at 21:45, Miao, Jun jun.miao@intel.com wrote:
Hi Guys,
AFAIK, the Yocto Project is an open source collaboration project that provides tools, templates, and methods to help developers create custom Linux-based systems for embedded devices.
My confusion is that:
- what`s the tool to make our Fedora Linux 38 released like Yocto?
- And Centos ?
- And Ubuntu ?
I believe the Yocto system is based on a similar system as Gentoo. It was chosen because most embedded teams at the time were fairly small (maybe 1-2 engineers per 'board') and setting up a multi-system pipeline was too much for getting a system which may only have a 'lifetime' of 1-2 years with a large amount of NDAs and other legal entanglements involved. It also came with the advantage that one could pick and choose layers to change overall design so a basic chipset could be found in a camera, phone, or washing machine but have different additions/subtractions done to make it work for each. As teams got larger, the tooling has become more complex so you can stand up dedicated build clusters and such but in general, a developer can figure out what will make something work themselves without bothering other developers.
My outsider look at Yocto OS development seems to have developers either focusing on specific problem sets: 'make this work on a refrigerator with this much memory/cpu/storage and these applications' or an overall 'layer' solution: 'Legal says no GPLv3 so please make sure none of them are included in an image' or 'we need to have one image with debug symbols and this debugger, and this image without', etc.
One problem with this style of development is that you can easily end up with 2 developers working on the same board but having chosen completely different optimizations and compile time choices that applications and libraries can not easily mix between the two.
The other operating systems you are looking at is aimed at a different problem where how do you get a large number of developers who may only focus on one bit be able to work with each other in a smooth as much manner. This means that the system is less about building the entire operating system in one location but more about makings sure that 400 developers can build stuff which will work together. This means that a developer builds only items which are for testing and developer, and then pushes changes to a central system which will attempt to rebuild things using choices (compiler options, disk layout, etc) set by either a Fedora Engineering Steering Committee decision or general agreement by the main maintainer of a subsystem.
The Fedora and CentOS build systems are fairly complicated with about 70 to 100 interrelated applications from when a developer does a build to when someone can download a finished package on a mirror. I have tried writing out a 'map' like a subway system but I usually get lost somewhere in the middle. The general solution is something like
[fedpkg] -> [Fedora Builders and Source Control] -> [Fedora QA] -> [Fedora Image Making (iso/containers/etc)] -> [Mirroring System] -> [User]
The above 'loses' a lot of granular tools in that set of items with various message buses, various side tools, etc. The lost granularity covers various attempts to deal with concurrency problems where group A is updating the entire python stack while group B is still building updates based on the old stack. Group A's work may take weeks so stopping group B during that time could lead to security issues. [There are many other concurrency which the community has found over the years but that was the first one which came to mind.]
Many kinds to me. Although I just maintain a few of small packages and only have a curiosity to this tools suddenly, I got so many people serious answers. Thanks again.
Warm regards Jun Miao
From: Stephen Smoogen ssmoogen@redhat.com Sent: Monday, July 17, 2023 7:21 PM To: Development discussions related to Fedora devel@lists.fedoraproject.org Cc: Miao, Jun jun.miao@intel.com Subject: Re: [Question] how to make Fedora linux os ?
On Thu, 13 Jul 2023 at 21:45, Miao, Jun <jun.miao@intel.commailto:jun.miao@intel.com> wrote: Hi Guys,
AFAIK, the Yocto Project is an open source collaboration project that provides tools, templates, and methods to help developers create custom Linux-based systems for embedded devices.
My confusion is that:
1. what`s the tool to make our Fedora Linux 38 released like Yocto? 2. And Centos ? 3. And Ubuntu ?
I believe the Yocto system is based on a similar system as Gentoo. It was chosen because most embedded teams at the time were fairly small (maybe 1-2 engineers per 'board') and setting up a multi-system pipeline was too much for getting a system which may only have a 'lifetime' of 1-2 years with a large amount of NDAs and other legal entanglements involved. It also came with the advantage that one could pick and choose layers to change overall design so a basic chipset could be found in a camera, phone, or washing machine but have different additions/subtractions done to make it work for each. As teams got larger, the tooling has become more complex so you can stand up dedicated build clusters and such but in general, a developer can figure out what will make something work themselves without bothering other developers.
My outsider look at Yocto OS development seems to have developers either focusing on specific problem sets: 'make this work on a refrigerator with this much memory/cpu/storage and these applications' or an overall 'layer' solution: 'Legal says no GPLv3 so please make sure none of them are included in an image' or 'we need to have one image with debug symbols and this debugger, and this image without', etc.
One problem with this style of development is that you can easily end up with 2 developers working on the same board but having chosen completely different optimizations and compile time choices that applications and libraries can not easily mix between the two.
The other operating systems you are looking at is aimed at a different problem where how do you get a large number of developers who may only focus on one bit be able to work with each other in a smooth as much manner. This means that the system is less about building the entire operating system in one location but more about makings sure that 400 developers can build stuff which will work together. This means that a developer builds only items which are for testing and developer, and then pushes changes to a central system which will attempt to rebuild things using choices (compiler options, disk layout, etc) set by either a Fedora Engineering Steering Committee decision or general agreement by the main maintainer of a subsystem.
The Fedora and CentOS build systems are fairly complicated with about 70 to 100 interrelated applications from when a developer does a build to when someone can download a finished package on a mirror. I have tried writing out a 'map' like a subway system but I usually get lost somewhere in the middle. The general solution is something like
[fedpkg] -> [Fedora Builders and Source Control] -> [Fedora QA] -> [Fedora Image Making (iso/containers/etc)] -> [Mirroring System] -> [User]
The above 'loses' a lot of granular tools in that set of items with various message buses, various side tools, etc. The lost granularity covers various attempts to deal with concurrency problems where group A is updating the entire python stack while group B is still building updates based on the old stack. Group A's work may take weeks so stopping group B during that time could lead to security issues. [There are many other concurrency which the community has found over the years but that was the first one which came to mind.]
-- Stephen Smoogen, Red Hat Automotive Let us be kind to one another, for most of us are fighting a hard battle. -- Ian MacClaren
-
Adam Williamson -
Christopher -
Kevin Kofler -
Leon Fauster -
Miao, Jun -
Stephen Smoogen