Hi, several months ago I talked to dgilmore and probinso and we talked about using DNF in Koji and start building F22 packages using DNF resolver. The conlusion was that it is quite late in cycle. And it is better to start with plain Mock and leave user to test it on their local workstatations. Mock is using DNF for fedora-rawhide-* targets for some time already. And I received none complaints. Therefore I'm raising question: Can we switch Koji to use DNF for building rawhide targets? Of course F22- will still use yum for building. It would be great if we can make this change before mass rebuils for F23.
On Wednesday, May 20, 2015 09:22:22 AM Miroslav Suchý wrote:
Hi, several months ago I talked to dgilmore and probinso and we talked about using DNF in Koji and start building F22 packages using DNF resolver. The conlusion was that it is quite late in cycle. And it is better to start with plain Mock and leave user to test it on their local workstatations. Mock is using DNF for fedora-rawhide-* targets for some time already. And I received none complaints. Therefore I'm raising question: Can we switch Koji to use DNF for building rawhide targets? Of course F22- will still use yum for building. It would be great if we can make this change before mass rebuils for F23.
We can not because koji does not have the ability to set the package backend depending on different targets. until koji supports being able to use dnf only for some targets we will still use yum everywhere.
Dennis
Dne 20.5.2015 v 16:52 Dennis Gilmore napsal(a):
We can not because koji does not have the ability to set the package backend depending on different targets. until koji supports being able to use dnf only for some targets we will still use yum everywhere.
Fair enough. Can we start with this patch? Not tested thou as I do not have staging Koji environment.
On Thursday, May 21, 2015 11:26:35 AM Miroslav Suchý wrote:
Dne 20.5.2015 v 16:52 Dennis Gilmore napsal(a):
We can not because koji does not have the ability to set the package backend depending on different targets. until koji supports being able to use dnf only for some targets we will still use yum everywhere.
Fair enough. Can we start with this patch? Not tested thou as I do not have staging Koji environment.
It will not be good enough, when the builders go to fedora 22 we will have to use yum-deprecated and not yum. this is all seriously a huge mess that needs to not be rushed. it needs to not be a True/False type setup it needs to be if there is a install command for the tag, that can be inherited use that, if not use yum. we also need to ensure that it will work with mock in rhel5 that does not support setting the backend command.
Dennis
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256
Dne 21.5.2015 v 21:02 Dennis Gilmore napsal(a):
It will not be good enough, when the builders go to fedora 22 we will have to use yum-deprecated and not yum. this is all seriously a huge mess that needs to not be rushed. it needs to not be a True/False type setup it needs to be if there is a install command for the tag, that can be inherited use that, if not use yum.
This is already handled by Mock. On F22 when Mock should use yum it use yum-deprecated. https://bugzilla.redhat.com/show_bug.cgi?id=1211978
we also need to ensure that it will work with mock in rhel5 that does not support setting the backend command.
We have some builder, which use rhel5? Mock are not released for RHEL5 for some time. And if you put into config some option, which mock does not recognize, then it will be simply ignored. It will be just unused variable. So unless you want to build F23 packages on RHEL5 host, then this will work.
Mirek
On Friday, May 22, 2015 01:26:16 PM Miroslav Suchy wrote:
Dne 21.5.2015 v 21:02 Dennis Gilmore napsal(a):
It will not be good enough, when the builders go to fedora 22 we will have to use yum-deprecated and not yum. this is all seriously a huge mess that needs to not be rushed. it needs to not be a True/False type setup it needs to be if there is a install command for the tag, that can be inherited use that, if not use yum.
This is already handled by Mock. On F22 when Mock should use yum it use yum-deprecated. https://bugzilla.redhat.com/show_bug.cgi?id=1211978
That is not a workable solution. we have to overwrite the site-defaults.cfg and have it work across rhel and fedora builders in fedora. the solution really needs to be a combination of things. I think the koji code needs to assume that there could be multiple possible backends, so we can set yum, yum- deprecated, dnf. but additionally mock needs to really know that if the host is f22 or newer or rhel8(big assumption here) or newer and you say yum is the backend you mean yum-deprecated. I am assuming yum-deprectaed will be in one of RHEL or EPEL.
we also need to ensure that it will work with mock in rhel5 that does not support setting the backend command.
We have some builder, which use rhel5? Mock are not released for RHEL5 for some time. And if you put into config some option, which mock does not recognize, then it will be simply ignored. It will be just unused variable. So unless you want to build F23 packages on RHEL5 host, then this will work.
yes, so long as mock does not choke on unkown config options we should be okay.
Dennis
Dne 23.5.2015 v 01:27 Dennis Gilmore napsal(a):
That is not a workable solution. we have to overwrite the site-defaults.cfg
I disagree here. But I'm not the one who operate the Koji.
and have it work across rhel and fedora builders in fedora. the solution really needs to be a combination of things. I think the koji code needs to assume that there could be multiple possible backends, so we can set yum, yum- deprecated, dnf. but additionally mock needs to really know that if the host is f22 or newer or rhel8(big assumption here) or newer and you say yum is the
The decision of mock is config driven. And mock ships different config for different distribution.
backend you mean yum-deprecated. I am assuming yum-deprectaed will be in one of RHEL or EPEL.
DNF (with yum-deprecated) is in EPEL (however just EPEL7), but there is no dnf-plugins-core, which is needed too.
Can you or somebody else from rel-engs alter my patch or come with different solution? I would like to remind that according https://fedorahosted.org/rel-eng/ticket/6162 we have just 3 weeks to do that in F23 time-frame. Then it will be again too late, and you will have to target for F24.
On Monday, May 25, 2015 08:45:21 AM Miroslav Suchý wrote:
Dne 23.5.2015 v 01:27 Dennis Gilmore napsal(a):
That is not a workable solution. we have to overwrite the site-defaults.cfg
I disagree here. But I'm not the one who operate the Koji.
I think I am doing a very poor job of explaining myself
today we have to set some options in the site-defaults.cfg file cat /etc/mock/site-defaults.cfg config_opts['plugin_conf']['package_state_enable'] = False config_opts['plugin_conf']['ccache_enable'] = False
maybe we do not need to explicitly disable ccache but we do. this is managed in ansible. so now we need to make two copies of the site-defaults.cfg file. first we have to know we need to do it(which we clearly do, but other koji users may not) then second whenever we update the site-defaults.cfg, which is not often we need to remember we have to make the change twice. unless of course we can use the dafult which we can not. the package_state_enable plugin was causing build failures and was duplicating work koji does internally, so we disabled it.
an example from fedora's koji setup, the arm, and x86 builders are all running Fedora 21, the ppc builders are running rhel. so for all the epel targets we would need to set the backend to be yum. but something on the fedora builders needs to know to translate yum to yum-deprecated.
I think that we have to assume that more backends will come along, apt or zypper or some other distros tool or a new one we write to replace dnf. So I strongly believe that the solution needs to be flexible to allow for future unknown changes.
I think it is reasonable to set the backend to yum-deprecated. To me the way to support this should be in mock based on the host os. we know there is a mapping between yum and yum-deprecated, if the host is f22 and newer or EL8 and newer when you say yum you really mean yum-deprecated and if you say yum- deprecated on something older you really mean yum. To me doing that outside of the code while it works for some cases it's not flexible.
and have it work across rhel and fedora builders in fedora. the solution really needs to be a combination of things. I think the koji code needs to assume that there could be multiple possible backends, so we can set yum, yum- deprecated, dnf. but additionally mock needs to really know that if the host is f22 or newer or rhel8(big assumption here) or newer and you say yum is the
The decision of mock is config driven. And mock ships different config for different distribution.
backend you mean yum-deprecated. I am assuming yum-deprectaed will be in one of RHEL or EPEL.
DNF (with yum-deprecated) is in EPEL (however just EPEL7), but there is no dnf-plugins-core, which is needed too.
Can you or somebody else from rel-engs alter my patch or come with different solution? I would like to remind that according https://fedorahosted.org/rel-eng/ticket/6162 we have just 3 weeks to do that in F23 time-frame. Then it will be again too late, and you will have to target for F24.
https://fedoraproject.org/wiki/FAD_Release_Tools_and_Infrastructure_2015#Del... it is one of the deliverables for the FAD next week. we do want a solution. I will try comeup with something I see as a workable option. but someone else is free to do something also. Thank you for taking a stab at it
Dennis
Dne 28.5.2015 v 20:29 Dennis Gilmore napsal(a):
an example from fedora's koji setup, the arm, and x86 builders are all running Fedora 21, the ppc builders are running rhel. so for all the epel targets we would need to set the backend to be yum. but something on the fedora builders needs to know to translate yum to yum-deprecated.
Easy. Instead of - name: mock site-defaults.cfg copy: src=builders/site-defaults.cfg dest=/etc/mock/site-defaults.cfg mode=0644 owner=root group=mock
you will use:
- name: mock site-defaults.cfg template: src=builders/site-defaults.cfg dest=/etc/mock/site-defaults.cfg mode=0644 owner=root group=mock --^^^^^^^^
and in that template use if-condition based on the value of {{ansible_distribution}}
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256
Dne 28.5.2015 v 20:29 Dennis Gilmore napsal(a):
and have it work across rhel and fedora builders in fedora. the solution
really needs to be a combination of things. I think the koji code needs to assume that there could be multiple possible backends, so we can set yum, yum- deprecated, dnf. but additionally mock needs to really know that if the host is f22 or newer or rhel8(big assumption here) or newer and you say yum is the
The decision of mock is config driven. And mock ships different config for different distribution.
backend you mean yum-deprecated. I am assuming yum-deprectaed will be in one of RHEL or EPEL.
DNF (with yum-deprecated) is in EPEL (however just EPEL7), but there is no dnf-plugins-core, which is needed too.
Can you or somebody else from rel-engs alter my patch or come with different solution? I would like to remind that according https://fedorahosted.org/rel-eng/ticket/6162 we have just 3 weeks to do that in F23 time-frame. Then it will be again too late, and you will have to target for F24.
https://fedoraproject.org/wiki/FAD_Release_Tools_and_Infrastructure_20
15#Deliverables
it is one of the deliverables for the FAD next week. we do want a solution. I will try comeup with something I see as a workable option. but someone else is free to do something also. Thank you for taking a stab at it
AFAIK the FAD is over, so do you have some solution?
Mirek
On Thu, Jun 11, 2015 at 8:08 AM, Miroslav Suchy msuchy@redhat.com wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256
Dne 28.5.2015 v 20:29 Dennis Gilmore napsal(a):
and have it work across rhel and fedora builders in fedora. the solution
really needs to be a combination of things. I think the koji code needs to assume that there could be multiple possible backends, so we can set yum, yum- deprecated, dnf. but additionally mock needs to really know that if the host is f22 or newer or rhel8(big assumption here) or newer and you say yum is the
The decision of mock is config driven. And mock ships different config for different distribution.
backend you mean yum-deprecated. I am assuming yum-deprectaed will be in one of RHEL or EPEL.
DNF (with yum-deprecated) is in EPEL (however just EPEL7), but there is no dnf-plugins-core, which is needed too.
Can you or somebody else from rel-engs alter my patch or come with different solution? I would like to remind that according https://fedorahosted.org/rel-eng/ticket/6162 we have just 3 weeks to do that in F23 time-frame. Then it will be again too late, and you will have to target for F24.
https://fedoraproject.org/wiki/FAD_Release_Tools_and_Infrastructure_20
15#Deliverables
it is one of the deliverables for the FAD next week. we do want a solution. I will try comeup with something I see as a workable option. but someone else is free to do something also. Thank you for taking a stab at it
AFAIK the FAD is over, so do you have some solution?
Still in process, there's a full understanding of the problem and Mike is working on the final implementation. There should be patches posted for review in the next week or so.
Peter
On 06/11/2015 05:01 AM, Peter Robinson wrote:
On Thu, Jun 11, 2015 at 8:08 AM, Miroslav Suchy msuchy@redhat.com wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256
Dne 28.5.2015 v 20:29 Dennis Gilmore napsal(a):
and have it work across rhel and fedora builders in fedora. the solution
> really needs to be a combination of things. I think the > koji code needs to assume that there could be multiple > possible backends, so we can set yum, yum- deprecated, dnf. > but additionally mock needs to really know that if the host > is f22 or newer or rhel8(big assumption here) or newer and > you say yum is the
The decision of mock is config driven. And mock ships different config for different distribution.
> backend you mean yum-deprecated. I am assuming > yum-deprectaed will be in one of RHEL or EPEL.
DNF (with yum-deprecated) is in EPEL (however just EPEL7), but there is no dnf-plugins-core, which is needed too.
Can you or somebody else from rel-engs alter my patch or come with different solution? I would like to remind that according https://fedorahosted.org/rel-eng/ticket/6162 we have just 3 weeks to do that in F23 time-frame. Then it will be again too late, and you will have to target for F24.
https://fedoraproject.org/wiki/FAD_Release_Tools_and_Infrastructure_20
15#Deliverables
it is one of the deliverables for the FAD next week. we do want a solution. I will try comeup with something I see as a workable option. but someone else is free to do something also. Thank you for taking a stab at it
AFAIK the FAD is over, so do you have some solution?
Still in process, there's a full understanding of the problem and Mike is working on the final implementation. There should be patches posted for review in the next week or so.
Sorry for the delay, I meant to finish these up last week.
Patches posted to koj-devel https://lists.fedorahosted.org/pipermail/koji-devel/2015-June/000034.html
One thing I'm not sure about is the releasever issue. https://bugzilla.redhat.com/show_bug.cgi?id=1173107
I have currently worked around it by setting config_opts['releasever'] = 'NULL' in site-defaults.cfg
Do we expect dnf to fix this bug, or will we be forced to work around it in a more permanent fashion?
Dne 17.6.2015 v 21:50 Mike McLean napsal(a):
Do we expect dnf to fix this bug, or will we be forced to work around it in a more permanent fashion?
I just talked to JSilhan and he told me that we can *not* expect the fix soon.
config_opts['releasever'] = 'NULL' Is good workaround from my POV.
Mirek
On Wed, May 20, 2015 at 8:22 AM, Miroslav Suchý msuchy@redhat.com wrote:
Hi, several months ago I talked to dgilmore and probinso and we talked about using DNF in Koji and start building F22 packages using DNF resolver. The conlusion was that it is quite late in cycle. And it is better to start with plain Mock and leave user to test it on their local workstatations. Mock is using DNF for fedora-rawhide-* targets for some time already. And I received none complaints.
One complaint I have is displaying of detailed dep-solving. To get the old style yum output would be very useful. I can get some of it back by adding -v to the cmd line but if a package adds a new dep for example it just says "adding blah" as opposed to what package is adding that requirement. Ultimately in builds we often need more rather than less debug so the end user can debug issues themselves via the logs that are output, without that we'll see quite a rise in requests for build debugs.
Peter
On Wed, May 20, 2015 at 1:22 AM, Miroslav Suchý msuchy@redhat.com wrote:
Hi, several months ago I talked to dgilmore and probinso and we talked about using DNF in Koji and start building F22 packages using DNF resolver. The conlusion was that it is quite late in cycle. And it is better to start with plain Mock and leave user to test it on their local workstatations. Mock is using DNF for fedora-rawhide-* targets for some time already. And I received none complaints.
I'm not sure if you've seen the issue described at https://bugzilla.redhat.com/1210862, but many rubygem packages are going to FTBFS if Koji switches to using DNF.
I don't know why DNF prefers JRuby over MRI, but it would be nice if DNF could preserve whatever Yum's behavior was for that.
- Ken
On Wed, May 20, 2015 at 04:18:14PM -0600, Ken Dreyer wrote:
I don't know why DNF prefers JRuby over MRI, but it would be nice if DNF could preserve whatever Yum's behavior was for that.
Possibly related: https://fedorahosted.org/fpc/ticket/331
This could end up causing significant problems beyond the build system.
buildsys@lists.fedoraproject.org
-
Dennis Gilmore -
Ken Dreyer -
Matthew Miller -
Mike McLean -
Miroslav Suchý -
Peter Robinson