Hi Kamil,
Thank you for the reply.
On Wed, Apr 13, 2016 at 7:34 PM, Kamil Paral kparal@redhat.com wrote:
Hello Sinny, thanks for your work and sorry for late response. I'll review your taskotron task and let you know if there's something that should be changed or not. Afterwards, we will start mirroring your git repo on our taskotron servers, and patch our taskotron-trigger to know about task-libabigail. We can then execute it on every new Koji build (or Bodhi update, your choice). Your task will probably be the first one that we will execute regularly while not being written and maintained directly by us, so if there are any rough edges in the process, I apologize in advance :-)
Okay, sounds good to me.
You'll need to have two branches in your git: master - this will be used on our production server https://taskotron.fedoraproject.org/ develop - this will be used on our dev and staging server http://taskotron-dev.fedoraproject.org/ and https://taskotron.stg.fedoraproject.org/
Okay, works for me.
You need to decide whether it is better to run libabigain against every new
Koji build, or just against every new Bodhi update. From a quick look, I think it makes more sense to run libabigail on every new Koji build, so that people can see the results even before creating the update (that requires looking into ResultsDB manually at the moment). If we run it on every Koji build, the results will still show up in Bodhi - Bodhi should query ResultsDB and show the results for those particular builds present in the update. (We might need to teach Bodhi about libabigail existence, I'm not sure). Ultimately it's your choice, what makes more sense for your check.
I believe that when we say every new Koji build, we are talking about non-scratch build which doesn't include scratch build done by anyone. If my assumption is right, then yes running libabigail task on each koji build will be good. It is possible to do that with current implementation since libabigail task look for a koji build-id to download corresponding rpms.
Please also create a wiki page at https://fedoraproject.org/wiki/Taskotron/Tasks/libabigail similar to these https://fedoraproject.org/wiki/Taskotron/Tasks/depcheck https://fedoraproject.org/wiki/Taskotron/Tasks/upgradepath linked from https://fedoraproject.org/wiki/Taskotron/Tasks .
Sure, I will create one.
We try to have at least some basic documentation and FAQ for our checks in there. Currently it's not very discoverable (we should link to it at least from ResultsDB, which we currently don't) and the location can change, but at least it's a link we can give to people when asking basic questions about one of our tasks. Also, since you're going to maintain the task and not us, please include some "Contact" section where to post feedback or report bugs (e.g. github issues page). If people ask us about the task and we don't know the answer, we will point them to that wiki page.
Will add contact section in wiki page. I wonder if it is better to have this included with other fedora qa tasks?
Can we please continue this discussion in qa-devel [1] mailing list? We can discuss more implementation details in there, and I'll post my review findings in there as well.
done!
On Wed, 13 Apr 2016 21:14:19 +0530 Sinny Kumari ksinny@gmail.com wrote:
Thank you for getting this done, I'm excited to start it running in Taskotron.
<snip>
You need to decide whether it is better to run libabigain against every new
Koji build, or just against every new Bodhi update. From a quick look, I think it makes more sense to run libabigail on every new Koji build, so that people can see the results even before creating the update (that requires looking into ResultsDB manually at the moment). If we run it on every Koji build, the results will still show up in Bodhi - Bodhi should query ResultsDB and show the results for those particular builds present in the update. (We might need to teach Bodhi about libabigail existence, I'm not sure). Ultimately it's your choice, what makes more sense for your check.
I believe that when we say every new Koji build, we are talking about non-scratch build which doesn't include scratch build done by anyone. If my assumption is right, then yes running libabigail task on each koji build will be good. It is possible to do that with current implementation since libabigail task look for a koji build-id to download corresponding rpms.
I'd advocate for running libabigail against koji builds, if possible. There are no updates in rawhide and I think it'd be useful to have abi checks running on rawhide builds.
<snip>
We try to have at least some basic documentation and FAQ for our checks in there. Currently it's not very discoverable (we should link to it at least from ResultsDB, which we currently don't) and the location can change, but at least it's a link we can give to people when asking basic questions about one of our tasks. Also, since you're going to maintain the task and not us, please include some "Contact" section where to post feedback or report bugs (e.g. github issues page). If people ask us about the task and we don't know the answer, we will point them to that wiki page.
Will add contact section in wiki page. I wonder if it is better to have this included with other fedora qa tasks?
By "have this included wiht other fedora qa tasks", do you mean have the git repo hosted in the same place?
If so, I'd say that's up to you but I expect to be moving all our repos off of bitbucket and into something hosted in Fedora in the next month or so. If you do decide that you want the task hosted that way, I'd suggest waiting until we move all of our current repos so that there's only one change.
If I misunderstood, would you rephrase the question?
Again, thanks for getting this done.
Tim
On Thu, Apr 14, 2016 at 2:18 AM, Tim Flink tflink@redhat.com wrote:
I'd advocate for running libabigail against koji builds, if possible. There are no updates in rawhide and I think it'd be useful to have abi checks running on rawhide builds.
Indeed, running abi checks on rawhide builds are important.
By "have this included wiht other fedora qa tasks", do you mean have the git repo hosted in the same place?
Yes
If so, I'd say that's up to you but I expect to be moving all our repos off of bitbucket and into something hosted in Fedora in the next month or so. If you do decide that you want the task hosted that way, I'd suggest waiting until we move all of our current repos so that there's only one change.
Sure, for now I will maintain source code on github and would love to get it moved with other fedora qa repos when ready.
Thanks, Sinny
Hi Sinny,
I have created a review here: https://phab.qadevel.cloud.fedoraproject.org/D813
I used our Phabricator, because it makes it easy to comment on particular lines. You can log in using your FAS email [1] and reply there, or here in the mailing list, it doesn't matter. I'm pleasantly surprised that despite our very lacking documentation, you managed to create a very good Taskotron task :-)
One additional thing that I noticed today - when I ran the abipkgdiff comparison on lyx package, it used up to 6 GB RAM. That is really high and I believe our test clients don't have that many available. I don't know how abipkgdiff works and whether it is able to adjust its memory requirements based on the amount of available system memory (so that it would use less memory on a less equipped system and still work properly). Do you know if this is going to be a problem or not? And do you know what is the peak memory usage for very big packages (e.g. what about comparing two kernel packages)? Is there any solution that we could use to limit the memory consumption of abipkgdiff?
Thanks, Kamil
[1] Please note we've been experiencing some problems with authentication recently, so it might not work for you: https://phab.qadevel.cloud.fedoraproject.org/T761
Hi Kamil,
On Thu, Apr 14, 2016 at 8:40 PM, Kamil Paral kparal@redhat.com wrote:
Hi Sinny,
I have created a review here: https://phab.qadevel.cloud.fedoraproject.org/D813
I used our Phabricator, because it makes it easy to comment on particular lines. You can log in using your FAS email [1] and reply there, or here in the mailing list, it doesn't matter. I'm pleasantly surprised that despite our very lacking documentation, you managed to create a very good Taskotron task :-)
Thanks for the quick review. I have addressed review comments and added new diff at https://phab.qadevel.cloud.fedoraproject.org/D817?id=2081 . I am not sure if I updated diff in right way but this is what I ended up by following option "Update diff" available in right side :)
One additional thing that I noticed today - when I ran the abipkgdiff comparison on lyx package, it used up to 6 GB RAM. That is really high and I believe our test clients don't have that many available. I don't know how abipkgdiff works and whether it is able to adjust its memory requirements based on the amount of available system memory (so that it would use less memory on a less equipped system and still work properly). Do you know if this is going to be a problem or not? And do you know what is the peak memory usage for very big packages (e.g. what about comparing two kernel packages)? Is there any solution that we could use to limit the memory consumption of abipkgdiff?
I had a talk about this with libabigail's maintainer (Dodji). If a binary is taking more RAM then right now, we can't do anything about it. This is because ELF information for binaries to be compared and corresponding debuginfo gets loaded while performing ABI comparison. Also, right now libabigail doesn't support comparison of two kernel version.
I noticed that binary files available in lyx package are PIE (Position Independent Executable) files which are basically executable files. As we know, an application mainly consumes shared libraries from its dependent packages. ABI change of shared libraries are what really matter in packages. Therefore, I am now calling abipkgdiff with --dso-only option. This will avoid unnecessary comparison between executable files. This reduces overall comparison time of a koji build update and hopefully reduces memory consumption since most of shared libraries are smaller in size (few MBs).
Right now there is a bug in libabigail that it doesn't distinguish between PIE files and shared libraries since both are of type DSO(Dynamic Shared object). So, right now abipkgdiff with --dso-only option does ABI comparison for PIE files as well. Upstream bug has been reported to distinguish between Shared libraries and PIE files https://sourceware.org/bugzilla/show_bug.cgi?id=19961. This will be fixed soon and will be available in next libabigail release.
Thanks, Sinny
Thanks for the quick review. I have addressed review comments and added new diff at https://phab.qadevel.cloud.fedoraproject.org/D817?id=2081 .
Thanks, looks good. I have no further concerns regarding the task code.
I am not sure if I updated diff in right way but this is what I ended up by following option "Update diff" available in right side :)
Whatever works. Phabricator is not completely suited for reviewing patches from unknown projects, so it can be a bit cumbersome. But I needed a place where I can add comments to any line of the script (I think it's not possible to do on github). If we need to review any further changes, we can use Phab or github, doesn't matter.
On Thu, Apr 21, 2016 at 8:00 PM, Kamil Paral kparal@redhat.com wrote:
Thanks for the quick review. I have addressed review comments and added new diff at https://phab.qadevel.cloud.fedoraproject.org/D817?id=2081 .
Thanks, looks good. I have no further concerns regarding the task code.
Thanks again for the review. Happy to see it going one step further to get it integrated with taskotron :)
I am not sure if I updated diff in right way but this is what I ended up by following option "Update diff" available in right side :)
Whatever works. Phabricator is not completely suited for reviewing patches from unknown projects, so it can be a bit cumbersome. But I needed a place where I can add comments to any line of the script (I think it's not possible to do on github). If we need to review any further changes, we can use Phab or github, doesn't matter.
I also don't use github a lot. So, just checked whether inline comments in a commit is possible or not. Yes, we can do it! for reference attached a screenshot in which + icon appears on a line with mouse hover. Clicking on + icon leads to a comment box.
Hello Kamil,
Sorry for replying late. I subscribed to this list, but for a reason the emails are still not being delivered to me. I clicked on some more buttons in the interface right now, so we'll see. In the mean time, I am using the web interface to reply, so please forgive the awkward formatting that might come out of this.
One additional thing that I noticed today - when I ran the abipkgdiff comparison on lyx package, it used up to 6 GB RAM. That is really high and I believe our test clients don't have that many available.
Ah.
I think the test systems must have that kind of amount of memory. For most packages, the memory consumption is reasonable. But for some packages where the combination of ELF binary size and uncompressed debug info size is big, then, well, the tool is going to require a lot of memory. For instance, it takes 13+GB of memory to compare the ABI of two instances of the libjvm.so library.
Really, abipkgdiff uses the usual "static analysis" techniques (it's only that it acts on binaries rather than source code), so it is expected to have the same kind of memory requirements.
I don't know how abipkgdiff works and whether it is able to adjust its memory requirements based on the amount of available system memory (so that it would use less memory on a less equipped system and still work properly).
abipkgdiff performs static analysis of the binary. For a given library, it loads the type information of each one of the compilation units that make up the final library. There can be hundreds of thousands of those types in big libraries. abipkgdiff has to represent them all to analyse them. It needs to have them all in memory because it analyses the relationships between all the types. And unlike a compiler that generally sees only one translation unit at a time, abipkgdiff sees the type information of all the translation units at a time. Just like what a debugger has to do. And abipkgdiff has to do this *twice*, because it needs to load two libraries to compare the types of the functions and variables in that library. So yes, that can take a lot of memory. We are working on optimizations, of course, but the reality is that there are going to be binaries for which we'll need a lot of memory to operate on. And right now, we cannot do this incrementally.
Do you know if this is going to be a problem or not?
If the machine doesn't have enough memory, then yes it's going to be a problem. Our testing showed that in practice, for the vast majority of packages, the memory consumption is OK. But there are going to be some packages that will require much much more memory.
And do you know what is the peak memory usage for very big packages (e.g. what about comparing two kernel packages)?
As Sinny said in another thread, abipkgdiff doesn't support the kernel yet. So we haven't tested that.
Is there any solution that we could use to limit the memory consumption of abipkgdiff?
In general, I'd say there is no good solution beside us optimizing the code more to make it consume less over time. And this is something I am really interested in doing. I have started already and I intend to keep doing that as part of the normal development process.
As Sinny pointed out, in the particular case of the lyx package, the tool should *not* have performed any comparison because that package only contains an executable. The tool should only act on libraries, at least in a first instalment. The reason why it tried to compare two lyx executable is because it mistakenly considered that executable as a shared library. This mistake can happen on Position Independent Executables, due to a bug in Libabigail. This bug is fixed now, though. So on lyx, the memory consumption should be very minimal now, as abipkgdiff won't do anything ;-)
I hope this helps.
Hello Kamil,
Sorry for replying late. I subscribed to this list, but for a reason the emails are still not being delivered to me. I clicked on some more buttons in the interface right now, so we'll see. In the mean time, I am using the web interface to reply, so please forgive the awkward formatting that might come out of this.
Hello Dodji, welcome.
I think the test systems must have that kind of amount of memory. For most packages, the memory consumption is reasonable. But for some packages where the combination of ELF binary size and uncompressed debug info size is big, then, well, the tool is going to require a lot of memory. For instance, it takes 13+GB of memory to compare the ABI of two instances of the libjvm.so library.
I've read the backlog of your discussion with tflink, but I'm not sure what the conclusion is. I believe we can't assign 15GB RAM to all our test VMs (and we can't tie a specific task to a specific set of VMs at the moment). So I guess we will increase our test VMs memory to some "reasonable" amount and let the few extremely large packages crash with OOM. (That reminds me we should make sure to disable swap in VMs, otherwise that would kill the host). Do you have any advice what this "reasonable" amount of RAM should be? 4 GB? 6 GB? So that 95% of tasks work OK, and just the extreme ones crash. We can then add those to a blacklist.
Speaking of lists, you and Tim were mentioning white/blacklists, also critpath set. So what is the ideal set of packages we should run on initially? All packages? Only critpath packages? Only libraries included in critpath? In case we should run just on libraries, any good recommendation how to recognize that (better than matching "lib*" in package name)? We would need to make the decision without downloading the package, but I guess we could query koji or repo metadata if necessary.
We will need to implement white/blacklisting, ideally I'd like you to have the control over it, not us (so e.g. defining that in task.yaml formula). We can probably implement critpath support (recognizing what is in critpath and what is not), I'm sure it will be needed in other tasks in the future as well.
Kamil
On Thu, Apr 21, 2016 at 8:27 PM, Kamil Paral kparal@redhat.com wrote:
Hello Kamil,
Sorry for replying late. I subscribed to this list, but for a reason the emails are still not being delivered to me. I clicked on some more buttons in the interface right now, so we'll see. In the mean time, I am using the web interface to reply, so please forgive the awkward formatting that might come out of this.
Hello Dodji, welcome.
I think the test systems must have that kind of amount of memory. For most packages, the memory consumption is reasonable. But for some packages where the combination of ELF binary size and uncompressed debug info size is big, then, well, the tool is going to require a lot of memory. For instance, it takes 13+GB of memory to compare the ABI of two instances of the libjvm.so library.
I've read the backlog of your discussion with tflink, but I'm not sure what the conclusion is. I believe we can't assign 15GB RAM to all our test VMs (and we can't tie a specific task to a specific set of VMs at the moment). So I guess we will increase our test VMs memory to some "reasonable" amount and let the few extremely large packages crash with OOM. (That reminds me we should make sure to disable swap in VMs, otherwise that would kill the host). Do you have any advice what this "reasonable" amount of RAM should be? 4 GB? 6 GB? So that 95% of tasks work OK, and just the extreme ones crash. We can then add those to a blacklist.
Speaking of lists, you and Tim were mentioning white/blacklists, also critpath set. So what is the ideal set of packages we should run on initially? All packages? Only critpath packages? Only libraries included in critpath? In case we should run just on libraries, any good recommendation how to recognize that (better than matching "lib*" in package name)? We would need to make the decision without downloading the package, but I guess we could query koji or repo metadata if necessary.
With latest commit [1] in libabigail taskotron task, ABI comparison will occur only on shared libraries because we call abipkgdiff with --dso-only option. So, if a package doesn't contains any library file, it won't run abi checks on files available in package.There is a related bug[2] which has already been fixed and will be available in next libabigail release. So, there is no need to look specifically for lib* package. Also, packages providing libraries may not start/contain *lib* (e.g. elfutils) in its name.
So, let's say we initially start with packages available in critpath[3], then we don't have to worry whether a package provides any shared library or not. Our main concern would be which library in any of the critpath package consumes more memory (usage of memory is greater than available in spawned VM).
[1] https://github.com/sinnykumari/task-libabigail/commit/752ace2c74b2ccc5aa37a3... [2] https://sourceware.org/bugzilla/show_bug.cgi?id=19961 [3] https://admin.fedoraproject.org/pkgdb/api/critpath
Speaking of lists, you and Tim were mentioning white/blacklists, also critpath set. So what is the ideal set of packages we should run on initially? All packages? Only critpath packages? Only libraries included in critpath? In case we should run just on libraries, any good recommendation how to recognize that (better than matching "lib*" in package name)? We would need to make the decision without downloading the package, but I guess we could query koji or repo metadata if necessary.
With latest commit [1] in libabigail taskotron task, ABI comparison will occur only on shared libraries because we call abipkgdiff with --dso-only option. So, if a package doesn't contains any library file, it won't run abi checks on files available in package.There is a related bug[2] which has already been fixed and will be available in next libabigail release. So, there is no need to look specifically for lib* package. Also, packages providing libraries may not start/contain *lib* (e.g. elfutils) in its name.
I don't see any libraries in elfutils, they are in elfutils-libs, byt yes, you're right, we can't rely on package names.
So, let's say we initially start with packages available in critpath[3], then we don't have to worry whether a package provides any shared library or not.
Let me try this another way. Let's assume we can detect whether a certain package contains an *.so (or *.so.N+) file. Can we use this to decide which packages to run libabigail on? Or does libabigail run checks on more files than just *.so (now that we're using --dso-only)?
I'm asking, because I expect this request ("run my task on packages containing shared libraries/files or certain other kind") is going to be quite common in the future and I think we should cover it somehow (and I have an idea how). I understand that currently we can run libabigail on everything, and if there are no libraries inside, nothing bad happens. But it's not very efficient, and that's why I'm interested to learn how exactly to distinguish packages libabigail is useful for from packages libabigail just skips.
Thanks.
Hello,
Sorry for the late reply.
Kamil Paral kparal@redhat.com a écrit:
So, let's say we initially start with packages available in critpath[3], then we don't have to worry whether a package provides any shared library or not.
Let me try this another way. Let's assume we can detect whether a certain package contains an *.so (or *.so.N+) file. Can we use this to decide which packages to run libabigail on?
If by "we" you mean taskotron, then the answer to your question is yes.
Or does libabigail run checks on more files than just *.so (now that we're using --dso-only)?
Now that the taskotron task is invoking abipkgdiff with the --dso-only option, if the package contains shared libraries and and other types of binaries, then only the shared libraries are going to be ABI-compared.
I'm asking, because I expect this request ("run my task on packages containing shared libraries/files or certain other kind") is going to be quite common in the future and I think we should cover it somehow (and I have an idea how). I understand that currently we can run libabigail on everything, and if there are no libraries inside, nothing bad happens. But it's not very efficient, and that's why I'm interested to learn how exactly to distinguish packages libabigail is useful for from packages libabigail just skips.
I understand. I agree that being able to skip packages early based on their content would likely be more efficent.
I hope this helps.
Cheers,
So, let's say we initially start with packages available in critpath[3], then we don't have to worry whether a package provides any shared library or not.
Let me try this another way. Let's assume we can detect whether a certain package contains an *.so (or *.so.N+) file. Can we use this to decide which packages to run libabigail on?
If by "we" you mean taskotron, then the answer to your question is yes.
Or does libabigail run checks on more files than just *.so (now that we're using --dso-only)?
Now that the taskotron task is invoking abipkgdiff with the --dso-only option, if the package contains shared libraries and and other types of binaries, then only the shared libraries are going to be ABI-compared.
I'm asking, because I expect this request ("run my task on packages containing shared libraries/files or certain other kind") is going to be quite common in the future and I think we should cover it somehow (and I have an idea how). I understand that currently we can run libabigail on everything, and if there are no libraries inside, nothing bad happens. But it's not very efficient, and that's why I'm interested to learn how exactly to distinguish packages libabigail is useful for from packages libabigail just skips.
I understand. I agree that being able to skip packages early based on their content would likely be more efficent.
I hope this helps.
Thanks, I reported an RFE about this: https://phab.qadevel.cloud.fedoraproject.org/T811
Kamil Paral kparal@redhat.com a écrit:
Thanks, I reported an RFE about this: https://phab.qadevel.cloud.fedoraproject.org/T811
Thanks Kamil,
Unfortunately, I still cannot log into https://phab.qadevel.cloud.fedoraproject.org/T811 using my dodji@fedoraproject.org persona, so I cannot comment on the issue. When I try to log in there, I am getting this error:
[HTTP/500] Internal Server Error Internal Server Error
But I can log into the Fedora infrastracture just fine, in general. It's just the https://phab.qadevel.cloud.fedoraproject.org/T811 thingy that I am having issue with. I am not sure what I am doing wrong.
I wish all the tickets handling gets migrated to pagure.io one day :-)
Cheers,
qa-devel@lists.fedoraproject.org
-
Dodji Seketeli -
Kamil Paral -
Sinny Kumari -
Tim Flink