Highlights from the latest Copr release

[Pavel Raiskup]

Hello,

recently (on Feb 06, 2020) a new Copr release landed production.  Here is
the list of visible changes:

- Users now can build packages against explicitly enabled modules.  Go to
  `Project -> Settings -> Build Options -> [Edit] button (near _enabled_
  chroot) -> Enable module: textarea`.  Note that fedora modular
  repositories do not behave entirely good in mock (and copr) so we had to
  disable before by default.  So to actually bring the modules into copr,
  you need to also configure the "external repositories" textarea for the
  fedora-modular repo.

- Module builds were historically done only against Fedora dist-git.  Now
  the `copr-cli build-module` command was enhanced to accept `--distgit`
  option which allows users to specify which dist-git instance to build
  the module against.  For now we have only the "fedora" distgit
  configured.  If you have ideas what external dist-git we should add to
  our configuration, let us know.

- The EOL chroot policy scripts responsible for notifying users about
  upcoming removals were fixed, users should now always be notified - and
  if for some reason they are not, the EOL chroot will not be removed.

- Several fixes were done in copr-rpmbuild so the builds should be more
  reliable, especially when the builder VM is heavily re-used for many
  builds.

- News from yesterday - @msuchy enabled CDN for copr backend repositories.
  You, or your users, likely want to re-enable the copr repositories
  (dnf copr enable <owner/project>).

Happy building!
Pavel

[John M. Harris Jr]

On Thursday, February 6, 2020 8:42:56 AM MST Pavel Raiskup wrote:
>  Hello,
>  
>  recently (on Feb 06, 2020) a new Copr release landed production.  Here is
>  the list of visible changes:
>  
>  - Users now can build packages against explicitly enabled modules.  Go to
>    `Project -> Settings -> Build Options -> [Edit] button (near _enabled_
>    chroot) -> Enable module: textarea`.  Note that fedora modular
>    repositories do not behave entirely good in mock (and copr) so we had to
>    disable before by default.  So to actually bring the modules into copr,
>    you need to also configure the "external repositories" textarea for the
>    fedora-modular repo.
>  
>  - Module builds were historically done only against Fedora dist-git.  Now
>    the `copr-cli build-module` command was enhanced to accept `--distgit`
>    option which allows users to specify which dist-git instance to build
>    the module against.  For now we have only the "fedora" distgit
>    configured.  If you have ideas what external dist-git we should add to
>    our configuration, let us know.
>  
>  - The EOL chroot policy scripts responsible for notifying users about
>    upcoming removals were fixed, users should now always be notified - and
>    if for some reason they are not, the EOL chroot will not be removed.
>  
>  - Several fixes were done in copr-rpmbuild so the builds should be more
>    reliable, especially when the builder VM is heavily re-used for many
>    builds.
>  
>  - News from yesterday - @msuchy enabled CDN for copr backend repositories.
>    You, or your users, likely want to re-enable the copr repositories
>    (dnf copr enable <owner/project>).
>  
>  Happy building!
>  Pavel 
It looks like this is mostly module related stuff, which is absolutely useless 
for most packagers.. I hope it's good for RHEL at least, but it seems to be 
more of a nightmare there (for sysadmins, rather than packagers).

-- 
John M. Harris, Jr.
Splentity

[Kevin Kofler]

Pavel Raiskup wrote:
>  - The EOL chroot policy scripts responsible for notifying users
> about
>    upcoming removals were fixed, users should now always be notified - and
>    if for some reason they are not, the EOL chroot will not be removed. 
This is still inherently flawed and unsafe because you have absolutely no 
way of guaranteeing that the notification mail actually reached the 
maintainer, only that it has been sent. E-mail is not a certified delivery 
mechanism.

The whole concept of opt-out deletion is inherently flawed. It is 
unacceptable to delete user data without explicit confirmation.

And of course, the fix comes too late for the repositories that you folks 
already irremediably destroyed with your deletion rampage. E.g., the latest 
release of Kannolo, Kannolo 27, is now missing one of its default 
repositories and I have no way to fix that (and neither do you, for that 
matter – the only way the problem could be solved would be to allow building 
for Fedora 27 again, then I could simply build the packages again).

        Kevin Kofler

[Jakub Kadlcik]

Hello Kevin,
thank you for your feedback.

>  This is still inherently flawed and unsafe because you have
> absolutely no
>  way of guaranteeing that the notification mail actually reached the
>  maintainer, only that it has been sent. E-mail is not a certified delivery
>  mechanism. 
True, that's why we wanted to have a 180 days preservation period.
The email accounts are not Copr-specific either, they are linked with
FAS. If somebody's mailbox can't be reached by any Fedora service
for half a year, there is something wrong. And I don't think we can do
anything about it.

Though, we are considering implementing some notification icon directly
on the Copr website. Is this something you would like to see?


>  The whole concept of opt-out deletion is inherently flawed. It is
>  unacceptable to delete user data without explicit confirmation. 
I agree, that in the ideal world, it should be that way. But in the meantime,
Copr is not a paid service and has limited resources - mainly disk space.
Not removing any user data, would mean, that by now it wouldn't be possible
to build any package anymore. We would run out of space months ago.
We got promised new storage, but in the meantime, we need to remove
_outdated_ user data, just to keep the service working. Moreover, once
a new fedora is branched from rawhide, we create the chroots with
packages, that was build in rawhide chroots. This eats a lot (I can check
for the numbers) of space immediately after branching.

You can read more about the disk space issues in the original blog post
http://frostyx.cz/posts/copr-removing-outdated-chroots

We needed to figure out, what data we can remove without affecting the
most users. For years, we are removing old build results for packages
whose newer (or same) version was successfully built in the project.
Complying with "It is unacceptable to delete user data without explicit
confirmation", we wouldn't be allowed to do even this. But thanks to
this feature, we were able to not run out of storage for many months,
maybe even years. But lately, this mechanism wasn't sufficient, so we
needed to save disk space somewhere else. Our original idea was to
announce beforehand but then remove all the outdated chroots. Then
we downgraded the idea a little bit and we came with the concept of
notification emails and once an owner doesn't care about the outdated
chroots for his project, removing them.

If you have any other suggestions how to save storage space, let us
know, please. We will look at it.


>  And of course, the fix comes too late for the repositories that you
> folks
>  already irremediably destroyed with your deletion rampage. E.g., the latest
>  release of Kannolo, Kannolo 27, is now missing one of its default
>  repositories and I have no way to fix that (and neither do you, for that
>  matter – the only way the problem could be solved would be to allow building
>  for Fedora 27 again, then I could simply build the packages again). 
What happened, was very unfortunate, by any means intentional, and
we are very sorry about it. A bug slipped into production and broke the
notification command, so any email couldn't be sent. After you reported
the issue, we fixed the bug, added several unit tests to cover any possible
regressions in the future, and updated the code so a chroot that we haven't
sent a notification about, can't be removed. I am aware that it won't un-remove
the data, that were already lost, so it is not that comforting for you, but
unfortunately, we have no way to restore the data.

Temporarily enabling outdated chroots for people who care, to rebuild
their data, wouldn't be that easy too. There is no guarantee that
end of life mock chroots still work.


Jakub


On Mon, Feb 10, 2020 at 1:36 PM Kevin Kofler <kevin.kofler(a)chello.at&gt; wrote:
>
> Pavel Raiskup wrote:
> > - The EOL chroot policy scripts responsible for notifying users about
> >   upcoming removals were fixed, users should now always be notified - and
> >   if for some reason they are not, the EOL chroot will not be removed.
>
>  This is still inherently flawed and unsafe because you have
> absolutely no
>  way of guaranteeing that the notification mail actually reached the
>  maintainer, only that it has been sent. E-mail is not a certified delivery
>  mechanism. 
>
>  The whole concept of opt-out deletion is inherently flawed. It is
>  unacceptable to delete user data without explicit confirmation. 
>
>  And of course, the fix comes too late for the repositories that you
> folks
>  already irremediably destroyed with your deletion rampage. E.g., the latest
>  release of Kannolo, Kannolo 27, is now missing one of its default
>  repositories and I have no way to fix that (and neither do you, for that
>  matter – the only way the problem could be solved would be to allow building
>  for Fedora 27 again, then I could simply build the packages again). 
>
>         Kevin Kofler
> _______________________________________________
> devel mailing list -- devel(a)lists.fedoraproject.org
> To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
> Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
https://lists.fedoraproject.org/archives/list/[e-mail]