August 2020 - infrastructure - Fedora Mailing-Lists

Delta updates for Flatpaks - implementing in Fedora infrastructure

by Owen Taylor

Introduction ========= The original way that Flatpaks were updated was via ostree repositories. More recently, the ability was added to distribute them as container images, and that is what we do for the Fedora build Flatpaks via registry.fedoraproject.org. The one big gap we have between the two ways of distributing Flatpaks is delta updates - if you have downloaded a Flatpak from an ostree repository, you can update to the next version via the raw ostree protocol - downloading only changed files one, and even better, if "ostree static deltas" have been properly computed and stored in the upstream repository, you can get one big blob that uses bsdiff and other techniques to efficiently compress the differences This spring, Alex Larsson came up with a way of updating containers with deltas *in general* and implemented that in podman (https://github.com/containers/image/pull/902, still pending) and in Flatpak (support released with Flatpak 1.8.). See: https://blogs.gnome.org/alexl/2020/05/13/putting-container-updates-on-a-d... This is inspired by ostree static deltas, but modified to fit into the container world - the end result is both super simple and works remarkably well. The question then becomes: how do we generate deltas for the Flatpaks we ship for Fedora and make them available to users? Generating static deltas for Flatpaks =========================== There are currently 3 different codebases I maintain to wrangle Flatpak metadata: regindexer: script that queries registry.fedoraproject.org and writes an index of Flatpaks for use by Flatpak clients (deployed in Fedora infrastructure currently) https://pagure.io/regindexer flatpak-status: daemon that queries bodhi and koji, figures out what Flatpaks are out-of-date, and generates a JSON file used to create a web user interface https://fedora.fishsoup.net/flatpak-status/ https://github.com/owtaylor/flatpak-status flatpak-indexer: daemon that queries the Red Hat container api, and internal Koji instance, and writes an index of Flatpaks for use by Flatpak clients (e.g., https://flatpaks.redhat.io/rhel/index/static?label:org.flatpak.ref:exists...) My path forward here was: take the code from flatpak-status that queries bodhi and koji, use it to teach flatpak-indexer how to handle Fedora Flatpaks as well as RHEL flatpaks, then add the capability to generate deltas. The result of this can be found: https://github.com/owtaylor/flatpak-indexer it seems to work fine - it generates index and deltas for Fedora Flatpaks that work in limited testing. (The download for updating berusky from the last stable version in Fedora to the current one was reduced from 5.5MB to 18k) Distributing static deltas for Flatpaks =========================== The eventual goal of our delta project is to upload them to container registries - as described in Alex's blog post - and we've been in discussion with the Quay.io folks to figure out the best way to do this. But we didn't want to block static deltas on a) having OCI artifact support on quay.io b) having a finalized way to do delta updates as OCI artifacts and a white-listed MIME type c) getting Fedora switched to quay.io d) having Red Hat built containers hosted on a container registry. So we added a second path to Flatpak - the image index that Flatpak consumes can point to a "delta manifest" as an HTTP URL, and that can point to the individual layer deltas also by HTTP URL. For now, flatpak-indexer doesn't upload the delta manifests or layer deltas - it just writes them into the static data along with the indexes and icons, and the points to them from the index. Architecture of flatpak-indexer ======================= flatpak-indexer shares a property with the currently deployed regindexer - it is entirely generating static content. The overall components are: redis: used to cache data retrieved from "upstream" (koji and bodhi for Fedora), and to communicate between the indexer and differ containers indexer: the main container - it periodically retrieves data from upstream sources, determines what layers need deltas, queues them up for the differ containers, collects the results, and writes the delta manifests and indexes differ: containers that execute the expensive 'tar-diff' operation to compute the layer deltas. The number of containers can potentially be scaled based on the number of queued layer deltas frontend: an apache server that serves up the generated data - with appropriate redirects and headers This is set up in openshift internally, and would presumably be done the same way in Fedora infrastructure, though we could potentially leave the "frontend" role to sundries as it is currently with the regindexer generated index and icons. Resource consumption ================== The 'redis' and 'indexer' containers are lightweight - intermittent usage of 1 cpu, maybe 256MB of memory. The differ containers need to be more beefy - they could use 1-2 cpus and 2-3GB of memory. But they are only needed intermittently and are otherwise idle. With some added complexity, they could be scaled to zero, and only scaled up when there are tar-diffs to process. The disk usage for index+icons+deltas for the current Fedora Flatpak set is 347MB - this will increase proportional to the number of Flatpaks in Fedora, but I wouldn't expect it to ever be *much* bigger - deltas pointing to old versions of images will be cleaned up. Future work ========= The current status is good enough to deploy as a delta solution, but some more future possibilities: * Teach flatpak-indexer how to upload deltas to a container registry as OCI artifacts * Use flatpak-indexer to generate indexes for Fedora containers that are not Flatpaks, useful for * Make flatpak-indexer the backend to the Flatpak status web page, move that inside Fedora infrastructure

3 years, 8 months

1
0
0 / 0

Fedora Infra Doc hackfest

by Mark O'Brien

Hi All, As previously proposed we would like to have a hackfest to get our documentation in order. With some considerable help form siddharthvipul we have created an agenda of what we think should be worked on during the hackfest. The agends is located at https://board.net/p/infra-doc-hackfest. This is a working board so if anyone would like to add topics to it please feel encouraged to do so. Petr Bokoc has kindly agreed to give a demo at the start of the hackfest on the use of antora and how the docs make their way to docs.fp.o. He has also shared some links if people would like to brush up on asciidoc https://docs.antora.org/antora/2.3/, learn about antora https://docs.antora.org/antora/2.3/, or about the contributing https://docs.fedoraproject.org/en-US/fedora-docs/ The event will take place on Friday 4th September 13:00-16:00 UTC. The event is on the cpe calendar. here https://bit.ly/3lnTUbH Regards, Mark

3 years, 8 months

1
1
0 / 0

[Fedocal] Reminder meeting : Fedora Infrastructure

by smooge＠gmail.com

Dear all, You are kindly invited to the meeting: Fedora Infrastructure on 2020-08-27 from 15:00:00 to 16:00:00 UTC At fedora-meeting-1(a)irc.freenode.net The meeting will be about: Weekly Fedora Infrastructure meeting. See infrastructure list for agenda a day before. Source: https://apps.fedoraproject.org/calendar/meeting/9675/

3 years, 8 months

1
0
0 / 0

Hosting Options for QA Projects

by Tim Flink

From what I understand, communishift as it was originally planned is not likely to be a thing going forward. We had been planning to use communishift to host a number of QA projects and we're looking for new options going forward. Those projects are: Testdays [1] is an app that we use to help run test days. It used to live on the old cloud and is currently hosted outside Fedora's infra on a machine we have access to; the plan was to move it to communishift once that was available. The packager dashboard [2] was recently announced for testing on devel@. It and its backend service, oraculum, live on the same external machine as the testdays app. We're also looking into a TCMS to replace our mediawiki-based setup - this isn't an immediate need but our plan was to host test instances on communishift for our evaluation. Do folks here have suggestions on what we could do about hosting for projects like these? Thanks, Tim [1] https://testdays.fedorainfracloud.org/events [2] https://packager.fedorainfracloud.org/

3 years, 8 months

7
10
0 / 0

Meeting Agenda Item: Introduction Ahmed Elmaghallawy

by ahmedelmaghallawy＠protonmail.com

dear infrastructure team, my IRC handle is : elmaghallawy my skills: - Python/Flask/Django - JavaScript-CSS-HTML - SQL basics - Git done some projects with Django/ Flask, have been using Linux for two years and on that period I learned a lot and I am eager to give back by contributing, some certificates I earned: - Python for Everybody Specialization - Coursera - Introduction to Data Science Specialization - Coursera - Introduction to the Internet of Things and Embedded Systems - Coursera - Introduction to Programming with MATLAB - Coursera - Introduction to Philosophy - Coursera - Blockchain Basics - Coursera I want to learn more about how your team works and i am looking forward to solving bugs and problems! Hope we'll be in touch soon, Looking forward to involve in such interesting community and projects! Regards, Ahmed

3 years, 8 months

3
2
0 / 0

CPE Feedback Survey

by Ant Carroll

Hey folks, CPE <https://docs.fedoraproject.org/en-US/cpe/>need your help :) Over the last several months we've been trying to improve how we interact and share information with you all. From the blog posts, to mails and how we work on the tickets you send us. Here is a link to a very short survey we've put together to learn how we can give you the best experience possible going forward by understanding the experiences you've had recently. If you could take the time (5mins max) to complete it for us it would be hugely valuable as we work on this continuous improvement - https://fedoraproject.limequery.com/696793?lang=en <https://meet.google.com/linkredirect?authuser=0&dest=https%3A%2F%2Ffedora...> Cheers, Ant -- Ant Carroll Associate Manager, CPE Team Red Hat Waterford <http://www.redhat.com> Communications House Cork Road, Waterford City ancarrol(a)redhat.com M: +353876213163 IM: ancarrol @redhatjobs <https://twitter.com/redhatjobs> redhatjobs <https://www.facebook.com/redhatjobs> @redhatjobs <https://instagram.com/redhatjobs> <https://www.redhat.com/>

3 years, 8 months

1
1
0 / 0

[Fedocal] Reminder meeting : Fedora Infrastructure

by smooge＠gmail.com

Dear all, You are kindly invited to the meeting: Fedora Infrastructure on 2020-08-20 from 15:00:00 to 16:00:00 UTC At fedora-meeting-1(a)irc.freenode.net The meeting will be about: Weekly Fedora Infrastructure meeting. See infrastructure list for agenda a day before. Source: https://apps.fedoraproject.org/calendar/meeting/9675/

3 years, 8 months

1
0
0 / 0

Congrats to our new sysadmin-mainers

by Kevin Fenzi

I'm happy to announce that We have approved several folks into the sysadmin-main group: mobrien - Mark O'Brien abompard - Aurelien Bompard This is the core group of trusted folks that high level access to most everything in fedora infrastructure. Mark has been deploying proxies to aws and working through all the associated issues successfully. Aurelien has been around for many years and intends to deploy our new account system interface noggin, which this access to help with. They have both proved their dedication, trustworthiness, and ability. Congrats to them both! Use your powers for good! :) kevin

3 years, 8 months

9
8
0 / 0

Planned Outage - Blockerbugs - 2020-08-18 17:00 UTC

by Pierre-Yves Chibon

There will be an outage starting at 2020-08-18 17:00UTC, which will last approximately 3 hours. To convert UTC to your local time, take a look at http://fedoraproject.org/wiki/Infrastructure/UTCHowto or run: date -d '2020-08-18 17:00UTC' Reason for outage: Blockerbugs host VM upgrade from EL7 to FC32. Affected Services: https://qa.fedoraproject.org/blockerbugs/ Ticket Link: https://pagure.io/fedora-infrastructure/issue/9244 Please join #fedora-admin or #fedora-noc on irc.freenode.net or add comments to the ticket for this outage above.

3 years, 8 months

1
0
0 / 0

Meeting Agenda Item: Introduction Jeffrey Serio

by Jeffrey Serio

IRC: hyperreal Keybase: https://keybase.io/hyperreal2 TZ/Country: US-Central/Chicago I have a Bachelor of Science degree in computer science from the University of Illinois-Springfield (2018). I'm familiar with and have used Python, Golang, Bash, Java, Podman/Docker, Git, and the GNU userland utilities, but am by no means an expert in any of them. I love Fedora, so I want to give back to the project and gain experience in the open source ecosystem. I'm particularly interested in system administration, DevOps, projects using Python and Golang, Ansible, automation, and containerized workflows. So I would like to work on tasks and issues related to those technologies. Issues #9236, #9231, #9164, #9161 on https://pagure.io/fedora-infrastructure/issues seem interesting. As I'm currently unemployed, I have /at least/ 4 hours per week to contribute. I also think this would be good experience for my resume/portfolio. Cheers :-) Jeffrey Serio <hyperreal64(a)pm.me> GPG = BDB9 E728 4107 317D AE0D A261 9129 BD07 C350 9CED

3 years, 8 months

1
0
0 / 0

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

infrastructure August 2020