Meeting agenda: RATS vs. librat
by Randy Barlow
Greetings!
I'd like to have a discussion during this week's infrastructure meeting
about whether we need the project that will re-run failed gating tests
to be a service, or whether a library will do. There has been some
lively debate about this in IRC, but it didn't seem like there was
resolution and it would be helpful if we could make a decision together
so we can proceed with implementation. Would this week's infra meeting
be a good time to discuss this? Alternatively, we could discuss it on
this mailing list if desired.
6 years
Planned MirrorManager changes
by Adrian Reber
I would like to change the setup of our mirror crawler and just wanted
to mention my planned changes here before working on them.
Currently we have two VMs which are crawling our mirrors. Each of the
machine is responsible for one half of the active mirrors. The crawl
starts every 12 hours on the first crawler and 6 hours later on the
second crawler. So every 6 hours one crawler is accessing the database.
Currently most of the crawling time is not spent crawling but updating
the database about which host has which directory up to date. With a
timeout of 4 hours per host we are hitting that timeout on some hosts
regularly and most of the time the database access is the problem.
What I would like to change is to crawl each category (Fedora Linux,
Fedora Other, Fedora EPEL, Fedora Secondary Arches, Fedora Archive)
separately and at different times and intervals.
We would not hit the timeout as often as now as only the information for
a single category has to be updated. We could scan 'Fedora Archive' only
once per day or every second day. We can scan 'Fedora EPEL' much more
often as it is usually really fast and get better data about the
available mirrors.
My goal would be to distribute the scanning in such a way to decrease
the load on the database and to decrease the cases of mirror
auto-deactivation due to slow database accesses.
Let me know if you think that these planned changes are the wrong
direction of if you have other ideas how to improve the mirror crawling.
Adrian
6 years
Proposed zchunk file format - V2
by Jonathan Dieter
Neal, thanks for the feedback. After taking your comments into
consideration, here's version 2.
+-+-+-+-+-+------------------+-+-+-+-+-+-+-+-+
| ID | Compression type | Index size |
+-+-+-+-+-+------------------+-+-+-+-+-+-+-+-+
+==================+=================+
| Compressed Index | Compressed Dict |
+==================+=================+
+===========+===========+
| Chunk | Chunk | ==> More chunks
+===========+===========+
ID
'\0ZCK1', identifies file as zchunk version 1 file
Compression type
Type of compression used to compress dict and chunks
Current values:
0 - Uncompressed
2 - zstd
Index size
This is a 64-bit unsigned integer containing the size of compressed
index.
Compressed Index
This is the index, which is described in the next section. The index
is compressed without a custom dictionary.
Compressed Dict (optional)
This is a custom dictionary used when compressing each chunk.
Because each chunk is compressed completely separately from the
others, the custom dictionary gives us much better overall
compression. The custom dictionary is compressed without a custom
dictionary (for obvious reasons).
Chunk
This is a chunk of data, compressed with the custom dictionary
provided above.
The index:
+---------------+======================+
| Checksum type | Checksum of all data |
+---------------+======================+
+================+-+-+-+-+-+-+-+-+
| Dict checksum | End of dict |
+================+-+-+-+-+-+-+-+-+
+================+-+-+-+-+-+-+-+-+
| Chunk checksum | End of chunk | ==> More
+================+-+-+-+-+-+-+-+-+
Checksum type
This is the type of checksum used to generate the checksums in the
index.
Current values:
0 = SHA-256
Checksum of all data
This is the checksum of the compressed dict and all the compressed
chunks, used to verify that the file is actually the same, even in
the unlikely event of a hash collision for one of the chunks
Dict checksum
This is the checksum of the compressed dict, used to detect whether
two dicts are identical. If there is no dict, the checksum must be
all zeros.
End of dict
This is the location of the end of the dict starting from the end of
the index. This gives us the information we need to find and
decompress the dict. If there is no dict, the checksum must be all
zeros.
Chunk checksum
This is the checksum of the compressed chunk, used to detect whether
any two chunks are identical.
End of chunk
This is the location of the end of the chunk starting from the end of
the index. This gives us the information we need to find and
decompress each chunk.
The index is designed to be able to be extracted from the file on the
server and downloaded separately, to facilitate downloading only the
parts of the file that are needed, but must then be re-embedded when
assembling the file so the user only needs to keep one file.
6 years
Changes in fedora-infrastructure pagure issues
by Kevin Fenzi
Greetings.
One of the (many) things that came out of the infra hackfest last week
was a few changes to our issue tracker. If you have ticket perms on the
fedora-infrastructure pagure instance, please follow these guidelines:
We have added several 'priority's:
🔥 URGENT 🔥
Needs Review
Next Meeting
Waiting on Asignee
Waiting on Reporter
Waiting on External
By default, when a new issue is filed, it will get the "Needs Review" pri.
Once anyone with ticket privs touches the issue, they should change that
pri to the correct other state:
🔥 URGENT 🔥 - This should be set only for those things that are really
urgent and need immediate attention. Like a high SLE service down or
something preventing a release from happening or the like. Please do not
misuse this for low SLE things or things that don't need "all hands on
deck" to fix.
Needs Review - Default state before anyone has looked at the issue.
Next Meeting - Something that is to be discussed in the next weekly
meeting.
Waiting on Asignee - This means we have accepted the ticket and have all
needed info to work on it, but it's waiting for cycles to actually do
the needed work. If the assignee is not set it's waiting for someone
with cycles to take it on and do it.
Waiting on Reporter - This means that we have asked the reporter for
more info or need something from them to move the task forward. The task
is stalled until the Reporter provides that.
Waiting on External - This means that the task/issue is waiting until
something else is done first. When using this pri we should be as
explicit as possible about what the thing is, ie 'Waiting for Fedora 28
GA release and this can be done the day after' or 'Waiting until FESCo
rules on this and gives the ok'.
Please if you touch any tickets set the pri accordingly and it will help
us out. I am also going to process the existing tickets and get them in
the right states.
Additionally, I created a default template for new issues. It reads:
* Describe what you need us to do:
* When do you need this? (YYYY/MM/DD)
* When is this no longer needed or useful? (YYYY/MM/DD)
* If we cannot complete your request, what is the impact?
These questions will help us know about the issue, but if anyone can
think of a clearer way to word them or things to add, please let me know.
Many thanks to mattdm for the ideas from the Council issue tracker,
hopefully it will help us set expectations better.
kevin
6 years
new openstack instance?
by Miroslav Suchý
Hi,
what is the status of the new openstack instance? The old one is becoming pain with every new day.
Miroslav
6 years
[release] fedimg 1.1.0
by Sayan Chowdhury
Hi,
I have cut out a new release of fedimg: 1.1.0
Changelog for the release:
1.1.0
-----
Pull Requests
- (@sayanchowdhury) #66, consumer: fedfind has deprecated
`get_release_cid` method.
https://github.com/fedora-infra/fedimg/pull/66
- (@sayanchowdhury) #67, scripts: Fix the manual upload trigger script
https://github.com/fedora-infra/fedimg/pull/67
- (@sayanchowdhury) #68, fedimg: Remove redundant ec2.py file
https://github.com/fedora-infra/fedimg/pull/68
- (@sayanchowdhury) #76, scripts: Update the trigger_upload, and
remove redundant code
https://github.com/fedora-infra/fedimg/pull/76
- (@sayanchowdhury) #75, Initial tests for fedimg
https://github.com/fedora-infra/fedimg/pull/75
- (@euank) #77, Fix numerous dependency issues, fix broken
unit tests, add travis ci config
https://github.com/fedora-infra/fedimg/pull/77
- (@sayanchowdhury) #79, services.ec2: Change the bucket name to more
related to fedimg
https://github.com/fedora-infra/fedimg/pull/79
- (@puiterwijk) #80, Error out if starting the task failed
https://github.com/fedora-infra/fedimg/pull/80
- (@sayanchowdhury) #70, fedimg.ec2: Add metadata to the `image.copy`
fedmsg message
https://github.com/fedora-infra/fedimg/pull/70
- (@sayanchowdhury) #81, services.ec2: Deprecate the PV images
https://github.com/fedora-infra/fedimg/pull/81
- (@sayanchowdhury) #69, fedimg.ec2: Add the support for Elastic Network Adapter
https://github.com/fedora-infra/fedimg/pull/69
- (@sayanchowdhury) #82, uploader: Set push_notifications to True when
automatic upload
https://github.com/fedora-infra/fedimg/pull/82
- (@sayanchowdhury) #84, Update the trigger_upload.py script to add
push_notifications
https://github.com/fedora-infra/fedimg/pull/84
Commits
- 84d0d69ef consumer: fedfind has deprecated `get_release_cid` method.
https://github.com/fedora-infra/fedimg/commit/84d0d69ef
- 7bdd06f56 scripts: Fix the manual upload trigger script
https://github.com/fedora-infra/fedimg/commit/7bdd06f56
- 33a86b79b fedimg: Remove redundant ec2.py file
https://github.com/fedora-infra/fedimg/commit/33a86b79b
- b0fa1c4f9 tests: Fix the test for consumers
https://github.com/fedora-infra/fedimg/commit/b0fa1c4f9
- e082ad464 tests: Add tests for the fedimg.uploader
https://github.com/fedora-infra/fedimg/commit/e082ad464
- 1788e9e6c tests: Add the tests for the fedimg.utils
https://github.com/fedora-infra/fedimg/commit/1788e9e6c
- ed5ddf5bf tests: Add a few more tests for utils. utils cov at 78%
https://github.com/fedora-infra/fedimg/commit/ed5ddf5bf
- 57d4414c0 tests: Add tests for fedimg.utils, coverage 100%
https://github.com/fedora-infra/fedimg/commit/57d4414c0
- 5ccc75652 scripts: Remove redundant imports in the trigger_upload script
https://github.com/fedora-infra/fedimg/commit/5ccc75652
- 84cf48443 docs: Update the README.md for the trigger upload script
https://github.com/fedora-infra/fedimg/commit/84cf48443
- 93e2358fc tests: Fix the copyright years in the test files
https://github.com/fedora-infra/fedimg/commit/93e2358fc
- 0bcb54661 tests: Use assertIs method to check for boolean
https://github.com/fedora-infra/fedimg/commit/0bcb54661
- b6b651f8a tests: Remove the redundant code
https://github.com/fedora-infra/fedimg/commit/b6b651f8a
- 57a2eec93 tests: Make a stronger assertion if the urls is made to
atomic and cloud base
https://github.com/fedora-infra/fedimg/commit/57a2eec93
- 9b77f95f9 tests: Change the assertions to use self.assertIs
https://github.com/fedora-infra/fedimg/commit/9b77f95f9
- 0f3056847 tests: include 'vcr' dependency in setup.py
https://github.com/fedora-infra/fedimg/commit/0f3056847
- 1f29c9389 setup.py: specify test suite to use
https://github.com/fedora-infra/fedimg/commit/1f29c9389
- 9c29204c1 setup.py: use consistent quoting for dependencies
https://github.com/fedora-infra/fedimg/commit/9c29204c1
- 326ab9ef8 setup.py: add 'toml' dependency
https://github.com/fedora-infra/fedimg/commit/326ab9ef8
- b206d76da setup.py: add 'fedfind' dependency
https://github.com/fedora-infra/fedimg/commit/b206d76da
- 32533b038 tests: don't validate signatures for mockhub
https://github.com/fedora-infra/fedimg/commit/32533b038
- 4dc97ac3f tests: add travis.yml
https://github.com/fedora-infra/fedimg/commit/4dc97ac3f
- 5adef1a75 docs/devel: update test running instructions
https://github.com/fedora-infra/fedimg/commit/5adef1a75
- 87f470edb fedimg.ec2: Add metadata to the `image.copy` fedmsg message
https://github.com/fedora-infra/fedimg/commit/87f470edb
- 0ddfd41e9 fedimg.ec2: Add the support for Elastic Network Adapter
https://github.com/fedora-infra/fedimg/commit/0ddfd41e9
- 44ac7b8b3 services.ec2: Change the bucket name to more related to fedimg
https://github.com/fedora-infra/fedimg/commit/44ac7b8b3
- e5df4686f Error out if starting the task failed
https://github.com/fedora-infra/fedimg/commit/e5df4686f
- 744b729ce services.ec2: Deprecate the PV images
https://github.com/fedora-infra/fedimg/commit/744b729ce
- e1607cb5a uploader: Set push_notifications to True when automatic upload
https://github.com/fedora-infra/fedimg/commit/e1607cb5a
- 82b63d886 fedimg: Fix the trigger_upload script to include
push_notifications arg
https://github.com/fedora-infra/fedimg/commit/82b63d886
- 0f8d6c08f readme: Update the trigger_upload usage in README file
https://github.com/fedora-infra/fedimg/commit/0f8d6c08f
- 0dc560f13 scripts: Make the -p arg optional
https://github.com/fedora-infra/fedimg/commit/0dc560f13
- 43103f59b scripts: Move logic inside the main function
https://github.com/fedora-infra/fedimg/commit/43103f59b
--
Sayan Chowdhury <https://sayanchowdhury.dgplug.org/>
Senior Software Engineer, Fedora Engineering - Emerging Platform
GPG Fingerprint : 0F16 E841 E517 225C 7D13 AB3C B023 9931 9CD0 5C8B
Proud to work at The Open Organization!
6 years
Libravatar shutting down on 2018-09-01
by Justin W. Flory
Hi all,
Libravatar announced this morning they are shutting down their service
on 2018 September 1; this affects applications that use Libravatar, such
as Pagure, Tahrir (Fedora Badges), and possibly others.
https://blog.libravatar.org/posts/Libravatar.org_is_shutting_down_on_2018...
In their post, they mentioned they do not know of another FOSS image
hosting service like Libravatar. If hosting images ourselves is not
desired, Gravatar is the best viable alternative I know of.
Since the Infrastructure hackathon is next week, I thought this might be
timely to mention, since it affects most Fedora applications that use a
profile picture for contributors.
I will try and file RFEs on Pagure and Tahrir today, but other affected
services don't immediately come to mind. Feel free to file some too if
there are more.
--
Cheers,
Justin W. Flory
jflory7(a)gmail.com
6 years
Interviews on the Fedora Infrastructure Hackathon 2018
by Justin W. Flory
Hi all!
This week, the Fedora Infrastructure team is convening for a Hackathon
from April 9-13 at Fredericksburg, VA. The hackathon is intended to help
the team leap ahead for several critical Fedora and CentOS initiatives.
Dennis Otugo, a member of the CommOps team, interviewed members of the
Fedora Infrastructure team to ask what the goals for the hackathon are
and why it is needed.
Read more on the Community Blog:
https://communityblog.fedoraproject.org/fedora-infrastructure-hackathon-2...
Thanks Dennis for putting the interview together, and thanks to the
interviewees for their answers on short notice. :-)
--
Cheers,
Justin W. Flory
jflory7(a)gmail.com
6 years
