Improve compression ratio of SquashFS filesystem on the installation media.
Name: Bohdan Khomutskyi
Targeted release: I propose this change for Fedora 32
Last updated: Jan 5 2020
Pagure.io issue: https://pagure.io/releng/issue/9127
I was unable to create an article in Fedora wiki system.
As of Fedora 31, the LiveOS/squashfs.img file on the installation image, is
compressed with default settings of mksquashfs. The standard configuration
is set to XZ algorithm with block size of 128k and BCJ filter enabled.
Those parameters can be adjusted which will lead to a better compression
ratio and/or reduction of the CPU usage at build time.
This is simple to achieve. Recently, Lorax has gotten support for
adjusting the compression options for mksquashfs via the configuration
file. The file should be altered as following:
bcj = yes
args = -b 1M -Xdict-size 1M -no-recovery
Where -b 1M and -Xdict-size 1M are block and dictionary sizes respectively.
Could be adjusted.
Benefit to Fedora
Reduction of the installation media size and the cost of storing and
Reduction of the CPU usage at build time. Depending on which compression
See a graphical detail at https://pagure.io/releng/issue/9127.
The build environment should have support for adjusting the Lorax
configuration file.. Lorax is a program that produces the
LiveOS/squashfs.img file on the installation media.
One of the way to allow for such customization, is to add a feature in
Pungi, to allow for passing -c option to Lorax.
Release engineering: #9127 <https://pagure.io/releng/issue/9127>
Policies and guidelines: N/A
Trademark approval: N/A
This change comes at a cost of higher memory usage during the
installation. Based on my personal estimations, this should not be the
issue. Since the decompression should require up to 1MiB per thread.
Increasing the block size on the current configuration with EXT4 file
system, should increase latency while accessing the EXT4 filesystem. The
exact impact is to be evaluated.
The impact of latency will be reduced, if the plain SquashFS option is
Release notes See also
Release Configuration Management engineer
== Summary ==
Change format of the RPM database from Berkeley DB to a new Sqlite format.
== Owner ==
* Name: [[User:pmatilai| Panu Matilainen]] [[User:ffesti|Florian Festi]]
* Email: pmatilai(a)redhat.com ffesti(a)redhat.com
== Detailed Description ==
The current rpm database implementation is based on Berkeley DB 5.x, a
version which is unmaintained upstream for several years now. Berkeley
DB 6.x is license incompatible so moving to that is not an option. In
addition, the existing rpmdb implementation is notoriously unreliable
as it's not transactional and has no other means to detect
Changing to a more sustainable database implementation is long
overdue. We propose to change the default rpmdb format to the new
sqlite based implementation. Support for current BDB format will be
retained in Fedora 33, and phased out to read-only support in Fedora
== Benefit to Fedora ==
* A far more robust rpm database implementation
* Getting rid of Berkeley DB dependency in one of the core components
== Scope ==
* Proposal owners:
** Once [[Changes/RPM-4.16|RPM 4.16]] lands and passes initial
shakedown, change the default rpmdb configuration to sqlite
** Address any bugs and issues in the database backend found by wider
** Help other developers to address Berkeley DB dependencies
* Other developers:
** Test for hidden Berkeley DB dependencies in other software, address
them as found and needed
* Release engineering: [https://pagure.io/releng/issue/9308 #9308]
* Policies and guidelines: Policies and guidelines are not affected
* Trademark approval: N/A (not needed for this Change)
== Upgrade/compatibility impact ==
=== Upgrading ===
* Ability to upgrade is not affected
* After upgrade completes, manual action (rpmdb --rebuilddb) will
probably be needed to convert to sqlite. Alternatively user can change
configuration to stay on BDB.
=== Compatibility ===
* Container/chroot use-cases will be affected: older rpm versions will
be unable to query/manipulate the rpmdb from outside the chroot
* Koji/COPR may need to override the database format (back to) BDB for
the time being
== How To Test ==
* Rpmdb gets thoroughly exercised as a matter of normal system
operation, performing installs, updates, package builds etc
* Of specific interest here is torture testing: forcibly killing rpm
in various stages of execution - database should stay consistent and
operational (other system state is out of scope)
* Test database conversions from one backend to another (rpmdb
--rebuilddb --define "_db_backend <backend>")
== User Experience ==
* In normal operation, users should see little or no change
* Behavior in error situations is much more robust: forcibly killed
transaction no longer causes database inconsistency or corruption
== Dependencies ==
* This change depends on [[Changes/RPM-4.16|RPM 4.16]], support for
sqlite rpmdb is not present in older versions
* RPM will grow a new dependency on sqlite-libs
* Technically the rpmdb format is an internal implementation detail of
RPM and the data is only accessible through the librpm API, but some
software is making assumptions both about the format and/or in
particular, file naming. These are being tracked at
* Upgrade tooling could/should perform rpmdb rebuild at end, this
would be a good thing to do regardless of this change
== Contingency Plan ==
* Contingency mechanism:
** Revert the default database back to Berkeley DB backend in the
package. Running 'rpmdb --rebuilddb' on hosts is currently required to
actually convert the database, but means to automate conversion in
specific conditions is being discussed upstream.
** The rpm-team does not expect problems with the database backend
itself, but we are aware that postponing may be needed due to
infrastructure or other tooling not being ready, primarily due to
inability to access the database from older releases.
* Contingency deadline: Beta freeze
* Blocks release? Yes
== Documentation ==
* [https://rpm.org/wiki/Releases/4.16.0 | RPM 4.16 release notes]
== Release Notes ==
* After upgrading from an older release, rpm operations will issue
warnings about database backend configuration not matching what's on
disk. Users should run 'rpmdb --rebuilddb' at earliest opportunity, or
change configuration to stay on Berkeley DB backend (eg 'echo
%_db_backend bdb > /etc/rpm/macros.db')
* The details are subject to change, the database rebuild may be done
by upgrade tooling
He / Him / His
Senior Program Manager, Fedora & CentOS Stream
Almost a week ago, I built cmpfit and fityk in side tags on F31, F32
and F33. While the builds for F33 moved directly to stable - as
expected - the other two got stuck for 4 days. I noticed that I could
push them manually to testing, which I did a little over two days ago,
but they seem to be stuck again, transitioning from pending to
testing. Updates for other packages I built around the same time and
later are already in testing. These are the updates in question:
Hello fellow java package maintainers!
We are planning to bump the JDK from java-1.8.0-openjdk to java-11-openjdk for F33. Please see
* if you have some java package, be aware that we are bumping JDK in rawhide
* Ensure your package builds and runs fine with JDK11 (see the
* there is special tooling ready for this, before mass rebuild is launched
** See https://fedoraproject.org/wiki/Changes/Java11#copr_preliminary_rebuild
* If you do not want Fedora rotten with JDK8 for ever, continue reading
We ran a preliminary mass rebuild of javastack in copr repo
https://copr.fedorainfracloud.org/coprs/jvanek/java11/builds/ (select "all" instead of "25" at the
bottom), on packages requiring java,javac, java-devel, maven-local, ant, ivy & comp for build. You
can see, the result was quite dramatic:
1225 total; attempted to rebuild
483 failed; from those 191 are trivial failures (but if you fix it, there is no guarantee real
troubles are not hidden behind that)
556 orphans or dead or otherwise tragic so the build did not even start
I would kindly ask you to search yourself in this list: https://jvanek.fedorapeople.org/java11/people
If you are here, please check status of your package in https://jvanek.fedorapeople.org/java11/init
(pain text of https://copr.fedorainfracloud.org/coprs/jvanek/java11/builds).
* If your package is "succeeded", congratulations nothing to do, and just keep en eye on JDK bump
* If there is "failed" but contains "- -" then it is probably orphan. If you wish to resurrect it,
please ensure it runs against JDK11 (see lower)
* If there is "failed" but failed in "seconds", then those packages failed so quickly, that the
build was in initial phases. That usually mean that you build with source/target lower then 1.6
JDK11 supports 1.6 and up. We recommend to bump the source/target to 1.8, to allow existence of
compact 1.8 packages alongside main javastack. See
https://fedoraproject.org/wiki/Changes/Java11#Wrong_source.2Ftarget_version. Don't forget to
upstream the patch, or maybe it is enough to update to more fresh upstream release which supports
JDK11? it may happen, that after the fix, your build will fail in more terrible way (see below)
* If there is "failed", and its none of above, then your package simply failed. Very often the
scary error may be fixed by bump to latest upstream version. JDK 11 is out for several years.
Please, try to fix the package. Don't hesitate to ask on devel(a)fedoraproject.org or
java-devel(a)fedoraproject.org or directly to me jvanek(a)redhat.com. If you fix the fail, feel free to
share your fix, it may help others.
We are trying to gather the most common issues at
Feel free to enhance the page, or write us your case (possibly both with solution and without) so
we can add it here.
Debugging Your failures.
The copr repo we maintain, contains builds of java-11-openjdk as system JDK, javapackages-tools
honoring that, and java-1.8.0-openjdk as non system JDK. Also it contains successfully rebuilt
packages. You can directly use this copr repo in several ways.
* first glance on error. On https://copr.fedorainfracloud.org/coprs/jvanek/java11/builds/ find your
build (select "all" instead of "25" at the bottom),
** Click its number, select chroot (currently fedora-32-x86_64 ) and check the logs. Main log is
* anything you push to rawhide, will automatically rebuild here in f32 chroot (we have a JDK in
rawhide broken a bit currently)
** It is the best approach. If you can fix your package in rawhide directly, without breaking the
rawhide too much, go for it
** If yo need to experiment, I have a mock config for you (generated from copr-cli mock-config
jvanek/java11 fedora-32-x86_64) which you can copy to your /etc/mock and use -
https://jvanek.fedorapeople.org/java11/jvanek-java11-fedora-32-x86_64.cfg . Eg:
sudo cp downloaded-fedora-32-x86_64.cfg /etc/mock/jvanek-java11-fedora-32-x86_64.cfg
# change spec, bump sources, apply patches
mock -r jvanek-java11-fedora-32-x86_64 *.src.rpm
Or any other packaging workflow you use, and you can use against the copr repo.
Thank you very much for your help, there are 500 failures, and 1000 java packagers, but only 2
active members of java sig. Without your help, the JDK bump will be very hard.
On behalf of Fedora java group
devel-announce mailing list -- devel-announce(a)lists.fedoraproject.org
To unsubscribe send an email to devel-announce-leave(a)lists.fedoraproject.org
Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: https://firstname.lastname@example.org...
the semiannual exercise is upon us. FESCo candidates must submit an
"interview" in which they answer a set of questions (but can also add whatever they want).
The question whether we should have a new set of questions needs to be answered.
Currently we have the following:
Mandatory Question #1: Describe some of the important technical issues
you foresee affecting the Fedora community. What insight do you bring
to these issues?
Mandatory Question #2: What objectives or goals should FESCo focus on
to help keep Fedora on the cutting edge of open source development?
Mandatory Question #3: What are the areas of the distribution and our
processes that, in your opinion, need improvement the most? Do you
have any ideas how FESCo would be able to help in those "trouble
Please answer with any proposals. If there is sufficient support for
change, I'll gather a list and submit this for some kind of poll
(details to be figured out...).
On 29. 04. 20 21:42, Lloyd Kvam wrote:
>> What you say is true. I still don't agree that "python3.9" as a package name
>> annoys humans.
> I am not a package pro, but simply reading along as an interested human user. To me, adding
> periods in package names can be confusing.
My sentence was about "python3.9" not being more annoying than "python-3.9".
I wonder, why do you consider periods in names confusing?
We have around ~100 source package names with dot. Most of them have versions, e.g.:
Some use dot as a separator, e.g.:
> I will adjust to whatever you decide to do, and I am not informed enough to want a vote in how
> this decision comes down, but I do not see an advantage to this sort of change.
The biggest advantage I see is getting closer to upstream.
The command that the user executes is "python3.9", not "python39".
I know no other place in the Python ecosystem where Python 3.9 is called
"python39" than the names of RPM packages (or other Linux distro packages).
I've googled "python36", "python37" etc. and all I could find was
Fedora/RHEL/CentOS related (or AUR). We have invented that naming ourselves and
we don't like being different :)
(There is the "py39" short identifier used e.g. in tox, but not "python39".)