Automatic virtual provides for RPM macros?
by Miro Hrončok
Hello,
today at Nest, somebody said "unfortunately, there is no way to tell what
package to install to get a particular RPM macro".
I think that having an RPM provides generator for "rpm-macro(__python3)" or
similar should be a fairly simple exercise.
Would you folks consider that useful?
--
Miro Hrončok
--
Phone: +420777974800
IRC: mhroncok
2 months, 3 weeks
RPM-level auto release and changelog bumping - Fedora 33 System-Wide Change proposal
by Ben Cotton
https://fedoraproject.org/wiki/Changes/rpm_level_auto_release_and_changel...
== Summary ==
redhat-rpm-config will be updated so users of the auto framework get
automated release and changelog bumping.
== Owner ==
* Name: [[User:nim| Nicolas Mailhot]]
* Email: <nicolas.mailhot at laposte.net>
== Detailed Description ==
This is a system-wide change because all packages build with
redhat-rpm-config, but it only concerns packages that opted to use
this part of redhat-rpm-config (auto framework).
The change will make those packages auto-bump and auto-changelog at
the rpm level, in an infrastructure-independent way.
== Benefit to Fedora ==
Autobumping removes a huge packager shore and makes timestamping in
changelogs more reliable.
== Scope ==
* Proposal owners: The feature is coded and works at the rpm level.
Unfortunately, mock filters away the srpms containing the bump state,
so it does not work in upper layers.
* Other developers: The feature requires buy-in by mock developers
(and probably koji developers) to lift the restrictions that block it
above the rpm level. Also, it requires a mechanism to pass the user
name and email that will be used in bumped changelogs (defining two
variables in ~/.rpmmacros is sufficient at rpm level)
* Mock issue: https://github.com/rpm-software-management/mock/issues/599
* Release engineering: https://pagure.io/releng/issue/9567
* Policies and guidelines: maybe eventually if things work out on the
technical level
* FPC issue: https://pagure.io/packaging-committee/issue/998
* Trademark approval: N/A (not needed for this Change)
== Upgrade/compatibility impact ==
This is a pure build tooling update, it changes how things are built
not what is built.
== How To Test ==
A redhat-rpm-config packages with the changes and some example
packages are available in
https://copr.fedorainfracloud.org/coprs/nim/refactoring-forge-patches-aut...
Since the mock/copr layer is currently blocking the feature, you need
to install the redhat-rpm-config and forge macro packages available in
this repo locally. Afterwards you can take any of the example packages
in the repo and rebuild them with rpmbuild -ba to your heart content,
and see the releases bump and the changelogs being updated
accordingly.
To get beautiful changelogs, you also need to add
<pre>
%buildsys_name Your name
%buildsys_email Your email
</pre>
in ~/.rpmmacros
== User Experience ==
N/A Packager experience change only
== Dependencies ==
The change is a spin-off of
https://fedoraproject.org/wiki/Changes/Patches_in_Forge_macros_-_Auto_mac...
Therefore, it depends on the success of that other change and will
probably need rebasing if the code in this other change evolves during
the redhat-rpm-config merge.
It also depends on mock / copr/ koji buy-in and changes, that may add
their own requirements.
== Contingency Plan ==
There is no contingency plan because the change will happen or not at all.
== Documentation ==
There is as much documentation as the average redhat-rpm-config change
(ie comments in the macro files themselves)
== Release Notes ==
N/A Packager productivity change only
--
Ben Cotton
He / Him / His
Senior Program Manager, Fedora & CentOS Stream
Red Hat
TZ=America/Indiana/Indianapolis
9 months, 2 weeks
Mangling shebangs in text files: How to detect them, bug in the current implementation and possible solutions
by Miro Hrončok
Hello,
for many releases, Fedora has the brp-mangle-sehbangs BuildRoot Policy Script
that does the following:
1) Gets all executable files in the buildroot
2) Gets all "text" files from those
3a) Mangles shebangs that are "wrong"
(e.g. #!/usr/bin/env node -> #!/usr/bin/node)
3b) Removes executable bits from "text" files without shebangs
The idea behind this is that all "text" files that are executable need a
shebang and if they don't have it, something is wrong. OTOH files that are
"binary" don't need it.
I intentionally put the terms "text" and "binary" in quotation marks, as the
definition is somewhat fuzzy. Up until now, the script did the detection by
utilizing the file tool to get the MIME type. If the MIME type starts with
text/, it considered the executable to be a text file.
However, a bug [1] has been discovered. Some obvious text files, such as
executable JavaScript scripts, are detected as application/ (e.g.
application/javascript), and hence are not considered "text". If a JavaScript
executable script has the #!/usr/bin/env node shebang, the brp-mangle-sehbangs
script does not mangle it.
One possible solution [2] to this problem is to limit the number of bytes the
MIME detection reads. My experiments showed that limiting the number of bytes
to 8 always recognizes JavaScript (and other scripting languages) files as
text/plain and binary files as application/octet-stream. As a side effect, it
might make the BRP script faster. However, I am not sure if this approach is
deterministic enough.
Another solution, suggested by Florian Weimer [3], is to not detect MIME type
at all, but use eu-elfclassify instead. The idea is quite simple: If (and only
if) the executable file is ELF [4], it does not require a shebang. Instead of
some fragile idea about what files are text and what files are binary, this is
quite deterministic. It allows mangling shebangs of executable ZIP files etc.
I've drafted the eu-elfclassify solution in a pull request [5]. However, we
have discovered that several non-elf binary formats in Fedora are possibly
legitimately executable. E.g. .exe files (for mono or wine) or other formats
registered with the kernel [6].
We are presented with 3 possible actions:
1) Keep the script as it is, say the text/ MIME type limitation is how this BRP
script was scoped. Affected packages would need to correct shebangs manually.
2) Limit the MIME type detection to 8 bytes and hope it will not yield
incorrect results.
3) Use eu-elfclassify. Consider non-ELF executables without shebangs bogus and
document this. Packages that are affected would need to opt-out.
What do you think?
[1] https://bugzilla.redhat.com/1998924
[2] https://bugzilla.redhat.com/1998924#c3
[3] https://bugzilla.redhat.com/1998924#c4
[4] https://en.wikipedia.org/wiki/Executable_and_Linkable_Format
[5] https://src.fedoraproject.org/rpms/redhat-rpm-config/pull-request/145
[6] https://www.kernel.org/doc/html/latest/admin-guide/binfmt-misc.html
--
Miro Hrončok
--
Phone: +420777974800
IRC: mhroncok
1 year, 8 months
contacting package owners
by cheese＠nosuchhost.net
hi
i tried to contact package owners of the package "mysqltuner" and thought
it would be sufficiant to mail to
PACKAGE-owner(a)fedoraproject.org and there tried
mysqltuner-owner(a)fedoraproject.org
but this did now work. i think that used to work some years ago.
i than tried:
mysqltuner-owners(a)fedoraproject.org
mysqltuner-owner(a)lists.fedoraproject.org
did not work
i knew that we *had* some address for contaction all owners/packagers, but
i did not find a solution via google (maybe my fault) and i miss a search
over our documentation at
https://docs.fedoraproject.org/en-US/packaging-guidelines/
therefore:
*) what is currently the correct way to contact package-owners?
*) we should have some simple search-box for our own documentation
yours
josef
1 year, 8 months
Sphinx/Doxygen HTML documentation and bundling guidelines
by Ben Beasley
It was recently brought up in a package review
(https://bugzilla.redhat.com/show_bug.cgi?id=2006555) that a
Sphinx-generated HTML documentation package contained several JavaScript
libraries, and that:
1. their licenses should be accounted for and added to the License
field for the -doc subpackage
2. they should be treated as bundled libraries and marked with virtual
Provides
Note that the JavaScript is inserted by the Sphinx documentation build
system, and is not present in the source tarball. Your “typical” Sphinx
documentation has minified and unminified copies of JQuery (js-jquery)
and Underscore (js-underscore), both MIT-licensed, plus several
unminified utility libraries that belong to Sphinx itself (doctools.js,
language_data.js, searchtools.js). The overall Sphinx project is
BSD-licensed. The details vary by theme, but this is the most common case.
To me, it seems that this feedback is a reasonable strict reading of the
relevant packaging guidelines. If this is the right interpretation of
Fedora policy, what should be done about it in general?
-----
Based on looking for installed files named “searchtools.js” in packages
named -doc or -docs, there are about 600 documentation packages
generated with Sphinx. This might miss some that have exotic themes.
As far as I can tell, only one existing package in the distribution,
python-murano-package-check, tries to use virtual Provides to track the
libraries that belong to Sphinx itself:
Provides: bundled(js-doctools)
Provides: bundled(js-jquery) = 3.1.0
Provides: bundled(js-searchtools)
Provides: bundled(js-underscore) = 1.3.1
Provides: bundled(js-websupport)
and this package does not actually build its documentation subpackage.
Only the following packages have virtual Provides for js-jquery and
js-underscore. In most cases these lack the version numbers.
- arb-doc
- mpdecimal-doc
- python-BTrees-doc
- python-latexcodec-doc
- python-networkx-doc
- python3-persistent-doc
- sympy-doc
Of the packages in the list above, only mpdecimal-doc has a License
field that seems to try to account for the licenses of the JavaScript
libraries.
-----
Personally, I can add virtual Provides and adjust the License on the
-doc subpackage for my package under review, and it’s even possible for
me to replace the Underscore and JQuery libraries with symbolic links to
those installed by js-jquery and js-underscore. However, changing one
package, or even every package I control, is a tiny drop in a vast ocean
of documentation packages. Consider, for example, that Doxygen also
includes JQuery and several Doxygen-specific libraries in its output, so
the same arguments about licenses and virtual Provides apply to it. A
search for “dynsections.js” in packages named *-doc or *-docs gives
nearly 400 Doxygen-based packages.
If this is a real problem, it seems like it needs to be handled and/or
documented in a way that can potentially scale to the rest of those 600
Sphinx-based packages, to the nearly 400 Doxygen-based packages, and
beyond that to the output of other documentation systems. I am not sure
what that would look like.
All thoughts are welcome.
– Ben Beasley (FAS: music)
1 year, 8 months
Re: Mangling shebangs in text files: How to detect them, bug in the current implementation and possible solutions
by Miro Hrončok
On 23. 09. 21 1:40, Josh Stone wrote:
> On 9/22/21 4:21 AM, Miro Hrončok wrote:
>> Hello,
>>
>> for many releases, Fedora has the brp-mangle-sehbangs BuildRoot Policy Script
>> that does the following:
>>
>> 1) Gets all executable files in the buildroot
>> 2) Gets all "text" files from those
>> 3a) Mangles shebangs that are "wrong"
>> (e.g. #!/usr/bin/env node -> #!/usr/bin/node)
>> 3b) Removes executable bits from "text" files without shebangs
>
> While we're at it, can we teach the script to ignore Rust attributes?
> They're written like #![attr...], and when that's on the first line some
> editors try to be helpful and make the file executable. That's
> considered an error with the current script since the "shebang" doesn't
> start with '/', but it would be best IMO to have it remove the
> executable bit.
I believe that currently the script would error:
ERROR: $f has shebang which doesn't start with '/' (#![attr...])
Have you ever seen that in a Fedora package?
--
Miro Hrončok
--
Phone: +420777974800
IRC: mhroncok
1 year, 8 months
Re: Mangling shebangs in text files: How to detect them, bug in the current implementation and possible solutions
by Petr Pisar
V Wed, Sep 22, 2021 at 12:47:07PM -0400, Steve Grubb napsal(a):
> I find the file utility to be almost reliable. It changes how it identifies ELF
> files every couple releases. So, to stabilize this, fapolicyd-cli uses it's
> own logic to determine what kind of ELF file it finds. I also regularly find
> text/plain files where it cannot identify the language and files that are
> application/octet-stream which are also misidentified.
>
File's libmagic will always misdetect some files.
I'd like to see rpmbuild to prefer user.mime_type extended attribute over
libmagic guess. That way packagers could override the MIME type directly from
a spec file:
%install
setfattr -n 'user.mime_type' -v 'text/x-perl' %{buildroot}%{_bindir}/GET
If rpmbuild carried that attribute to RPM archive, rpm would set the attribute
when installing that package and it would become available to other tools like
fapolicyd-cli.
Technically we could patch libmagic to do that, but I feel that libmagic
upstream wouldn't like that. Maybe a place for an intermediate wrapper.
-- Petr
1 year, 8 months
Re: Mangling shebangs in text files: How to detect them, bug in the current implementation and possible solutions
by Miro Hrončok
On 22. 09. 21 22:48, Steve Grubb wrote:
> On Wednesday, September 22, 2021 4:26:49 PM EDT Miro Hrončok wrote:
>>> By chance do you have a pointer to one of those javascript files that is
>>> misidentified? (Or any other for that matter). I'd like to see what's
>>> going on and get a fix in place.
>>
>> yarnpkg package, %prepped
>>
>> $ file --mime-type yarn-1.22.10/bin/yarn.js
>> yarn-1.22.10/bin/yarn.js: application/javascript
>
> application/javascript is, unfortunately, correct. This one is governed by
> RFC 4329 which makes it official.
>
> From all the scan that we've done on fullish installs in the past, there's
> only 2 others that you might run across: application/x-elc (lisp) and
> application/x-java-applet.
>
> Maybe you just build in logic to workaround these 3 types? application/
> javascript is really the only one I can think of that is common.
Yeah, maybe we should just do that. However, that would not cleanup the
executable pngs.
--
Miro Hrončok
--
Phone: +420777974800
IRC: mhroncok
1 year, 8 months
Re: Mangling shebangs in text files: How to detect them, bug in the current implementation and possible solutions
by Miro Hrončok
On 22. 09. 21 18:47, Steve Grubb wrote:
> 4) maybe fapolicyd-cli has better detection? Or at least, its more closely
> maintained. It also has it's own ELF detection so that it's stable from
> release to release.
Not checked whether it has better detection or not, but I see it would pull the
following packages into the default buildroot:
acl
cryptsetup-libs
dbus-broker
dbus-common
dbus
device-mapper-libs
device-mapper
expat
fapolicyd-dnf-plugin
fapolicyd
iptables-legacy-libs
json-c
kmod-libs
libargon2
libibverbs
libnl3
libpcap
libseccomp
lmdb-libs
mpdecimal
python-pip-wheel
python-setuptools-wheel
python3-libs
python3
systemd-pam
systemd-rpm-macros
systemd
Installed size: 54 M
I don't think this is acceptable. Even systemd and Python alone are not
supposed to be there.
--
Miro Hrončok
--
Phone: +420777974800
IRC: mhroncok
1 year, 8 months
