koji in PiPy/support of setup.py
by Pavol Babincak
Hi,
I'm using koji as a python package in my scripts. Usually I develop them
inside of a virtual environment. I do not use system packages[1] in
virtualenv.
Currently I'm not able to install koji and it dependencies inside
virtualenv easily. Having koji in PiPy would help me. That way I can
just specify it in my requirements.txt or in install_requires of my
setup.py. I *guess*[2] one way to do that is to have setup.py in koji
code and add some extra steps as part of the release process to publish
it there.
What do you think about this?
For some time now I use my own setup.py. I guess it is not ready for PR
so I'd love to see comments on this as well:
https://pagure.io/fork/pbabinca/koji/blob/setup_WIP/f/setup.py
[1] Among other reasons I'd like to have environment isolated from the
system as much as possible.
[2] I have never maintained package on PiPy.
--
Pavol Babincak
6 years, 1 month
Re: New release: mock 1.4.1
by Dennis Gregorovic
I have filed https://pagure.io/koji/issue/398 as an RFE for koji to provide support for choosing between chroot/nspawn.
Cheers
-- Dennis
On 04/26/2017 02:35 PM, Dennis Gilmore wrote:
> Please do not push this stable until there has been sufficient time to
> test in koji and possibly develop new features in koji to make
> supporting chroot or systemd-nspawn configurable per tag/target
>
> Dennis
>
> El mié, 26-04-2017 a las 15:52 +0200, Miroslav Suchý escribió:
>> Hi,
>> I just released new version of Mock.
>>
>> I submitted it to updates-testing F26, F25 and EPEL-7 so please test
>> it
>> before it hit stable.
>>
>> From the release notes:
>> https://github.com/rpm-software-management/mock/wiki/Release-Notes
>> -1.4.1
>>
>> There are new features:
>>
>> * Mock previously used chroot technology. Few past releases Mock
>> offered
>> systemd-nspawn which is modern container technology for better
>> isolation. This release use systemd-nspawn as default. If you want
>> to
>> preserve previous behaviour you can use `--old-chroot` option.
>> * Mock now uses bootstrap chroot to install target chroot. This is
>> big
>> change and see special paragraph at the bottom of this release notes.
>> * Chroot now contains `/dev/hwrng` and `/dev/prandom` when they
>> exists
>> in host [[#33](https://github.com/rpm-software-management/mock/issues
>> /33)]
>> * We added %distro_section macro to Mageia configs
>>
>> There are some bugfixes:
>>
>> * Resultdir is now chowned to user who executed mock so they can
>> delete
>> the files.
>> * Previously we declared that package state plugin is enabled by
>> default, but the plugin was in fact disabled. It is now enabled by
>> default (as stated in mock documentation)
>> [[RHBZ#1277187](https://bugzilla.redhat.com/show_bug.cgi?id=1277187)]
>> .
>> * Creating directories for mount points have been delayed after mount
>> of
>> tmpfs [[#57](https://github.com/rpm-software-management/mock/issues/5
>> 7)]
>> * Exit code of machinectl is now ignored as machinectl set non-zero
>> code
>> even for non-fatal errors. Errors which are quite often not relevant
>> nor
>> important for mock.
>> * hw_info plugin does not crash when output contains non-ASCII
>> characters
>> [[#68](https://github.com/rpm-software-management/mock/issues/68)]
>>
>> Notes:
>> * This version has not been released for EL6. If you are using EL6
>> and
>> you want to use latest Mock, please upgrade you infrastructure to
>> EL7.
>> * Configs for s390 architecture has been removed as it is not
>> supported
>> any more.
>> * Configs for aarch64 and ppc64le now use different GPG key as those
>> architectures has been moved from Secondary to Primary.
>> * Epel5 config points now to Vault.centos.org. Note that EL5 has
>> been
>> EOLed. We will keep epel-5 config for some time. But any issue with
>> building for epel-5 target will not be fixed.
>>
>> ## Bootstrap chroot
>>
>> Mock is calling `dnf --installroot` to install packages for target
>> architecture into target directory. This works. Mostly. The only
>> problem
>> that use host DNF and rpm to install packages. But this can cause
>> problem when new RPM feature is introduces. Like Soft dependencies
>> or
>> Rich dependencies. When you have EL6 host and try to install Fedora
>> rawhide package with Rich dependency then rpm will fail and you
>> cannot
>> do anything about it. You can upgrade your build machine to Fedora
>> rawhide, but that is often not possible when it is part of critical
>> infrastructure.
>>
>> So we introduced Boostrap chroot. And 'we' actually means Michael
>> Cullen
>> who implement it. And Igor Gnatenko who proposed this idea. Big
>> kudos
>> for both of them.
>>
>> Bootstrap chroot means that we first create very minimal chroot for
>> target platform and we call DNF/YUM from that platform. For example:
>> when you are on RHEL7 and you want to build package for
>> `fedora-26-x86_64`, mock will first create chroot called
>> `fedora-26-x86_64-bootstrap`, it will install DNF and rpm there
>> (fc26
>> versions). Then it will call DNF from `fedora-26-x86_64-bootstrap`
>> to
>> install all needed packages to `fedora-26-x86_64` chroot.
>>
>> The disadvantage is that you will need more storage in
>> `/var/lib/mock`,
>> the build is little bit slower. But you will hardly notice that
>> unless
>> you disabled `yum_cache` and `root_cache` plugins for some reasons.
>>
>> The advantage is that you can use stable version of OS to build
>> packages
>> for even most recent OS. And vice versa.
>>
>>
>>
>> If you want to preserve previous behaviour you can use
>> `--no-bootstrap-chroot` command line option or set:
>>
>> ```
>> config_opts['use_bootstrap_container'] = False
>> ```
>>
>> in your configuration.
>>
>>
>> --
>> Miroslav Suchy, RHCA
>> Red Hat, Senior Software Engineer, #brno, #devexp, #fedora-buildsys
>> _______________________________________________
>> buildsys mailing list -- buildsys(a)lists.fedoraproject.org
>> To unsubscribe send an email to buildsys-leave(a)lists.fedoraproject.or
>> g
>>
>>
>> _______________________________________________
>> buildsys mailing list -- buildsys(a)lists.fedoraproject.org
>> To unsubscribe send an email to buildsys-leave(a)lists.fedoraproject.org
--
Dennis Gregorovic
Associate Manager, PnT DevOps
Red Hat
dgregor(a)redhat.com T: +1 978.392.3112 M: +1 617.901.9799
6 years, 7 months
RFC: Content generator for module build service (v2)
by Stanislav Ochotnicky
Hi folks,
Note: my first email was sent to @fedoraproject.org so sorry for
duplication for some of you. This has been tweaked so the output is
actually importable into koji test instance with minor tweaks.
I'm working on module build service change that would make it into a
content generator similar to what OSBS is doing.
Quick intro around current state:
* MBS orchestrates koji builds, tags, etc
* Koji AFAIK knows nothing about mbs itself
* after finishing the build - mbs creates a module-specific tag and
tags every relevant build into it
This is the second stab and what the mbs CG metadata output might look
like:
https://paste.fedoraproject.org/paste/GM00iYtq55HXCQyXjbEVZV5M1UNdIGYhyRL...
I was actually able to import the above into a test koji instance (after
some SQL additions of new types etc).
Notable things:
* outputs - missing logs. Do we need them? Each module is basically a
collection (repository) of rpms built in koji which each have their
own logs. We could probably add orchestration logs later?
* tools, components - Not sure there's any set of specific tools worth
mentioning. Maybe version of koji client library?
* container - MBS makes no use of container so the "type" is none, but
we still have to specify the architecture or koji complains on import.
* "typeinfo" keys in build and output fields - I am a little
confused. When I tried to copy what OSBS is doing in the extra
fields koji refused to import the metadata. Current metadata is
based on some experimentation and adjustments based on error
messages. I think some guidance on the structure of the types and
typeinfo(s) would be nice since they don't seem to be up to date
with documentation.
And most importantly: the modulemd type output
- this would be a modulemd.yaml file that contains the mmd content [1] once
downloaded
- component metadata list would include all the rpms that are tagged in
the final module tag - all architectures, subpackages etc
- notably no signatures would ever be mentioned here since they are
not fixed at the end of module build unlike container builds
There was an idea to split the modulemd into separate arch-specific
pieces, but it didn't bring anything useful to the table so was
scratched.
Would the above approach be generally acceptable? Is there any concern
about adding components for file that is just metadata? Anything else I
might have missed?
Thanks,
[1] can be seen as "modulemd" in
https://mbs.fedoraproject.org/module-build-service/1/module-builds/199?ve...
[2] WIP PR for the MBS change is here: https://pagure.io/fm-orchestrator/pull-request/522#
--
Stanislav Ochotnicky <sochotnicky(a)redhat.com>
Business System Analyst, PnT DevOps - Brno
PGP: F434 2286 27DC 7D9B 2B64 0866 BCBD 752E 7B08 7241
Red Hat Inc. http://www.redhat.com
6 years, 7 months
How to best troubleshoot the kojid error "koji.AuthError: User 3 is
not a host"
by Tengiz Dawkins
Hi all,
For the lack of koji-users@ list I dared to ask koji-devel@.
I am very new to koji and I setup a testing stack using
http://www.devops-blog.net/koji/koji-rpm-build-system-installation-part-1
Currently I am stuck at koji builder authentication. The validation user vs
host is not quite clear to me:
kojiHUB@root@/etc/kojid>/usr/sbin/kojid -f
Traceback (most recent call last):
File "/usr/sbin/kojid", line 5140, in <module>
main(options, session)
File "/usr/sbin/kojid", line 97, in main
tm = TaskManager(options, session)
File "/usr/lib/python2.7/site-packages/koji/daemon.py", line 480, in
__init__
self.host_id = self.session.host.getID()
File "/usr/lib/python2.7/site-packages/koji/__init__.py", line 1945, in
__call__
return self.__func(self.__name, args, opts)
File "/usr/lib/python2.7/site-packages/koji/__init__.py", line 2381, in
_callMethod
raise err
koji.AuthError: User 3 is not a host
Should user ever be a host?
I used --debug-xmlrpc but it doesn't give any useful info. Bumping up the
koji-hub debug level is not useful neither (KojiDebug=on).
Any pointers are appreciated.
The OS is centos7, stock koji (i guess 1.11, but user agent claims it is
1.7).
6 years, 7 months
Re: RFC: Content generator for module build service
by Dennis Gregorovic
On 04/24/2017 09:25 AM, Ralph Bean wrote:
> On Mon, Apr 24, 2017 at 01:51:39PM +0200, Stanislav Ochotnicky wrote:
>> This is the first stab and what the mbs CG metadata output might look like:
>> https://paste.fedoraproject.org/paste/5ix-EZVVRkmqpSvCQKOw715M1UNdIGYhyRL...
>
> Just to clarify - in the "build" entry here, this is a build of the
> "nginx module". The "version" (which is 1) and the "release" (which is
> 2) are in koji terms, not MBS terms.
>
> The mapping between MBS and koji terms should look something like:
>
> MBS -> Koji
> ---------------
> name -> name
> stream -> version
> version -> release
>
> The only kicker I can see at the moment is the datatype of stream(MBS)
> versus version(Koji). The MBS stream is a string (master, or f26, or
> whatever). The Koji version... does it have to be an integer?
>
Is there potential for nvr conflict? For example if you have a nginx-1-2 RPM and a nginx-1-2 module
6 years, 7 months
Koji 1.12 Release
by Mike McLean
The 1.12 release of Koji landed last week. This release is tagged as
koji-1.12.0 in git, and you can download a tarball here:
https://releases.pagure.org/koji/
The full list of changes is at the end of this message. Highlights include:
• Dist repos (formerly known as signed repos)
• Rewritten clone-tag command (better, stronger, faster)
• Honor excludearch and exclusivearch for noarch builds
• Convert from pygresql to psycopg2
• Option to allow image subtasks to fail
• Plugin for saving failed build trees
As usual, we have included a migration doc:
https://docs.pagure.org/koji/migrating_to_1.12/
In the past, Koji releases have been often been very far apart, but that
is changing. We are currently aiming for a two-month release cycle, so
you should expect the next release sometime in June.
Here is the full changelog:
$ git log --first-parent --format=%s koji-1.11.0..koji-1.12.0
PR#361 1.12.0 release
PR#373 backward-compatible try/except
PR#365 handle buildroots with state=None
PR#367 play nice with older hubs and new volume options
PR#359 Add koji-tools link to docs
PR#318 Signed repos, take two [dist repos]
PR#200 Saving failed build trees
PR#354 more runroot tests
PR#232 Allow uploading files to non-default volumes
PR#350 cli: clarify some "mismatch" warnings
PR#351 cli: check # of args in handle_set_build_volume()
PR#358 jenkins configuration update
PR#260 Add debug and debug_xmlrpc to default koji config
PR#304 unify KeyboardInterrupt behaviour for watch commands
PR#341 Some more 2to3 python2.4 safe results
PR#345 support removing extra values from tags
PR#295 Set compatrequests defaults same as requests
PR#348 remove unused function parse_timestamp
PR#347 Return datetime objects in iso string format
PR#343 Handle empty file upload
PR#337 cli: move list-permissions to info category
PR#332 remove has_key (not working in python3)
PR#336 use alabaster theme for docs
PR#333 Fix README link to mash project
PR#331 use new exception syntax
PR#330 formatting typo
PR#226 print statement -> print function
PR#319 Added support for CG provided owner
PR#324 jenkins' docs
PR#326 use multicall for clone tag
PR#283 wrap sending email in try except
PR#323 Honor excludearch and exclusivearch for noarch builds
Merge #322 `fix encoding when parsing json data on the hub`
PR#278 mock_output.log not included with logs when importing rpm builds
Merge #321 `hub: enforce strict in get_user()`
PR#309 Make --can-fail option working for make-image
Merge #243 `add TrustForwardedIP and CheckClientIP for hubs behind proxies`
PR#307 Fix options.force in import_comps
PR#308 fix a syntax error introduced by commit 6f4c576
PR#303 check http request status before attempting to decode response
PR#317 docs update - krbV configuration
PR#310 Fix koji-devel mailing list address
PR#311 Add indirectionimage to pull-down menu in webui
PR#313 docs typo
PR#316 update test requirements docs
Merge #281 `web.conf options for specifying which methods will appear in
filter`
Merge #291 `Missing --can-fail option for spin-appliance`
Merge PR#209 add disttag handling to get_next_release
Merge PR#262 koji-shadow: allow use without certs
Merge #297 `Fixed minor typo in writing koji code doc`
Merge PR#289 Don't fail on unimported krbV
Merge PR#287 Update content generator metadata documentation
Merge PR#223 convert the packages page to use paginateMethod()
PR#240 Convert from pygresql to psycopg2
Merge PR#239 Allow principal and keytab in cli config
Merge PR#263 Error message for missing certificates
Merge PR#274 Fix kojiweb error using getfile to download non-text files
Merge PR#177 allow tasks to fail on some arches for images/lives/appliances
Merge PR#264 unify CLI parsing of multiple architectures
Merge #265 `fix poll_interval ref in list-history cmd`
Merge #272 `fix default values for buildroot.container_type`
Merge PR#242 Make tests compatible with rhel7/centos7
Merge #267 `more direct tag functions for the hub`
Merge #256 `update url and source in spec`
Merge #257 `Clarify purpose of cfgmap`
Merge #245 `Rewrite koji.util.rmtree to avoid forming long paths`
Merge PR#244 Add krb_rdns to koji-shadow
Merge PR#246 Revert "default krb_rdns to True"
Merge PR#248 Make koji-gc also work with principal and keytab
Merge PR#253 Updated links in docs/code
Merge #254 `Extended clone-tag`
PR#83 add support for putting scripts just before the closing </body> tag
Merge PR#141 Don't hide results in kojiweb
Merge #225 `Also set WSGIApplicationGroup to %{GLOBAL} for the web`
Merge #238 `make the tlstimeout class compatible with newer versions of
qpid`
fix version in docs
6 years, 7 months
koji ssl support
by skbrash＠gmail.com
As of December 16'th, 2016 fedora has started to support only Kerberos credentials, they have even removed the SSL setup from /etc/koji.conf in koji version 1.10. Does koji plan on continuing support for both security systems?
We are a small company with 12 developers creating packages for Fedora, RedHat, and Centos and currently use koji with external repo's to package our product. Switching to a Kerberos security system would be difficult and time consuming. Hopefully, koji will continue to support SSL.
6 years, 7 months
Re: RFC: Content generator for module build service
by Mike Bonnet
Re-added koji-devel.
On 04/24/2017 06:25 AM, Ralph Bean wrote:
> On Mon, Apr 24, 2017 at 01:51:39PM +0200, Stanislav Ochotnicky wrote:
>> This is the first stab and what the mbs CG metadata output might look like:
>> https://paste.fedoraproject.org/paste/5ix-EZVVRkmqpSvCQKOw715M1UNdIGYhyRL...
>
> Just to clarify - in the "build" entry here, this is a build of the
> "nginx module". The "version" (which is 1) and the "release" (which is
> 2) are in koji terms, not MBS terms.
>
> The mapping between MBS and koji terms should look something like:
>
> MBS -> Koji
> ---------------
> name -> name
> stream -> version
> version -> release
>
> The only kicker I can see at the moment is the datatype of stream(MBS)
> versus version(Koji). The MBS stream is a string (master, or f26, or
> whatever). The Koji version... does it have to be an integer?
No, the Koji version and release are also strings. However, because of
the way NVRs are parsed, neither version nor release can contain a
hyphen (-). Not sure if MBS has the same constraint, but if not, we may
need some mapping between the two. Maybe a new build type (and
associated metadata).
6 years, 7 months
