8:27 a.m.
sochotni reported a new issue against the project: `releng` that you are following:
``
We will need changes/additions to koji configuration to enable imports of Modules through
content generator.
Expect future updates once since this is a bit off the top of my head.
1. Add content generator "module-build-service"
2. Add btype "module"
3. Configure whatever user is used by modulebuild service to have permission for the
content generator APIs
``
To reply, visit the link below or just reply to this email
https://pagure.io/releng/issue/6799
9:33 a.m.
ralph added a new comment to an issue you are following:
``
Configure whatever user is used by modulebuild service to have
permission for the content generator APIs
The user is `mbs/mbs.fedoraproejct.org`.
For more info on what this is for, see the koji-devel thread(s):
https://lists.fedoraproject.org/archives/search?mlist=koji-devel%40lists....
``
To reply, visit the link below or just reply to this email
https://pagure.io/releng/issue/6799
10:54 a.m.
ralph added a new comment to an issue you are following:
``
FYI, this came up in the bodhi stakeholders meeting today:
https://meetbot.fedoraproject.org/fedora-meeting-1/2017-05-23/bodhi_stake...
``
To reply, visit the link below or just reply to this email
https://pagure.io/releng/issue/6799
6:33 a.m.
sochotni added a new comment to an issue you are following:
``
So a summary of assumed required calls:
koji grant-cg-access mbs/mbs.fedoraproejct.org module-build-service --new
koji call addBType module
That should be all that's needed from MBS side to work as content generator
``
To reply, visit the link below or just reply to this email
https://pagure.io/releng/issue/6799
8:24 a.m.
ralph added a new comment to an issue you are following:
``
Pull request submitted to Bodhi to look for markers provided by this content generator:
https://github.com/fedora-infra/bodhi/pull/1543
``
To reply, visit the link below or just reply to this email
https://pagure.io/releng/issue/6799
9:39 a.m.
ralph added a new comment to an issue you are following:
``
Blocking https://pagure.io/fm-orchestrator/pull-request/569
``
To reply, visit the link below or just reply to this email
https://pagure.io/releng/issue/6799
11:37 a.m.
ausil added a new comment to an issue you are following:
``
@rlaliber FYI this is a module config issue we have open.
``
To reply, visit the link below or just reply to this email
https://pagure.io/releng/issue/6799
12:42 p.m.
ralph added a new comment to an issue you are following:
``
Adding the meeting label to see if there's anything that needs to be discussed here.
``
To reply, visit the link below or just reply to this email
https://pagure.io/releng/issue/6799
10:52 a.m.
ralph added a new comment to an issue you are following:
``
FYI, this was discussed today:
https://meetbot.fedoraproject.org/fedora-meeting-2/2017-06-12/releng.2017...
My read of the conversation is that this request is "approved", and that this
ticket has to date been stuck on a lack of explicit policy for new content generators.
releng took an action item to work on documenting that for the future.
``
To reply, visit the link below or just reply to this email
https://pagure.io/releng/issue/6799
7:30 a.m.
ralph added a new comment to an issue you are following:
``
Linking some more information about what we want to do with this:
https://fedoraproject.org/wiki/Infrastructure/Factory2/Focus/Bodhi
``
To reply, visit the link below or just reply to this email
https://pagure.io/releng/issue/6799
7:42 a.m.
sochotni added a new comment to an issue you are following:
``
Also - offered to write to SOP to @katec, added @ausil to that mail waiting to hear back
whether that would help
``
To reply, visit the link below or just reply to this email
https://pagure.io/releng/issue/6799
12:31 p.m.
sochotni added a new comment to an issue you are following:
``
Added the SOP PR: https://pagure.io/releng/pull-request/6849
I am just referencing the koji documentation for content generator requirements since they
seem aligned with what I'd expect rel-eng to require.
Alternatively koji documentation could be copied and tweaked but unless specific
reservations are made it would be a waste of time.
``
To reply, visit the link below or just reply to this email
https://pagure.io/releng/issue/6799
10:03 a.m.
mohanboddu added a new comment to an issue you are following:
``
As per the policy we have set
https://docs.pagure.org/releng/sop_adding_content_generator.html
Can you answer the following questions:
https://docs.pagure.org/releng/sop_adding_content_generator.html#question...
And one other question I would like to know, where can we access the logs?
``
To reply, visit the link below or just reply to this email
https://pagure.io/releng/issue/6799
10:45 a.m.
sochotni added a new comment to an issue you are following:
``
I set myself up didn't I? :-)
The content generator is module build service - mbs.fedoraproject.org
I am not 100% sure what the support agreement is around it but given it's running in
the infra and there's playbooks I'd say it's in a decent state:
https://infrastructure.fedoraproject.org/cgit/ansible.git/tree/roles/mbs
Dev team is decently spread between continents at the moment and technically if mbs is
down it's only affecting module builds, not rest of koji workflows.
The module build service itself doesn't actually do any builds - it just orchestrates
koji so all logs are available in there. There's a possibility that we'd improve
logging of the orchestration itself and make it part of the module import itself in the
future.
New content type: We are adding "module" content type which is only a modulemd
[1] yaml file. File size should be in the order of KBs. The binary content (module rpms)
are already in koji and are just referenced in the build metadata.
As for the generic requirements from koji docs:
* We record rpms installed on the module build system at the time of the module
orchestration
* We record version of the MBS itself
* Since we call koji to do actual binary builds we don't use host software
* For the same reason as above - we won't allow network during builds or use of pip
or other content directly
Mocked content generator that would be imported into Koji looks like this:
https://fedorapeople.org/~sochotni/mbs-content-generator-v2.json
In addition the mentioned modulemd file would be included as output and all rpms from the
module referenced as belonging to this modulemd file.
[1] https://pagure.io/modulemd
``
To reply, visit the link below or just reply to this email
https://pagure.io/releng/issue/6799
10:46 a.m.
sochotni added a new comment to an issue you are following:
``
I should have mentioned - basic information about state of orchestration is visible in mbs
itself. For example:
https://mbs.fedoraproject.org/module-build-service/1/module-builds/640
``
To reply, visit the link below or just reply to this email
https://pagure.io/releng/issue/6799
7:44 a.m.
sochotni added a new comment to an issue you are following:
``
I forgot that in addition to the buildroot info and host rpms etc on MBs, we also provide
dist-git source URL/hash. This leads to modulemd yaml file which was used by MBS to
orchestrate the builds - same way you have source url for rpm builds
Linking of the module rpms into this module build itself provides way to collect logs and
track which specific branches/hashes module builds were done.
``
To reply, visit the link below or just reply to this email
https://pagure.io/releng/issue/6799
1:26 p.m.
ralph added a new comment to an issue you are following:
``
A picture of MBS in action! http://threebean.org/img/mbs-with-mohan.png
``
To reply, visit the link below or just reply to this email
https://pagure.io/releng/issue/6799
1:28 p.m.
mohanboddu added a new comment to an issue you are following:
``
Some final quick questions : where does the json file thats imported into koji comes
from?
How is it generated?
And where are the logs on its generation?
``
To reply, visit the link below or just reply to this email
https://pagure.io/releng/issue/6799
1:32 p.m.
ralph added a new comment to an issue you are following:
``
where does the json file thats imported into koji comes from?
It gets generated on the fly by the MBS in this file
https://pagure.io/fm-orchestrator/blob/master/f/module_build_service/buil...
How is it generated?
From the modulemd that the user used to drive the module build as a whole (plus
information from koji during the module build process).
And where are the logs on its generation?
The logs can be found in the journal on mbs-backend01.phx2.fedoraproject.org. See the
'fedmsg-hub' unit. (that's where the mbs backend runs).
Importantly, see also
https://fedora-infra-docs.readthedocs.io/en/latest/sysadmin-guide/sops/mb...
``
To reply, visit the link below or just reply to this email
https://pagure.io/releng/issue/6799
7:20 a.m.
ralph added a new comment to an issue you are following:
``
Any other questions we can answer @mohanboddu?
``
To reply, visit the link below or just reply to this email
https://pagure.io/releng/issue/6799
9:55 a.m.
The issue: `[koji] Add configuration for module build service content generator` of
project: `releng` has been assigned to `mohanboddu` by ausil.
https://pagure.io/releng/issue/6799
9:55 a.m.
ausil added a new comment to an issue you are following:
``
```
14:47 < dgilmore> I think the only change I would like to propose here is that mbs
also uploads teh logs on teh json generation
14:47 < dgilmore> not sure how invasive that would be
14:47 < dgilmore> threebean: any idea?
14:48 * threebean reads up
14:50 < threebean> eh, should be possible.
14:50 < threebean> but will take a code change.
14:51 < dgilmore> threebean: cool
14:51 < threebean> will file an issue to track it.
14:51 < threebean> can we add it in later?
14:51 < dgilmore> I think it wou,ld be usefult to have as part of the auditability
14:51 < dgilmore> I think so
14:51 < threebean> (for now at least, the logs are accessible on mbs-backend01 in
the journal)
14:51 < dgilmore> threebean: thats not public and open and transparent :D
14:52 < threebean> yup, yup.
14:52 < threebean> but it is auditable!
14:53 < dgilmore> #agreed that under the condition we add the logs creating the json
to the upload we can add the content generator
```
``
To reply, visit the link below or just reply to this email
https://pagure.io/releng/issue/6799
6:14 a.m.
sochotni added a new comment to an issue you are following:
``
MBS pull request with the addition of build logs to the content generator outputs
section:
https://pagure.io/fm-orchestrator/pull-request/601
``
To reply, visit the link below or just reply to this email
https://pagure.io/releng/issue/6799
2:03 a.m.
sochotni added a new comment to an issue you are following:
``
FYI the PR is now merged and @jkaluza is working on releasing new MBS version that will be
importing the logs during CG imports
``
To reply, visit the link below or just reply to this email
https://pagure.io/releng/issue/6799
5:54 a.m.
sochotni added a new comment to an issue you are following:
``
FYI the MBS is now deployed - cg imports are disabled until Koji configuration is changed.
So ... shall we?
``
To reply, visit the link below or just reply to this email
https://pagure.io/releng/issue/6799
3:45 a.m.
sochotni added a new comment to an issue you are following:
``
Can we get an update/plan for this? Boltron is out so I assume folks should have some time
now?
``
To reply, visit the link below or just reply to this email
https://pagure.io/releng/issue/6799
9:46 a.m.
mohanboddu added a new comment to an issue you are following:
``
@sochotni Just for an example, can you link one of the logs here?
``
To reply, visit the link below or just reply to this email
https://pagure.io/releng/issue/6799
10:13 a.m.
sochotni added a new comment to an issue you are following:
``
By default the build logs are stored in /tmp and it doesn't seem to be overridden in
the infra repo. I don't have permissions to ssh into mbs.fedoraproject.org but if you
do you should find logs in /tmp/build-X.log
``
To reply, visit the link below or just reply to this email
https://pagure.io/releng/issue/6799
1:50 p.m.
sochotni added a new comment to an issue you are following:
``
Correction, I looked at the wrong default config variable - by default we don't
generate per-build logs unless it's enabled by changing BUILD_LOGS_DIR from empty
string. Which would happen once we enabled CG code but it's not there yet.
``
To reply, visit the link below or just reply to this email
https://pagure.io/releng/issue/6799
12:53 p.m.
mohanboddu added a new comment to an issue you are following:
``
$ koji grant-cg-access mbs/mbs.fedoraproject.org module-build-service --new
$ koji call addBType module
None
$
The content generator has been added. We still need the logs which requires a change in
koji config. Please let us know when its done or what needs to be done to get the logs and
then we can close the ticket.
Thanks.
``
To reply, visit the link below or just reply to this email
https://pagure.io/releng/issue/6799
2452
days inactive
2529
days old
rel-eng@lists.fedoraproject.org
29 comments
4 participants
participants (4)
-
Dennis Gilmore -
Mohan Boddu
-
Ralph Bean
-
Stanislav Ochotnicky