Jenkins migration to the new cloud

[Mikolaj Izdebski]

As briefly discussed today during the meeting, I would like to evaluate
the possibility to use our packaged Jenkins in Fedora infrastructure,
instead of upstream binaries. (Jenkins is available[1] in Fedora 21 and
later.)

To get started with development of new Jenkins machines I would need
someone to create 2 new cloud machines: master with Fedora 22 and with
any OS (RHEL 6 would be my choice), each machine with at least 1 CPU, 2
GB RAM, public IP and root access for me (FAS: mizdebsk).

> From that I will try to come with my proof-of-concept of "the new
Jenkins". If people like it then old data can be migrated and it can
become the new production instance. If not we can just scratch these
machines and keep using upstream binaries.

So if you don't mind I'd start working on this.
Should I open a ticket for creating the new cloud instances?

Some more technical notes:

1) It is possible to use third-party plugins with packaged Jenkins,
which means that missing plugins can be installed as binary blobs until
they are packaged in Fedora.

2) Jenkins RPMs must be installed only on master node. All slave nodes
can connect to master and download Jenkins code from there. Slaves still
need to have basic environment installed (such as Java, git, mock), but
not Jenkins itself. This means that only the master node must be Fedora
21+, slaves can be anything (RHEL 6/7, older Fedoras).

3) Michal Srb is currently looking into packaging Jenkins as software
collection for RHEL 6 and 7. Once (and if) done this could allow having
RHEL 7 master, with the disadvantage of using "unofficial" RPMs from
softwarecollections.org. This is just the beginning and we can evaluate
having non-Fedora master later, if needed.

[1] https://fedoraproject.org/wiki/Changes/Jenkins

-- 
Mikolaj Izdebski
Software Engineer, Red Hat
IRC: mizdebsk

[Mikolaj Izdebski]

On 06/25/2015 09:45 PM, Mikolaj Izdebski wrote:
>  To get started with development of new Jenkins machines I would need
>  someone to create 2 new cloud machines: master with Fedora 22 and with
>  any OS (RHEL 6 would be my choice) 
It should read "[...] and *slave* with any OS [...]"

-- 
Mikolaj Izdebski
Software Engineer, Red Hat
IRC: mizdebsk

[Kevin Fenzi]

On Thu, 25 Jun 2015 21:45:10 +0200
Mikolaj Izdebski <mizdebsk(a)redhat.com&gt; wrote:

>  As briefly discussed today during the meeting, I would like to
>  evaluate the possibility to use our packaged Jenkins in Fedora
>  infrastructure, instead of upstream binaries. (Jenkins is
>  available[1] in Fedora 21 and later.)
>  
>  To get started with development of new Jenkins machines I would need
>  someone to create 2 new cloud machines: master with Fedora 22 and with
>  any OS (RHEL 6 would be my choice), each machine with at least 1 CPU,
>  2 GB RAM, public IP and root access for me (FAS: mizdebsk). 
Also, probibly we want a persistent volume for data?

>  From that I will try to come with my proof-of-concept of "the
> new
>  Jenkins". If people like it then old data can be migrated and it can
>  become the new production instance. If not we can just scratch these
>  machines and keep using upstream binaries.
>  
>  So if you don't mind I'd start working on this.
>  Should I open a ticket for creating the new cloud instances? 
Sure. :) 
 
>  Some more technical notes:
>  
>  1) It is possible to use third-party plugins with packaged Jenkins,
>  which means that missing plugins can be installed as binary blobs
>  until they are packaged in Fedora. 
Yeah. 
 
>  2) Jenkins RPMs must be installed only on master node. All slave
> nodes
>  can connect to master and download Jenkins code from there. Slaves
>  still need to have basic environment installed (such as Java, git,
>  mock), but not Jenkins itself. This means that only the master node
>  must be Fedora 21+, slaves can be anything (RHEL 6/7, older Fedoras). 
 
Which is great, as we don't need then to package it for everything
also. I guess it also makes the slaves simpler. 

>  3) Michal Srb is currently looking into packaging Jenkins as
> software
>  collection for RHEL 6 and 7. Once (and if) done this could allow
>  having RHEL 7 master, with the disadvantage of using "unofficial"
>  RPMs from softwarecollections.org. This is just the beginning and we
>  can evaluate having non-Fedora master later, if needed.
>  
>  [1] https://fedoraproject.org/wiki/Changes/Jenkins 
I'd prefer the Fedora master over using SCL's personally. 

Of course it means we will need to upgrade the master every 6 months or
year, but hopefully that won't be too bad if everything is packaged in
fedora. 

kevin

[Mikolaj Izdebski]

On 06/25/2015 09:55 PM, Kevin Fenzi wrote:
>  On Thu, 25 Jun 2015 21:45:10 +0200
>  Mikolaj Izdebski <mizdebsk(a)redhat.com&gt; wrote:
>  
> > As briefly discussed today during the meeting, I would like to
> > evaluate the possibility to use our packaged Jenkins in Fedora
> > infrastructure, instead of upstream binaries. (Jenkins is
> > available[1] in Fedora 21 and later.)
> >
> > To get started with development of new Jenkins machines I would need
> > someone to create 2 new cloud machines: master with Fedora 22 and with
> > any OS (RHEL 6 would be my choice), each machine with at least 1 CPU,
> > 2 GB RAM, public IP and root access for me (FAS: mizdebsk).
>  
>  Also, probibly we want a persistent volume for data? 
Persistent storage would be needed in production (and nice to have
earlier), but at this point I think we can live with default cloud image
without extra storage.

> > Should I open a ticket for creating the new cloud instances?
>  
>  Sure. :)  
Ok, I'll open a ticket.

> > 3) Michal Srb is currently looking into packaging Jenkins as
> software
> > collection for RHEL 6 and 7. Once (and if) done this could allow
> > having RHEL 7 master, with the disadvantage of using "unofficial"
> > RPMs from softwarecollections.org. This is just the beginning and we
> > can evaluate having non-Fedora master later, if needed.
> >
> > [1] https://fedoraproject.org/wiki/Changes/Jenkins
>  
>  I'd prefer the Fedora master over using SCL's personally. 
>  
>  Of course it means we will need to upgrade the master every 6 months or
>  year, but hopefully that won't be too bad if everything is packaged in
>  fedora.  
I rather agree, but this can be decided later.

-- 
Mikolaj Izdebski
Software Engineer, Red Hat
IRC: mizdebsk

[Pierre-Yves Chibon]

On Thu, Jun 25, 2015 at 09:45:10PM +0200, Mikolaj Izdebski wrote:
>  As briefly discussed today during the meeting, I would like to
> evaluate
>  the possibility to use our packaged Jenkins in Fedora infrastructure,
>  instead of upstream binaries. (Jenkins is available[1] in Fedora 21 and
>  later.)
>  
>  To get started with development of new Jenkins machines I would need
>  someone to create 2 new cloud machines: master with Fedora 22 and with
>  any OS (RHEL 6 would be my choice), each machine with at least 1 CPU, 2
>  GB RAM, public IP and root access for me (FAS: mizdebsk).
>  
>  From that I will try to come with my proof-of-concept of "the new
>  Jenkins". If people like it then old data can be migrated and it can
>  become the new production instance. If not we can just scratch these
>  machines and keep using upstream binaries.
>  
>  So if you don't mind I'd start working on this.
>  Should I open a ticket for creating the new cloud instances?
>  
>  Some more technical notes:
>  
>  1) It is possible to use third-party plugins with packaged Jenkins,
>  which means that missing plugins can be installed as binary blobs until
>  they are packaged in Fedora.
>  
>  2) Jenkins RPMs must be installed only on master node. All slave nodes
>  can connect to master and download Jenkins code from there. Slaves still
>  need to have basic environment installed (such as Java, git, mock), but
>  not Jenkins itself. This means that only the master node must be Fedora
>  21+, slaves can be anything (RHEL 6/7, older Fedoras).
>  
>  3) Michal Srb is currently looking into packaging Jenkins as software
>  collection for RHEL 6 and 7. Once (and if) done this could allow having
>  RHEL 7 master, with the disadvantage of using "unofficial" RPMs from
>  softwarecollections.org. This is just the beginning and we can evaluate
>  having non-Fedora master later, if needed.
>  
>  [1] https://fedoraproject.org/wiki/Changes/Jenkins 
This sounds all quite nice, thanks for picking it up!

Just a side question, if we were to use an EL for the master node, is there
an advantage of using SCLs over using upstream's repo as we do now?

Otherwise, +1 on the Fedora master and let me know if I can help.
We would probably have to re-adjust the jenkins playbook, likely in something
like: roles/jenkins/{master,slaves} instead of the big blob we currently have.


Pierre

[Michal Srb]

On 06/26/2015 10:18 AM, Pierre-Yves Chibon wrote:
>  On Thu, Jun 25, 2015 at 09:45:10PM +0200, Mikolaj Izdebski wrote:
> > As briefly discussed today during the meeting, I would like to evaluate
> > the possibility to use our packaged Jenkins in Fedora infrastructure,
> > instead of upstream binaries. (Jenkins is available[1] in Fedora 21 and
> > later.)
> >
> > To get started with development of new Jenkins machines I would need
> > someone to create 2 new cloud machines: master with Fedora 22 and with
> > any OS (RHEL 6 would be my choice), each machine with at least 1 CPU, 2
> > GB RAM, public IP and root access for me (FAS: mizdebsk).
> >
> >  From that I will try to come with my proof-of-concept of "the new
> > Jenkins". If people like it then old data can be migrated and it can
> > become the new production instance. If not we can just scratch these
> > machines and keep using upstream binaries.
> >
> > So if you don't mind I'd start working on this.
> > Should I open a ticket for creating the new cloud instances?
> >
> > Some more technical notes:
> >
> > 1) It is possible to use third-party plugins with packaged Jenkins,
> > which means that missing plugins can be installed as binary blobs until
> > they are packaged in Fedora.
> >
> > 2) Jenkins RPMs must be installed only on master node. All slave nodes
> > can connect to master and download Jenkins code from there. Slaves still
> > need to have basic environment installed (such as Java, git, mock), but
> > not Jenkins itself. This means that only the master node must be Fedora
> > 21+, slaves can be anything (RHEL 6/7, older Fedoras).
> >
> > 3) Michal Srb is currently looking into packaging Jenkins as software
> > collection for RHEL 6 and 7. Once (and if) done this could allow having
> > RHEL 7 master, with the disadvantage of using "unofficial" RPMs from
> > softwarecollections.org. This is just the beginning and we can evaluate
> > having non-Fedora master later, if needed.
> >
> > [1] https://fedoraproject.org/wiki/Changes/Jenkins
>  This sounds all quite nice, thanks for picking it up!
>
>  Just a side question, if we were to use an EL for the master node, is there
>  an advantage of using SCLs over using upstream's repo as we do now? 
All the dependencies in SCL will be built from sources. I basically plan 
to take spec files from Fedora, SCL-ize them (hopefully automatically), 
build them in Copr and then make the SCL available via [1]. Upstream 
releases new RPM literally every week. It basically contains what's 
currently in master branch in git. Stability is usually not bad, but it 
may vary. Then there are LTS releases. These receive updates for ~3 
months. Jenkins in SCL would receive security updates even after this 
period.

Michal

[1]: https://www.softwarecollections.org/en/

>  Otherwise, +1 on the Fedora master and let me know if I can help.
>  We would probably have to re-adjust the jenkins playbook, likely in something
>  like: roles/jenkins/{master,slaves} instead of the big blob we currently have.
>
>
>  Pierre
>  _______________________________________________
>  infrastructure mailing list
>  infrastructure(a)lists.fedoraproject.org
>  https://admin.fedoraproject.org/mailman/listinfo/infrastructure

[Mikolaj Izdebski]

Status update:

Initial Jenkins master and slave instances are deployed in the new
cloud, thanks to Kevin. Currently jenkins service is stopped to prevent
abuse as security is not yet configured due to missing OpenID plugin.

I've packaged jenkins-openid-plugin for Fedora. You can help by
reviewing the package and its dependency:

https://bugzilla.redhat.com/show_bug.cgi?id=1236224
https://bugzilla.redhat.com/show_bug.cgi?id=1236225

-- 
Mikolaj Izdebski
Software Engineer, Red Hat
IRC: mizdebsk