Jenkins in new cloud - status update

[Mikolaj Izdebski]

Overall status: most of things are configured and working.  New Jenkins
should be ready for data migration from old instance.  I've migrated two
projects for now and they seem to be working.

Deployment details: There are: 1 master and 3 slaves (workers) running
in the new cloud.  All instances are of type "m1.small" (1 CPU, 2 GB
RAM, 20 GB disk).

master:
  jenkins.fedorainfracloud.org (Fedora 22)
slaves:
  jenkins-slave-el6.fedorainfracloud.org (Centos 6.6)
  jenkins-slave-el7.fedorainfracloud.org (RHEL 7.1)
  jenkins-slave-f22.fedorainfracloud.org (Fedora 22)

Some notes or questions, in random order:

Slaves have public IP assigned.  It's not needed by Jenkins itself,
but I think it is required to be able to manage them through ansible.

Master is running official jenkins package from Fedora 22, with
several core plugins also from Fedora.  Plugins which were not
available in Fedora are packaged in Copr repo and installed as RPMs
too.  Packages in Copr are just wrappers around upstream binaries -
they are not built from source in Copr.

Jenknis runs as unprivileged user, so it can't listen on port 80 - it
listens on port 8080.  Old Jenkins was running httpd proxy, which
forwarded incoming requests from port 80 to 8080.  New Jenkins
forwards ports using netfilter (aka iptables).  Was there any other
reason for running httpd proxy?

Only members of sysadmin-jenkins group have admin access through web
interface.  Should sysadmin-main be added there too?

Subversion plugin is currently disabled due to unresolved problem.
Projects using subversion SCM (if any) won't work until this is fixed.

Slaves have the same set of extra packages installed as in the old
cloud.  Like in the old Jenkins instance, jenkins user is member of
mock group, which opens possibility for arbitrary code execution on
any of Jenkins hosts (on slaves as root, on master as jenkins user)
for anyone with write access to any project.  I'm not saying it's
necessairly a problem, but I wanted to point it out clearly.

Current Jenkins master doesn't have persistent storage, but I would
like to add it when moving to production.

Any questions or comments?

-- 
Mikolaj Izdebski
Software Engineer, Red Hat
IRC: mizdebsk

[Joshua hoblitt]

On 07/30/2015 10:39 AM, Mikolaj Izdebski wrote:
>  Jenknis runs as unprivileged user, so it can't listen on port 80
> - it
>  listens on port 8080.  Old Jenkins was running httpd proxy, which
>  forwarded incoming requests from port 80 to 8080.  New Jenkins
>  forwards ports using netfilter (aka iptables).  Was there any other
>  reason for running httpd proxy? 
At $day_job, I've been using nginx as a reverse proxy in front of the 
jenkins master to handle TLS termination and inject HTTP headers (HSTS, 
key pinning).  In theory, it also allows us to show a help page instead 
of timeout / negative http status codes from jenkins during a restart or 
other problem.

-Josh

--

[Kevin Fenzi]

On Thu, 30 Jul 2015 19:39:12 +0200
Mikolaj Izdebski <mizdebsk(a)redhat.com&gt; wrote:

>  Overall status: most of things are configured and working.  New
>  Jenkins should be ready for data migration from old instance.  I've
>  migrated two projects for now and they seem to be working. 
Great news. Thanks so much for working on this. ;) 

>  Deployment details: There are: 1 master and 3 slaves (workers)
> running
>  in the new cloud.  All instances are of type "m1.small" (1 CPU, 2 GB
>  RAM, 20 GB disk). 
In the old cloud some of the builders were bigger, should we bump
these up some when we move to production? 
 
>  master:
>    jenkins.fedorainfracloud.org (Fedora 22)
>  slaves:
>    jenkins-slave-el6.fedorainfracloud.org (Centos 6.6)
>    jenkins-slave-el7.fedorainfracloud.org (RHEL 7.1)
>    jenkins-slave-f22.fedorainfracloud.org (Fedora 22)
>  
>  Some notes or questions, in random order:
>  
>  Slaves have public IP assigned.  It's not needed by Jenkins itself,
>  but I think it is required to be able to manage them through ansible. 
Yeah, easiest to just leave this as is. We could proxy to them via the
master, but it would be a hassle to get all working right. 

>  Master is running official jenkins package from Fedora 22, with
>  several core plugins also from Fedora.  Plugins which were not
>  available in Fedora are packaged in Copr repo and installed as RPMs
>  too.  Packages in Copr are just wrappers around upstream binaries -
>  they are not built from source in Copr.
>  
>  Jenknis runs as unprivileged user, so it can't listen on port 80 - it
>  listens on port 8080.  Old Jenkins was running httpd proxy, which
>  forwarded incoming requests from port 80 to 8080.  New Jenkins
>  forwards ports using netfilter (aka iptables).  Was there any other
>  reason for running httpd proxy?
>  
>  Only members of sysadmin-jenkins group have admin access through web
>  interface.  Should sysadmin-main be added there too? 
We could, but I think all the sysadmin-main folks that are interested
are already in sysadmin-jenkins and/or we could add them easily. 

>  Subversion plugin is currently disabled due to unresolved problem.
>  Projects using subversion SCM (if any) won't work until this is fixed. 
Do we have any SVN projects in the old cloud?
 
>  Slaves have the same set of extra packages installed as in the old
>  cloud.  Like in the old Jenkins instance, jenkins user is member of
>  mock group, which opens possibility for arbitrary code execution on
>  any of Jenkins hosts (on slaves as root, on master as jenkins user)
>  for anyone with write access to any project.  I'm not saying it's
>  necessairly a problem, but I wanted to point it out clearly. 
Yep. :( 

>  Current Jenkins master doesn't have persistent storage, but I
> would
>  like to add it when moving to production. 
Easy to do anytime. 

Thanks again for working on it. 

kevin

[Mikolaj Izdebski]

On 07/30/2015 07:46 PM, Joshua hoblitt wrote:
>  On 07/30/2015 10:39 AM, Mikolaj Izdebski wrote:
> > Jenknis runs as unprivileged user, so it can't listen on port 80 - it
> > listens on port 8080.  Old Jenkins was running httpd proxy, which
> > forwarded incoming requests from port 80 to 8080.  New Jenkins
> > forwards ports using netfilter (aka iptables).  Was there any other
> > reason for running httpd proxy?
>  
>  At $day_job, I've been using nginx as a reverse proxy in front of the
>  jenkins master to handle TLS termination and inject HTTP headers (HSTS,
>  key pinning).  In theory, it also allows us to show a help page instead
>  of timeout / negative http status codes from jenkins during a restart or
>  other problem. 
That's a good point. It may be worth to have similar setup for Fedora
Jenkins.

-- 
Mikolaj Izdebski
Software Engineer, Red Hat
IRC: mizdebsk

[Mikolaj Izdebski]

On 07/30/2015 09:11 PM, Kevin Fenzi wrote:
> > Deployment details: There are: 1 master and 3 slaves (workers)
> running
> > in the new cloud.  All instances are of type "m1.small" (1 CPU, 2 GB
> > RAM, 20 GB disk).
>  
>  In the old cloud some of the builders were bigger, should we bump
>  these up some when we move to production?  
I will check instances on the old cloud, now I should be able to access
them. I wanted to avoid unnecessary resource consumption, given that
this is only dev instance.

> > Only members of sysadmin-jenkins group have admin access through
> web
> > interface.  Should sysadmin-main be added there too?
>  
>  We could, but I think all the sysadmin-main folks that are interested
>  are already in sysadmin-jenkins and/or we could add them easily.  
Ok, I'll leave it as-is (sysadmin-jenkins only).

> > Subversion plugin is currently disabled due to unresolved
> problem.
> > Projects using subversion SCM (if any) won't work until this is fixed.
>  
>  Do we have any SVN projects in the old cloud? 
I don't know, but there is subversion plugin installed.

I'm sure there is room for improvement - removing unneeded plugins and
slave packages, but I didn't manage to do that yet, give the short deadline.

-- 
Mikolaj Izdebski
Software Engineer, Red Hat
IRC: mizdebsk

[Pierre-Yves Chibon]

On Fri, Jul 31, 2015 at 12:24:50AM +0200, Mikolaj Izdebski wrote:
>  On 07/30/2015 09:11 PM, Kevin Fenzi wrote:
>  >> Deployment details: There are: 1 master and 3 slaves (workers) running
>  >> in the new cloud.  All instances are of type "m1.small" (1 CPU, 2 GB
>  >> RAM, 20 GB disk).
>  > 
>  > In the old cloud some of the builders were bigger, should we bump
>  > these up some when we move to production? 
>  
>  I will check instances on the old cloud, now I should be able to access
>  them. I wanted to avoid unnecessary resource consumption, given that
>  this is only dev instance. 
+1 for a larger disk size. We got bitten by this in the past that some projects
are quite demanding space wise and keeping the results of the builds also
increases the disk usage.

>  >> Only members of sysadmin-jenkins group have admin access
> through web
>  >> interface.  Should sysadmin-main be added there too?
>  > 
>  > We could, but I think all the sysadmin-main folks that are interested
>  > are already in sysadmin-jenkins and/or we could add them easily. 
>  
>  Ok, I'll leave it as-is (sysadmin-jenkins only).
>  
>  >> Subversion plugin is currently disabled due to unresolved problem.
>  >> Projects using subversion SCM (if any) won't work until this is fixed.
>  > 
>  > Do we have any SVN projects in the old cloud?
>  
>  I don't know, but there is subversion plugin installed. 
As far as I remember there are none, I added the plugin to be complete but I
don't know of any project using it.


Thanks for working on this!

Pierre