Also in the vein of automation, went ahead and installed an instance of Jenkins and connected it to SSG:
http://jenkins.ssgproject.org:8080/
Not meant to be permanent right now, but wanted to get something stood up for us to play with. Martin stood one up for the OpenSCAP interpreter, which was the source of thinking to stand up a sandbox for SSG too. There really isn't much to see in the UI, but one thing it does is integrate into GitHub. Whenever someone issues a pull request, Jenkins will automagically detect that, apply the patch(s), and run "make validate."
Check out https://github.com/OpenSCAP/scap-security-guide/pull/45 for an example. Or for the lazy :-), see below:
When you click on Details, you're brought to an overview of the patch and can even see console output of the build (in case there were errors): http://jenkins.ssgproject.org:8080/job/scap-security-guide/24/console
Having this in place should afford the peer reviewers an immediate answer to "does this validate?" Longer term, this could also enable unit checks of the OVAL and bash scripts.
Does anyone have experience with Jenkins, thoughts on how we could begin building out unit tests, or really any thoughts on usefulness beyond "make validate" on pull requests?
----- Original Message -----
From: "Shawn Wells" shawn@redhat.com To: "scap-security-guide" scap-security-guide@lists.fedorahosted.org Sent: Wednesday, September 3, 2014 6:31:00 AM Subject: Test run of Jenkins (CI tool)
Also in the vein of automation, went ahead and installed an instance of Jenkins and connected it to SSG:
http://jenkins.ssgproject.org:8080/
Not meant to be permanent right now, but wanted to get something stood up for us to play with. Martin stood one up for the OpenSCAP interpreter, which was the source of thinking to stand up a sandbox for SSG too. There really isn't much to see in the UI, but one thing it does is integrate into GitHub. Whenever someone issues a pull request, Jenkins will automagically detect that, apply the patch(s), and run "make validate."
I wanted to create a testing Jenkins instance for all things OpenSCAP. There are fixed costs in maintaining the master and all the slaves. You have to take care of the machines, keep them updated, etc... You have to keep updating Jenkins itself and be on top of plugin updates.
In my opinion it makes no sense to have 2 Jenkins servers for projects so closely related. I would prefer to pool resources.
Unfortunately we can't just host our Jenkins on OpenShift because we need multiple slaves for RHEL 5, 6, 7 and Fedora. And the storage space available is not enough for our use-case.
Check out https://github.com/OpenSCAP/scap-security-guide/pull/45 for an example. Or for the lazy :-), see below:
The integration is really cool. I planned to make the Jenkins instance public after a testing period and do exactly this.
Does anyone have experience with Jenkins, thoughts on how we could begin building out unit tests, or really any thoughts on usefulness beyond "make validate" on pull requests?
I was experimenting with docker for this purpose. The idea was to start with EL6 image, put ssg and a test script inside. Then you can call docker run and get the results. The docker instance is deleted after each run so there is no risk of side effects.
The idea is to do the following: 1) pull ssg HEAD
2) make content
3) generate a profile with just one Rule selected - the tested rule (a simple python script can easily do this)
4) evaluate that profile, compare the results with expected results on a freshly installed RHEL6
5) anti-remediate - break the configuration (if applicable)
6) evaluate again, check that the result is "fail" (if applicable)
7) remediate - fix the configuration (if applicable)
8) evaluate for the third time, check that the rule passes
*) as an added bonus we could run whatever we are testing, let it parse the config files and do another sanity check (sshd for example)
This should provide repeatable tests for all the rules.
Unfortunately we can't do this on OpenShift (AFAIK). We need real slaves with docker-io installed on them. Another problem is that we can not test every single rule with this approach. "process test" comes to mind. Or even systemd tests. However we should be able to test most of the rules.
It would be a lot of work, but we could write automated unit testing scripts using python unittest, python nose, or something else. Then, run those test with a command like 'run-tests'
On Wed, Sep 3, 2014 at 6:06 AM, Martin Preisler mpreisle@redhat.com wrote:
----- Original Message -----
From: "Shawn Wells" shawn@redhat.com To: "scap-security-guide" scap-security-guide@lists.fedorahosted.org Sent: Wednesday, September 3, 2014 6:31:00 AM Subject: Test run of Jenkins (CI tool)
Also in the vein of automation, went ahead and installed an instance of Jenkins and connected it to SSG:
http://jenkins.ssgproject.org:8080/
Not meant to be permanent right now, but wanted to get something stood up for us to play with. Martin stood one up for the OpenSCAP interpreter, which was the source of thinking to stand up a sandbox for SSG too. There really isn't much to see in the UI, but one thing it does is integrate into GitHub. Whenever someone issues a pull request, Jenkins will automagically detect that, apply the patch(s), and run "make validate."
I wanted to create a testing Jenkins instance for all things OpenSCAP. There are fixed costs in maintaining the master and all the slaves. You have to take care of the machines, keep them updated, etc... You have to keep updating Jenkins itself and be on top of plugin updates.
In my opinion it makes no sense to have 2 Jenkins servers for projects so closely related. I would prefer to pool resources.
Unfortunately we can't just host our Jenkins on OpenShift because we need multiple slaves for RHEL 5, 6, 7 and Fedora. And the storage space available is not enough for our use-case.
Check out https://github.com/OpenSCAP/scap-security-guide/pull/45 for an example. Or for the lazy :-), see below:
The integration is really cool. I planned to make the Jenkins instance public after a testing period and do exactly this.
Does anyone have experience with Jenkins, thoughts on how we could begin building out unit tests, or really any thoughts on usefulness beyond "make validate" on pull requests?
I was experimenting with docker for this purpose. The idea was to start with EL6 image, put ssg and a test script inside. Then you can call docker run and get the results. The docker instance is deleted after each run so there is no risk of side effects.
The idea is to do the following:
pull ssg HEAD
make content
generate a profile with just one Rule selected - the tested rule (a simple python script can easily do this)
evaluate that profile, compare the results with expected results on a freshly installed RHEL6
anti-remediate - break the configuration (if applicable)
evaluate again, check that the result is "fail" (if applicable)
remediate - fix the configuration (if applicable)
evaluate for the third time, check that the rule passes
*) as an added bonus we could run whatever we are testing, let it parse the config files and do another sanity check (sshd for example)
This should provide repeatable tests for all the rules.
Unfortunately we can't do this on OpenShift (AFAIK). We need real slaves with docker-io installed on them. Another problem is that we can not test every single rule with this approach. "process test" comes to mind. Or even systemd tests. However we should be able to test most of the rules.
-- Martin Preisler -- SCAP Security Guide mailing list scap-security-guide@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/scap-security-guide https://github.com/OpenSCAP/scap-security-guide/
----- Original Message -----
From: "Gabe Alford" redhatrises@gmail.com To: "SCAP Security Guide" scap-security-guide@lists.fedorahosted.org Sent: Wednesday, September 3, 2014 3:13:44 PM Subject: Re: Test run of Jenkins (CI tool)
It would be a lot of work, but we could write automated unit testing scripts using python unittest, python nose, or something else. Then, run those test with a command like 'run-tests'
The test harness is a non issue. There are tons of frameworks that would work well.
The issue in my opinion is how we run the tests themselves. That's the part that is hard to solve and we should focus on that first IMO. Unfortunately, it's hard to evaluation or remediate the system itself without potentially breaking it. This is simply unacceptable on the developer's workstation. However we can break docker instances all we want because we just delete them afterwards :-)
In the future we may have a test runner in upstream repo that would pull docker images and run all the test cases. This way contributors can run the tests themselves without having to depend on Jenkins. Because of the size of the images we shall not keep them in the ssg repo but we could keep the surrounding test harness, test cases and scripts in there.
Right. That makes sense. Docker wouldn't be a bad idea. Could you use a single image and have a bunch of dockerfiles or run 'docker run' commands for test cases?
On Wed, Sep 3, 2014 at 7:32 AM, Martin Preisler mpreisle@redhat.com wrote:
----- Original Message -----
From: "Gabe Alford" redhatrises@gmail.com To: "SCAP Security Guide" scap-security-guide@lists.fedorahosted.org Sent: Wednesday, September 3, 2014 3:13:44 PM Subject: Re: Test run of Jenkins (CI tool)
It would be a lot of work, but we could write automated unit testing scripts using python unittest, python nose, or something else. Then, run those test with a command like 'run-tests'
The test harness is a non issue. There are tons of frameworks that would work well.
The issue in my opinion is how we run the tests themselves. That's the part that is hard to solve and we should focus on that first IMO. Unfortunately, it's hard to evaluation or remediate the system itself without potentially breaking it. This is simply unacceptable on the developer's workstation. However we can break docker instances all we want because we just delete them afterwards :-)
In the future we may have a test runner in upstream repo that would pull docker images and run all the test cases. This way contributors can run the tests themselves without having to depend on Jenkins. Because of the size of the images we shall not keep them in the ssg repo but we could keep the surrounding test harness, test cases and scripts in there.
-- Martin Preisler -- SCAP Security Guide mailing list scap-security-guide@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/scap-security-guide https://github.com/OpenSCAP/scap-security-guide/
Seems like this might be a good case for a public OpenShift instance.
Shawn?
Trevor
On Wed, Sep 3, 2014 at 10:39 AM, Gabe Alford redhatrises@gmail.com wrote:
Right. That makes sense. Docker wouldn't be a bad idea. Could you use a single image and have a bunch of dockerfiles or run 'docker run' commands for test cases?
On Wed, Sep 3, 2014 at 7:32 AM, Martin Preisler mpreisle@redhat.com wrote:
----- Original Message -----
From: "Gabe Alford" redhatrises@gmail.com To: "SCAP Security Guide" scap-security-guide@lists.fedorahosted.org Sent: Wednesday, September 3, 2014 3:13:44 PM Subject: Re: Test run of Jenkins (CI tool)
It would be a lot of work, but we could write automated unit testing scripts using python unittest, python nose, or something else. Then, run those test with a command like 'run-tests'
The test harness is a non issue. There are tons of frameworks that would work well.
The issue in my opinion is how we run the tests themselves. That's the part that is hard to solve and we should focus on that first IMO. Unfortunately, it's hard to evaluation or remediate the system itself without potentially breaking it. This is simply unacceptable on the developer's workstation. However we can break docker instances all we want because we just delete them afterwards :-)
In the future we may have a test runner in upstream repo that would pull docker images and run all the test cases. This way contributors can run the tests themselves without having to depend on Jenkins. Because of the size of the images we shall not keep them in the ssg repo but we could keep the surrounding test harness, test cases and scripts in there.
-- Martin Preisler -- SCAP Security Guide mailing list scap-security-guide@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/scap-security-guide https://github.com/OpenSCAP/scap-security-guide/
-- SCAP Security Guide mailing list scap-security-guide@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/scap-security-guide https://github.com/OpenSCAP/scap-security-guide/
why not using travis? https://travis-ci.org/
You would simply need to create you unit tests and define travis test commands (a simple and single .travis.yml file), bind your github repo on your travis account and travis will stage and run you test on a temporary instance and destroy it when tests are over. It also enables you to automatically do the full testing and integretion testing on each push in a branch or pull request.
see this example file: https://github.com/savon-noir/python-libnmap/blob/master/.travis.yml
On Wed, Sep 3, 2014 at 5:19 PM, Trevor Vaughan tvaughan@onyxpoint.com wrote:
Seems like this might be a good case for a public OpenShift instance.
Shawn?
Trevor
On Wed, Sep 3, 2014 at 10:39 AM, Gabe Alford redhatrises@gmail.com wrote:
Right. That makes sense. Docker wouldn't be a bad idea. Could you use a single image and have a bunch of dockerfiles or run 'docker run' commands for test cases?
On Wed, Sep 3, 2014 at 7:32 AM, Martin Preisler mpreisle@redhat.com wrote:
----- Original Message -----
From: "Gabe Alford" redhatrises@gmail.com To: "SCAP Security Guide" scap-security-guide@lists.fedorahosted.org Sent: Wednesday, September 3, 2014 3:13:44 PM Subject: Re: Test run of Jenkins (CI tool)
It would be a lot of work, but we could write automated unit testing scripts using python unittest, python nose, or something else. Then,
run
those test with a command like 'run-tests'
The test harness is a non issue. There are tons of frameworks that would work well.
The issue in my opinion is how we run the tests themselves. That's the part that is hard to solve and we should focus on that first IMO. Unfortunately, it's hard to evaluation or remediate the system itself without potentially breaking it. This is simply unacceptable on the developer's workstation. However we can break docker instances all we want because we just delete them afterwards :-)
In the future we may have a test runner in upstream repo that would pull docker images and run all the test cases. This way contributors can run the tests themselves without having to depend on Jenkins. Because of the size of the images we shall not keep them in the ssg repo but we could keep the surrounding test harness, test cases and scripts in there.
-- Martin Preisler -- SCAP Security Guide mailing list scap-security-guide@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/scap-security-guide https://github.com/OpenSCAP/scap-security-guide/
-- SCAP Security Guide mailing list scap-security-guide@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/scap-security-guide https://github.com/OpenSCAP/scap-security-guide/
-- Trevor Vaughan Vice President, Onyx Point, Inc (410) 541-6699 tvaughan@onyxpoint.com
-- This account not approved for unencrypted proprietary information --
-- SCAP Security Guide mailing list scap-security-guide@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/scap-security-guide https://github.com/OpenSCAP/scap-security-guide/
----- Original Message -----
From: "Ronald" mini.pelle@gmail.com To: "SCAP Security Guide" scap-security-guide@lists.fedorahosted.org Sent: Wednesday, September 3, 2014 6:36:10 PM Subject: Re: Test run of Jenkins (CI tool)
why not using travis? https://travis-ci.org/
Because we need more than Travis can offer. Travis uses Ubuntu LTS AFAIK. You simply cannot get RHEL6 slaves there. Furthermore, I doubt they will let us use docker in there.
You would simply need to create you unit tests and define travis test commands (a simple and single .travis.yml file), bind your github repo on your travis account and travis will stage and run you test on a temporary instance and destroy it when tests are over. It also enables you to automatically do the full testing and integretion testing on each push in a branch or pull request.
Yeah, this works fine for many projects but scap-security-guide is more demanding in this regard. We need to evaluate real systems and we need root access.
On 9/3/14, 12:42 PM, Martin Preisler wrote:
----- Original Message -----
From: "Ronald" mini.pelle@gmail.com To: "SCAP Security Guide" scap-security-guide@lists.fedorahosted.org Sent: Wednesday, September 3, 2014 6:36:10 PM Subject: Re: Test run of Jenkins (CI tool)
why not using travis? https://travis-ci.org/
Because we need more than Travis can offer. Travis uses Ubuntu LTS AFAIK. You simply cannot get RHEL6 slaves there. Furthermore, I doubt they will let us use docker in there.
You would simply need to create you unit tests and define travis test commands (a simple and single .travis.yml file), bind your github repo on your travis account and travis will stage and run you test on a temporary instance and destroy it when tests are over. It also enables you to automatically do the full testing and integretion testing on each push in a branch or pull request.
Yeah, this works fine for many projects but scap-security-guide is more demanding in this regard. We need to evaluate real systems and we need root access.
fwiw, I ran across travis-ci and did start playing with it: https://travis-ci.org/ssg-buildbot/scap-security-guide
.... but then I RTFM'd and saw it only supports ubuntu hosts
........... but maybe someday we'll have Ubuntu LTS content (haaaayyyy daavvvee......where your patches be?)
On 9/3/14, 11:19 AM, Trevor Vaughan wrote:
Seems like this might be a good case for a public OpenShift instance.
Shawn?
yeah, we can likely snag a copy of openshift. sometimes having a @redhat.com affiliation is pretty cool :) openshift.open-scap.org or some such thing.
OpenShift does run on RHEL 6, I believe a formal rebase on RHEL7 is coming, and there's always OpenShift Origin for Fedora. We could create a custom cartridge with the various openscap-utils, python-lxml, and other build packages. This would get us to a "make validate" point, but as Martin pointed out, it wouldn't scale much beyond that -- functional unit tests (e.g. OVAL and remediation) wouldn't be configured easily.
Having a Jenkins script spin up/down the appropriate Docker instances does seem ideal.... but I've no idea how to set that up currently. There is a specific Jenkins Docker plugin: https://wiki.jenkins-ci.org/display/JENKINS/Docker+Plugin
Any Docker-ites interested in playing with this?
----- Original Message -----
From: "Shawn Wells" shawn@redhat.com To: scap-security-guide@lists.fedorahosted.org Sent: Thursday, September 4, 2014 5:51:12 AM Subject: Re: Test run of Jenkins (CI tool)
On 9/3/14, 11:19 AM, Trevor Vaughan wrote:
Seems like this might be a good case for a public OpenShift instance.
Shawn?
yeah, we can likely snag a copy of openshift. sometimes having a @redhat.com affiliation is pretty cool :) openshift.open-scap.org or some such thing.
OpenShift does run on RHEL 6, I believe a formal rebase on RHEL7 is coming, and there's always OpenShift Origin for Fedora. We could create a custom cartridge with the various openscap-utils, python-lxml, and other build packages. This would get us to a "make validate" point, but as Martin pointed out, it wouldn't scale much beyond that -- functional unit tests (e.g. OVAL and remediation) wouldn't be configured easily.
Having a Jenkins script spin up/down the appropriate Docker instances does seem ideal.... but I've no idea how to set that up currently. There is a specific Jenkins Docker plugin: https://wiki.jenkins-ci.org/display/JENKINS/Docker+Plugin
Any Docker-ites interested in playing with this?
Yes, that was actually my plan after performing the 0.19 release (to start building the scripts infrastructure for automated checks testing & set them up with Jenkins Docker plug-in).
Got a repository set up for it (https://git.fedorahosted.org/git/ssgtestsuite.git - currently empty) on Fedora infrastructure yet.
The plan is to create scripts handling management of Docker images (for RHEL-6, RHEL-7 & Fedora content => the host handling these would probably need to be RHEL-7 or Fedora based) to perform automated testing scenario as detailed by Martin in: https://lists.fedorahosted.org/pipermail/scap-security-guide/2014-September/...
Wrt to possibilities for Jenkins instances, the following capability: https://fedoraproject.org/wiki/Jenkins@infra
has been suggested to me by Petr Lautrbach (didn't get it more time just quick read through since Martin already has Jenkins instance able to handle all free types (RHEL6, RHEL7 & Fedora) of slaves (we might be interested in testing scripts at).
Thank you && Regards, Jan. -- Jan iankko Lieskovsky / Red Hat Security Technologies Team
-- SCAP Security Guide mailing list scap-security-guide@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/scap-security-guide https://github.com/OpenSCAP/scap-security-guide/
On 9/3/14, 8:06 AM, Martin Preisler wrote:
----- Original Message -----
From: "Shawn Wells" shawn@redhat.com To: "scap-security-guide" scap-security-guide@lists.fedorahosted.org Sent: Wednesday, September 3, 2014 6:31:00 AM Subject: Test run of Jenkins (CI tool)
Also in the vein of automation, went ahead and installed an instance of Jenkins and connected it to SSG:
http://jenkins.ssgproject.org:8080/
Not meant to be permanent right now, but wanted to get something stood up for us to play with. Martin stood one up for the OpenSCAP interpreter, which was the source of thinking to stand up a sandbox for SSG too. There really isn't much to see in the UI, but one thing it does is integrate into GitHub. Whenever someone issues a pull request, Jenkins will automagically detect that, apply the patch(s), and run "make validate."
I wanted to create a testing Jenkins instance for all things OpenSCAP. There are fixed costs in maintaining the master and all the slaves. You have to take care of the machines, keep them updated, etc... You have to keep updating Jenkins itself and be on top of plugin updates.
In my opinion it makes no sense to have 2 Jenkins servers for projects so closely related. I would prefer to pool resources.
Unfortunately we can't just host our Jenkins on OpenShift because we need multiple slaves for RHEL 5, 6, 7 and Fedora. And the storage space available is not enough for our use-case.
Check out https://github.com/OpenSCAP/scap-security-guide/pull/45 for an example. Or for the lazy :-), see below:
The integration is really cool. I planned to make the Jenkins instance public after a testing period and do exactly this.
Didn't mean to jump the gun to much. I'm just an impatient bastard and wanted to get the conversation started on the list (with full recognition it's just a test run, we'll figure out how to converge everything together, etc).
Does anyone have experience with Jenkins, thoughts on how we could begin building out unit tests, or really any thoughts on usefulness beyond "make validate" on pull requests?
I was experimenting with docker for this purpose. The idea was to start with EL6 image, put ssg and a test script inside. Then you can call docker run and get the results. The docker instance is deleted after each run so there is no risk of side effects.
The idea is to do the following:
pull ssg HEAD
make content
generate a profile with just one Rule selected - the tested rule (a simple python script can easily do this)
evaluate that profile, compare the results with expected results on a freshly installed RHEL6
anti-remediate - break the configuration (if applicable)
evaluate again, check that the result is "fail" (if applicable)
remediate - fix the configuration (if applicable)
evaluate for the third time, check that the rule passes
*) as an added bonus we could run whatever we are testing, let it parse the config files and do another sanity check (sshd for example)
This should provide repeatable tests for all the rules.
Unfortunately we can't do this on OpenShift (AFAIK). We need real slaves with docker-io installed on them. Another problem is that we can not test every single rule with this approach. "process test" comes to mind. Or even systemd tests. However we should be able to test most of the rules.
-- Martin Preisler -- SCAP Security Guide mailing list scap-security-guide@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/scap-security-guide https://github.com/OpenSCAP/scap-security-guide/
scap-security-guide@lists.fedorahosted.org
-
Gabe Alford -
Jan Lieskovsky -
Martin Preisler -
Ronald -
Shawn Wells -
Trevor Vaughan