SecState Evaluation

[Rodrian, Logan P (IS)]

Hello-

Along with the recent release of CLIP, I have begun using SecState to perform auditing and
remediation for my system.  In performing these tasks and viewing the reports, I found
that there were multiple issues with both the checks and the fixes being performed
incorrectly, along with some inconsistencies with the description versus the performed
check/fix.

> From the common profile, which is what I am running, I found the
> following: 
    Check Incorrect (Pre Remediation)     15
    Remediation Fix Broken                1
    Check Incorrect (Post Remediation)    15+19 (34)

I have compiled a spreadsheet documenting my findings.

1) What/who should the findings be submitted to?  What format?
2) When is the planned release or any fixes?


Logan Rodrian

[Francisco Slavin]

From: secstate-bounces(a)lists.fedorahosted.org
[mailto:[e-mail]] On Behalf Of Rodrian, Logan P (IS)
Sent: Thursday, December 06, 2012 12:30 PM
To: secstate(a)lists.fedorahosted.org
Subject: [SecState] SecState Evaluation

>  Hello- 
>  Along with the recent release of CLIP, I have begun using SecState to
>
>  perform auditing and remediation for my system.  In performing these 
>  tasks and viewing the reports, I found that there were multiple issues 
>  with both the checks and the fixes being performed incorrectly, along with some
> inconsistencies with the description versus the performed check/fix. 
Hi Logan.  It's good to see some early use of the latest SecState & CLIP releases.
 We appreciate the feedback and hopefully we can work to resolve the issues you've
encountered.
To give a brief overview of the current content included in CLIP and used by SecState:
	The Audit content (XCCDF and OVAL) is from the SCAP Security Guide (SSG) project.  See:
https://fedorahosted.org/scap-security-guide/
	The Remediation content (BASH) is from the Aqueduct project.  See:
https://fedorahosted.org/aqueduct/

If checks are reporting incorrect information it usually means one of two things: (1) the
check content itself has a bug, or (2) the code executing that check has a bug.
If fixes are behaving incorrectly it means the remediation content itself has a bug.  The
end goal is to have a set of content that can be run numerous times to dynamically
remediate a system.

>  From the common profile, which is what I am running, I found the
> following: 
>     Check Incorrect (Pre Remediation)     15
>     Remediation Fix Broken                1
>     Check Incorrect (Post Remediation)    15+19 (34) 
>  I have compiled a spreadsheet documenting my findings. 
>  1) What/who should the findings be submitted to?  What format?
For the remediation content related issues you described, please submit those findings
directly to the Aqueduct community.  They are the upstream maintainers of the BASH
remediation content we leverage.

For the strictly-documentation related issues you described in the audit content, please
submit those findings directly to the SSG community.  They are the upstream maintainers of
the OVAL & XCCDF content we leverage.

For the execution-related issues you described in the audit content, we can help look into
those further.  This mailing list should be configured to accept plain-text attachments. 
If you could export your spreadsheet to CSV you can attach it and send it over to us.  If
you have a file that's too large to be attached, putting it up on an ftp server and
providing a link is recommended.
If it turns out there are issues with the audit content, case (1) above, (XCCDF/OVAL) then
we can forward that information over to the SSG community.
If there are issues with the actual execution of the content, case (2) above, then we can
look deeper into our codebase.  It may indicate an issue with SecState or with the
underlying OpenSCAP library we use for probing the system.

>  2) When is the planned release or any fixes? 
This will depend
on where the issues are stemming from and the severity of the issues. 

We understand that this community separation is currently a bit confusing for end-users. 
There are ongoing discussions about merging different repos together to simplify this in
the future.

Thank you
 - Francisco Slavin

>  Logan Rodrian

[Rodrian, Logan P (IS)]

Francisco-

Attached are 4 CSV files, per your request. Their contents are as follows:

Bugs - Errors/inconsistencies with reporting or remediation
BrokenRemediation - Remediation fixes that do not perform what is described
AuditFalsePositivesPre - Audit report failures, pre-remediation, that incorrectly report
failure (check portion incorrectly discovering status of checked object)
AuditFalsePositivesPost - Audit report failures, post-remediation, that incorrectly report
failure (check portion incorrectly discovering status of checked object)

I believe most of these fall into your category (2), as there is an issue with the code
performing work.

Please let me know what additional information is needed/required.


Logan Rodrian


________________________________________
From: Francisco Slavin [fslavin(a)tresys.com]
Sent: Thursday, December 06, 2012 14:33
To: Rodrian, Logan P (IS); secstate(a)lists.fedorahosted.org
Subject: EXT :RE: SecState Evaluation

From: secstate-bounces(a)lists.fedorahosted.org
[mailto:[e-mail]] On Behalf Of Rodrian, Logan P (IS)
Sent: Thursday, December 06, 2012 12:30 PM
To: secstate(a)lists.fedorahosted.org
Subject: [SecState] SecState Evaluation

>  Hello- 
>  Along with the recent release of CLIP, I have begun using SecState
> to
>  perform auditing and remediation for my system. In performing these
>  tasks and viewing the reports, I found that there were multiple issues
>  with both the checks and the fixes being performed incorrectly, along with some
> inconsistencies with the description versus the performed check/fix. 
Hi Logan. It's good to see some early use of the latest SecState & CLIP releases.
We appreciate the feedback and hopefully we can work to resolve the issues you've
encountered.
To give a brief overview of the current content included in CLIP and used by SecState:
The Audit content (XCCDF and OVAL) is from the SCAP Security Guide (SSG) project. See:
https://fedorahosted.org/scap-security-guide/
The Remediation content (BASH) is from the Aqueduct project. See:
https://fedorahosted.org/aqueduct/

If checks are reporting incorrect information it usually means one of two things: (1) the
check content itself has a bug, or (2) the code executing that check has a bug.
If fixes are behaving incorrectly it means the remediation content itself has a bug. The
end goal is to have a set of content that can be run numerous times to dynamically
remediate a system.

>  From the common profile, which is what I am running, I found the
> following: 
>  Check Incorrect (Pre Remediation) 15
>  Remediation Fix Broken 1
>  Check Incorrect (Post Remediation) 15+19 (34) 
>  I have compiled a spreadsheet documenting my findings. 
>  1) What/who should the findings be submitted to? What format? 
For the remediation content related issues you described, please submit those findings
directly to the Aqueduct community. They are the upstream maintainers of the BASH
remediation content we leverage.

For the strictly-documentation related issues you described in the audit content, please
submit those findings directly to the SSG community. They are the upstream maintainers of
the OVAL & XCCDF content we leverage.

For the execution-related issues you described in the audit content, we can help look into
those further. This mailing list should be configured to accept plain-text attachments. If
you could export your spreadsheet to CSV you can attach it and send it over to us. If you
have a file that's too large to be attached, putting it up on an ftp server and
providing a link is recommended.
If it turns out there are issues with the audit content, case (1) above, (XCCDF/OVAL) then
we can forward that information over to the SSG community.
If there are issues with the actual execution of the content, case (2) above, then we can
look deeper into our codebase. It may indicate an issue with SecState or with the
underlying OpenSCAP library we use for probing the system.

>  2) When is the planned release or any fixes? 
This will depend
on where the issues are stemming from and the severity of the issues.

We understand that this community separation is currently a bit confusing for end-users.
There are ongoing discussions about merging different repos together to simplify this in
the future.

Thank you
- Francisco Slavin

>  Logan Rodrian

[Shawn Wells]

On 12/7/12 11:31 AM, Rodrian, Logan P (IS) wrote:
>  Attached are 4 CSV files, per your request. Their contents are as
> follows:
>
>  Bugs - Errors/inconsistencies with reporting or remediation
>  BrokenRemediation - Remediation fixes that do not perform what is described
>  AuditFalsePositivesPre - Audit report failures, pre-remediation, that incorrectly report
> failure (check portion incorrectly discovering status of checked object)
>  AuditFalsePositivesPost - Audit report failures, post-remediation, that incorrectly
> report failure (check portion incorrectly discovering status of checked object)
>
>  I believe most of these fall into your category (2), as there is an issue with the code
> performing work.
>
>  Please let me know what additional information is needed/required. 
I had a chance to step through your spreadsheet.

Many of these are bugs within the OVAL (checking) content of the SCAP 
Security Guide. I've created a number of tickets for these, located here:
https://fedorahosted.org/scap-security-guide/report/3

The remaining items on your "REMEDIATE - Broken" and "Bugs" tab appear

to be originating from the Aqueduct remediation scripts.

[Shawn Wells]

On 12/14/12 10:56 AM, Shawn Wells wrote:
>  On 12/7/12 11:31 AM, Rodrian, Logan P (IS) wrote:
> > Attached are 4 CSV files, per your request. Their contents are as 
> > follows:
> >
> > Bugs - Errors/inconsistencies with reporting or remediation
> > BrokenRemediation - Remediation fixes that do not perform what is 
> > described
> > AuditFalsePositivesPre - Audit report failures, pre-remediation, that 
> > incorrectly report failure (check portion incorrectly discovering 
> > status of checked object)
> > AuditFalsePositivesPost - Audit report failures, post-remediation, 
> > that incorrectly report failure (check portion incorrectly 
> > discovering status of checked object)
> >
> > I believe most of these fall into your category (2), as there is an 
> > issue with the code performing work.
> >
> > Please let me know what additional information is needed/required.
>
>  I had a chance to step through your spreadsheet.
>
>  Many of these are bugs within the OVAL (checking) content of the SCAP 
>  Security Guide. I've created a number of tickets for these, located here:
>  https://fedorahosted.org/scap-security-guide/report/3
>
>  The remaining items on your "REMEDIATE - Broken" and "Bugs" tab
> appear 
>  to be originating from the Aqueduct remediation scripts. 

Francisco - I'm unable to replicate this in the upstream SSG code. Do 
you know what version of SSG you cloned into SecState? Can you verify 
this behavior on your end? An example of what I'm doing to test is here: 
https://fedorahosted.org/scap-security-guide/ticket/196

Heading out on travel for a few days, should be back online Tuesday to 
continue looking at this.

[Francisco Slavin]

>  -----Original Message-----
>  From: secstate-bounces(a)lists.fedorahosted.org [mailto:secstate-
>  bounces(a)lists.fedorahosted.org] On Behalf Of Shawn Wells
>  Sent: Sunday, December 16, 2012 8:43 PM
>  To: secstate(a)lists.fedorahosted.org
>  Subject: Re: [SecState] SecState Evaluation
>  
>  On 12/14/12 10:56 AM, Shawn Wells wrote:
>  > On 12/7/12 11:31 AM, Rodrian, Logan P (IS) wrote:
>  >> Attached are 4 CSV files, per your request. Their contents are as
>  >> follows:
>  >>
>  >> Bugs - Errors/inconsistencies with reporting or remediation
>  >> BrokenRemediation - Remediation fixes that do not perform what is
>  >> described AuditFalsePositivesPre - Audit report failures,
>  >> pre-remediation, that incorrectly report failure (check portion
>  >> incorrectly discovering status of checked object)
>  >> AuditFalsePositivesPost - Audit report failures, post-remediation,
>  >> that incorrectly report failure (check portion incorrectly
>  >> discovering status of checked object)
>  >>
>  >> I believe most of these fall into your category (2), as there is an
>  >> issue with the code performing work.
>  >>
>  >> Please let me know what additional information is needed/required.
>  >
>  > I had a chance to step through your spreadsheet.
>  >
>  > Many of these are bugs within the OVAL (checking) content of the SCAP
>  > Security Guide. I've created a number of tickets for these, located here:
>  > https://fedorahosted.org/scap-security-guide/report/3
>  >
>  > The remaining items on your "REMEDIATE - Broken" and "Bugs" tab
> appear
>  > to be originating from the Aqueduct remediation scripts.
>  
>  
>  Francisco - I'm unable to replicate this in the upstream SSG code. Do you
>  know what version of SSG you cloned into SecState? Can you verify this
>  behavior on your end? An example of what I'm doing to test is here:
>  https://fedorahosted.org/scap-security-guide/ticket/196
>  
>  
Shawn,
SecState doesn't have a cloned copy of SSG; the CLIP project pulls in both the
SecState tool and the SSG content.
I believe Logan is using CLIP for RHEL 6.2.  Based on a checkout of the CLIP repo, at the
tag CLIP_RHEL_6.2_Final, it looks like the corresponding SSG git-ish is: 4d28ff3
This is from the RELEASE variable in the clip/packages/scap-security-guide/Makefile

Thanks
 - Francisco

>  Heading out on travel for a few days, should be back online Tuesday
> to
>  continue looking at this.
>  _______________________________________________
>  SecState mailing list
>  SecState(a)lists.fedorahosted.org
>  https://lists.fedorahosted.org/mailman/listinfo/secstate
>