Several participants in the thread "Re: New report and guide in openscap 1.1.0" raised concerned over a language "The system is not compliant!" in the report.
I agree. "not compliant" ideally should only be done against a policy established by an organization for that system. The assessment and passing of any set of controls != compliance, it is a step toward compliance.
Public profiles are, generally speaking, against *baselines*. It would be great if the use of OpenSCAP and SSG inspired/encouraged Departments and Agencies to develop and manage their own profiles that build on top of the baseline profiles.
For example, if a test is run against a default profile, feedback should emphasize the profile is a default profile and encourage organization to maintain their own organizational profiles.
Imagine if by default OpenSCAP tried to find to an agency's default repository of profiles. If no agency-defined repository existed, OpenSCAP would communicate it "No agency-specific profiles available. To create agency specific profiles blah, blah...OpenSCAP using default SCAP-Security-Guide baseline profile blah, blah, blah." This would constantly remind users they should be managing their own set of profiles.
Of course, managing drift among agency specific profiles vs the baselines becomes another issue -- but that is something further for OpenSCAP and SSG to assist.
Greg Elin http://govready.org - Making FISMA compliance easier for innovators
email: gregelin@gitmachines.com phone: 917-304-3488
----- Original Message -----
From: "Greg Elin" gregelin@gitmachines.com To: "SCAP Security Guide" scap-security-guide@lists.fedorahosted.org Sent: Sunday, August 31, 2014 2:43:43 PM Subject: Best ways to say this system is not compliant
Several participants in the thread "Re: New report and guide in openscap 1.1.0" raised concerned over a language "The system is not compliant!" in the report.
I decided to avoid using the word compliant at all in this case. XCCDF spec defines what it means on the report but people may think the word has a different meaning and may be shocked.
So instead I decided to explicitly say how many rules failed or were inconclusive.
For example: "The target system did not satisfy conditions of 131 rules! Furthermore, the results of 2 rules were inconclusive. Please review rule results and consider applying remediation."
See https://git.fedorahosted.org/cgit/openscap.git/commit/?id=6e622f7d86a1061ce0...
Does this help the situation? Is this a good summary of the TestResult?
On Tuesday, September 02, 2014 08:48:17 AM Martin Preisler wrote:
Several participants in the thread "Re: New report and guide in openscap 1.1.0" raised concerned over a language "The system is not compliant!" in the report.
I decided to avoid using the word compliant at all in this case. XCCDF spec defines what it means on the report but people may think the word has a different meaning and may be shocked.
So instead I decided to explicitly say how many rules failed or were inconclusive.
For example: "The target system did not satisfy conditions of 131 rules! Furthermore, the results of 2 rules were inconclusive. Please review rule results and consider applying remediation."
Inconclusive is a word that I would not use. It prejudices the results by indicating that something may or may not be wrong and it can't really tell. That's not the message I want from any tool. Imagine running badblocks, aide, or find and the message output is that it's inconclusive. :-)
I'd simply say something to the effect that the scan has findings that need to be reviewed. Short & simple.
-Steve
----- Original Message -----
From: "Steve Grubb" sgrubb@redhat.com To: scap-security-guide@lists.fedorahosted.org Cc: "Martin Preisler" mpreisle@redhat.com Sent: Tuesday, September 2, 2014 5:33:05 PM Subject: Re: Best ways to say this system is not compliant
On Tuesday, September 02, 2014 08:48:17 AM Martin Preisler wrote:
Several participants in the thread "Re: New report and guide in openscap 1.1.0" raised concerned over a language "The system is not compliant!" in the report.
I decided to avoid using the word compliant at all in this case. XCCDF spec defines what it means on the report but people may think the word has a different meaning and may be shocked.
So instead I decided to explicitly say how many rules failed or were inconclusive.
For example: "The target system did not satisfy conditions of 131 rules! Furthermore, the results of 2 rules were inconclusive. Please review rule results and consider applying remediation."
Inconclusive is a word that I would not use. It prejudices the results by indicating that something may or may not be wrong and it can't really tell.
That is exactly the case. It's not prejudice, it's fact.
That's not the message I want from any tool. Imagine running badblocks, aide, or find and the message output is that it's inconclusive. :-)
I'd simply say something to the effect that the scan has findings that need to be reviewed. Short & simple.
The report says "inconclusive" for rules 'error' or 'unknown' as cdf:result.
Excerpt from XCCDF 1.2 spec:
error - The checking engine could not complete the evaluation, therefore the status of the target's compliance with the rule is not certain. This could happen, for example, if a testing tool was run with insufficient privileges and could not gather all of the necessary information.
unknown - The testing tool encountered some problem and the result is unknown. For example, a result of 'unknown' might be given if the testing tool was unable to interpret the output of the checking engine (the output has no meaning to the testing tool).
I am not a native speaker but I am pretty sure inconclusive is the right word for this. If not, please provide an alternative.
On Tuesday, September 02, 2014 11:43:14 AM Martin Preisler wrote:
So instead I decided to explicitly say how many rules failed or were inconclusive.
For example: "The target system did not satisfy conditions of 131 rules! Furthermore, the results of 2 rules were inconclusive. Please review rule results and consider applying remediation."
Inconclusive is a word that I would not use. It prejudices the results by indicating that something may or may not be wrong and it can't really tell.
That is exactly the case. It's not prejudice, it's fact.
Apologies. I think I misread the output. For those two items I think you are right.
-Steve
On 9/2/14, 11:43 AM, Martin Preisler wrote:
----- Original Message -----
From: "Steve Grubb" sgrubb@redhat.com To: scap-security-guide@lists.fedorahosted.org Cc: "Martin Preisler" mpreisle@redhat.com Sent: Tuesday, September 2, 2014 5:33:05 PM Subject: Re: Best ways to say this system is not compliant
On Tuesday, September 02, 2014 08:48:17 AM Martin Preisler wrote:
> Several participants in the thread "Re: New report and guide in openscap > 1.1.0" > raised concerned over a language "The system is not compliant!" in the > report.
I decided to avoid using the word compliant at all in this case. XCCDF spec defines what it means on the report but people may think the word has a different meaning and may be shocked.
So instead I decided to explicitly say how many rules failed or were inconclusive.
For example: "The target system did not satisfy conditions of 131 rules! Furthermore, the results of 2 rules were inconclusive. Please review rule results and consider applying remediation."
Inconclusive is a word that I would not use. It prejudices the results by indicating that something may or may not be wrong and it can't really tell.
That is exactly the case. It's not prejudice, it's fact.
That's not the message I want from any tool. Imagine running badblocks, aide, or find and the message output is that it's inconclusive. :-)
I'd simply say something to the effect that the scan has findings that need to be reviewed. Short & simple.
The report says "inconclusive" for rules 'error' or 'unknown' as cdf:result.
Excerpt from XCCDF 1.2 spec:
error - The checking engine could not complete the evaluation, therefore the status of the target's compliance with the rule is not certain. This could happen, for example, if a testing tool was run with insufficient privileges and could not gather all of the necessary information.
unknown - The testing tool encountered some problem and the result is unknown. For example, a result of 'unknown' might be given if the testing tool was unable to interpret the output of the checking engine (the output has no meaning to the testing tool).
I am not a native speaker but I am pretty sure inconclusive is the right word for this. If not, please provide an alternative.
Would it be possible to add a tooltip/hover so that when people mouse over the "Rule Overview" check boxes they would see these definitions?
Many find the differences between notchecked, notselected, and notapplicable confusing... and really, to someone who hasn't read the xccdf spec, that's entirely understandable.
I would always highlight with a bright green that all rules pass and not use any coloring for a failing system. Getting everything green is an achievement. Anything else is just typical rather than failing.
So. For 100% passing (pseudo code):
<green> Target device "my-server" is passing 223 of 223 rules defined in profile "usgcb-rhel6-server"! </green>
And for less than 100% passing (pseudo code):
<light gray background> Target device "my-server" is passing 166 of 223 rules defined in profile "usgcb-rhel6-server"!
Results show "my-server" failing <green>0 high severity rules</green>, 16 medium severity rules, and 31 low severity rules of the profile. Also, there were 10 rules indicating a known checking engine "error" or an "unknown" problem. </light gray background>
Other thoughts: - It would be nice if the text block was easy to copy and paste to share with someone. Which also makes me wonder if unique report ID can be generated somehow to link back to this report.
Greg
On Tue, Sep 2, 2014 at 1:54 PM, Shawn Wells shawn@redhat.com wrote:
On 9/2/14, 11:43 AM, Martin Preisler wrote:
----- Original Message -----
From: "Steve Grubb" sgrubb@redhat.com sgrubb@redhat.com> To: scap-security-guide@lists.fedorahosted.org> Cc: "Martin Preisler" mpreisle@redhat.com mpreisle@redhat.com> Sent: Tuesday, September 2, 2014 5:33:05 PM> Subject: Re: Best ways to say this system is not compliant> > On Tuesday, September 02, 2014 08:48:17 AM Martin Preisler wrote:
Several participants in the thread "Re: New report and guide in openscap> > > 1.1.0"> > > raised concerned over a language "The system is not compliant!" in the> > > report.
I decided to avoid using the word compliant at all in this case.> > XCCDF spec defines what it means on the report but people may think> > the word has a different meaning and may be shocked.> > > > So instead I decided to explicitly say how many rules failed or were> > inconclusive.> > > > For example:> > "The target system did not satisfy conditions of 131 rules! Furthermore,> > the results of 2 rules were inconclusive. Please review rule results> > and consider applying remediation."
Inconclusive is a word that I would not use. It prejudices the results by> indicating that something may or may not be wrong and it can't really tell.
That is exactly the case. It's not prejudice, it's fact.
That's not the message I want from any tool. Imagine running badblocks, aide,> or find and the message output is that it's inconclusive. :-)> > I'd simply say something to the effect that the scan has findings that need> to> be reviewed. Short & simple.
The report says "inconclusive" for rules 'error' or 'unknown' as cdf:result.
Excerpt from XCCDF 1.2 spec:
error - The checking engine could not complete the evaluation, therefore the status of the target's compliance with the rule is not certain. This could happen, for example, if a testing tool was run with insufficient privileges and could not gather all of the necessary information.
unknown - The testing tool encountered some problem and the result is unknown. For example, a result of 'unknown' might be given if the testing tool was unable to interpret the output of the checking engine (the output has no meaning to the testing tool).
I am not a native speaker but I am pretty sure inconclusive is the right word for this. If not, please provide an alternative.
Would it be possible to add a tooltip/hover so that when people mouse over the "Rule Overview" check boxes they would see these definitions?
Many find the differences between notchecked, notselected, and notapplicable confusing... and really, to someone who hasn't read the xccdf spec, that's entirely understandable.
-- SCAP Security Guide mailing list scap-security-guide@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/scap-security-guide https://github.com/OpenSCAP/scap-security-guide/
----- Original Message -----
From: "Greg Elin" gregelin@gitmachines.com To: "SCAP Security Guide" scap-security-guide@lists.fedorahosted.org Sent: Tuesday, September 2, 2014 8:23:30 PM Subject: Re: Best ways to say this system is not compliant
I would always highlight with a bright green that all rules pass and not use any coloring for a failing system. Getting everything green is an achievement. Anything else is just typical rather than failing.
So. For 100% passing (pseudo code):
<green> Target device "my-server" is passing 223 of 223 rules defined in profile "usgcb-rhel6-server"! </green>
And for less than 100% passing (pseudo code):
<light gray background> Target device "my-server" is passing 166 of 223 rules defined in profile "usgcb-rhel6-server"!
Results show "my-server" failing <green>0 high severity rules</green>, 16 medium severity rules, and 31 low severity rules of the profile. Also, there were 10 rules indicating a known checking engine "error" or an "unknown" problem. </light gray background>
No, just no. I am drawing the line right there.
Other thoughts:
- It would be nice if the text block was easy to copy and paste to share
with someone. Which also makes me wonder if unique report ID can be generated somehow to link back to this report.
Sure but openscap is too low-level for this feature. We have plans for features like that in various SCAP integrations - cockpit, satellite 6, ...
----- Original Message -----
From: "Shawn Wells" shawn@redhat.com To: scap-security-guide@lists.fedorahosted.org Sent: Tuesday, September 2, 2014 7:54:15 PM Subject: Re: Best ways to say this system is not compliant
[snip]
Would it be possible to add a tooltip/hover so that when people mouse over the "Rule Overview" check boxes they would see these definitions?
Many find the differences between notchecked, notselected, and notapplicable confusing... and really, to someone who hasn't read the xccdf spec, that's entirely understandable.
We have tooltips over the rule results themselves but you are right that having them in the box in Rule Overview would be better.
I'll make a patch.
On 9/3/14, 7:38 AM, Martin Preisler wrote:
----- Original Message -----
From: "Shawn Wells" shawn@redhat.com To: scap-security-guide@lists.fedorahosted.org Sent: Tuesday, September 2, 2014 7:54:15 PM Subject: Re: Best ways to say this system is not compliant
[snip]
Would it be possible to add a tooltip/hover so that when people mouse over the "Rule Overview" check boxes they would see these definitions?
Many find the differences between notchecked, notselected, and notapplicable confusing... and really, to someone who hasn't read the xccdf spec, that's entirely understandable.
We have tooltips over the rule results themselves but you are right that having them in the box in Rule Overview would be better.
I'll make a patch.
Thanks!
Absolutely no rush or anything: could you re-spin the samples off your fedorapeople page? It'd be great to see how all these changes have been incorporated (the code presents some insight, but it's funner to play around on the html page).
btw - I was at a customer site today who made a comment that they've been watching this thread and really loving where the report is heading. these design improvements are really going far to show responsiveness to the user community, and generally making OpenSCAP reports badass :)
----- Original Message -----
From: "Shawn Wells" shawn@redhat.com To: scap-security-guide@lists.fedorahosted.org Sent: Thursday, September 4, 2014 6:12:20 AM Subject: Re: Best ways to say this system is not compliant
On 9/3/14, 7:38 AM, Martin Preisler wrote:
----- Original Message -----
From: "Shawn Wells" shawn@redhat.com To: scap-security-guide@lists.fedorahosted.org Sent: Tuesday, September 2, 2014 7:54:15 PM Subject: Re: Best ways to say this system is not compliant
[snip]
Would it be possible to add a tooltip/hover so that when people mouse over the "Rule Overview" check boxes they would see these definitions?
Many find the differences between notchecked, notselected, and notapplicable confusing... and really, to someone who hasn't read the xccdf spec, that's entirely understandable.
We have tooltips over the rule results themselves but you are right that having them in the box in Rule Overview would be better.
I'll make a patch.
Thanks!
Absolutely no rush or anything: could you re-spin the samples off your fedorapeople page? It'd be great to see how all these changes have been incorporated (the code presents some insight, but it's funner to play around on the html page).
Done -> https://mpreisle.fedorapeople.org/openscap/1.1.0_xslt/
btw - I was at a customer site today who made a comment that they've been watching this thread and really loving where the report is heading. these design improvements are really going far to show responsiveness to the user community, and generally making OpenSCAP reports badass :)
Cool. That's nice to hear.
scap-security-guide@lists.fedorahosted.org
-
Greg Elin -
Martin Preisler -
Shawn Wells -
Steve Grubb