Waiver support in HTML report

[Martin Preisler]

Hi,
I wrote a short blog post about waivers in HTML report.
These changes are coming in 1.2.0 so we would like to gather
some feedback before the release.

Suggestions welcome!

http://martin.preisler.me/2014/11/waivers-in-openscap-html-report/

[Shawn Wells]

The post mentions users won't be able to create waivers in the HTML report itself. What's the workflow to introduce a waiver?

--
Shawn Wells
Director, Innovation Programs
[e-mail] | 443.534.0130
@shawndwells
> On Nov 6, 2014, at 10:49 AM, Martin Preisler  wrote:
>
> Hi,
> I wrote a short blog post about waivers in HTML report.
> These changes are coming in 1.2.0 so we would like to gather
> some feedback before the release.
>
> Suggestions welcome!
>
> http://martin.preisler.me/2014/11/waivers-in-openscap-html-report/
>
> -- 
> Martin Preisler
> _______________________________________________
> scap-workbench mailing list
> [e-mail]
> https://lists.fedorahosted.org/mailman/listinfo/scap-workbench

[Open-scap] Waiver support in HTML report

[Šimon Lukašík]

On 11/06/2014 05:33 PM, Shawn Wells wrote:
> The post mentions users won't be able to create waivers in the HTML report itself. What's the workflow to introduce a waiver?

Hey Shawn,

The waiver creation will be first available through Foreman interface 
using recently announced project SCAPtimony. This Martin's work is very 
first step needed to make it happen.

[Martin Preisler]

----- Original Message -----
> From: "Shawn Wells" 
> To: "Martin Preisler" 
> Cc: "open-scap-list" , [e-mail], "SCAP Security Guide"
>
> Sent: Thursday, November 6, 2014 5:33:43 PM
> Subject: Re: Waiver support in HTML report
>
> The post mentions users won't be able to create waivers in the HTML report
> itself. What's the workflow to introduce a waiver?

There is no nice way to introduce a waiver at this point.

In the future the way to do this would be with openscap integration of your
choice - scap-workbench, sat5, 6, cockpit, ... I may add some javascript hooks
to the HTML report to allow integrations to listen to waiver requests or some
such. But the HTML report itself is static, it can't change the XML which
stores the results.

[Martin Preisler]

----- Original Message -----
> From: "Shawn Wells" 
> To: "Martin Preisler" 
> Cc: "open-scap-list" , [e-mail], "SCAP Security Guide"
>
> Sent: Thursday, November 6, 2014 5:33:43 PM
> Subject: Re: Waiver support in HTML report
>
> The post mentions users won't be able to create waivers in the HTML report
> itself. What's the workflow to introduce a waiver?

I have done initial research on how to implementing adding waivers from
the report.

See the prototype:
https://mpreisle.fedorapeople.org/openscap/interactive_waiver.html

Please note that this is not committed yet. It needs quite some cleanup.
As discussed previously this allows integrations of openscap to preprocess
the HTML report and insert their own callback. That's how they get notified
about added waivers.

A nice accidental feature I discovered today is that you can waive the report
in firefox and when you "Save as" it saves the report with the waivers.
Could be useful in small deployments.

Known issues:
- score not recomputed
- number of rules that fail, pass, ... isn't recomputed

Feedback appreciated!

[Jan Ruzicka]

Hi

The report looks great.

Inconsistency of graph direction Rule result breakdown vs Failed rules by severity breakdown caught my eye.
The Rule result breakdown shows more important issues to the right, but the Failed rules by severity breakdown to the left.
The whole document seems to be bringing focus to the right side.

Jan

On Nov 12, 2014, at 11:36, Martin Preisler wrote:

----- Original Message -----
From: "Shawn Wells" <[e-mail]>
To: "Martin Preisler" <[e-mail]>
Cc: "open-scap-list" <[e-mail]>, [e-mail], "SCAP Security Guide"
<[e-mail]>
Sent: Thursday, November 6, 2014 5:33:43 PM
Subject: Re: Waiver support in HTML report

The post mentions users won't be able to create waivers in the HTML report
itself. What's the workflow to introduce a waiver?

I have done initial research on how to implementing adding waivers from
the report.

See the prototype:
https://mpreisle.fedorapeople.org/openscap/interactive_waiver.html

Please note that this is not committed yet. It needs quite some cleanup.
As discussed previously this allows integrations of openscap to preprocess
the HTML report and insert their own callback. That's how they get notified
about added waivers.

A nice accidental feature I discovered today is that you can waive the report
in firefox and when you "Save as" it saves the report with the waivers.
Could be useful in small deployments.

Known issues:
- score not recomputed
- number of rules that fail, pass, ... isn't recomputed

Feedback appreciated!

--
Martin Preisler
--
SCAP Security Guide mailing list
[e-mail]
https://lists.fedorahosted.org/mailman/listinfo/scap-security-guide
https://github.com/OpenSCAP/scap-security-guide/

Jan Ruzicka
Senior Software Engineer
Comtech Mobile Datacom Corporation
20430 Century Blvd, Germantown, MD 20874
Office: 240-686-3300
Fax: [phone]

The information contained in this message may be privileged and/or confidential. If you are not the intended recipient, or responsible for delivering this message to the intended recipient, any review, forwarding, dissemination, distribution or copying of this communication or any attachment(s) is strictly prohibited. If you have received this message in error, please so notify the sender immediately, and delete it and all attachments from your computer and network.

[Martin Preisler]

----- Original Message -----
> From: "Jan Ruzicka" 
> To: "SCAP Security Guide" 
> Sent: Wednesday, November 12, 2014 6:51:21 PM
> Subject: Re: Waiver support in HTML report
>
> Hi
>
> The report looks great.
>
> Inconsistency of graph direction Rule result breakdown vs Failed rules by
> severity breakdown caught my eye.
> The Rule result breakdown shows more important issues to the right, but the
> Failed rules by severity breakdown to the left.
> The whole document seems to be bringing focus to the right side.

So the failed rules should be on the left and passed on the right?
Or can you suggest another modification that would fix it?

[Shawn Wells]

On 11/13/14, 5:19 AM, Martin Preisler wrote:
> ----- Original Message -----
> > > From: "Jan Ruzicka" 
> > > To: "SCAP Security Guide" 
> > > Sent: Wednesday, November 12, 2014 6:51:21 PM
> > > Subject: Re: Waiver support in HTML report
> > >
> > > Hi
> > >
> > > The report looks great.
> > >
> > > Inconsistency of graph direction Rule result breakdown vs Failed rules by
> > > severity breakdown caught my eye.
> > > The Rule result breakdown shows more important issues to the right, but the
> > > Failed rules by severity breakdown to the left.
> > > The whole document seems to be bringing focus to the right side.
>
> So the failed rules should be on the left and passed on the right?
> Or can you suggest another modification that would fix it?

It's more about consistency.

The "Rule result breakdown" goes from green (passed) to red (failed) to
orange ("other"), which is fine. Having the passed rules first seems
highly ideal -- I'd rather have control assessors see green before red.
This makes the report less alarming.

Immediately below, under "Failed rules by severity breakdown," some of
the same colors are reused from the "rule result breakdown." This
section uses red for "high", the same color of orange for "medium."

Because the same colors are used, it took a double take to recognize the
"1 high" severity finding doesn't correlate to the "11 failed" rules
immediately above. Slightly changing the colors should take care of this.

[Jan Ruzicka]

The high severity should be on the right and low on the left.

That way reader can skim the report looking mainly on the right side.

--
Jan Ruzicka

On Nov 13, 2014, at 05:19, Martin Preisler wrote:
> ----- Original Message -----
> > From: "Jan Ruzicka" 
> > To: "SCAP Security Guide" 
> > Sent: Wednesday, November 12, 2014 6:51:21 PM
> > Subject: Re: Waiver support in HTML report
> >
> > Hi
> >
> > The report looks great.
> >
> > Inconsistency of graph direction Rule result breakdown vs Failed rules by
> > severity breakdown caught my eye.
> > The Rule result breakdown shows more important issues to the right, but the
> > Failed rules by severity breakdown to the left.
> > The whole document seems to be bringing focus to the right side.
>
> So the failed rules should be on the left and passed on the right?
> Or can you suggest another modification that would fix it?
>
> -- 
> Martin Preisler

[Martin Preisler]

----- Original Message -----
> From: "Jan Ruzicka" 
> To: "Martin Preisler" 
> Cc: "SCAP Security Guide" 
> Sent: Thursday, November 13, 2014 7:14:30 PM
> Subject: Re: Waiver support in HTML report
>
> The high severity should be on the right and low on the left.
>
> That way reader can skim the report looking mainly on the right side.

Sorry this took so long but it's fixed now.

See:
https://github.com/OpenSCAP/openscap/commit/b7b2f16e4f29c343847658f69ed253e7...
https://fedorahosted.org/openscap/ticket/428

Thanks for your feedback!

[Gabe Alford]

Really like the new feature! One thing is how do I remove a waiver, e.g.
what if I accidentally add a waiver to the wrong rule?

Gabe

On Wed, Nov 12, 2014 at 9:36 AM, Martin Preisler 
wrote:
> ----- Original Message -----
> > From: "Shawn Wells" 
> > To: "Martin Preisler" 
> > Cc: "open-scap-list" ,
>
> [e-mail], "SCAP Security Guide"
> > Sent: Thursday, November 6, 2014 5:33:43 PM
> > Subject: Re: Waiver support in HTML report
> >
> > The post mentions users won't be able to create waivers in the HTML
>
> report
> > itself. What's the workflow to introduce a waiver?
>
> I have done initial research on how to implementing adding waivers from
> the report.
>
> See the prototype:
> https://mpreisle.fedorapeople.org/openscap/interactive_waiver.html
>
> Please note that this is not committed yet. It needs quite some cleanup.
> As discussed previously this allows integrations of openscap to preprocess
> the HTML report and insert their own callback. That's how they get notified
> about added waivers.
>
> A nice accidental feature I discovered today is that you can waive the
> report
> in firefox and when you "Save as" it saves the report with the waivers.
> Could be useful in small deployments.
>
> Known issues:
> score not recomputed
> number of rules that fail, pass, ... isn't recomputed
>
> Feedback appreciated!
>
> --
> Martin Preisler
> --
> SCAP Security Guide mailing list
> [e-mail]
> https://lists.fedorahosted.org/mailman/listinfo/scap-security-guide
> https://github.com/OpenSCAP/scap-security-guide/

[Martin Preisler]

----- Original Message -----
> From: "Gabe Alford" 
> To: "SCAP Security Guide" 
> Sent: Thursday, November 13, 2014 2:05:40 PM
> Subject: Re: Waiver support in HTML report
>
> Really like the new feature! One thing is how do I remove a waiver, e.g.
> what if I accidentally add a waiver to the wrong rule?

Right now you can't but this is a planned feature. There will be a waiver
removal callback that integrations can set. I will most likely not add
waiver modification, if you need waiver modification you can always remove
a waiver and add a new one.

[Shawn Wells]

On 11/13/14, 8:52 AM, Martin Preisler wrote:
> ----- Original Message -----
> > > From: "Gabe Alford" 
> > > To: "SCAP Security Guide" 
> > > Sent: Thursday, November 13, 2014 2:05:40 PM
> > > Subject: Re: Waiver support in HTML report
> > >
> > > Really like the new feature! One thing is how do I remove a waiver, e.g.
> > > what if I accidentally add a waiver to the wrong rule?
>
> Right now you can't but this is a planned feature. There will be a waiver
> removal callback that integrations can set. I will most likely not add
> waiver modification, if you need waiver modification you can always remove
> a waiver and add a new one.

In an earlier thread someone mentioned setting a "waiver expiration"
concept. This would be INCREDIBLY useful, but would this be better
discussed for SCAPtimony integration?

As a sample use case.... During many C&A efforts, I've had a control
assessor find something I've overlooked. An example would be setting the
system login banners -- sometimes on small, compartmentalized networks,
setting the login banner is more a formality to pass a compliance check
than a meaningful legal countermeasure. So they grant ATO given that I
must fix the finding within 5-10 days.  In such a scenario, I load up
the SCAP report and click "add waiver." I select a "expires on" date,
which somehow integrates into SCAPtimony. As that date approaches I get
nag screens.

Would something like this be achievable? And if so, should an RFE be
filed to the SCAPtimony GitHub page or somewhere else?

[Šimon Lukašík]

On 11/13/2014 06:20 PM, Shawn Wells wrote:
> On 11/13/14, 8:52 AM, Martin Preisler wrote:
> > ----- Original Message -----
> > > > From: "Gabe Alford" 
> > > > To: "SCAP Security Guide" 
> > > > Sent: Thursday, November 13, 2014 2:05:40 PM
> > > > Subject: Re: Waiver support in HTML report
> > > >
> > > > Really like the new feature! One thing is how do I remove a waiver, e.g.
> > > > what if I accidentally add a waiver to the wrong rule?
> >
> > Right now you can't but this is a planned feature. There will be a waiver
> > removal callback that integrations can set. I will most likely not add
> > waiver modification, if you need waiver modification you can always remove
> > a waiver and add a new one.
>
> In an earlier thread someone mentioned setting a "waiver expiration"
> concept. This would be INCREDIBLY useful, but would this be better
> discussed for SCAPtimony integration?
>
> As a sample use case.... During many C&A efforts, I've had a control
> assessor find something I've overlooked. An example would be setting the
> system login banners -- sometimes on small, compartmentalized networks,
> setting the login banner is more a formality to pass a compliance check
> than a meaningful legal countermeasure. So they grant ATO given that I
> must fix the finding within 5-10 days.  In such a scenario, I load up
> the SCAP report and click "add waiver." I select a "expires on" date,
> which somehow integrates into SCAPtimony. As that date approaches I get
> nag screens.
>
> Would something like this be achievable? And if so, should an RFE be
> filed to the SCAPtimony GitHub page or somewhere else?

Very nice idea Shawn!

I have added it to the list of planned features.

https://github.com/OpenSCAP/scaptimony/commit/6caf6b39c3771b1cecc2f67ec656ee...

Thanks!

[Shawn Wells]

On 11/12/14, 11:36 AM, Martin Preisler wrote:
> ----- Original Message -----
> > > From: "Shawn Wells" 
> > > To: "Martin Preisler" 
> > > Cc: "open-scap-list" , [e-mail], "SCAP Security Guide"
> > >
> > > Sent: Thursday, November 6, 2014 5:33:43 PM
> > > Subject: Re: Waiver support in HTML report
> > >
> > > The post mentions users won't be able to create waivers in the HTML report
> > > itself. What's the workflow to introduce a waiver?
>
> I have done initial research on how to implementing adding waivers from
> the report.
>
> See the prototype:
> https://mpreisle.fedorapeople.org/openscap/interactive_waiver.html
>
> Please note that this is not committed yet. It needs quite some cleanup.
> As discussed previously this allows integrations of openscap to preprocess
> the HTML report and insert their own callback. That's how they get notified
> about added waivers.
>
> A nice accidental feature I discovered today is that you can waive the report
> in firefox and when you "Save as" it saves the report with the waivers.
> Could be useful in small deployments.
>
> Known issues:
> score not recomputed
> number of rules that fail, pass, ... isn't recomputed
>
> Feedback appreciated!

A popup/callout screen appears when opening a rule. It'd be great if the
"rule details" callout could be escaped by clicking elsewhere outside
the window, or simply hitting the Escape key.

[Open-scap] Waiver support in HTML report

[Kayse, Josh]

> On Nov 6, 2014, at 10:49 AM, Martin Preisler  wrote:
>
> Hi,
> I wrote a short blog post about waivers in HTML report.
> These changes are coming in 1.2.0 so we would like to gather
> some feedback before the release.
>
> Suggestions welcome!
>
> http://martin.preisler.me/2014/11/waivers-in-openscap-html-report/

This is awesome. I’ll echo Shawn Wells question about generating waivers. Additionally, does a waived rule still impact the score of the system?

We would like to be able to use the STIG from SSG with minimal modifications (changing variables in the STIG only). Unfortunately this results in several rules failing that we have obtained waivers for. Therefore we currently modify the weight of the known failures such that they do not impact the score of the system. This allows us to confidently monitor the posture of the system by monitoring the score. If the known failures were integrated in to the score we would not know whether the score is due to a known failure or a new failure.

For example, say the score is 90 because prelink is enabled. An admin comes in and changes prelink to be disable but also causes another rule, such as password complexity, to fail which leaves the score at 90.

-josh

[Open-scap] Waiver support in HTML report

[Martin Preisler]

----- Original Message -----
> From: "Josh Kayse" 
> To: "Martin Preisler" 
> Cc: "open-scap-list" , [e-mail], "SCAP Security Guide"
>
> Sent: Thursday, November 6, 2014 6:58:33 PM
> Subject: Re: [Open-scap] Waiver support in HTML report
> > On Nov 6, 2014, at 10:49 AM, Martin Preisler  wrote:
> >
> > Hi,
> > I wrote a short blog post about waivers in HTML report.
> > These changes are coming in 1.2.0 so we would like to gather
> > some feedback before the release.
> >
> > Suggestions welcome!
> >
> > http://martin.preisler.me/2014/11/waivers-in-openscap-html-report/
>
> This is awesome. I’ll echo Shawn Wells question about generating waivers.

Replied about this to Shawn.
> Additionally, does a waived rule still impact the score of the system?

It does not. For all intents and purposes it behaves like a rule of the result
the waiver set it to. So if you waive a failed rule and make it "pass" you
basically make it behave exactly like a passed rule.