Hi all,
Martin Zember kindly let us torture him a bit about SCAP scanning and tailoring so we can get a better idea about what we should document or completely change. We explicitly asked him not to study anything in advance so that we can figure out what's "easy" and what's not. And more importantly, where are users looking for information.
I am cross-posting to share the results of this test because it involves all the SCAP projects.
The test we performed is described at http://open-scap.org/page/TestPlan
Please note that this is the first iteration and we will likely tune the test process in each iteration. Suggestions are of course welcome.
Notable general pain points:
* USGCB? What is it? Is it on paper or is it machine readable?
* What does benchmark mean in this context?
* How do I change a guidance? (tailoring confusion)
* What is a profile? A lot of confusion about content vs profile.
* It is easy to find some info about RHEL5 USGCB, the assumption was that RHEL6 should be similar which is sadly not the case.
* usgcb.nist.gov provides content for RHEL5, doesn't even mention anything about RHEL6
* In general it's hard to figure out that SCAP is the way to automate paper guidances. Perhaps we want to use other keywords.
* A lot of confusion about the difference between openscap and scap-security-guide.
* xccdf Value vs xccdf Rule, which Rules use which Values.
openscap specific:
* openscap doesn't include any content, yet it was expected of it - user tried to look for "some version of openscap that can do USGCB"
* if user selects a profile ID that's not in the content, openscap should either print `oscap info` on the file to help or at least hint the existence of `oscap info`.
* ssg custom repo in documentation, outdated...
* user didn't notice a report has already been generated using --report and generated it again
* confusion about OVAL results and check system details
scap-security-guide specific:
* Upstream web mentions EPEL which is outdated and confusing - note: it is not actually outdated yet but will be soon
* Upstream doesn't mention profiles that ssg provides
* It is entirely ungooglable
- user expected to find it with: "automate usgcb linux", "usgcb linux tool", "check linux against usgcb", "linux security compliance", "usgcb audit linux".
- the only way to find it is to find openscap page first and then look at related projects, unfortunately openscap doesn't mention that scap-security-guide has USGCB...
* stig-rhel6-server was renamed to stig-rhel6-server-upstream, documentation mentions stig-rhel6-server. Why would an ID like this change?
scap-workbench specific:
* user@target placeholder text disappears when you paste IP there. Then user doesn't realize that user@ can also go there.
* Where does workbench get ssh password? User first searched for input boxes or settings before starting the scan and filling out the ssh-askpass box.
* Ctrl+F or / was expected to search in both main window and tailoring window.
* Enter doesn't confirm search in tailoring window, user had to click
* No visualization of dependencies between rules and values
* Save content vs Save tailoring, what is the difference, what do I want to use?
* The user manual was not used unless explicitly talked about.
* Opening tailoring file directly doesn't automatically open the content and then the tailoring file.
* scap-workbench is not all that much advertised on scap-security-guide or openscap web pages. It is therefore quite hard to discover it.
----- Original Message -----
From: "Martin Preisler" mpreisle@redhat.com To: "open-scap-list" open-scap-list@redhat.com, scap-workbench@lists.fedorahosted.org, "SCAP Security Guide" scap-security-guide@lists.fedorahosted.org Cc: "Martin Zember" mzember@redhat.com Sent: Thursday, September 18, 2014 3:22:05 PM Subject: [Open-scap] SCAP ecosystem test day results
[snip]
openscap specific:
openscap doesn't include any content, yet it was expected of it
- user tried to look for "some version of openscap that can do USGCB"
if user selects a profile ID that's not in the content, openscap should either print `oscap info` on the file to help or at least hint the existence of `oscap info`.
https://fedorahosted.org/openscap/ticket/395
- ssg custom repo in documentation, outdated...
Fixed.
- user didn't notice a report has already been generated using --report and generated it again
Not sure what to do about this, the docs are fairly clear already :-/
- confusion about OVAL results and check system details
Should be much improved with https://fedorahosted.org/openscap/ticket/396
[snip]
scap-workbench specific:
- user@target placeholder text disappears when you paste IP there. Then user doesn't realize that user@ can also go there.
No idea what to do about this :-(
- Where does workbench get ssh password? User first searched for input boxes or settings before starting the scan and filling out the ssh-askpass box.
No idea what to do about this either. I want to avoid workbench storing or even handling any ssh credentials anywhere.
- Ctrl+F or / was expected to search in both main window and tailoring window.
https://fedorahosted.org/scap-workbench/ticket/234
- Enter doesn't confirm search in tailoring window, user had to click
https://fedorahosted.org/scap-workbench/ticket/235
- No visualization of dependencies between rules and values
https://fedorahosted.org/scap-workbench/ticket/229
- Save content vs Save tailoring, what is the difference, what do I want to use?
https://fedorahosted.org/scap-workbench/ticket/236
- The user manual was not used unless explicitly talked about.
No idea how to convince users to use the manual, workbench even has Help -> User Manual which opens it in the web browser.
Any suggestions welcome here.
- Opening tailoring file directly doesn't automatically open the content and then the tailoring file.
We should be able to fix this but it will require changes in both openscap and scap-workbench.
https://fedorahosted.org/scap-workbench/ticket/237
- scap-workbench is not all that much advertised on scap-security-guide or openscap web pages. It is therefore quite hard to discover it.
I added scap-workbench to Documentation introduction but the truth is that users probably skip past that. Not sure what else to do, it makes no sense to hint scap-workbench in every example listed there.
Comments Inline....
[snip]
- user didn't notice a report has already been generated using --report and generated it again
Not sure what to do about this, the docs are fairly clear already :-/
Can you detect previously generated reports and have something obvious to open them?
- user@target placeholder text disappears when you paste IP there. Then user doesn't realize that user@ can also go there.
No idea what to do about this :-(
Tooltip?
- Where does workbench get ssh password? User first searched for input boxes or settings before starting the scan and filling out the ssh-askpass box.
No idea what to do about this either. I want to avoid workbench storing or even handling any ssh credentials anywhere.
Honestly, most scanning apps let you do this. Store it in memory and make sure that it's not cached I suppose.
- The user manual was not used unless explicitly talked about.
No idea how to convince users to use the manual, workbench even has Help -> User Manual which opens it in the web browser.
Any suggestions welcome here.
This is probably a good thing. If the manual wasn't used, it indicates that the interface was reasonably intuitive for general use. The only other thing that you could do is to add 'help' buttons beside each input item that jump you to the relevant sections of the docs. This is how most tools that I've used do it.
- scap-workbench is not all that much advertised on scap-security-guide or openscap web pages. It is therefore quite hard to discover it.
I added scap-workbench to Documentation introduction but the truth is that users probably skip past that. Not sure what else to do, it makes no sense to hint scap-workbench in every example listed there.
In any example code, add something like 'Generated with scap-workbench <link>
Trevor Vaughan Vice President, Onyx Point, Inc (410) 541-6699 tvaughan@onyxpoint.com
-- This account not approved for unencrypted proprietary information --
scap-security-guide@lists.fedorahosted.org