New subject: [Open-scap] SCAP ecosystem test day results

18 Sep 2014


      Hi all,
Martin Zember kindly let us torture him a bit about SCAP scanning and
tailoring so we can get a better idea about what we should document
or completely change. We explicitly asked him not to study anything
in advance so that we can figure out what's "easy" and what's not.
And more importantly, where are users looking for information.
I am cross-posting to share the results of this test because it
involves all the SCAP projects.
The test we performed is described at
http://open-scap.org/page/TestPlan
Please note that this is the first iteration and we will likely
tune the test process in each iteration. Suggestions are of course
welcome.
Notable general pain points:
* USGCB? What is it? Is it on paper or is it machine readable?
* What does benchmark mean in this context?
* How do I change a guidance? (tailoring confusion)
* What is a profile? A lot of confusion about content vs profile.
* It is easy to find some info about RHEL5 USGCB, the assumption
   was that RHEL6 should be similar which is sadly not the case.
* usgcb.nist.gov provides content for RHEL5, doesn't even mention
   anything about RHEL6
* In general it's hard to figure out that SCAP is the way to
   automate paper guidances. Perhaps we want to use other keywords.
* A lot of confusion about the difference between openscap and
   scap-security-guide.
* xccdf Value vs xccdf Rule, which Rules use which Values.
openscap specific:
* openscap doesn't include any content, yet it was expected of it
   - user tried to look for "some version of openscap that can do
     USGCB"
* if user selects a profile ID that's not in the content, openscap
   should either print `oscap info` on the file to help or at
   least hint the existence of `oscap info`.
* ssg custom repo in documentation, outdated...
* user didn't notice a report has already been generated using
   --report and generated it again
* confusion about OVAL results and check system details
scap-security-guide specific:
* Upstream web mentions EPEL which is outdated and confusing
   - note: it is not actually outdated yet but will be soon
* Upstream doesn't mention profiles that ssg provides
* It is entirely ungooglable
- user expected to find it with: "automate usgcb linux",
     "usgcb linux tool", "check linux against usgcb",
     "linux security compliance", "usgcb audit linux".
- the only way to find it is to find openscap page first
     and then look at related projects, unfortunately openscap
     doesn't mention that scap-security-guide has USGCB...
* stig-rhel6-server was renamed to stig-rhel6-server-upstream,
   documentation mentions stig-rhel6-server. Why would an ID
   like this change?
scap-workbench specific:
* user@target placeholder text disappears when you paste IP
   there. Then user doesn't realize that user@ can also go there.
* Where does workbench get ssh password? User first searched
   for input boxes or settings before starting the scan and
   filling out the ssh-askpass box.
* Ctrl+F or / was expected to search in both main window
   and tailoring window.
* Enter doesn't confirm search in tailoring window, user had
   to click
* No visualization of dependencies between rules and values
* Save content vs Save tailoring, what is the difference,
   what do I want to use?
* The user manual was not used unless explicitly talked about.
* Opening tailoring file directly doesn't automatically open
   the content and then the tailoring file.
* scap-workbench is not all that much advertised on
   scap-security-guide or openscap web pages. It is therefore
   quite hard to discover it.
-- 
Martin Preisler