Survey stuff

Mon Jul 14 14:55:51 UTC 2014

----- Original Message -----
> From: "Eric H. Christensen" <sparks at fedoraproject.org>
> To: "Fedora community advisory board" <board-discuss at lists.fedoraproject.org>
> Sent: Wednesday, July 9, 2014 7:33:55 AM
> Subject: Re: Survey stuff
> 
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA512
> 
> On Wed, Jul 09, 2014 at 07:53:44AM -0600, Pete Travis wrote:
> > On Jul 8, 2014 6:41 AM, "Eric H. Christensen" <sparks at fedoraproject.org>
> > wrote:
> > The primary language question could be "what languages do you use with
> > Fedora? (Select all)" or have a followup question to otherwise reflect the
> > degree of multilingual use.
> 
> This could be difficult to consume.  How many multi-lingual people could
> breakout their use of each language by %age?  The intent behind the question
> is to determine what languages they use within Fedora.  This question can
> likely be presented better.
> 
> > It would also be interesting to get FAS IDs to correlate survey responses
> > with the degree of participation.  Even "most responders do not contribute,
> > have not contributed recently, or declined to provide FASid" would be
> > useful.  This would make participation at least appear to be a lot less
> > anonymous, but doing so would reveal the degree of bias in the results.
> 
> So, the original plan was to provide each contributor with a unique code
> (part of the URL) so we could individualize the inputs and provide
> accountablility.  We would also have specific numbers on how many people we
> have participating versus not (and maybe we provide a badge for
> participating).  That said, we should still separate the data collected from
> the people providing the data and not provide a means of tracing back.
> People should feel free to provide whatever feedback they wish without fear
> of it coming back to haunt them later.

A few things to note (assuming the plan is still to use limesurvey):

* If we can generate a list of FAS Account IDs/names/email addresses from FAS (in .csv or via ldap query) then we can generate a unique token for each address. Each address would get a unique link mailed to them. (And we can also send follow-up mails to those who have not yet used their tokens to complete the survey, and/or mails to folks who started but have not yet completed the survey.) 
* If the survey is set to anonymous then there is no link between the addresses/token and the responses, other than whether or not a person completed the survey. (Tokens can also be used to figure out whether or not a person completed the survey if the survey is not-anonymous; either way, when the survey closes, we could export the list of fas accounts for "who responded" into a list for getting badges, or alternately, provide a generic link or QR code for people to click on at the end of the survey that could be used for them to get their badge.)

For the question above re: FAS IDs/survey responses and frequency of participation - if anonymous, all we could get would be the total # of respondents and what their answer to the question "how frequently do you contribute" (or however it is worded). Unless you meant "we would determine each FAS IDs frequency of participation independently of the survey" and ... well, that's a tough one to tackle on its own, and I don't know that I'd be in favor of labelling people in that way when there is a lot of activity that can't be tracked/categorized/etc. 

If *not* anonymous - we can add in data points that can be correlated to each answer - basically anything in FAS (region, language, what FAS groups they are part of) could be loaded into their response and then wouldn't be questions folks would even have to answer.  And we could do a lot of additional analysis in that way, if we wanted. Those fields could also be used to determine whether or not to ask folks additional sets of questions, ie: "If folks are in the Docs FAS group, ask them these additional 2 questions."  But, like Eric: I think people respond best and most freely, and in the highest frequency, when they know they can respond without fear of it it coming back to haunt them later.

> 
> So I guess we *could* get statistics on the people that did respond to the
> survey based on their FAS information (again, removing identifying features
> wherever possible).  This could be difficult to accomplish in reality.

I guess in short (since I answered this above): We can do this with an anonymous survey, but only in a limited fashion. We'll have the complete set of answers, and we'll have the complete list of FAS accounts who completed the survey. But tying any of the answers to any further identifying information in the FAS account (region, language, time zone, FAS groups) wouldn't be possible.

> 
> > I'd also like to see a primary/secondary/other OS question.  Quantifying
> > the degree of multiboot activity, and making a distinction between "have
> > other platforms to check my code" and "can haz every distro", would be a
> > valuable insight.
> 
> I welcome specific questions or chains of questions (if answer is 3 then ask
> this question too) for addition to the survey.  I'll caution that we don't
> want to make this survey too long.  The questions should be repeatable over
> the course of years and should have some strategic purpose.

+1. The longer the survey gets, it becomes increasingly likely that people won't respond or finish. We can generate something that shows the total number of questions at the beginning, and/or show a progress bar. And ++ on repeatability / strategic purpose too.

And on that note: I've seen a couple folks mention asking about gender in the survey. My thoughts are as follows:

* If the survey is non-anonymous - I would consider omitting that question altogether. I'm not particularly in favor of having a results list with my FAS ID and/or email address linked to the fact that I am female being published. (Note: I am assuming that the results will be freely viewable on the wiki or downloadable as an .ods.) Not that this isn't something largely obvious if you look on the internet and see that I refer to myself as a mother and etc., but giant lists of names and addresses are probably more easily digested and turned into targeted spam or dating solicitations or prescription ads or etc. (Though I assume if we publish anything we would strip out email addresses at the bare minimum, if not also FAS ids, even if not-anonymous.)
* If the survey is anonymous - I'm still not quite sure what we get out of the question. It doesn't necessarily measure anything (we don't track sex in FAS) - more respondents as F over M from one year to the next doesn't reflect anything other than just that - number of respondents identifying that way. And unless that answer is planned to be correllated against other responses - ie, "the number of female respondents who answered 'yes' to [some other question]" - I'm having a hard time figuring out what is actionable with it. 
* On actionability - this may also apply to some of the other demographic information as well (not sure if anyone suggested age); other bits, like language, can at least be useful in some way. (Even if it is things like, "Comparing the number of completed surveys vs. started-but-not-completed surveys for people speaking english vs. other languages might show that we really need to have translations available for future surveys.") 

> 
> I'm hoping to spend some time on this on Friday or over the weekend and have
> some sort of mock up ready for Monday.
> 
> - -- Eric
> 
> - --------------------------------------------------
> Eric "Sparks" Christensen
> Fedora Project
> 
> sparks at fedoraproject.org - sparks at redhat.com
> 097C 82C3 52DF C64A 50C2  E3A3 8076 ABDE 024B B3D1
> - --------------------------------------------------
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1
> 
> iQGcBAEBCgAGBQJTvVLNAAoJEB/kgVGp2CYvYr8L/RHZfc1I7vJ3KEN/KPzszYjo
> +rFZMVWIPGN/fK80+tmPBHVqjEF9BCVoJCoTkCKBSHfYgqqozXisVreq0bnAoOWy
> vh7xUqswBc2vkYqX2zWW2nAANbb54ybxPJd8DwrO1MKNMwTDgIgyP++cFnoQYsVC
> qZZW8DVC6k19Wc/6fnkqfVM++UmxQ5x8ovA5fnceEFq9ci0QPYiS/Au0OjFK96Ml
> gOVURFBTMldoOLtmC+JYX9/8qYyOof7pOnvHnS/kDD9CLc4t9MVwrWBHmRWNo8tm
> hwvomXHdTFGuzzbkL3gUr9WCwK78I7x1GTGkmsL9Y6cuvyVVlP8+5+343FPGMIZ6
> oTMWs+C300NQmSBB7VLH5aFmAVJhy9j05RlmTKUtXRn9RNfPI2dtxULTtwZ4tS4H
> 0H03Tc1RKjWxk89za3brJ1LvpLr3naVfswusAhxDki63IZn7q3QFTUmFvp9+P7Yl
> QdW0FiMopPHLFyZe/4NC5vT5Xp5UVltnqj17MHs7tA==
> =0POs
> -----END PGP SIGNATURE-----
> _______________________________________________
> board-discuss mailing list
> board-discuss at lists.fedoraproject.org
> https://admin.fedoraproject.org/mailman/listinfo/board-discuss