User Survey

Fri May 3 09:26:57 UTC 2013

Le jeudi 02 mai 2013 à 20:43 -0400, Eric H. Christensen a écrit :
> To follow up on today's Board meeting discussion regarding the user 
> survey, I've got an install of LimeSurvey working and am playing with 
> the logic to build the questions and answers.  There are a few things 
> to consider when creating this type of survey and I wanted to get input 
> from everyone.

Quick summary for non board people and for people who were not present,
yesterday we had a phone conference call to really assess if we are
going in the right direction with the userbase discussion. However, due
to technical constraints ( ie most of us being unprepared for that ), we
do not have detailed enough meetings notes. We brainstormed and expanded
on the points made earlier ( plateform, does default desktop offering
mean something, how to manage more than 1, etc ), and in the end, it was
proposed to see if a survey would help us to have more data and a better
view of the landscape.

( complete me if I am wrong, my low level of glycogen at that time may
have provoked a temporary loss of memory )

> 
> Public or Directed?
> -------------------
> We can either make the survey open to the public or we can directly email people an invitation 
> to take the survey.  There are pros and cons to each but here are my thoughts on the subject.
> 
> A public survey will likely include people that are users of Fedora that aren't contributors.  
> That's okay because we assume that not all Fedora users have signed up in FAS (where our directed 
> survey data comes from).  This, of course, also opens the door to non-Fedora users and people that 
> don't have anything better to do with their day than to screw around online.  
> 
> A directed survey would be sent to the ~9500 people that have signed up in FAS.  They would receive 
> a more personalized letter via email (perhaps in their native language if we can swing that) and we 
> would be able to track the responses (i.e. what percentage of contributors have completed the survey).  
> This type of survey would limit our audience to only those that have a FAS account.  I guess we could do
>  a campaign before the survey to have users sign up for a FAS account so we can get a survey to them.
> 
> Personally, I'd prefer to do a directed survey as we can do a more scientific analysis of the results.  
> This will also hopefully reduce the amount of junk responses.

As we can do more than 1 survey and with regard to the first discussion
we had, I would propose that we try to have a survey for github users
( ie, "log in with github credentials" ) trying to see why Fedora is not
suitable for them, or if they tried and left, and why.

In fact, I would :
- ask basic demography questions
- if they know what is Fedora
- if they use it
- if tried it 

then if they use, why, so we know what are our strong points for that
community, and pass the word so the community is aware and maybe develop
it. 

if they no longer/don't, why, so we have data on what we should improve
( and I say we as "we the community", we cannot tell to people what to
do, but I think that saying "we would attract more coders if we do this"
will be something others may be interested to do ).

I fully agree with Robyn when she said that this is one of the biggest
contributors pool we could find. And maybe we can find another ones :
- wikipedia contributors for wiki/documentation
- deviant art for artist 
- transifex community for i18n
etc, etc.

And then do the same type of survey ( either the same, or tailored to
the community ), so we know what we could push and suggest to attract
people who are likely to contribute.

> Questions
> ---------
> What do we need/want to know?  I've started a wiki page[0] where we can collect questions and I can start putting together the 
> logic.  I'd appreciate if people would start putting their questions there so I can start building this thing.

There is a whole methodology to make a valid survey ( or maybe more than
one, but I only know one ), and from what I remember from my university
years, we first start by a few interviews of the people we want to
survey so we can , then we need to express the hypothesis we want to
validate or invalidate, and then we start doing questions.

I think we do all have already enough discussions with people so we know
the field, but I do not think we have clearly expressed the hypothesis
we want to validate if we want the survey to be useful.

So what do we want to verify or what do we want to get from the survey
in the end ?

I would also make people aware of the numerous bias we can introduce in
the survey questions, and so we should think on a way to
detect/avoid/reduce them ( like 1 person dedicated on this topic, or
trying to get in touch with someone who do surveys on a more regular
basis for validation ). 
Or maybe ask te community at large to specifically look at this, and
suppose that if no one complain, then we did our best ?

( no, I don't suggest making a survey on the survey, even if
"surveyception" would be a nice project name )

> Hosting
> -------
> I mentioned today that I use a web hosting provider that allows LimeSurvey installs (actually makes it easy to deploy LimeSurvey). 
>  I have no problems hosting this thing and will happily give others admin access so they can audit the results and have access
>  to the system during the survey.  The other solution that Robyn mentioned was cutting a check to LimeSurvey to have them host
>  the survey.  Personally I just assume do it myself where we'll have more control over the data.

Not wanting to be fanboyish, but why not openshift ?
https://github.com/openshift-quickstart/limesurvey-quickstart

And as a last point, I would propose that we make clear in all our
communication that a survey is just that, a survey, ie a approximation
of a self selected set of users, which may not reflect as accurately as
people tend to believe the reality, and that we may have introduced
bias, etc.

I also think that this would requires some specific treatments with
regard to personal data, especially since the law in EU are slightly
much more stringent in the EU ( especially in Germany ) than in the USA.
I would find value to have email of people if we need a follow up of
some points, but maybe we will not have time, and so not need it, and
should bother.

-- 
Michael Scherer