Hey everyone! I happened by chance to meet someone at Red Hat who works on Red Hat's Net Promoter Score program. NPS, if you're not familiar, is a metric based on the simple question "How likely are you to recommend this?" on a scale from 0-10.
I think this can be very useful, especially as tracked over time. And it turns out that the group at Red Hat who does this would be happy to provide this to Fedora as a service.
I'd like to make a link to this survey available on some of the various Fedora websites. (Get Fedora, Fedora Magazine, Ask Fedora...)
We'll ask that basic question and a couple of followups like "Do you participate in the Fedora community" and "What are your top challenges?" (This is different from the demographic / contributor survey and won't ask any of those sorts of questions.)
For, like, a hotel chain, you usually get this survey as an email to some customer list. While we have email addresses of Fedora contributors, for obvious reasons we aren't gonna be using those for this in bulk. I also don't feel good about using the announce list — I will probably include a link to the survey in future release announcements, but just sending mail about the survey itself doesn't feel like what people signed up to that list for.
I am thinking, though, of creating a "please survey me" mailing list that people can opt-in to — if you sign up, we'll email you once every three months or six months or so.
What do you think of this idea overall? We also have the option of creating more specific surveys focused on an edition or spin, if your team would be interested in such a thing.
+1, the thing, the release notes, and the survey list. I like surveys.
P
On Tue, Nov 12, 2019 at 5:18 PM Matthew Miller mattdm@fedoraproject.org wrote:
Hey everyone! I happened by chance to meet someone at Red Hat who works on Red Hat's Net Promoter Score program. NPS, if you're not familiar, is a metric based on the simple question "How likely are you to recommend this?" on a scale from 0-10.
I think this can be very useful, especially as tracked over time. And it turns out that the group at Red Hat who does this would be happy to provide this to Fedora as a service.
I'd like to make a link to this survey available on some of the various Fedora websites. (Get Fedora, Fedora Magazine, Ask Fedora...)
We'll ask that basic question and a couple of followups like "Do you participate in the Fedora community" and "What are your top challenges?" (This is different from the demographic / contributor survey and won't ask any of those sorts of questions.)
For, like, a hotel chain, you usually get this survey as an email to some customer list. While we have email addresses of Fedora contributors, for obvious reasons we aren't gonna be using those for this in bulk. I also don't feel good about using the announce list — I will probably include a link to the survey in future release announcements, but just sending mail about the survey itself doesn't feel like what people signed up to that list for.
I am thinking, though, of creating a "please survey me" mailing list that people can opt-in to — if you sign up, we'll email you once every three months or six months or so.
What do you think of this idea overall? We also have the option of creating more specific surveys focused on an edition or spin, if your team would be interested in such a thing.
-- Matthew Miller mattdm@fedoraproject.org Fedora Project Leader _______________________________________________ council-discuss mailing list -- council-discuss@lists.fedoraproject.org To unsubscribe send an email to council-discuss-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/council-discuss@lists.fedorapr...
On Tue, Nov 12, 2019, 14:06 Petr Šabata psabata@redhat.com wrote:
+1, the thing, the release notes, and the survey list. I like surveys.
+1 to the general idea. We'll want to give some thought about how to put it in front of people such that we get meaningful answers. Which I suppose means we should think about what it is that we're asking. Essentially NPS comes down to either "yay Fedora" or "boo Fedora", but where we put the survey will depend on what we're specifically trying to find out.
On Tue, Nov 12, 2019 at 02:11:42PM -0500, Ben Cotton wrote:
+1 to the general idea. We'll want to give some thought about how to put it in front of people such that we get meaningful answers. Which I suppose means we should think about what it is that we're asking. Essentially NPS comes down to either "yay Fedora" or "boo Fedora", but where we put the survey will depend on what we're specifically trying to find out.
We will also be able to categorize the responses by where they came from, so we can look at, for example, different response from Ask Fedora vs. the opt-in mailing list.
Le 2019-11-12 17:17, Matthew Miller a écrit :
I think this can be very useful, especially as tracked over time. And it turns out that the group at Red Hat who does this would be happy to provide this to Fedora as a service.
Hello Matthew,
I'm not a huge fan of surveys as I doubt the value we can get from it.
Is our survey need so big we need a complex tool requiring to depend on an external team?
In terms of tooling, wouldn't our vote tooling do the job here? We'll have control over the data and opt in.
In terms of governance, it's the responsibility of Commops to deliver such insight of our community. https://docs.fedoraproject.org/en-US/commops/
Use metrics and data to improve understanding of Fedora community
Support development of metrics tooling
Can't we do this inside our community?
In addition, how will this team validate the statistical validity of the survey without using data collected for other means? Which will also leads us to GDPR issues.
Jean-Baptiste
On Wed, Nov 13, 2019 at 11:15 AM Jean-Baptiste Holcroft jean-baptiste@holcroft.fr wrote:
Le 2019-11-12 17:17, Matthew Miller a écrit :
I think this can be very useful, especially as tracked over time. And it turns out that the group at Red Hat who does this would be happy to provide this to Fedora as a service.
Hello Matthew,
I'm not a huge fan of surveys as I doubt the value we can get from it.
Is our survey need so big we need a complex tool requiring to depend on an external team?
In terms of tooling, wouldn't our vote tooling do the job here? We'll have control over the data and opt in.
I would actually prefer a third party to do this, the community as a whole has too many tools and related things to manage and third parties have good process for dealing with things like GDPR. I feel if it's deemed that this is necessary I think a specialist tool is the better way to get the value rather than putting the load onto already overloaded teams.
In terms of governance, it's the responsibility of Commops to deliver such insight of our community. https://docs.fedoraproject.org/en-US/commops/
Use metrics and data to improve understanding of Fedora community
Support development of metrics tooling
Can't we do this inside our community?
In addition, how will this team validate the statistical validity of the survey without using data collected for other means? Which will also leads us to GDPR issues.
Jean-Baptiste _______________________________________________ council-discuss mailing list -- council-discuss@lists.fedoraproject.org To unsubscribe send an email to council-discuss-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/council-discuss@lists.fedorapr...
Matthew Miller mattdm@fedoraproject.org wrote: ...
Hey everyone! I happened by chance to meet someone at Red Hat who works on Red Hat's Net Promoter Score program. NPS, if you're not familiar, is a metric based on the simple question "How likely are you to recommend this?" on a scale from 0-10.
I think this can be very useful, especially as tracked over time.
How do you think that it might be useful? This seems like the most important question to answer: if we know what we want to do with the data, that will enable us to effectively design the research exercise.
Allan
On Wed, Nov 13, 2019 at 11:51:45AM +0000, Allan Day wrote:
I think this can be very useful, especially as tracked over time.
How do you think that it might be useful? This seems like the most important question to answer: if we know what we want to do with the data, that will enable us to effectively design the research exercise.
Net Promoter Score is a gauge of loyalty and engagement. Having that as a running metric lets us know in a broad sense whether our outreach and mindshare efforts are working.
Matthew Miller mattdm@fedoraproject.org wrote: ...
Net Promoter Score is a gauge of loyalty and engagement. Having that as a running metric lets us know in a broad sense whether our outreach and mindshare efforts are working.
Personally I'd have expected NPS to be more strongly influenced by someone's experience of using Fedora, rather than outreach and mindshare efforts. If someone recommends a product, isn't it usually because they've used it and liked it?
If measuring outreach and mindshare is the goal, wouldn't more direct measures be more effective, like social media metrics, website data, press coverage and user numbers? Better still would be research targeted at new users, to find out what drew them to Fedora.
Allan
On Thu, Nov 14, 2019 at 02:40:23PM +0000, Allan Day wrote:
Net Promoter Score is a gauge of loyalty and engagement. Having that as a running metric lets us know in a broad sense whether our outreach and mindshare efforts are working.
Personally I'd have expected NPS to be more strongly influenced by someone's experience of using Fedora, rather than outreach and mindshare efforts. If someone recommends a product, isn't it usually because they've used it and liked it?
That's an excellent question and the answer is: I don't know. But in our current draft of the survey, the question _after_ the NPS one asks why the respondant chose that value.
If measuring outreach and mindshare is the goal, wouldn't more direct measures be more effective, like social media metrics, website data, press coverage and user numbers? Better still would be research targeted at new users, to find out what drew them to Fedora.
Those are all valuable too. We can definitely also use this platform to run specific targeted surveys, maybe even with a link at the end of GNOME initial setup or something like that.
Matthew Miller mattdm@fedoraproject.org wrote: ...
Net Promoter Score is a gauge of loyalty and engagement. Having that as a running metric lets us know in a broad sense whether our outreach and mindshare efforts are working.
Personally I'd have expected NPS to be more strongly influenced by someone's experience of using Fedora, rather than outreach and mindshare efforts. If someone recommends a product, isn't it usually because they've used it and liked it?
That's an excellent question and the answer is: I don't know. But in our current draft of the survey, the question _after_ the NPS one asks why the respondant chose that value.
I'd personally be careful about asking "why": I think the common research view is that you don't ask respondents to supply the analysis themselves (peoples' self-understandings are often wrong!)
From a research design perspective, it would likely be better to ask a range of more specific questions, rather than having an open question which you then have to try and interpret afterwards. This would make it simpler to analyse results, and we could be more confident in how the respondents understand the questions.
Other things I wonder about NPS:
- Will it help us understand why activities are successful or unsuccessful, and how to improve them? - How long will it take to see trends over time, and by the time we see them, will it be too late to change the outcome?
...
We can definitely also use this platform to run specific targeted surveys, maybe even with a link at the end of GNOME initial setup or something like that.
Yep, I'm definitely interested in that!
Allan
Hi,
Fedora is lucky to have great metrics and data about our community; however, as others mentioned, we have not used it as well as we could.
A good goal for community metrics to get direct feedback from the Fedora contributor community. This was the goal of the demographic / contributor survey planning in 2017 because the Diversity & Inclusion Team recognized the importance of creating more bidirectional feedback loops between the Fedora community and Fedora leadership.
On 11/12/19 11:17 AM, Matthew Miller wrote:
I am thinking, though, of creating a "please survey me" mailing list that people can opt-in to — if you sign up, we'll email you once every three months or six months or so.
Note there is sampling bias (convenience sampling) with this idea. Those who opt in to this list and participate are people who are especially motivated to subscribe and share feedback with Fedora leadership. This will get some feedback, but you will likely get feedback from a specific subset (perhaps an already vocal subset) of the community.
If directly emailing is off the table (which personally does not irritate me as a community contributor), then I think we could be more creative about where to get people's attention. For example, Elections are an activity typically done by an active, dedicated group of contributors. Embedding a link to the survey in the Elections site might solicit feedback from more types of contributors than an opt-in list.
On Mon, Dec 02, 2019 at 11:36:09AM -0500, Justin W. Flory wrote:
A good goal for community metrics to get direct feedback from the Fedora contributor community. This was the goal of the demographic / contributor survey planning in 2017 because the Diversity & Inclusion Team recognized the importance of creating more bidirectional feedback loops between the Fedora community and Fedora leadership.
Yeah, and I still would love that. But I also understand the difficulties it presents. This new approach sidesteps two things: first, we're avoiding demographic info for now, and second, since it is Red Hat's NPS team running the survey and managing the data, compliance and GDPR and etc. are handled for us.
people can opt-in to — if you sign up, we'll email you once every three months or six months or so.
Note there is sampling bias (convenience sampling) with this idea. Those who opt in to this list and participate are people who are especially motivated to subscribe and share feedback with Fedora leadership. This will get some feedback, but you will likely get feedback from a specific subset (perhaps an already vocal subset) of the community.
Oh, absolutely. We will have the ability to see where people came to the survey from, and can take this into account.
If directly emailing is off the table (which personally does not irritate me as a community contributor), then I think we could be more creative about where to get people's attention. For example, Elections are an activity typically done by an active, dedicated group of contributors. Embedding a link to the survey in the Elections site might solicit feedback from more types of contributors than an opt-in list.
I agree -- that's a great idea. Thanks!
The idea was something like this?
https://ubuntu.com/blog/the-ubuntu-20-04-lts-pre-release-survey
On Thu, Dec 19, 2019 at 07:42:53PM -0000, Alessio Ciregia wrote:
The idea was something like this? https://ubuntu.com/blog/the-ubuntu-20-04-lts-pre-release-survey
We could do some surveys like that, although we plan to start with something much more general.
council-discuss@lists.fedoraproject.org
-
Alessio Ciregia -
Allan Day -
Ben Cotton -
Jean-Baptiste Holcroft -
Justin W. Flory -
Matthew Miller -
Peter Robinson -
Petr Šabata