Hi folks,
Stephanie is going to be working on the final touches (fixing typos, adjusting typography, etc.) of the survey for the anaconda usability tests this week - thank you very much for all of the helpful feedback on list.
I wanted to share with you a potential rough plan for the usability test at Devconf, and I'd like to ask for your feedback on that as well.
Stephanie, Filip, and I have talked about given the time we have allotted at Devconf, we'll probably run two rounds of tests, with four users in each round. So we'll have 8 usability testers total. We talked a little bit about the type of users we'd like to have test anaconda, which I think was cc'ed to this list, but for easy reference they are as follows:
1) Novice users who may be new to linux who just want to install a desktop system 2) Experienced linux users who may dabble in system administration, are very technical, and like trying out different technologies 3) Professional system administrators who work with advanced storage devices and configurations, typically on the job, not for fun
Since installing Fedora without customization is the most basic thing a user can do, and it is so critical that any user be able to do that, we talked about recruiting 4 "novice" (#1 above) users to do basically just that - we'll hand them a Fedora disk (or point them at an ISO file) and simply ask that they install Fedora, with no special requirements or customizations needed.
We also talked about recruiting more technical users ("experienced" and "professional" / #2 & #3 above) for the remaining 4 slots. Since there are a lot of features and paths someone could take through Anaconda without unlimited time and testers, I would like to propose giving each of the 4 'technical' testers each their own version of the test, each one dealing with a particular feature of Anaconda:
1) Dual-boot: provide the user with a VM that has Windows installed, and see if the user can install Fedora alongside it without wiping Windows. How we do this depends on whether or not the shrink/resize partitions UI is available - did we drop it for final?
2) Customized LVM: provide the user with a diagram and writeup of a specific LVM layout we'd like him or her to use while installing Fedora.
3) Customized RAID: provide the user with a diagram and writeup of a specific RAID configuration we'd like him or her to use while installing Fedora.
4) Preserve /home: provide the user with a VM that has Fedora 17 installed on it with a separate /home directory, and see if the user can install Fedora 18 on top of Fedora 17 without wiping the /home directory from Fedora 17.
Some scenarios we are leaving out here:
- BTRFS configuration - there's actually a bug that is in F18 gold that complicates BTRFS installations (https://bugzilla.redhat.com/show_bug.cgi?id=893758) so it might not be a good idea to include this anyway.
- text mode UI - I think having 4 folks who are not American & native English speakers (the default setting for the UI now is English / us-en keyboard) is much more valuable than having one or more of them split off and do a text mode UI. We can test the text mode UI when we run tests where I am here in Boston if need be.
- no network case - Another lower priority test case that I can cover in Boston later on.
In case this explanation is confusing, here's a user-by-user rundown of what the usability tests at Devconf should look like:
- User #1: novice user, given free reign of installer with simple task of installing Fedora 18 to a VM. - User #2: novice user, given free reign of installer with simple task of installing Fedora 18 to a VM. - User #3: novice user, given free reign of installer with simple task of installing Fedora 18 to a VM. - User #4: novice user, given free reign of installer with simple task of installing Fedora 18 to a VM. - User #5: professional/experienced linux user, asked to install Fedora 18 alongside pre-existing Windows install. - User #6: professional/experienced linux user, asked to install Fedora 18 using a specific LVM configuration. - User #7: professional/experienced linux user, asked to install Fedora 18 using a specific RAID configuration. - User #8: professional/experienced linux user, asked to install Fedora 18 while preserving the /home from a previous Fedora 17 install.
Do you think this high-level, basic test plan makes sense? Are there higher-priority test cases we're missing here, do you think? Do you have ideas on how we might be able to run some of these tests using a VM, or do you foresee any issues with making some of these test cases possible at Devconf?
As always, feedback/advice very much appreciated. ~m
- no network case - Another lower priority test case that I can cover
in Boston later on.
I assumed all installations at DevConf would be DVD installs, because we don't really want to depend on network speed (or having network at all). Is that what you meant? Or have you meant "custom network configuration"? Because I agree that custom networking (custom repositories etc) should not be part of DevConf testing. It's a too specific area.
On 2013-01-14 11:53, Kamil Paral wrote:
- no network case - Another lower priority test case that I can
cover in Boston later on.
I assumed all installations at DevConf would be DVD installs, because we don't really want to depend on network speed (or having network at all). Is that what you meant? Or have you meant "custom network configuration"? Because I agree that custom networking (custom repositories etc) should not be part of DevConf testing. It's a too specific area.
I meant there would be no network access at all - but, it's not a problem if there isn't network access at Devconf, I guess that would take care of the no-network case as well. :) The point was I didn't think we should do a separate 'no network' case on top of everything else, but it's fine I think if there's no network for any of these cases.
I definitely think we shouldn't rely on network for grabbing the packages for install either - I agree that using DVD (or full DVD isos already available on the test laptops) is the right thing to do.
~m
On Mon, Jan 14, 2013 at 11:34:11AM -0500, Máirín Duffy wrote:
- Dual-boot: provide the user with a VM that has Windows installed,
and see if the user can install Fedora alongside it without wiping Windows. How we do this depends on whether or not the shrink/resize partitions UI is available - did we drop it for final?
We dropped the shrink button since all it did was minimized your windows partition, not leaving much space for actually using it.
- BTRFS configuration - there's actually a bug that is in F18 gold
that complicates BTRFS installations (https://bugzilla.redhat.com/show_bug.cgi?id=893758) so it might not be a good idea to include this anyway.
This really only hits VM use with a tiny disk, doesn't it? As long as you have more than enough space to do the install you won't have space problems.
On 2013-01-14 15:05, Brian C. Lane wrote:
We dropped the shrink button since all it did was minimized your windows partition, not leaving much space for actually using it.
Okay, so if we do dual-boot, we'll have to make sure that the VM image we give the users has enough free space to install Fedora without shrinking Windows. That should still be doable though, right?
- BTRFS configuration - there's actually a bug that is in F18 gold
that complicates BTRFS installations (https://bugzilla.redhat.com/show_bug.cgi?id=893758) so it might not be a good idea to include this anyway.
This really only hits VM use with a tiny disk, doesn't it? As long as you have more than enough space to do the install you won't have space problems.
By default virt-manager gives you 8 GB, so we'd have to be sure during any test involving BTRFS that we provide more space than that. I'm not sure how much it needs though, maybe 20GB would be safe?
~m
On Mon, Jan 14, 2013 at 03:27:42PM -0500, Máirín Duffy wrote:
On 2013-01-14 15:05, Brian C. Lane wrote:
We dropped the shrink button since all it did was minimized your windows partition, not leaving much space for actually using it.
Okay, so if we do dual-boot, we'll have to make sure that the VM image we give the users has enough free space to install Fedora without shrinking Windows. That should still be doable though, right?
Should be.
- BTRFS configuration - there's actually a bug that is in F18 gold
that complicates BTRFS installations (https://bugzilla.redhat.com/show_bug.cgi?id=893758) so it might not be a good idea to include this anyway.
This really only hits VM use with a tiny disk, doesn't it? As long as you have more than enough space to do the install you won't have space problems.
By default virt-manager gives you 8 GB, so we'd have to be sure during any test involving BTRFS that we provide more space than that. I'm not sure how much it needs though, maybe 20GB would be safe?
Should be more than enough, as long as the machine doesn't have a huge amount of ram, driving the swap size up.
Do you think this high-level, basic test plan makes sense? Are there higher-priority test cases we're missing here, do you think? Do you have ideas on how we might be able to run some of these tests using a VM, or do you foresee any issues with making some of these test cases possible at Devconf?
I think this plan is pretty good. We get the most coverage of what we believe is the most common scenario, and we get some coverage of the slightly more exotic cases. To me, I think the biggest opportunity here is that we can take advantage of a lot of i18n testing. I still don't feel like we have very good data on that at all.
- Chris
----- Original Message -----
Hi folks,
Stephanie is going to be working on the final touches (fixing typos, adjusting typography, etc.) of the survey for the anaconda usability tests this week - thank you very much for all of the helpful feedback on list.
I wanted to share with you a potential rough plan for the usability test at Devconf, and I'd like to ask for your feedback on that as well.
Stephanie, Filip, and I have talked about given the time we have allotted at Devconf, we'll probably run two rounds of tests, with four users in each round.
Hi! We got scheduled to Meeting point 2 on Saturday, from 12:30 PM to 3:40 PM, there are currently free slots after the lab but also OpenLMI guys would like to share our machines :(
How many machines I should ask for? Last time I heard about 4 machines, so to have some reserve 6 would be enough I guess (and OpenLMI guys are going to use them too).
- Dual-boot: provide the user with a VM that has Windows installed,
and see if the user can install Fedora alongside it without wiping Windows. How we do this depends on whether or not the shrink/resize partitions UI is available - did we drop it for final?
Btw. not only shrink was dropped but we would not be able to provide Windows at all (licenses etc.).
- Customized LVM: provide the user with a diagram and writeup of a
specific LVM layout we'd like him or her to use while installing Fedora.
- Customized RAID: provide the user with a diagram and writeup of a
specific RAID configuration we'd like him or her to use while installing Fedora.
- Preserve /home: provide the user with a VM that has Fedora 17
installed on it with a separate /home directory, and see if the user can install Fedora 18 on top of Fedora 17 without wiping the /home directory from Fedora 17.
Some scenarios we are leaving out here:
- BTRFS configuration - there's actually a bug that is in F18 gold
that complicates BTRFS installations (https://bugzilla.redhat.com/show_bug.cgi?id=893758) so it might not be a good idea to include this anyway.
- text mode UI - I think having 4 folks who are not American & native
English speakers (the default setting for the UI now is English / us-en keyboard) is much more valuable than having one or more of them split off and do a text mode UI. We can test the text mode UI when we run tests where I am here in Boston if need be.
- no network case - Another lower priority test case that I can cover
in Boston later on.
In case this explanation is confusing, here's a user-by-user rundown of what the usability tests at Devconf should look like:
- User #1: novice user, given free reign of installer with simple
task of installing Fedora 18 to a VM.
- User #2: novice user, given free reign of installer with simple
task of installing Fedora 18 to a VM.
- User #3: novice user, given free reign of installer with simple
task of installing Fedora 18 to a VM.
- User #4: novice user, given free reign of installer with simple
task of installing Fedora 18 to a VM.
- User #5: professional/experienced linux user, asked to install
Fedora 18 alongside pre-existing Windows install.
- User #6: professional/experienced linux user, asked to install
Fedora 18 using a specific LVM configuration.
- User #7: professional/experienced linux user, asked to install
Fedora 18 using a specific RAID configuration.
- User #8: professional/experienced linux user, asked to install
Fedora 18 while preserving the /home from a previous Fedora 17 install.
Do you think this high-level, basic test plan makes sense? Are there higher-priority test cases we're missing here, do you think? Do you have ideas on how we might be able to run some of these tests using a VM, or do you foresee any issues with making some of these test cases possible at Devconf?
As always, feedback/advice very much appreciated. ~m
- Dual-boot: provide the user with a VM that has Windows
installed, and see if the user can install Fedora alongside it without wiping Windows. How we do this depends on whether or not the shrink/resize partitions UI is available - did we drop it for final?
Btw. not only shrink was dropped but we would not be able to provide Windows at all (licenses etc.).
We can use this: http://msdn.microsoft.com/en-us/evalcenter/jj554510.aspx
I have the images locally and I have confirmed with Richard Fontana (RH lawyer) that they can be used for dual-boot testing purposes (I assume this is essentially the same use case as our Fedora QA use case). The only problem is that KVM support needs to be verified (according to [1] we might need Fedora 18), or we need to use VirtualBox. I quickly tried KVM and the Windows installer doesn't boot, but maybe it just needs some tweaking.
Btw. not only shrink was dropped but we would not be able to provide Windows at all (licenses etc.).
At the very least, we could just make a big NTFS filesystem, put some stuff in it, and pretend that's Windows. It might not allow testing of whether Windows boots afterwards or not, but I don't think that's the real focus here.
- Chris
----- Original Message -----
Btw. not only shrink was dropped but we would not be able to provide Windows at all (licenses etc.).
At the very least, we could just make a big NTFS filesystem, put some stuff in it, and pretend that's Windows. It might not allow testing of whether Windows boots afterwards or not, but I don't think that's the real focus here.
Yep, that's our intention now. If it actually boots is really out of scope of usability testing.
Jaroslav
- Chris
Anaconda-devel-list mailing list Anaconda-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/anaconda-devel-list
On Mon 18 Feb 2013 08:04:47 AM EST, Jaroslav Reznik wrote:
At the very least, we could just make a big NTFS filesystem, put some stuff in it, and pretend that's Windows. It might not allow testing of whether Windows boots afterwards or not, but I don't think that's the real focus here.
Yep, that's our intention now. If it actually boots is really out of scope of usability testing.
If we don't know if it boots or not though, how do we know if the user got it to work and went thru the UI the intended way or not?
~m
On Mon, 2013-02-18 at 09:44 -0500, Máirín Duffy wrote:
On Mon 18 Feb 2013 08:04:47 AM EST, Jaroslav Reznik wrote:
At the very least, we could just make a big NTFS filesystem, put some stuff in it, and pretend that's Windows. It might not allow testing of whether Windows boots afterwards or not, but I don't think that's the real focus here.
Yep, that's our intention now. If it actually boots is really out of scope of usability testing.
If we don't know if it boots or not though, how do we know if the user got it to work and went thru the UI the intended way or not?
I believe preserving NTFS partition (shrinked or not) is what the user should achieve. Whether the system boots or not depends on whether there are bugs or not in the way we create grub.cfg on in the way grub is installed.
On Feb 19, 2013, at 3:54 AM, Vratislav Podzimek vpodzime@redhat.com wrote:
I believe preserving NTFS partition (shrinked or not) is what the user should achieve. Whether the system boots or not depends on whether there are bugs or not in the way we create grub.cfg on in the way grub is installed.
It's expected to work on BIOS hardware. On UEFI a limitation of os-prober apparently causes Windows to not be added as a menu entry, but Windows should still be bootable through the built-in boot manager.
Chris Murphy
Hi Máirín,
we are going to perform pilot usability tests (tomorrow) but we don't know how testers should be assigned to specific use cases. Whether you think that advanced user should perform ONE of the advanced installation test (one test case for one tester) or all advanced testers should perform all (four) advanced use cases (4 tests for one tester). Could you clarify it please?
Thanks
Filip
Hi Filip!
On 02/20/2013 02:26 PM, Filip Kosík wrote:
we are going to perform pilot usability tests (tomorrow) but we don't know how testers should be assigned to specific use cases. Whether you think that advanced user should perform ONE of the advanced installation test (one test case for one tester) or all advanced testers should perform all (four) advanced use cases (4 tests for one tester). Could you clarify it please?
Awesome!!! Let me know how it goes!
The tests are supposed to be one person, one test case. So one advanced user should only perform one of the four advanced installation tests, not all four. Try to assign the advanced testers tests that fit their experience, if that makes sense. E.g., try to give the RAID test to the person who filled out the most RAID experience in the survey (does that make sense?)
If the advanced users want to do more than one and you're willing to let them, there's no problem with that, but Stephanie and I only meant for each person to take just one test case.
Does this make it clearer? Sorry for the confusion!
~m
Dne 20.2.2013 20:50, Máirín Duffy napsal(a):
Hi Filip!
On 02/20/2013 02:26 PM, Filip Kosík wrote:
we are going to perform pilot usability tests (tomorrow) but we don't know how testers should be assigned to specific use cases. Whether you think that advanced user should perform ONE of the advanced installation test (one test case for one tester) or all advanced testers should perform all (four) advanced use cases (4 tests for one tester). Could you clarify it please?
Awesome!!! Let me know how it goes!
The tests are supposed to be one person, one test case. So one advanced user should only perform one of the four advanced installation tests, not all four. Try to assign the advanced testers tests that fit their experience, if that makes sense. E.g., try to give the RAID test to the person who filled out the most RAID experience in the survey (does that make sense?)
Thanks for clarifying.
If the advanced users want to do more than one and you're willing to let them, there's no problem with that, but Stephanie and I only meant for each person to take just one test case.
We had similar idea. We will ask them to do some extra test if there is any time left.
Filip
Hi,
15.1.2013 10:53, Jaroslav Reznik wrote:> ----- Original Message -----
Hi folks,
Stephanie is going to be working on the final touches (fixing typos, adjusting typography, etc.) of the survey for the anaconda usability tests this week - thank you very much for all of the helpful feedback on list.
I wanted to share with you a potential rough plan for the usability test at Devconf, and I'd like to ask for your feedback on that as well.
Stephanie, Filip, and I have talked about given the time we have allotted at Devconf, we'll probably run two rounds of tests, with four users in each round.
Hi! We got scheduled to Meeting point 2 on Saturday, from 12:30 PM to
3:40 PM,
there are currently free slots after the lab but also OpenLMI guys would like to share our machines :(
How many machines I should ask for? Last time I heard about 4 machines, so to have some reserve 6 would be enough I guess (and OpenLMI guys are going to use them too).
The last plan is: 4 participants for each round of usability tests. There will be 2 rounds. I think that there should be enough time (between these rounds) to prepare the same 4 machines for the next round. I think that 6 machines would be fine.
- Dual-boot: provide the user with a VM that has Windows installed,
and see if the user can install Fedora alongside it without wiping Windows. How we do this depends on whether or not the shrink/resize partitions UI is available - did we drop it for final?
Btw. not only shrink was dropped but we would not be able to provide Windows at all (licenses etc.).
What about using the license under MSDN Academic Alliance? I have this license available for study purposes and my work on usability tests (among others) is part of my master thesis. Should I check the conditions of use?
Filip
anaconda-devel@lists.fedoraproject.org
-
Brian C. Lane -
Chris Lumens -
Chris Murphy -
Filip Kosík -
Jaroslav Reznik -
Kamil Paral -
Máirín Duffy -
Vratislav Podzimek