Fedora 15: a NFSv4 to glusterfs migration HOW-TO

[Dr. Michael J. Chudobiak]

As detailed in another thread, we upgraded a few test machines on our 
LAN to Fedora 15 (with gnome-shell and firefox), with user folders 
served from a NFSv4 server (F14 originally, then F15).

It just didn't work. The F15 desktops would freeze frequently. And 
worse, this would freeze ALL desktops on the LAN intermittently, as the 
NFS server struggled with client flakiness.

When it did work, Firefox would lose authenticated logins randomly, 
presumably due to corruption of its cookies.sqlite file. sqlite and NFS 
seems to be a nightmare, for both NFSv3 and NFSv4.

Moving from a NFSv4 server to a glusterfs server solved all of these 
problems, and sped up boot times significantly too. glusterfs looks 
intimidating at first, because of all its fancy replicating features and 
what-not, but it turns out to be trivially easy to set up a simple 
server than will replace 95% of the NFS installations out there.

Luckily, you can easily point both the NFS daemon and the glusterfs 
daemon at the same export folder, so you can migrate clients slowly over 
time.

This HOW-TO is intended to document the process. There are other similar 
HOW-TOs out there, but they are all a little out-of-date or don't show 
how to enable locking correctly, which is critical for Firefox.

In this example, we export the server's /fileserver folder, and mount it 
on /fileserver on the clients. In my server, /fileserver was already 
being served by the NFSv4 server, which is fine.

--------------------------------------------
1. On the server:
- yum install glusterfs-server

--------------------------------------------
2. On each client:
- yum install glusterfs-fuse
- mkdir /etc/glusterfs/
- mkdir /fileserver

--------------------------------------------
3. On the server, edit the volume configuration file
(/etc/glusterfs/glusterfsd.vol) so that it looks like this:

volume raw
   type storage/posix
   option directory /fileserver
end-volume

volume brick
   type features/posix-locks
   subvolumes raw
end-volume

volume server
   type protocol/server
   option transport-type tcp
   subvolumes brick
   option auth.addr.brick.allow *
end-volume


The first stanza selects the basic folder to export.

The second stanza adds file locking to it. This is required to support 
Firefox, and some other applications.

The third stanza authorizes everyone to access this file-locked export 
over the network.

There is also a /etc/glusterfs/glusterfsd.vol file on the system, for 
configuring the management interface. For this simple installation it 
does not need to be modified.

--------------------------------------------
4. Restart the server services:
- service glusterd restart
- service glusterfsd restart

I believe the first service is a management service, and the second is 
the actual file-export service.
--------------------------------------------

--------------------------------------------
5. On the client, create the /etc/glusterfs/glusterfs.vol configuration 
file, which should look like this:

volume client
   type protocol/client
   option transport-type tcp
   option remote-host 192.168.0.3  # use YOUR server IP here
   option remote-subvolume brick
end-volume


--------------------------------------------
6. On the client, add this line to the end of /etc/rc.d/rc.local:

mount -t glusterfs /etc/glusterfs/glusterfs.vol /fileserver

--------------------------------------------
7. On the client, reboot, and check /var/log/messages for errors. On one 
machine, we had an selinux problem that was flagged in the logs. We had 
to manually create the logging file using:

touch /var/log/glusterfs/fileserver.log; reboot

--------------------------------------------
8. On the client, see if you can access the files in /fileserver. If 
not, read the /var/log/glusterfs/* files on both the client and the server.

At this point, everything should work!



Weird things and gotchas:

A. You need the file-locking option to make Firefox work properly.

B. LibreOffice wouldn't start on one system, until we did:
rm ~/.libreoffice
rm ~/.openoffice.org

C. selinux prevented the creation of log files on one client, which 
prevented the filesystem from mouting. The manual fix noted above fixed 
that.

D. This HOWTO mounts the glusterfs from /etc/rc.d/rc.local, which is the 
last step in the boot process. In theory, you can mount it from 
/etc/fstab or using autofs. However, we found that autofs mounting just 
didn't work - not sure why. fstab mounting didn't work either - I 
suspect it occurred too early in the boot process. /etc needs to be up 
and running so glusterfs can read the config file, and I don't think the 
current init/systemd files handle this correctly. There are some Debian 
bug reports about this that you can google.


I hope this is useful to someone, and that we can finally drive a stake 
through the heart of NFS...


- Mike

[Dr. Michael J. Chudobiak]

On 10/06/2011 06:02 AM, Jan Engelhardt wrote:
> > 3. On the server, edit the volume configuration file
> > (/etc/glusterfs/glusterfsd.vol) so that it looks like this:
> >
> > volume raw
> >    type storage/posix
> >    option directory /fileserver
> > end-volume
> > [...]
>
>  Hm, the setup of volumes is very different from what is done in the
>  Gluster Administration Guide. The latter does not touch plain files at
>  all but consistently uses the gluster CLI. There is no need to deal with
>  "volume raw", "volume brick" or so, and above all there is no
>  modification to glusterfsd.vol.
>  Why the deviation? 
It's simple; it works. I was following various recipes collected from 
the web.

Also, I had a problem starting the management service (which provides 
the CLI functionality), so I couldn't use the CLI tools. I don't recall 
the exact problem now.

I guess you can configure things "statically" from the config files, or 
"dynamically" from the CLI. I like the config file approach, personally.

- Mike

[Jan Engelhardt]

> 3. On the server, edit the volume configuration file
> (/etc/glusterfs/glusterfsd.vol) so that it looks like this:
>
> volume raw
>    type storage/posix
>    option directory /fileserver
> end-volume
> [...] 
Hm, the setup of volumes is very different from what is done in the 
Gluster Administration Guide. The latter does not touch plain files at 
all but consistently uses the gluster CLI. There is no need to deal with 
"volume raw", "volume brick" or so, and above all there is no 
modification to glusterfsd.vol.
Why the deviation?

[Dr. Michael J. Chudobiak]

On 08/25/2011 08:59 AM, Marcos Luis Ortiz Valmaseda wrote:
>  Excellent how to Dr.
>  Can you add this to the Fedora wiki? 
If someone wants to stick it in an appropriate place on a Fedora wiki, 
please go ahead. I'll help maintain and edit it.

- Mike

[Marcos Luis Ortiz Valmaseda]

Excellent how to Dr.
Can you add this to the Fedora wiki?

2011/8/25 Dr. Michael J. Chudobiak <mjc(a)avtechpulse.com&gt;

>  As detailed in another thread, we upgraded a few test machines on
> our
>  LAN to Fedora 15 (with gnome-shell and firefox), with user folders
>  served from a NFSv4 server (F14 originally, then F15).
>
>  It just didn't work. The F15 desktops would freeze frequently. And
>  worse, this would freeze ALL desktops on the LAN intermittently, as the
>  NFS server struggled with client flakiness.
>
>  When it did work, Firefox would lose authenticated logins randomly,
>  presumably due to corruption of its cookies.sqlite file. sqlite and NFS
>  seems to be a nightmare, for both NFSv3 and NFSv4.
>
>  Moving from a NFSv4 server to a glusterfs server solved all of these
>  problems, and sped up boot times significantly too. glusterfs looks
>  intimidating at first, because of all its fancy replicating features and
>  what-not, but it turns out to be trivially easy to set up a simple
>  server than will replace 95% of the NFS installations out there.
>
>  Luckily, you can easily point both the NFS daemon and the glusterfs
>  daemon at the same export folder, so you can migrate clients slowly over
>  time.
>
>  This HOW-TO is intended to document the process. There are other similar
>  HOW-TOs out there, but they are all a little out-of-date or don't show
>  how to enable locking correctly, which is critical for Firefox.
>
>  In this example, we export the server's /fileserver folder, and mount it
>  on /fileserver on the clients. In my server, /fileserver was already
>  being served by the NFSv4 server, which is fine.
>
>  --------------------------------------------
>  1. On the server:
>  - yum install glusterfs-server
>
>  --------------------------------------------
>  2. On each client:
>  - yum install glusterfs-fuse
>  - mkdir /etc/glusterfs/
>  - mkdir /fileserver
>
>  --------------------------------------------
>  3. On the server, edit the volume configuration file
>  (/etc/glusterfs/glusterfsd.vol) so that it looks like this:
>
>  volume raw
>    type storage/posix
>    option directory /fileserver
>  end-volume
>
>  volume brick
>    type features/posix-locks
>    subvolumes raw
>  end-volume
>
>  volume server
>    type protocol/server
>    option transport-type tcp
>    subvolumes brick
>    option auth.addr.brick.allow *
>  end-volume
>
>
>  The first stanza selects the basic folder to export.
>
>  The second stanza adds file locking to it. This is required to support
>  Firefox, and some other applications.
>
>  The third stanza authorizes everyone to access this file-locked export
>  over the network.
>
>  There is also a /etc/glusterfs/glusterfsd.vol file on the system, for
>  configuring the management interface. For this simple installation it
>  does not need to be modified.
>
>  --------------------------------------------
>  4. Restart the server services:
>  - service glusterd restart
>  - service glusterfsd restart
>
>  I believe the first service is a management service, and the second is
>  the actual file-export service.
>  --------------------------------------------
>
>  --------------------------------------------
>  5. On the client, create the /etc/glusterfs/glusterfs.vol configuration
>  file, which should look like this:
>
>  volume client
>    type protocol/client
>    option transport-type tcp
>    option remote-host 192.168.0.3  # use YOUR server IP here
>    option remote-subvolume brick
>  end-volume
>
>
>  --------------------------------------------
>  6. On the client, add this line to the end of /etc/rc.d/rc.local:
>
>  mount -t glusterfs /etc/glusterfs/glusterfs.vol /fileserver
>
>  --------------------------------------------
>  7. On the client, reboot, and check /var/log/messages for errors. On one
>  machine, we had an selinux problem that was flagged in the logs. We had
>  to manually create the logging file using:
>
>  touch /var/log/glusterfs/fileserver.log; reboot
>
>  --------------------------------------------
>  8. On the client, see if you can access the files in /fileserver. If
>  not, read the /var/log/glusterfs/* files on both the client and the server.
>
>  At this point, everything should work!
>
>
>
>  Weird things and gotchas:
>
>  A. You need the file-locking option to make Firefox work properly.
>
>  B. LibreOffice wouldn't start on one system, until we did:
>  rm ~/.libreoffice
>  rm ~/.openoffice.org
>
>  C. selinux prevented the creation of log files on one client, which
>  prevented the filesystem from mouting. The manual fix noted above fixed
>  that.
>
>  D. This HOWTO mounts the glusterfs from /etc/rc.d/rc.local, which is the
>  last step in the boot process. In theory, you can mount it from
>  /etc/fstab or using autofs. However, we found that autofs mounting just
>  didn't work - not sure why. fstab mounting didn't work either - I
>  suspect it occurred too early in the boot process. /etc needs to be up
>  and running so glusterfs can read the config file, and I don't think the
>  current init/systemd files handle this correctly. There are some Debian
>  bug reports about this that you can google.
>
>
>  I hope this is useful to someone, and that we can finally drive a stake
>  through the heart of NFS...
>
>
>  - Mike
>  --
>  users mailing list
>  users(a)lists.fedoraproject.org
>  To unsubscribe or change subscription options:
>  https://admin.fedoraproject.org/mailman/listinfo/users
>  Guidelines: http://fedoraproject.org/wiki/Mailing_list_guidelines
>  


-- 
-- 
Marcos Luis Ortíz Valmaseda
 Software Engineer (UCI)
 Linux User # 418229
 http://marcosluis2186.posterous.com
 http://www.linkedin.com/in/marcosluis2186
 https://fedoraproject.org/wiki/User:Marcosluis