Use iSCSI for remote /home?

[Richard Shaw]

I have a Nvidia Jetson Nano w/ Fedora 33 Workstation I'm using for some
aarch64 package development.

Right now I have duplicate users on it and use scp to push files over to it
but had a thought...

Could I completely replace /home with my desktop F33 system using iSCSI?

I like this idea for two reasons:

1. I would have access to the identical files on both systems without
having to scp files around, including not having to do duplicate "fedpkg
clone...".

2. A lot of the time a package is building the CPU is waiting on the SD
card even though it's a class 10. I'm betting IO would be faster over my
gigabit Ethernet. AND save all the writing to the SD card, extending it's
life.

Thoughts?

Thanks,
Richard

[Qiyu Yan]

Why not try nfs, which would be easier to set up?

[Jorge Fábregas]

On 12/25/20 10:44 AM, Richard Shaw wrote:
>  Could I completely replace /home with my desktop F33 system using
>  iSCSI? 
Hi,

Assuming you'll have /home mounted on both systems *at the same
time* then you can't use iSCSI unless you use a cluster filesystem like
GFS2 or OCFS2 on top of the LUN (plus you'll need some clusterware with
that).

If you really want to use iSCSI then you won't be able to mount the
filesystem on both systems at the same time (technically you could but
you *will* corrupt the filesystem). You just need to make sure you have
it unmounted in one system before mounting it in the other... (don't
even attempt to mount it read-only in
one side while mounted in the other side..)

If you're using a standard filesystem like ext4, xfs or btrfs then you
need to work a the file-level layer  (not block-level like you suggest with
iSCSI). I suggest you use NFS v4.1.  You simply ake an export (an NFS
share) on the desktop and import in the other side.

-- 
Jorge

[John Mellor]

On 2020-12-25 9:44 a.m., Richard Shaw wrote:
>  Right now I have duplicate users on it and use scp to push files over
>
>  to it but had a thought...
>
>  Could I completely replace /home with my desktop F33 system using iSCSI?
>
>  I like this idea for two reasons:
>
>  1. I would have access to the identical files on both systems without 
>  having to scp files around, including not having to do duplicate 
>  "fedpkg clone...".
>
>  2. A lot of the time a package is building the CPU is waiting on the 
>  SD card even though it's a class 10. I'm betting IO would be faster 
>  over my gigabit Ethernet. AND save all the writing to the SD card, 
>  extending it's life.
>  
Sure, why not?  Three caveats:

1. iSCSI is extremely sensitive to latency and somewhat sensitive to 
bandwidth, so make sure they are network "close".

2. The UID and GID will be important in this setup, so make sure that 
they are the same for the user(s) that you want to have common on the 
two machines.

3. The selinux policy for the home disk had better be pretty much the 
same for the two machines.  Any differences between the capabilities on 
disk will produce strange blocking behaviours.

--

John Mellor

[Richard Shaw]

Thanks for all the input!

Based on the feedback, I settled on NFS for /home sharing because I
really just want to share files, but my point #2 still stands.

I may try setting up a dedicated /var/lb/mock using iSCSI to prevent SD
card degradation and improve build performance.

Thanks,
Richard

[Cameron Simpson]

On 25Dec2020 13:53, Richard Shaw <hobbes1069(a)gmail.com&gt; wrote:
> Based on the feedback, I settled on NFS for /home sharing because I
> really just want to share files, [...] 
I just want to reiterate what Jorge Fábregas said: DO NOT use iscsi to 
_share_ a filesystem. I know you're not, but it isn't clear to me that 
you realise that this is a direct recipe for corruption and disaster.

A filesystem expects sole access to whatever block device it stores its 
data on. Two systems with the same block device mounted (i.e. via iscsi) 
_will_ walk all over its data structures without coordination, and 
_will_ mangle your data.

NFS or equivalent is definitely the go: you want _file_ level sharing, 
not _block_ level sharing, because that way the system managing "files" 
out of a block device is just one system (the remote system holding the 
device).

An NFS (or equivalent - cifs, sshfs etc) mount is essentially a file 
based proxy to a remote mounted tree.

Cheers,
Cameron Simpson <cs(a)cskk.id.au&gt;

[Chris Murphy]

On Fri, Dec 25, 2020 at 7:45 AM Richard Shaw <hobbes1069(a)gmail.com&gt; wrote:

>  2. A lot of the time a package is building the CPU is waiting on the
> SD card even though it's a class 10. I'm betting IO would be faster over my
> gigabit Ethernet. AND save all the writing to the SD card, extending it's life.
USB stick and SD Card flash media are simultaneously remarkably cool
technologies, very inexpensive, but also unreliable. They have
firmware on them just like any other SSD, except it's quite
unsophisticated. I've never seen either one report a read error, they
just return garbage or zeros when blocks begin to fail. And these
failures can even be transient. The write performance is pretty
horrible too. Even the ones that can do decent writes are designed for
single queue, heavy sequential writes like you'd expect from a camera.
They are horrible at random writes, or even a mix of random and
sequential.

And yes they are much more write sensitive than non-removable SSD and
NVMe. They will eventually fail. So far I go through a USB stick or SD
Card used as sysroot about once every 7 months. They're that bad. HN
is full of these kinds of experiences, and the best work around is to
get so called "industrial" SD Cards specifically for the embedded use
case. There's nothing that special about them other than they're using
more expensive single layer NAND with higher wear capacity. So they
write even slower than the consumer SSDs but they last a lot longer.

My workaround is I use Btrfs with compression, and with cheap
snapshots and btrfs send/receive it's easy to keep backups. Eventually
I'll start to see Btrfs report csum errors (corrupt data) and then I
know I'm getting close to the time when the media will pancake itself.
The most typical scenario I've seen is the media goes read-only. Not
the file system, but the media itself. Write to a block, no error.
Read that block, get back old data. File systems will get deeply
confused by this eventually. But the errors they report will be
non-obvious because all it's saying is "haha wut?! that ain't right!"
over and over again. I didn't figure out the media had gone read-only
until I did my own write and read tests independent from the file
system.

The first time it happened I even reformatted one of the partitions
with a different file system, wrote stuff, and no errors. Then
unmounted it, mounted it again, and saw it was the old file system and
old data. i.e. the media is accepting write commands without error,
basically saying "yeah I've got your data" but then failing to
actually commit it to the media. Without a single error reported. It's
really funny how bad the error reporting is for this class of media.
Plausibly I could write some important files to such a device and have
no idea at all it was not committed.


-- 
Chris Murphy