What is better a 2nd drive for Raid 1 or a backup one?

[Javier Perez]

Hi
I have my /home partition on a 2TB HDD drive about half full. Should be
doing regular backups but not in the habit.

What is better?
A. Purchase a second 2TB to create a Raid1 mirror or
B. Purchase a second 4TB drive for backup purposes.

If case B, should I get a USB3.0 adapter or similar so that I could just
plug the HDD, let it do the backup and unplug and store it? Or is it better
to install it on another networked pc and do it through the network?

Is there a better, easier, less troublesome option to keep the data safe
that does not involve the cloud?

This is a home network

Thanks


-- 
------------------------------
 /\_/\
 |O O|  pepebuho(a)gmail.com
 ~~~~     Javier Perez
 ~~~~          While the night runs
 ~~~~          toward the day...
  m m       Pepebuho watches
                from his high perch.

[Patrick O'Callaghan]

On Tue, 2019-12-24 at 14:47 -0500, Javier Perez wrote:
>  Hi
>  I have my /home partition on a 2TB HDD drive about half full. Should be
>  doing regular backups but not in the habit.
>  
>  What is better?
>  A. Purchase a second 2TB to create a Raid1 mirror or
>  B. Purchase a second 4TB drive for backup purposes. 
These are two different things with different objectives. RAID is not a
backup solution, it's a data-loss recovery solution (i.e. for hardware-
related problems). If you accidentally delete an important file, RAID
in itself is not going to help you recover it (though it might if you
have a journalled filesystem and take regular snapshots), but a backup
will do that if it's configured properly.

>  If case B, should I get a USB3.0 adapter or similar so that I could
> just
>  plug the HDD, let it do the backup and unplug and store it? Or is it better
>  to install it on another networked pc and do it through the network? 
Either. I use a NAS on my LAN, but it's an old machine with only
100Mbps Ethernet and an obsolete version of Debian. I don't want to buy
a new NAS so I'm seriously thinking of just using a local disk, which
would also be easier to configure.

Your preferred backup software may be important, e.g. s/w that assumes
the backup is on a directly-accesible filesystem can really hammer your
network if you just mount the remote drive with NFS, even if it's just
verifying that nothing has changed.

>  Is there a better, easier, less troublesome option to keep the data
> safe
>  that does not involve the cloud? 
You need to give more detail of your exact requirements, e.g. how much
data? How often do you want to back it up? How long do you want to keep
it? Are you backing up multiple machines or just one? If the former,
are some of these running Windows or other non-Linux software?

poc

[Javier Perez]

On Tue, Dec 24, 2019, 15:29 Patrick O'Callaghan <pocallaghan(a)gmail.com&gt;
wrote:

>  On Tue, 2019-12-24 at 14:47 -0500, Javier Perez wrote:
>  > Hi
>  > I have my /home partition on a 2TB HDD drive about half full. Should be
>  > doing regular backups but not in the habit.
>  >
>  > What is better?
>  > A. Purchase a second 2TB to create a Raid1 mirror or
>  > B. Purchase a second 4TB drive for backup purposes.
>
>  These are two different things with different objectives. 

>  Understood. But in order of importance which would go first? Seems to
> me 
like backup should take precedence over Mirroring the data if I look
forward to preserve it.


>  If case B, should I get a USB3.0 adapter or similar so that I could
> just
>  > plug the HDD, let it do the backup and unplug and store it? Or is it
>  better
>  > to install it on another networked pc and do it through the network?
>
>  Either. I use a NAS on my LAN, but it's an old machine with only
>  100Mbps Ethernet and an obsolete version of Debian. I don't want to buy
>  a new NAS so I'm seriously thinking of just using a local disk, which
>  would also be easier to configure.
>
>  Your preferred backup software may be important, e.g. s/w that assumes
>  the backup is on a directly-accesible filesystem can really hammer your
>  network if you just mount the remote drive with NFS, even if it's just
>  verifying that nothing has changed.
>
>  > Is there a better, easier, less troublesome option to keep the data safe
>  > that does not involve the cloud?
>
>  You need to give more detail of your exact requirements, e.g. how much
>  data? How often do you want to back it up? How long do you want to keep
>  it? Are you backing up multiple machines or just one? If the former,
>  are some of these running Windows or other non-Linux software?
>
>  poc
>  
Mine is an all linux network.
The data exist on two machines. Machine A has about 500GB
Machine B has about 1T but it is mostly my CD and DVD collection. if
something happens to it I can always rip it again (assuming the original
media did not go kaput).
Machine B is a "like to have". Machine A I'd rather try not to lose.
JP

_______________________________________________
>  users mailing list -- users(a)lists.fedoraproject.org
>  To unsubscribe send an email to users-leave(a)lists.fedoraproject.org
>  Fedora Code of Conduct:
>  https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>  List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives:
>  https://lists.fedoraproject.org/archives/list/[e-mail]
>

[Gordon Messmer]

On 12/24/19 11:47 AM, Javier Perez wrote:
>  I have my /home partition on a 2TB HDD drive about half full. Should
>
>  be doing regular backups but not in the habit.
>
>  What is better?
>  A. Purchase a second 2TB to create a Raid1 mirror or
>  B. Purchase a second 4TB drive for backup purposes. 

That depends on whether you want a) that computer to continue working 
without interruption if a hard drive fails, or b) you want to be able to 
rebuild the computer if a hard drive fails, or to be able to recover 
data if it's accidentally removed, or to restore an old version of data 
for any reason.

If you don't have regular, scheduled, automatic backups, then I would 
personally prioritize B much higher than A.


>  If case B, should I get a USB3.0 adapter or similar so that I could 
>  just plug the HDD, let it do the backup and unplug and store it? Or is 
>  it better to install it on another networked pc and do it through the 
>  network? 

A directly-attached backup can be deleted by some types of accidental 
removal of the data you want ("rm -rf /*").  A computer system should 
have at least one backup that isn't mounted in the local filesystem.


>  Is there a better, easier, less troublesome option to keep the data 
>  safe that does not involve the cloud? 

Backups are probably the only purpose where I recommend "the cloud" 
without reservation.  Storage costs are typically lower than you'll get 
on purchases of your own, and backups are more likely to survive events 
like fire and theft.

[Cameron Simpson]

On 24Dec2019 14:47, Javier Perez <pepebuho(a)gmail.com&gt; wrote:
> I have my /home partition on a 2TB HDD drive about half full. Should
> be
> doing regular backups but not in the habit.
>
> What is better?
> A. Purchase a second 2TB to create a Raid1 mirror or
> B. Purchase a second 4TB drive for backup purposes.
>
> If case B, should I get a USB3.0 adapter or similar so that I could just
> plug the HDD, let it do the backup and unplug and store it? Or is it better
> to install it on another networked pc and do it through the network?
>
> Is there a better, easier, less troublesome option to keep the data safe
> that does not involve the cloud?
>
> This is a home network 
As pointed out, these are 2 different requirements. My recommendation: 
do both. (Yes, more $s, alas.)

Let me describe our home setup.

Our personal machines are Macs which we backup with TimeMachine to 
drives which generally live in a drawer (i.e. not online, and less 
visible to theft).

Our home server and media machine is an Ubuntu Linux box. The main 
storage area is a pair of 8TB drives in RAID1. We like RAID1 because it 
means that a single drive works standalone, even without RAID - RAID5 
etc distributes the data more complexly. This makes RAID1 handy for 
recovery in another machine (eg plugged into a USB cradle).

If you have RAID1 I recommend you have a cron job to monitor it. I run 
my chkmdstat script once every 5 minutes; it is silent when things are 
good and thus generates an email if a drive fails:

  https://bitbucket.org/cameron_simpson/css/src/tip/bin/chkmdstat

We back up the main area to external USB3 drives (we like WD 
MyPassports, but there are equivalent products for similar prices; a 
nice USB powered 2TB or 4TB drive). We keep 2 backups: an "online" one 
which is updated regularly and an "offline" one which lives in a drawer 
where accidents won't happen to it.

You'll notice that 2TB < 8TB. (WD now sell a 4TB MyPassport, but that is 
still <8TB.)

We divide the main area into different areas for backup (I'm in the 
midst of changing this to use a tagging scheme instead of directory 
layout) and backup some areas to backup A, some to backup B and so 
forth. This manages the space requirement. It sounds like your home 
machine is small enough to fit in a single external drive, so you are 
spared this pain.

I use histbackup for these backups; it is an rsync wrapper which keeps 
multiple historic backups as hardlinked trees named after the backup 
date.  So a new backup goes: hard link the last tree to the new tree and 
rsync into it. Since rsync makes new files for changes this keeps the 
hard links for unchanged files and makes shiny new files for changes, so 
the cost is incremental.

Histbackup is here:

  https://bitbucket.org/cameron_simpson/css/src/tip/bin/histbackup

There are plenty of other similar tools.

These days, to reduce the likelihood of accidental damage to the backups 
I like to have the "A1" backup plugged in but not mounted ("A2" is the

second copy which lives in a drawer).  This also has the advantage that 
provided a backup is not running you are free to unplug or plug in the 
external drive at will, because it is unmounted.

The backup script has the form:

    if mount /mnt/backupA
    then
        histbackup into /mnt/backupA/area from /app8tb/area
        umount /mnt/backupA
    fi

(Obviously the real script reads mountpoints and areas in a loop to do 
several backups.)

My fstab has lines for each backup drive, eg:

    LABEL=ARCHIVE_1A       /mnt/archive1_a       xfs     user,noauto,noatime        0 0

so that the mounts work; you just have to label the USB drive 
filesystems when you make them. I also put a little paper sticker on 
each drive to describe it.

Finally, if you have multiple external drives (eg multiple backup drives 
for space reasons, or both the 1 and 2 copies plugged in at once) I 
recommend doing the backups in series, not in parallel; I've a little 4 
port USB4 hub and have had bad experiences trying to run both drives at 
once - might be lack of power, might be dodgy hardware, might be USB 
driver issues. Anyway, I/O errors ensued.

Cheers,
Cameron Simpson <cs(a)cskk.id.au&gt;

[Patrick O'Callaghan]

On Wed, 2019-12-25 at 08:51 +1100, Cameron Simpson wrote:
>  Histbackup is here:
>  
>    https://bitbucket.org/cameron_simpson/css/src/tip/bin/histbackup
>  
>  There are plenty of other similar tools. 
I use rsnapshot, which seems quite similar. Any thoughts on pros/cons
versus histbackup?

poc

[Cameron Simpson]

On 25Dec2019 00:22, Patrick O'Callaghan <pocallaghan(a)gmail.com&gt; wrote:
> On Wed, 2019-12-25 at 08:51 +1100, Cameron Simpson wrote:
> > Histbackup is here:
> >
> >   https://bitbucket.org/cameron_simpson/css/src/tip/bin/histbackup
> >
> > There are plenty of other similar tools.
>
> I use rsnapshot, which seems quite similar. Any thoughts on pros/cons
> versus histbackup? 
I wrote histbackup, so it does what I want with some handy (to me) 
features (symlink-if-unchanged, "named" symlink to latest backup and 
specific backups, second pass if we hit the hardlink limit, etc).

I haven't an opinion about rsnapshot, as I've not had cause to use it.

Is there anything you wish rsnapshot did?

Cheers,
Cameron Simpson <cs(a)cskk.id.au&gt;

[Patrick O'Callaghan]

On Wed, 2019-12-25 at 13:29 +1100, Cameron Simpson wrote:
>  On 25Dec2019 00:22, Patrick O'Callaghan
> <pocallaghan(a)gmail.com&gt; wrote:
>  > On Wed, 2019-12-25 at 08:51 +1100, Cameron Simpson wrote:
>  > > Histbackup is here:
>  > > 
>  > >   https://bitbucket.org/cameron_simpson/css/src/tip/bin/histbackup
>  > > 
>  > > There are plenty of other similar tools.
>  > 
>  > I use rsnapshot, which seems quite similar. Any thoughts on pros/cons
>  > versus histbackup?
>  
>  I wrote histbackup, so it does what I want with some handy (to me) 
>  features (symlink-if-unchanged, "named" symlink to latest backup and 
>  specific backups, second pass if we hit the hardlink limit, etc).
>  
>  I haven't an opinion about rsnapshot, as I've not had cause to use it.
>  
>  Is there anything you wish rsnapshot did? 
Not really. The documentation could be a little clearer in some
respects but it makes sense when you understand the implicit assumption
that you're pulling files from the backup client to the target rather
than the other way round. This makes a huge difference when using NFS
(by about 10 to 1 in completion times in my case) but once over that
hurdle I just let it run.

poc

[Tim]

On Wed, 2019-12-25 at 08:51 +1100, Cameron Simpson wrote:
>  if you have multiple external drives (eg multiple backup drives 
>  for space reasons, or both the 1 and 2 copies plugged in at once) I 
>  recommend doing the backups in series, not in parallel; I've a little
>  4 port USB4 hub and have had bad experiences trying to run both
>  drives at once - might be lack of power, might be dodgy hardware,
>  might be USB driver issues. Anyway, I/O errors ensued. 
I've always had foulups when doing lots of data across USB (especially
when that was masses of small files).  Partway through something would
stuff up, transfers would jam or abort, and there'd be a pile of
corrupted data.  I've come to a few conclusions:

Drives powered from your computer's USB socket are a bad idea.  Drives
with their own good quality power supplies are better (though there's a
lot with crappy supplies).

Drives in tiny cases, drives with bad heat dissipation, are bad news. 
The drives overheat when continuously churning data through for a
while.

If I ever had to copy a lot of data from one USB drive to another USB
drive, that nearly always ended in tears.  I don't think some computers
are capable of prolonged throughput of data across USB.
 
-- 
 
uname -rsvp
Linux 3.10.0-1062.9.1.el7.x86_64 #1 SMP Fri Dec 6 15:49:49 UTC 2019 x86_64
 
Boilerplate:  All unexpected mail to my mailbox is automatically deleted.
I will only get to see the messages that are posted to the mailing list.

[Ed Greshko]

On 2019-12-25 05:51, Cameron Simpson wrote:
>  We keep 2 backups: an "online" one which is updated
> regularly and an "offline" one which lives in a drawer
> where accidents won't happen to it.  
What does "online" (in quotes) mean?

I ask since, until now, I don't recall anyone mentioning the 2 most important words
when it comes
to backups.  "Off-Site".

I know the chances are small, but not zeo, that one's house could catch fire or suffer
another catastrophe.
But if it does, that "drive in a drawer" is not likely to survive.

-- 
The key to getting good answers is to ask good questions.

[Cameron Simpson]

On 28Dec2019 07:32, Ed Greshko <ed.greshko(a)greshko.com&gt; wrote:
> On 2019-12-25 05:51, Cameron Simpson wrote:
> > We keep 2 backups: an "online" one which is updated regularly and an
> "offline" one which lives in a drawer where accidents won't happen to it.
>
> What does "online" (in quotes) mean? 
Unmounted, but routinely plugged in so that it can be updated with cron, 
or just on a whim.

The "offline" one lives in a drawer, as a hedge against accidental 
damage and casual theft. The scenario here, aside from accidents, is 
malicious programmes when eat not just your primary data but accessable 
backups.

> I ask since, until now, I don't recall anyone mentioning the 2
> most important words when it comes
> to backups.  "Off-Site". 
Oh, aye. Something i need to fix. My source repo is off site, but the 
bulky stuff isn't.

> I know the chances are small, but not zeo, that one's house could
> catch fire or suffer another catastrophe.
> But if it does, that "drive in a drawer" is not likely to survive. 
Correct.

I had neighbours once who worked from home. They were almost always 
around.  However, they were renting and the owner put the place on the 
market, and thus put interior shots on websites. During that time, in a 
window of about an hour when they were both out, they got cleaned out, 
including their backup drives.

Cheers,
Cameron Simpson <cs(a)cskk.id.au&gt;

[Cameron Simpson]

Oh yes, one more thing. If you do the RAID1 thing: either make a shiny 
new RAID1 and copy to it, or practice the transition with test drives.  
Do not risk your high value data by trying to "in place RAID1" it.

Cheers,
Cameron Simpson <cs(a)cskk.id.au&gt;

[Javier Perez]

Thanks !

On Tue, Dec 24, 2019, 16:56 Cameron Simpson <cs(a)cskk.id.au&gt; wrote:

>  Oh yes, one more thing. If you do the RAID1 thing: either make a
> shiny
>  new RAID1 and copy to it, or practice the transition with test drives.
>  Do not risk your high value data by trying to "in place RAID1" it.
>
>  Cheers,
>  Cameron Simpson <cs(a)cskk.id.au&gt;
>  _______________________________________________
>  users mailing list -- users(a)lists.fedoraproject.org
>  To unsubscribe send an email to users-leave(a)lists.fedoraproject.org
>  Fedora Code of Conduct:
>  https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>  List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives:
>  https://lists.fedoraproject.org/archives/list/[e-mail]
>

[Chris Murphy]

On Tue, Dec 24, 2019 at 2:56 PM Cameron Simpson <cs(a)cskk.id.au&gt; wrote:
>  Oh yes, one more thing. If you do the RAID1 thing: either make a shiny
>  new RAID1 and copy to it, or practice the transition with test drives.
>  Do not risk your high value data by trying to "in place RAID1" it. 
I'm not sure if this is even possible with mdadm or lvm. For sure they
have no way of knowing which mirror is correct. But even if it's
possible, it's a bit complicated because it implies repartitioning in
order to make room for the necessary metadata area.

Whereas converting a Btrfs volume from single to raid1 uses balance
code, so it isn't any riskier than a full balance.


-- 
Chris Murphy

[Roberto Ragusa]

On 12/26/19 10:54 PM, Chris Murphy wrote:
>  On Tue, Dec 24, 2019 at 2:56 PM Cameron Simpson <cs(a)cskk.id.au&gt;
> wrote:
> >
> > Oh yes, one more thing. If you do the RAID1 thing: either make a shiny
> > new RAID1 and copy to it, or practice the transition with test drives.
> > Do not risk your high value data by trying to "in place RAID1" it.
>  
>  I'm not sure if this is even possible with mdadm or lvm. For sure they
>  have no way of knowing which mirror is correct. But even if it's
>  possible, it's a bit complicated because it implies repartitioning in
>  order to make room for the necessary metadata area. 
The "which copy is correct" problem is solvable:
1) you can create a 1-disk RAID1 on the partition with the correct data
(it will tell you your config is stupid but you can force it)
2) you then tell mdadm to change the number of drives to 2 (mdadm --grow);
this will be a 2-disk RAID1 working in 1-disk degraded mode
3) you then tell mdadm you have a new drive for that RAID1 (mdadm --add);
sync from 1st disk to 2nd will begin

The metadata area problem is a bit tricky, but it is not necessary
to repartition, just make your filesystem a bit smaller than it is now.
So:
a) unmount the filesystem
b) resize the filesystem to 100MB smaller (actually you need just a few kB,
but let's play very very safe, we will get the space back later)
c) create the RAID etc., the steps described above, so 1) 2) 3)
d) resize the filesystem without any size parameter (i.e. let it expand to
occupy the 99.9MB of extra space you have on the RAID device)

In any phase after 1) you can also mount the filesystem again (from /dev/md*),
since all the rest can be done on a mounted filesystem (no problem with d) too).

There are some things to notice:
- step b) requires a filesystem that supports shrinking; this can be done with
ext4 but it is not supported on xfs (BTW, this is why I refuse to consider xfs
a serious filesystem)
- step d) can be done on both a mounted or unmounted filesystem for ext4
but can only be done on a mounted fileystem ox xfs (another reason why I do not
like xfs)
- RAID creation in step 1) must be done with a --metadata option that forces the
metadata at the END of the space (so 0.90 or 1.0), since you are not going
to shift all your data forward to have space at the beginning

At the end of day, it can be done, but you really have to know what you are doing,
a small error can lead to a disaster. I would do that only on data I have a backup
already, or at least I would try all the procedure on a small test filesystem
before doing it on the real stuff.

Regards.

-- 
    Roberto Ragusa    mail at robertoragusa.it

[Chris Murphy]

On Sat, Dec 28, 2019 at 2:01 PM Roberto Ragusa <mail(a)robertoragusa.it&gt; wrote:
>  On 12/26/19 10:54 PM, Chris Murphy wrote:
>  > On Tue, Dec 24, 2019 at 2:56 PM Cameron Simpson <cs(a)cskk.id.au&gt; wrote:
>  >>
>  >> Oh yes, one more thing. If you do the RAID1 thing: either make a shiny
>  >> new RAID1 and copy to it, or practice the transition with test drives.
>  >> Do not risk your high value data by trying to "in place RAID1" it.
>  >
>  > I'm not sure if this is even possible with mdadm or lvm. For sure they
>  > have no way of knowing which mirror is correct. But even if it's
>  > possible, it's a bit complicated because it implies repartitioning in
>  > order to make room for the necessary metadata area.
>
>  The "which copy is correct" problem is solvable:
>  1) you can create a 1-disk RAID1 on the partition with the correct data
>  (it will tell you your config is stupid but you can force it)
>  2) you then tell mdadm to change the number of drives to 2 (mdadm --grow);
>  this will be a 2-disk RAID1 working in 1-disk degraded mode
>  3) you then tell mdadm you have a new drive for that RAID1 (mdadm --add);
>  sync from 1st disk to 2nd will begin
>
>  The metadata area problem is a bit tricky, but it is not necessary
>  to repartition, just make your filesystem a bit smaller than it is now.
>  So:
>  a) unmount the filesystem
>  b) resize the filesystem to 100MB smaller (actually you need just a few kB,
>  but let's play very very safe, we will get the space back later)
>  c) create the RAID etc., the steps described above, so 1) 2) 3)
>  d) resize the filesystem without any size parameter (i.e. let it expand to
>  occupy the 99.9MB of extra space you have on the RAID device) 
This requires mdadm metadata format 1.0 being explicitly chosen when
creating the array. The default format, 1.2, uses a 4K offset from the
start, the mdadm superblock is 256 bytes, and then there's a ~65MiB
gap before the start of the array.


>  In any phase after 1) you can also mount the filesystem again (from
> /dev/md*),
>  since all the rest can be done on a mounted filesystem (no problem with d) too).
>
>  There are some things to notice:
>  - step b) requires a filesystem that supports shrinking; this can be done with
>  ext4 but it is not supported on xfs (BTW, this is why I refuse to consider xfs
>  a serious filesystem)
>  - step d) can be done on both a mounted or unmounted filesystem for ext4
>  but can only be done on a mounted fileystem ox xfs (another reason why I do not
>  like xfs)
>  - RAID creation in step 1) must be done with a --metadata option that forces the
>  metadata at the END of the space (so 0.90 or 1.0), since you are not going
>  to shift all your data forward to have space at the beginning
>
>  At the end of day, it can be done, but you really have to know what you are doing,
>  a small error can lead to a disaster. I would do that only on data I have a backup
>  already, or at least I would try all the procedure on a small test filesystem
>  before doing it on the real stuff. 
It could be easier and safer to do this with LVM: pvcreate-> vgextend
-> lvconvert.

However, by default, the installer uses all the VG space for root,
home, and swap. And LVM itself doesn't hold any space in reserve for
future use by lvconvert, not for convert from thick to thin
provisioning, or from linear to raid type. The lvconvert to type raid1
needs to create metadata subvolumes on each physical device or the
lvconvert command fails. So yeah you probably end up needing to do an
fs shrink here too, and it rapidly gets esoteric. You could swapoff,
blow away the swap LV, do the lvconvert, and create a new (very
slightly smaller) swap LV and format it - that way you don't have to
unmount any ext4 volumes.

In comparison, this is a lot more straightforward on Btrfs.

-- 
Chris Murphy

[John M. Harris Jr.]

On Sunday, December 29, 2019 12:54:48 PM MST Chris Murphy wrote:
>  On Sat, Dec 28, 2019 at 2:01 PM Roberto Ragusa
> <mail(a)robertoragusa.it&gt;
>  wrote:
>  >
>  >
>  > On 12/26/19 10:54 PM, Chris Murphy wrote:
>  > 
>  > > On Tue, Dec 24, 2019 at 2:56 PM Cameron Simpson <cs(a)cskk.id.au&gt; wrote:
>  > > 
>  > >>
>  > >>
>  > >> Oh yes, one more thing. If you do the RAID1 thing: either make a shiny
>  > >> new RAID1 and copy to it, or practice the transition with test drives.
>  > >> Do not risk your high value data by trying to "in place RAID1"
> it.
>  > >
>  > >
>  > >
>  > > I'm not sure if this is even possible with mdadm or lvm. For sure they
>  > > have no way of knowing which mirror is correct. But even if it's
>  > > possible, it's a bit complicated because it implies repartitioning in
>  > > order to make room for the necessary metadata area.
>  >
>  >
>  >
>  > The "which copy is correct" problem is solvable:
>  > 1) you can create a 1-disk RAID1 on the partition with the correct data
>  > (it will tell you your config is stupid but you can force it)
>  > 2) you then tell mdadm to change the number of drives to 2 (mdadm
>  > --grow);
>  > this will be a 2-disk RAID1 working in 1-disk degraded mode
>  > 3) you then tell mdadm you have a new drive for that RAID1 (mdadm --add);
>  > sync from 1st disk to 2nd will begin
>  >
>  >
>  >
>  > The metadata area problem is a bit tricky, but it is not necessary
>  > to repartition, just make your filesystem a bit smaller than it is now.
>  > So:
>  > a) unmount the filesystem
>  > b) resize the filesystem to 100MB smaller (actually you need just a few
>  > kB, 
 but let's play very very safe, we will get the space back later)
>  > c) create the RAID etc., the steps described above, so 1) 2) 3)
>  > d) resize the filesystem without any size parameter (i.e. let it expand
>  > to
>  > occupy the 99.9MB of extra space you have on the RAID device)
>  
>  
>  This requires mdadm metadata format 1.0 being explicitly chosen when
>  creating the array. The default format, 1.2, uses a 4K offset from the
>  start, the mdadm superblock is 256 bytes, and then there's a ~65MiB
>  gap before the start of the array.
>  
>  
>  
>  > In any phase after 1) you can also mount the filesystem again (from
>  > /dev/md*), 
 since all the rest can be done on a mounted filesystem (no
>  > problem with d) too).>
>  >
>  >
>  > There are some things to notice:
>  > - step b) requires a filesystem that supports shrinking; this can be done
>  > with 
 ext4 but it is not supported on xfs (BTW, this is why I refuse to
>  > consider xfs a serious filesystem)
>  > - step d) can be done on both a mounted or unmounted filesystem for ext4
>  > but can only be done on a mounted fileystem ox xfs (another reason why I
>  > do not 
 like xfs)
>  > - RAID creation in step 1) must be done with a --metadata option
> that
>  > forces the 
 metadata at the END of the space (so 0.90 or 1.0), since you
>  > are not going to shift all your data forward to have space at
> the
>  > beginning
>  >
>  >
>  >
>  > At the end of day, it can be done, but you really have to know what you
>  > are doing, 
 a small error can lead to a disaster. I would do that only on
>  > data I have a backup already, or at least I would try all the
> procedure
>  > on a small test filesystem before doing it on the real stuff.
>  
>  
>  It could be easier and safer to do this with LVM: pvcreate-> vgextend
>  -> lvconvert.
>  
>  However, by default, the installer uses all the VG space for root,
>  home, and swap. And LVM itself doesn't hold any space in reserve for
>  future use by lvconvert, not for convert from thick to thin
>  provisioning, or from linear to raid type. The lvconvert to type raid1
>  needs to create metadata subvolumes on each physical device or the
>  lvconvert command fails. So yeah you probably end up needing to do an
>  fs shrink here too, and it rapidly gets esoteric. You could swapoff,
>  blow away the swap LV, do the lvconvert, and create a new (very
>  slightly smaller) swap LV and format it - that way you don't have to
>  unmount any ext4 volumes.
>  
>  In comparison, this is a lot more straightforward on Btrfs.
>  
>  -- 
>  Chris Murphy 
If that's the case for LVM, then it seems that mdadm would be the easier and 
safer option. You simply create an array with a missing disk, copy the data 
over, then add the existing disk to the array.

- -
John M. Harris, Jr.

[Bruno Wolff III]

On Tue, Dec 31, 2019 at 17:05:41 -0700,
  "John M. Harris Jr." <johnmh(a)splentity.com&gt; wrote:
> If that's the case for LVM, then it seems that mdadm would be the easier and
> safer option. You simply create an array with a missing disk, copy the data
> over, then add the existing disk to the array. 
If you copy block by block, you still need to make sure you have room for 
the metadata and the filesystem. If you are thinking about using cp, then 
you need to worry about sparse and special files and maybe hard links.

Note you can also use partitions for arrays rather than a whole disk.

Also note that raid doesn't replace backups. There are others ways to 
wreck your data than having a disk drive fail.

[John M. Harris Jr]

On Thursday, January 2, 2020 9:47:44 AM MST Bruno Wolff III wrote:
>  On Tue, Dec 31, 2019 at 17:05:41 -0700,
>  
>    "John M. Harris Jr." <johnmh(a)splentity.com&gt; wrote:
>  >If that's the case for LVM, then it seems that mdadm would be the easier
>  >and safer option. You simply create an array with a missing disk, copy the
>  >data over, then add the existing disk to the array.
>  
>  If you copy block by block, you still need to make sure you have room for
>  the metadata and the filesystem. If you are thinking about using cp, then
>  you need to worry about sparse and special files and maybe hard links. 
The way I'd personally handle it would depend on how the original system was 
partitioned. For example, you can safely take a few MiB off a new swap 
partition, or shrink a filesystem.

>  Note you can also use partitions for arrays rather than a whole
> disk.
>  
>  Also note that raid doesn't replace backups. There are others ways to
>  wreck your data than having a disk drive fail. 
I mentioned this in my earlier reply to the thread.

-- 
John M. Harris, Jr.
Splentity

[John M. Harris Jr.]

On Thursday, December 26, 2019 2:54:25 PM MST Chris Murphy wrote:
>  On Tue, Dec 24, 2019 at 2:56 PM Cameron Simpson <cs(a)cskk.id.au&gt;
> wrote:
>  
>  >
>  >
>  > Oh yes, one more thing. If you do the RAID1 thing: either make a shiny
>  > new RAID1 and copy to it, or practice the transition with test drives.
>  > Do not risk your high value data by trying to "in place RAID1" it.
>  
>  
>  I'm not sure if this is even possible with mdadm or lvm. For sure they
>  have no way of knowing which mirror is correct. But even if it's
>  possible, it's a bit complicated because it implies repartitioning in
>  order to make room for the necessary metadata area.
>  
>  Whereas converting a Btrfs volume from single to raid1 uses balance
>  code, so it isn't any riskier than a full balance. 
It is possible with mdadm, and is even easier with lvm as you're just 
converting to a mirror. With mdadm, you'd create the RAID1 with 'missing' for

the first disk, and the disk itself for the 2nd. It's not risky if you do it 
this way.

[Sam Varshavchik]

Javier Perez writes:

>  Hi
>  I have my /home partition on a 2TB HDD drive about half full. Should be doing  
>  regular backups but not in the habit.
>
>
>
>  What is better?
>  A. Purchase a second 2TB to create a Raid1 mirror or
>  B. Purchase a second 4TB drive for backup purposes. 
Since you already have an existing partition in use, using another drive for  
basic backup will be the path of least resistance. Others already explained  
that RAID and a backup drive are not the same thing; so this depends on what  
you want to do, but in this case adding a simple backup drive is the  
simplest one of the two.

I guess if you decide to go with RAID, you'd want to create a new RAID1  
volume on the new drive, with just one partition, a degraded RAID1 volume.  
It should still be mountable, so you'll copy your existing partition to the  
new RAID1 volume, drop your existing partition, then add it to the RAID1  
volume and have it sync back up. Just have to make sure that the new RAID1  
volume is not larger than your existing partition.

[Chris Murphy]

On Tue, Dec 24, 2019 at 12:48 PM Javier Perez <pepebuho(a)gmail.com&gt; wrote:
>  Hi
>  I have my /home partition on a 2TB HDD drive about half full. Should be doing regular
> backups but not in the habit.
>
>  What is better?
>  A. Purchase a second 2TB to create a Raid1 mirror or
>  B. Purchase a second 4TB drive for backup purposes. 
If you care about the data at all, then B is the top priority.

>  If case B, should I get a USB3.0 adapter or similar so that I could
> just plug the HDD, let it do the backup and unplug and store it? Or is it better to
> install it on another networked pc and do it through the network? 
Whichever is more reliably/consistently connected or available for
doing backups.

The network solution is a bit more complicated, but overall I find it
more convenient. USB enclosures are notoriously flaky with the myriad
bridge chipsets floating about; and even name brand USB bus support
doesn't help. I advise getting a good USB hub. Don't expect that a
direct connection between USB drive enclosure and the computer
provides a more reliable connection. I have a dyconn USB hub with a
power supply providing 2.5A, which for a 4-port hub. If you look at
the specs for laptop hard drives, all want upwards of 1A for spinup
and around 0.35A during use. So there's a power aspect to making sure
the drives have the power they need, without wigging out, and there's
a data stream rewrite aspect to the hub that in a sense "cleans up"
the flaky communication between host and device.



-- 
Chris Murphy

[John M. Harris Jr.]

On Thursday, December 26, 2019 11:32:53 AM MST Chris Murphy wrote:
>  On Tue, Dec 24, 2019 at 12:48 PM Javier Perez
> <pepebuho(a)gmail.com&gt; wrote:
>  
>  >
>  >
>  > Hi
>  > I have my /home partition on a 2TB HDD drive about half full. Should be
>  > doing regular backups but not in the habit.
>
>  >
>  >
>  > What is better?
>  > A. Purchase a second 2TB to create a Raid1 mirror or
>  > B. Purchase a second 4TB drive for backup purposes.
>  
>  
>  If you care about the data at all, then B is the top priority. 
This is not good advice, given the tone.

Option A is fine, if you're more worried about hardware faults with the 2TB HDD 
you currently have.

Option B is best if you're more concerned about other issues.

- -
John M. Harris, Jr.