I'm using shred on some 2Tb USB disk drive that I plan to give away. So far it has taken 8 hours to shred 50% of the drive, which implies that it will take about 16 hours to shred the whole drive. I have another 2 drives to go.
Is there a quicker way to protect my data when I give the drives away, other than smashing the drives to bits?
Thanks - jon
On 1/25/19 6:45 PM, Jonathan Ryshpan wrote:
I'm using shred on some 2Tb USB disk drive that I plan to give away. So far it has taken 8 hours to shred 50% of the drive, which implies that it will take about 16 hours to shred the whole drive. I have another 2 drives to go.
Is there a quicker way to protect my data when I give the drives away, other than smashing the drives to bits?
Thanks - jon
Hi Jon,
Figure out what your /dev/sdxx is.
as root (#): dd if=/dev/zero of=/dev/sdxx
This will also discharge all your cells too.
-T
On 1/25/19 6:45 PM, Jonathan Ryshpan wrote:
Is there a quicker way to protect my data when I give the drives away, other than smashing the drives to bits?
The quickest would be to encrypt the drives from the beginning. When you want to discard the drives, you just need to wipe the LUKS header. With the key wiped out, the rest of the drive is just random bits.
You can wipe your drive with:
dd if=/dev/zero of=/dev/[device to be wiped]
followed by:
dd if=/dev/urandom of=/dev/[device to be wiped]
repeat 7 times per device if you want the DoD standard for wiping a drive. If you would like to save time/trouble drop the sequence in a script. Plenty of examples on the web. One pass is usually good for most ppl tbh. Though I did do a recovery once that rebuilt the last 3 OS installs on the server... but if you trust the person you are giving them too not to care then one pass should do.
-- Fred
On Fri, Jan 25, 2019 at 8:25 PM Gordon Messmer gordon.messmer@gmail.com wrote:
On 1/25/19 6:45 PM, Jonathan Ryshpan wrote:
Is there a quicker way to protect my data when I give the drives away, other than smashing the drives to bits?
The quickest would be to encrypt the drives from the beginning. When you want to discard the drives, you just need to wipe the LUKS header. With the key wiped out, the rest of the drive is just random bits. _______________________________________________ users mailing list -- users@lists.fedoraproject.org To unsubscribe send an email to users-leave@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org
On 1/25/19 6:45 PM, Jonathan Ryshpan wrote:
I'm using shred on some 2Tb USB disk drive that I plan to give away. So far it has taken 8 hours to shred 50% of the drive, which implies that it will take about 16 hours to shred the whole drive. I have another 2 drives to go.
Is there a quicker way to protect my data when I give the drives away, other than smashing the drives to bits?
If you don't have state secrets on your hard drive and you don't expect that the NSA will be using million dollar equipment to scan your drive, a single pass with "dd if=/dev/urandom of=/dev/sdx bs=1M" should be sufficient and will write at the max speed of the hard drive.
On Fri, 2019-01-25 at 21:58 -0800, Samuel Sieb wrote:
On 1/25/19 6:45 PM, Jonathan Ryshpan wrote:
I'm using shred on some 2Tb USB disk drive that I plan to give away. So far it has taken 8 hours to shred 50% of the drive, which implies that it will take about 16 hours to shred the whole drive. I have another 2 drives to go.
Is there a quicker way to protect my data when I give the drives away, other than smashing the drives to bits?
If you don't have state secrets on your hard drive and you don't expect that the NSA will be using million dollar equipment to scan your drive, a single pass with "dd if=/dev/urandom of=/dev/sdx bs=1M" should be sufficieshred -v -n1 /dev/sdcnt and will write at the max speed of the hard drive.
I'll give dd a try; but I don't see offhand why dd if=/dev/urandom of=/dev/sdc bs=1M should be quicker than what I'm using now, namely shred -v -n1 /dev/sdc
Thanks to all for your help - jon
On 1/25/19 10:58 PM, Jonathan Ryshpan wrote:
I'll give dd a try; but I don't see offhand why dd if=/dev/urandom of=/dev/sdc bs=1M should be quicker than what I'm using now, namely shred -v -n1 /dev/sdc
I don't know how it generates the data to overwrite with. Does it use a lot of CPU? If not, then dd will probably not be faster, you might be at maximum speed for the drive. However, shred is intended for files, not partitions.
adding status=progress
to the dd command i believe will let you know how long it has so you can keep an eye on it in the terminal.
On Fri, Jan 25, 2019 at 11:32 PM Samuel Sieb samuel@sieb.net wrote:
On 1/25/19 10:58 PM, Jonathan Ryshpan wrote:
I'll give dd a try; but I don't see offhand why dd if=/dev/urandom of=/dev/sdc bs=1M should be quicker than what I'm using now, namely shred -v -n1 /dev/sdc
I don't know how it generates the data to overwrite with. Does it use a lot of CPU? If not, then dd will probably not be faster, you might be at maximum speed for the drive. However, shred is intended for files, not partitions. _______________________________________________ users mailing list -- users@lists.fedoraproject.org To unsubscribe send an email to users-leave@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org
On Fri, 2019-01-25 at 22:43 -0800, ToddAndMargo via users wrote:
On 1/25/19 9:38 PM, fred roller wrote:
You can wipe your drive with:
dd if=/dev/zero of=/dev/[device to be wiped]
/dev/zero is the fastest I have found.
All that does is set every bit to 0. Since magnetic domains have hysteresis, suitable equipment can recover the previous state at a physical level. That's why proper trashing takes several passes.
poc
On Fri, 25 Jan 2019 18:45:25 -0800, Jonathan Ryshpan wrote:
I'm using shred on some 2Tb USB disk drive that I plan to give away. So far it has taken 8 hours to shred 50% of the drive, which implies that it will take about 16 hours to shred the whole drive. I have another 2 drives to go.
Is there a quicker way to protect my data when I give the drives away, other than smashing the drives to bits?
I notice that no one even mentioned DBAN. Is it just not fast, or there some other consideration?
On Sat, Jan 26, 2019 at 05:50:20PM -0000, Beartooth wrote:
On Fri, 25 Jan 2019 18:45:25 -0800, Jonathan Ryshpan wrote:
I'm using shred on some 2Tb USB disk drive that I plan to give away. So far it has taken 8 hours to shred 50% of the drive, which implies that it will take about 16 hours to shred the whole drive. I have another 2 drives to go.
Is there a quicker way to protect my data when I give the drives away, other than smashing the drives to bits?
I notice that no one even mentioned DBAN. Is it just not fast, or there some other consideration?
Having used DBAN once, I'm not a very authoritative voice, but...
I found that depending on the options you choose, it can also be quite slow. one 250GB drive I wiped with it took more than 12 hours. So for the next one I wanted to wipe I just used /dev/zero, which should be adequate because I was just going to install something else over it anyway.
-- Beartooth Staffwright, Not Quite Clueless Power User Remember I know little (precious little!) of where up is.
users mailing list -- users@lists.fedoraproject.org To unsubscribe send an email to users-leave@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org
On 1/26/19 2:47 AM, Patrick O'Callaghan wrote:
On Fri, 2019-01-25 at 22:43 -0800, ToddAndMargo via users wrote:
On 1/25/19 9:38 PM, fred roller wrote:
You can wipe your drive with:
dd if=/dev/zero of=/dev/[device to be wiped]
/dev/zero is the fastest I have found.
All that does is set every bit to 0. Since magnetic domains have hysteresis, suitable equipment can recover the previous state at a physical level. That's why proper trashing takes several passes.
Sure, but do you expect someone to find your hard drive and put that much money and effort into extracting your data? For most people, it's just that you don't want some random person finding your hard drive in a second hand store and checking out your files.
On 1/26/19 9:50 AM, Beartooth wrote:
On Fri, 25 Jan 2019 18:45:25 -0800, Jonathan Ryshpan wrote:
I'm using shred on some 2Tb USB disk drive that I plan to give away. So far it has taken 8 hours to shred 50% of the drive, which implies that it will take about 16 hours to shred the whole drive. I have another 2 drives to go.
Is there a quicker way to protect my data when I give the drives away, other than smashing the drives to bits?
I notice that no one even mentioned DBAN. Is it just not fast, or there some other consideration?
DBAN is a serious wiping tool. It takes a long time since it uses multiple passes with different data patterns.
On 1/25/19 6:45 PM, Jonathan Ryshpan wrote:
I'm using shred on some 2Tb USB disk drive that I plan to give away. So far it has taken 8 hours to shred 50% of the drive, which implies that it will take about 16 hours to shred the whole drive. I have another 2 drives to go.
Is there a quicker way to protect my data when I give the drives away, other than smashing the drives to bits?
I just checked the numbers and I think you're erasing as fast as you can. That's a large hard drive and given the write speed of a spinning disk, that's the time it will take. Can you not write them all at the same time?
On Sat, 2019-01-26 at 11:04 -0800, Samuel Sieb wrote:
On 1/26/19 2:47 AM, Patrick O'Callaghan wrote:
On Fri, 2019-01-25 at 22:43 -0800, ToddAndMargo via users wrote:
On 1/25/19 9:38 PM, fred roller wrote:
You can wipe your drive with:
dd if=/dev/zero of=/dev/[device to be wiped]
/dev/zero is the fastest I have found.
All that does is set every bit to 0. Since magnetic domains have hysteresis, suitable equipment can recover the previous state at a physical level. That's why proper trashing takes several passes.
Sure, but do you expect someone to find your hard drive and put that much money and effort into extracting your data? For most people, it's just that you don't want some random person finding your hard drive in a second hand store and checking out your files.
I don't "expect" anything, but without knowing the OP's use-case, none of us can say what's cost-effective. I merely pointed out that a single pass with /dev/zero is not proof against even commercial-grade recovery, let alone military grade.
poc
On 01/26/2019 12:04 PM, Samuel Sieb wrote:
Sure, but do you expect someone to find your hard drive and put that much money and effort into extracting your data? For most people, it's just that you don't want some random person finding your hard drive in a second hand store and checking out your files.
My thought exactly. Just to put my two cents in, I'd start by using GParted to blow away any partitions on the drive, create four primary partitions and format them ext4. Then I'd blow them away, create one partition and format it NTFS, which is probably what whoever ends up with the drive will want. That won't remove the data, of course, but it will ensure that the average layman won't be able to recover it, and for most people, that's all you're likely to need.
maybe "hdparm security erase" is an option (don't know if disk content is afterwards still recoverable)
https://ata.wiki.kernel.org/index.php/ATA_Secure_Erase
It works with spinning drives too. At least bother of my Samsung drives are supported. (time: 500 GB 112 min)
I'm not quit sure, but I mean I did "security erase" with one of the Samsung HDD tools too - Samsung HDD Utility(HUTIL ? or - Samsung ESTOOL ?)
Also, I got a "security erase" option in my Bios (MSI B250-PC-MATE mainboard)
another tool: hdat2 https://www.hdat2.com/
On Fri, 2019-01-25 at 20:24 -0800, Gordon Messmer wrote:
On 1/25/19 6:45 PM, Jonathan Ryshpan wrote:
Is there a quicker way to protect my data when I give the drives away, other than smashing the drives to bits?
The quickest would be to encrypt the drives from the beginning. When you want to discard the drives, you just need to wipe the LUKS header. With the key wiped out, the rest of the drive is just random bits.
This, 100 times over!
Jonathan
On Sat, 2019-01-26 at 20:01 +0000, Jonathan Dieter wrote:
On Fri, 2019-01-25 at 20:24 -0800, Gordon Messmer wrote:
On 1/25/19 6:45 PM, Jonathan Ryshpan wrote:
Is there a quicker way to protect my data when I give the drives away, other than smashing the drives to bits?
The quickest would be to encrypt the drives from the beginning. When you want to discard the drives, you just need to wipe the LUKS header. With the key wiped out, the rest of the drive is just random bits.
This, 100 times over!
Undoubtedly, *if* the drive was previously encrypted. Nothing the OP has says indicates this is the case, and encrypting it now is not going to be faster than trashing it.
poc
Allegedly, on or about 26 January 2019, sixpack13 sent:
maybe "hdparm security erase" is an option (don't know if disk content is afterwards still recoverable)
Of course you could leave a drive filled with non-private files, just to lead them down the garden path. ;-)
On Sat, 2019-01-26 at 11:09 -0800, Samuel Sieb wrote:
On 1/25/19 6:45 PM, Jonathan Ryshpan wrote:
I'm using shred on some 2Tb USB disk drive that I plan to give away. So far it has taken 8 hours to shred 50% of the drive, which implies that it will take about 16 hours to shred the whole drive. I have another 2 drives to go.
Is there a quicker way to protect my data when I give the drives away, other than smashing the drives to bits?
I just checked the numbers and I think you're erasing as fast as you can. That's a large hard drive and given the write speed of a spinning disk, that's the time it will take. Can you not write them all at the same time?
I'm pretty sure you're right. Here's what I've found out from a small amount of research and experience.
(1) These two things $ dd if=/dev/urandom of=/dev/sdc bs=1M status=progress $ shred -v -n1 /dev/sdc do the same thing, which is to overwrite the drive with random data in a single pass.
They both do it at about the same speed, about 40Mbytes/sec., which is probably about the max transfer rate to the drive. Another instance of shred $ shred -v -n1 /dev/sdb applied to an internal drive (which I also plan to dispose of) ran at least twice as fast. Unfortunately I have no actual timings. Another indication that the shredding rate is bounded by the transfer rate.
(2) Shred uses less cpu power, as closely as I could judge from a small system monitor on my desktop.
(3) Since the time is bounded by the transfer rate, there's no advantage to using /dev/zero instead of /dev/urandom; /dev/urandom is slightly more secure.
(4) Repartitioning and reformatting the drive would almost surely be good enough for my purposes; nevertheless it makes me nervous.
(5) Bottom line. Repartition and reformat if you think that's good enough; otherwise use shred.
Thanks to all - jon
On Fri, Jan 25, 2019 at 06:45:25PM -0800, Jonathan Ryshpan wrote:
I'm using shred on some 2Tb USB disk drive that I plan to give away. So far it has taken 8 hours to shred 50% of the drive, which implies that it will take about 16 hours to shred the whole drive. I have another 2 drives to go.
Is there a quicker way to protect my data when I give the drives away, other than smashing the drives to bits?
I think, yes: simply encrypting the whole disk should do it: IIRC this should be *a lot* faster than piping /dev/urandom to a disk, or even using shred:
Excerpt from /usr/share/doc/cryptsetup/FAQ :
------------------------------------------------------------ * 2.19 How can I wipe a device with crypto-grade randomness?
The conventional recommendation if you want to not just do a zero-wipe is to use something like
cat /dev/urandom > <taget-device>
That is very slow and painful at 10-20MB/s on a fast computer. Using cryptsetup and a plain dm-crypt device with a random key, it is much faster and gives you the same level of security. The defaults are quite enough.
For device set-up, do the following:
cryptsetup open --type plain -d /dev/urandom /dev/<block-device> to_be_wiped
This maps the container as plain under /dev/mapper/to_be_wiped with a random password. For the actual wipe you have several options. Simple wipe without progress-indicator:
cat /dev/zero > /dev/mapper/to_be_wiped
Progress-indicator by dd_rescue:
dd_rescue -w /dev/zero /dev/mapper/to_be_wiped
Progress-indicator by my "wcs" stream meter (available from http://www.tansi.org/tools/index.html ):
cat /dev/zero | wcs > /dev/mapper/to_be_wiped
Remove the mapping at the end and you are done. ----------------------------------------------------------------
Last line above ("Remove the mapping ... ") simply means, IIUC: close the encrypted device:
cryptsetup -v close to_be_wiped
I think I even tested sth. like that at some point. Anyways: here is someone who seems to have done it actually, similary to the mentioned FAQ: https://www.redhat.com/archives/dm-devel/2016-July/msg00239.html
And sure: if you want an extra layer of security: disassembing the disk after encryption and smashing the platters won't hurt ... :)
HTH, Wolfgang
On 1/26/19 2:45 PM, Jonathan Ryshpan wrote:
They both do it at about the same speed, about 40Mbytes/sec., which is probably about the max transfer rate to the drive. Another instance of shred $ shred -v -n1 /dev/sdb applied to an internal drive (which I also plan to dispose of) ran at least twice as fast. Unfortunately I have no actual timings. Another indication that the shredding rate is bounded by the transfer rate.
Unless you're using esata or USB3, the internal one will be much faster.
On 1/26/19 5:29 PM, Wolfgang Pfeiffer wrote:
I think, yes: simply encrypting the whole disk should do it: IIRC this should be *a lot* faster than piping /dev/urandom to a disk, or even using shred:
Encrypting the whole disk involves writing the same amount of data, so it can't be faster.
Excerpt from /usr/share/doc/cryptsetup/FAQ :
- 2.19 How can I wipe a device with crypto-grade randomness?
The conventional recommendation if you want to not just do a zero-wipe is to use something like
cat /dev/urandom > <taget-device>
That is very slow and painful at 10-20MB/s on a fast computer. Using cryptsetup and a plain dm-crypt device with a random key, it is much faster and gives you the same level of security. The defaults are quite enough.
This must be a really old FAQ. My laptop can generate urandom at 275 MB/s, which is faster than even most SSDs could write. Also, I don't understand how a cryptographically secure process could be faster than urandom, unless it's using hardware cryptography.
Allegedly, on or about 26 January 2019, Patrick O'Callaghan sent:
Since magnetic domains have hysteresis, suitable equipment can recover the previous state at a physical level. That's why proper trashing takes several passes.
Some drives are supposed have a built in secure erase function. I wonder how effective it is.
On Sun, 2019-01-27 at 02:29 +0100, Wolfgang Pfeiffer wrote:
I think, yes: simply encrypting the whole disk should do it: IIRC this should be *a lot* faster than piping /dev/urandom to a disk,
Given that encrypting the disk means (at a minimum) reading the entire contents and rewriting it, even if the actual encryption costs nothing, this makes no sense.
poc
from a german IT magazine I got this:
https://www.heise.de/select/ct/2016/03/1454233793502849#titel_14542337935028...
openssl enc -aes-256-ctr -pass pass: "$(dd if=/dev/urandom bs=128 count=1 2>/dev/null | base64)" -nosalt < /dev/zero > /dev/sdX
I've never tested it, so I don't know if it works !
For non-german's the article sense in short:
- dd if=/dev/urandom of=/dev/sdX would need several days for 3 TB disk - the bottle leck is /dev/urandom cause it generates 15 MB/s only - to overwrite a disk there is no need to do it with a high value random number [*] - with the above command overwriting a 3 TB disk needs 5 h only
[*] the main tasks in this article is to prepare (by overwriting) a disk for encryption
P.S. in a previous contribution (Saturday, 26 January 8:52 p.m.) it should read
"(time: 500 GB *76* min)"
I looked at the wrong disk (my 320 GB disk needs more time: 112 min) !
On Sun, Jan 27, 2019 at 11:32:52AM +0000, Patrick O'Callaghan wrote:
On Sun, 2019-01-27 at 02:29 +0100, Wolfgang Pfeiffer wrote:
I think, yes: simply encrypting the whole disk should do it: IIRC this should be *a lot* faster than piping /dev/urandom to a disk,
Given that encrypting the disk means (at a minimum) reading the entire contents and rewriting it,
No. I don't think data is written and rewritten. See below.
First off: thanks a lot, poc, for writing: because although I can't agree with you so far on this, I found what I'd consider as being a mistake in the procedure I suggested.
My tests show, that an external 500 GB (USB 2) disk is encrypted in not even 10 secs (yes: 10. No typo). I think this is because cryptsetup doesn't rewrite data from non-encrypted to encrypted. Don't ask me why .. and the speed *might* have to do with the fact that only ~17% of the disk was filled with data
But thanks to your intervention, poc, I found that the initial command that I quoted from the cryptsetup FAQ doesn't seem to actually encrypt the disk. Bug? My mistake? No idea what went wrong. So I changed things.
My workaround, nearly as fast:
Create a randomly generated password key and pipe it into a file, max. is here 8192kB (people can find the max size on their systems with cryptsetup --help)
That's how it seems to work here - and careful everyone ofc: you destroy a disk with some of the following:
0: # Create the key-file: head -c 8192K /dev/urandom > lukskey
1: # Encrypt the disk:
cryptsetup -s 512 -d lukskey -c aes-xts-plain64 luksFormat /dev/sd[?]
# Again: didn't take 10 secs.
# Find your own favorite cipher (-c) and use it
# and re. /dev/sd[?] - yes: no partition seems being necessary to just # destroy the disk
2: # Check if the disk is encrypted:
cryptsetup luksDump /dev/sd[?]
3: # And you can open the container like so:
cryptsetup -v open -d lukskey /dev/sd[?] test Key slot 0 unlocked. Command successful.
# Close it after usage: cryptsetup -v close test
4:
Destroy the key, once you're finished (careful: the data on disk is useless without it .. )
That's it. To pipe /dev/zero into that disk (I used dd), as sht. like this was mentioned in the FAQ: with that USB2 disk here that took a long time: ~ 53 minutes for just ~ 130 GB.
Let me know, please, in case of a mistake ..
TIA.
HTH, Regards Wolfgang
On 20190127 10:56:25, Wolfgang Pfeiffer wrote:
On Sun, Jan 27, 2019 at 11:32:52AM +0000, Patrick O'Callaghan wrote:
On Sun, 2019-01-27 at 02:29 +0100, Wolfgang Pfeiffer wrote:
I think, yes: simply encrypting the whole disk should do it: IIRC this should be *a lot* faster than piping /dev/urandom to a disk,
Given that encrypting the disk means (at a minimum) reading the entire contents and rewriting it,
No. I don't think data is written and rewritten. See below.
First off: thanks a lot, poc, for writing: because although I can't agree with you so far on this, I found what I'd consider as being a mistake in the procedure I suggested.
My tests show, that an external 500 GB (USB 2) disk is encrypted in not even 10 secs (yes: 10. No typo). I think this is because cryptsetup doesn't rewrite data from non-encrypted to encrypted. Don't ask me why .. and the speed *might* have to do with the fact that only ~17% of the disk was filled with data
But thanks to your intervention, poc, I found that the initial command that I quoted from the cryptsetup FAQ doesn't seem to actually encrypt the disk. Bug? My mistake? No idea what went wrong. So I changed things.
My workaround, nearly as fast:
Create a randomly generated password key and pipe it into a file, max. is here 8192kB (people can find the max size on their systems with cryptsetup --help)
That's how it seems to work here - and careful everyone ofc: you destroy a disk with some of the following:
0: # Create the key-file: head -c 8192K /dev/urandom > lukskey
1: # Encrypt the disk:
cryptsetup -s 512 -d lukskey -c aes-xts-plain64 luksFormat /dev/sd[?]
# Again: didn't take 10 secs.
# Find your own favorite cipher (-c) and use it
# and re. /dev/sd[?] - yes: no partition seems being necessary to just # destroy the disk
2: # Check if the disk is encrypted:
cryptsetup luksDump /dev/sd[?]
3: # And you can open the container like so:
cryptsetup -v open -d lukskey /dev/sd[?] test Key slot 0 unlocked. Command successful.
# Close it after usage: cryptsetup -v close test
4:
Destroy the key, once you're finished (careful: the data on disk is useless without it .. )
That's it. To pipe /dev/zero into that disk (I used dd), as sht. like this was mentioned in the FAQ: with that USB2 disk here that took a long time: ~ 53 minutes for just ~ 130 GB.
Let me know, please, in case of a mistake ..
TIA.
HTH, Regards Wolfgang
If you really want a secure erase use dd. (bs=1M makes it very fast compared to leaving bs unset.) During operation most OSs use filesystems that do not rewrite using the existing blocks allocated to a file. So the old data is present on the disk if somebody goes looking for it. If you have data important enough to require removing it from the drive then be sure to get those blocks as well as the ones currently used. dd is about the only way to guarantee this.
If you have an SSD even dd is not sufficient because the blocks in the spares pool are used to replace active blocks to even out wear on the disk. Guaranteeing those blocks are gone requires specialized software.
10 seconds to encrypt a 500G disk is not a realistic number. Something else was going on you did not understand unless. That means the disk was reading 50 Gigabytes per second at the same time it was writing 50 Gigabytes per second. Does that sound at all reasonable to you? A key may have been set on the drive with the drive taking it upon itself to encrypt the data in the background.
If the cryptsetup command does not grab large chunks of disk at a time for reads and writes dd with /dev/random and 1M or even 64M block size (bs) may be faster. All it has to do is write with no reads. And the very large blocks pull I/O overhead, seek times, and rotational latency out of the picture. These can kill your speeds with a default block size of 512 bytes.
If the disk even MIGHT have an error on it and you use dd then add "conv=noerror" to the command string.
{^_^}
On Sun, 2019-01-27 at 19:56 +0100, Wolfgang Pfeiffer wrote:
Given that encrypting the disk means (at a minimum) reading the entire contents and rewriting it,
No. I don't think data is written and rewritten. See below.
If it's not being read and rewritten, it's not being encrypted. It's as simple as that. A cryptosystem that doesn't read the plaintext? How does that work?
poc
On 1/27/19 2:44 PM, Patrick O'Callaghan wrote:
If it's not being read and rewritten, it's not being encrypted. It's as simple as that. A cryptosystem that doesn't read the plaintext? How does that work?
The suggestion you're replying to didn't encrypt the drive in place. It read a stream of zeros from /dev/zero, encrypted that stream, and wrote that stream to the disk. Thus, nothing needed to be read from any disk.
However, this whole thread is questionable. It is predicated on the assumption that your CPU can encrypt a stream of zeros faster than it can generate random data, and also that it can write to its disk faster than it can generate random data. If either of those things is not true, then using an encrypted volume to "wipe" a drive will be slower and more complex, for no benefit.
On my Dell XPS 13, I can read from /dev/urandom at almost exactly the same speed that I can write to a dm-crypt block device, so there would be no reason to use dm-crypt over simply dd if=/dev/urandom to the drive.
(But the point that I was making when I replied to this thread to begin with is that if you are concerned with wiping your data from drives, it should never have been written to the drive in an unencrypted form to begin with. Encrypt your disks. When you want to get rid of them, they're already as secure as your passphrase, and you can irrecoverably wipe them by simply wiping the key header. It's nearly instantaneous.)
On 1/27/19 3:06 PM, Gordon Messmer wrote:
<snip/>
simply wiping the key header. Command example?
On Sun, 2019-01-27 at 15:06 -0800, Gordon Messmer wrote:
On 1/27/19 2:44 PM, Patrick O'Callaghan wrote:
If it's not being read and rewritten, it's not being encrypted. It's as simple as that. A cryptosystem that doesn't read the plaintext? How does that work?
The suggestion you're replying to didn't encrypt the drive in place. It read a stream of zeros from /dev/zero, encrypted that stream, and wrote that stream to the disk. Thus, nothing needed to be read from any disk.
I think the writer *thought* it was encrypting the disk. I may be wrong, but that's what I was responding to rather than the minutiae of the actual command.
Regardless, writing a bunch of encrypted zeroes is no better than writing plain zeroes or writing random noise, if it's just a single pass. I think the various answers here are actually addressing slightly different questions. In my case I'm talking about defence against physical-level analysis of the disk.
However, this whole thread is questionable. It is predicated on the assumption that your CPU can encrypt a stream of zeros faster than it can generate random data, and also that it can write to its disk faster than it can generate random data. If either of those things is not true, then using an encrypted volume to "wipe" a drive will be slower and more complex, for no benefit.
On my Dell XPS 13, I can read from /dev/urandom at almost exactly the same speed that I can write to a dm-crypt block device, so there would be no reason to use dm-crypt over simply dd if=/dev/urandom to the drive.
Of course.
(But the point that I was making when I replied to this thread to begin with is that if you are concerned with wiping your data from drives, it should never have been written to the drive in an unencrypted form to begin with. Encrypt your disks. When you want to get rid of them, they're already as secure as your passphrase, and you can irrecoverably wipe them by simply wiping the key header. It's nearly instantaneous.)
Again, I agree, but that's not what the OP asked. Telling him "you should have started with an encrypted disk" is like telling a traveller who's asking the way to Podunk "start from somewhere else".
poc
too bad it's not an old spnner... just leave it on your speaker magnet overnight. reformat in the morning.
:P Fred
On Sun, Jan 27, 2019 at 3:24 PM Patrick O'Callaghan pocallaghan@gmail.com wrote:
On Sun, 2019-01-27 at 15:06 -0800, Gordon Messmer wrote:
On 1/27/19 2:44 PM, Patrick O'Callaghan wrote:
If it's not being read and rewritten, it's not being encrypted. It's as simple as that. A cryptosystem that doesn't read the plaintext? How does that work?
The suggestion you're replying to didn't encrypt the drive in place. It read a stream of zeros from /dev/zero, encrypted that stream, and wrote that stream to the disk. Thus, nothing needed to be read from any disk.
I think the writer *thought* it was encrypting the disk. I may be wrong, but that's what I was responding to rather than the minutiae of the actual command.
Regardless, writing a bunch of encrypted zeroes is no better than writing plain zeroes or writing random noise, if it's just a single pass. I think the various answers here are actually addressing slightly different questions. In my case I'm talking about defence against physical-level analysis of the disk.
However, this whole thread is questionable. It is predicated on the assumption that your CPU can encrypt a stream of zeros faster than it can generate random data, and also that it can write to its disk faster than it can generate random data. If either of those things is not true, then using an encrypted volume to "wipe" a drive will be slower and more complex, for no benefit.
On my Dell XPS 13, I can read from /dev/urandom at almost exactly the same speed that I can write to a dm-crypt block device, so there would be no reason to use dm-crypt over simply dd if=/dev/urandom to the drive.
Of course.
(But the point that I was making when I replied to this thread to begin with is that if you are concerned with wiping your data from drives, it should never have been written to the drive in an unencrypted form to begin with. Encrypt your disks. When you want to get rid of them, they're already as secure as your passphrase, and you can irrecoverably wipe them by simply wiping the key header. It's nearly instantaneous.)
Again, I agree, but that's not what the OP asked. Telling him "you should have started with an encrypted disk" is like telling a traveller who's asking the way to Podunk "start from somewhere else".
poc _______________________________________________ users mailing list -- users@lists.fedoraproject.org To unsubscribe send an email to users-leave@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org
On Sun, Jan 27, 2019 at 10:44:52PM +0000, Patrick O'Callaghan wrote:
On Sun, 2019-01-27 at 19:56 +0100, Wolfgang Pfeiffer wrote:
Given that encrypting the disk means (at a minimum) reading the entire contents and rewriting it,
No. I don't think data is written and rewritten. See below.
If it's not being read and rewritten, it's not being encrypted.
Yes, something like that is what I suspect: The actual data on disk would be left untouched when the *disk/partition* is encrypted. I had a look through documents explaining luks, and again and again the topic is "disk" encryption, not "data" encryption. So maybe all that happens is that - to use a picture - a high security prison (some sort of crypto layer) is built around the data on disk, while leaving the actual data untouched, and non-encrypted.
In other words: It seems the file system is encrypted, not the data: see the already mentioned FAQ: "Create the LUKS container" - at the end of the section: "Done. You can now use the encrypted file system to store data"
And what also comes to mind is that when I open (unencrypt) a luks device, only a second or two after providing the password for the encrypted disk/partition the partition is unencrypted. I doubt that this could happen if e.g. a 1 TB partition had all data stored encrypted on it .. Or - another possibility - the data stay encrypted after opening the luks device and will be decrypted only on the fly, just as needed.
No idea, not yet ...
It's as simple as that. A cryptosystem that doesn't read the plaintext?
You mean, without actually encrypting the data, and instead of that just some sort of crypto layer around it?
How does that work?
I'd like to know, yes ... :)
Wolfgang
On 20190127 14:44:52, Patrick O'Callaghan wrote:
On Sun, 2019-01-27 at 19:56 +0100, Wolfgang Pfeiffer wrote:
Given that encrypting the disk means (at a minimum) reading the entire contents and rewriting it,
No. I don't think data is written and rewritten. See below.
If it's not being read and rewritten, it's not being encrypted. It's as simple as that. A cryptosystem that doesn't read the plaintext? How does that work?
poc
Give me an image if the disk and after suitable time and effort I have access to the whole thing if only the directory entries are encrypted. So the tool in question gives a false sense of security at best. {^_^}
On 1/27/19 6:47 PM, Wolfgang Pfeiffer wrote:
On Sun, Jan 27, 2019 at 10:44:52PM +0000, Patrick O'Callaghan wrote:
If it's not being read and rewritten, it's not being encrypted.
Yes, something like that is what I suspect: The actual data on disk would be left untouched when the *disk/partition* is encrypted. I had a look through documents explaining luks, and again and again the topic is "disk" encryption, not "data" encryption. So maybe all that happens is that - to use a picture - a high security prison (some sort of crypto layer) is built around the data on disk, while leaving the actual data untouched, and non-encrypted.
In other words: It seems the file system is encrypted, not the data: see the already mentioned FAQ: "Create the LUKS container" - at the end of the section: "Done. You can now use the encrypted file system to store data"
From your first message where you described this, the first part is setting up the container. That doesn't encrypt any data at first, only when something is written does it get encrypted. Further down in your email, you describe the command of writing /dev/zero to the encrypted disk container. That is what writes the "random" data to the disk and would not require reading first. But in any case, as has already been mentioned multiple times, this still is not going to be faster than using /dev/urandom.
On 1/27/19 7:34 AM, sixpack13 wrote:
from a german IT magazine I got this:
https://www.heise.de/select/ct/2016/03/1454233793502849#titel_14542337935028...
openssl enc -aes-256-ctr -pass pass: "$(dd if=/dev/urandom bs=128 count=1 2>/dev/null | base64)" -nosalt < /dev/zero > /dev/sdX
I've never tested it, so I don't know if it works !
For non-german's the article sense in short:
- dd if=/dev/urandom of=/dev/sdX would need several days for 3 TB disk > - the bottle leck is /dev/urandom cause it generates 15 MB/s only
This is repeating really old information. On my not new laptop /dev/urandom can provide over 250MB/s. And I was surprised that the openssl command is faster, but still urandom is faster than you can write to the disk anyway.
Btw, there is an error in that openssl command. There should be no space after "pass:".
On Sun, 2019-01-27 at 20:46 -0800, jdow wrote:
On 20190127 14:44:52, Patrick O'Callaghan wrote:
On Sun, 2019-01-27 at 19:56 +0100, Wolfgang Pfeiffer wrote:
Given that encrypting the disk means (at a minimum) reading the entire contents and rewriting it,
No. I don't think data is written and rewritten. See below.
If it's not being read and rewritten, it's not being encrypted. It's as simple as that. A cryptosystem that doesn't read the plaintext? How does that work?
poc
Give me an image if the disk and after suitable time and effort I have access to the whole thing if only the directory entries are encrypted. So the tool in question gives a false sense of security at best. {^_^}
If that's indeed what it does, I agree.
Another point: several people have mentioned using /dev/urandom. It's important to note that this is a *pseudo-random* generator. It starts from a random seed, but from that generates a completely deterministic pattern. If you have the seed, you have everything. And since the idea here is to overwrite the disk, the first part of which contains "plaintext" that follows a regular layout (partition table etc.) it makes the task of decoding the disk even easier as that's the only part you would actually have to analyse at a physical level.
poc
On 2019-01-28 at 10:12:28 Patrick O'Callaghan wrote:
Another point: several people have mentioned using /dev/urandom. It's important to note that this is a *pseudo-random* generator. It starts from a random seed, but from that generates a completely deterministic pattern. If you have the seed, you have everything. And since the idea here is to overwrite the disk, the first part of which contains "plaintext" that follows a regular layout (partition table etc.) it makes the task of decoding the disk even easier as that's the only part you would actually have to analyse at a physical level.
But it is just a little more random than all zeroes.
-- Regards, Erik P. Olsen
On Mon, 28 Jan 2019 at 10:13, Patrick O'Callaghan pocallaghan@gmail.com wrote:
On Sun, 2019-01-27 at 20:46 -0800, jdow wrote:
On 20190127 14:44:52, Patrick O'Callaghan wrote:
On Sun, 2019-01-27 at 19:56 +0100, Wolfgang Pfeiffer wrote:
Given that encrypting the disk means (at a minimum) reading the entire contents and rewriting it,
No. I don't think data is written and rewritten. See below.
If it's not being read and rewritten, it's not being encrypted. It's as simple as that. A cryptosystem that doesn't read the plaintext? How does that work?
poc
Give me an image if the disk and after suitable time and effort I have access to the whole thing if only the directory entries are encrypted. So the tool in question gives a false sense of security at best. {^_^}
If that's indeed what it does, I agree.
Another point: several people have mentioned using /dev/urandom. It's important to note that this is a *pseudo-random* generator. It starts from a random seed, but from that generates a completely deterministic pattern. If you have the seed, you have everything. And since the idea here is to overwrite the disk, the first part of which contains "plaintext" that follows a regular layout (partition table etc.) it makes the task of decoding the disk even easier as that's the only part you would actually have to analyse at a physical level.
/dev/urandom isn't purely pseudorandom, it does use entropy from the pool, it will become pseudorandom when entropy runs out and become random again when more entropy arrives. Unless you have a hardware entropy source I suspect using /dev/random will take a much longer time to produce enough data even for a single over-write. However it's worth noting that the over-write process is not simply XOR with the written bytes (in which case enough plaintext to work out the sequence would definitely let you recover the rest of the data), it's attempting to randomise the hysteresis effect on the media, which is why multiple writes are good, because though they theoretically just increase the number of possible residual states each time you do it the residual size of the signal you're over-writing is reduced, and attempting to get the sequences applied to a known plaintext will become harder. I wouldn't recommend just doing /dev/zero if the CIA, or even a moderately funded newspaper might specifically be after your data, but it would certainly be enough to stop a casual user plugging it in and getting anything back, a few urandom writes is enough to stop anyone without serious cryptographic expertise and access to the equipment necessary to read the platters directly getting anything off the disc.
On Mon, 2019-01-28 at 14:03 +0000, Ian Malone wrote:
Another point: several people have mentioned using /dev/urandom. It's important to note that this is a *pseudo-random* generator. It starts from a random seed, but from that generates a completely deterministic pattern. If you have the seed, you have everything. And since the idea here is to overwrite the disk, the first part of which contains "plaintext" that follows a regular layout (partition table etc.) it makes the task of decoding the disk even easier as that's the only part you would actually have to analyse at a physical level.
/dev/urandom isn't purely pseudorandom, it does use entropy from the pool, it will become pseudorandom when entropy runs out and become random again when more entropy arrives.
Good to know.
Unless you have a hardware entropy source I suspect using /dev/random will take a much longer time to produce enough data even for a single over-write.
Of course.
However it's worth noting that the over-write process is not simply XOR with the written bytes (in which case enough plaintext to work out the sequence would definitely let you recover the rest of the data), it's attempting to randomise the hysteresis effect on the media, which is why multiple writes are good, because though they theoretically just increase the number of possible residual states each time you do it the residual size of the signal you're over-writing is reduced, and attempting to get the sequences applied to a known plaintext will become harder.
I assume you mean 'shred' does this. 'dd if=/dev/urandom ...' won't.
I wouldn't recommend just doing /dev/zero if the CIA, or even a moderately funded newspaper might specifically be after your data, but it would certainly be enough to stop a casual user plugging it in and getting anything back, a few urandom writes is enough to stop anyone without serious cryptographic expertise and access to the equipment necessary to read the platters directly getting anything off the disc.
Indeed.
poc
Zit: "This is repeating really old information. On my not new laptop /dev/urandom can provide over 250MB/s."
yep, I thought this already. I better had mention the date of the article.
lesson learned !
On 1/28/19 8:28 AM, sixpack13 wrote:
Zit: "This is repeating really old information. On my not new laptop /dev/urandom can provide over 250MB/s."
yep, I thought this already. I better had mention the date of the article.
lesson learned !
I was referring to the website post, not you. I can't read it, but from the URL, that suggests that it's from 2016 which is not very long ago.
On Mon, 28 Jan 2019 at 15:43, Patrick O'Callaghan pocallaghan@gmail.com wrote:
On Mon, 2019-01-28 at 14:03 +0000, Ian Malone wrote:
/dev/urandom isn't purely pseudorandom, it does use entropy from the pool, it will become pseudorandom when entropy runs out and become random again when more entropy arrives.
Good to know.
Unless you have a hardware entropy source I suspect using /dev/random will take a much longer time to produce enough data even for a single over-write.
Of course.
However it's worth noting that the over-write process is not simply XOR with the written bytes (in which case enough plaintext to work out the sequence would definitely let you recover the rest of the data), it's attempting to randomise the hysteresis effect on the media, which is why multiple writes are good, because though they theoretically just increase the number of possible residual states each time you do it the residual size of the signal you're over-writing is reduced, and attempting to get the sequences applied to a known plaintext will become harder.
I assume you mean 'shred' does this. 'dd if=/dev/urandom ...' won't.
Yes, default is 3 writes. But since I'm checking, it's been a while since I read the shred manpage... It turns out the default source is actually not discussed, so may be a purely pseudorandom algorithm, rather than /dev/urandom. It's after midnight here, so I'm not inclined to dig into the source right now to find out. It is possible to specify urandom if wished by using the --random-source argument. One old piece of advice that may be outdated now, multiple writes are a good idea, because the residual signal on magnetic media is less than the current value, my understanding is that at one point you could effectively substract out the ideal current signal to retrieve the last value, which would work even against a random overwrite. Of course disc technology has moved on a lot and technologies like shingling may have turned that into an extra data channel by now. The truely paranoid might still want to use a physical shredder rather than a software one.
On 1/28/19 8:03 AM, Ian Malone wrote:
I wouldn't recommend just doing /dev/zero if the CIA, or even a moderately funded newspaper might specifically be after your data,
I would be interested to know if you can name any data recovery service that has ever demonstrated the ability to recover data from a reasonably modern hard disk that has been overwritten once with zeros.
run photorec and see what it pulls.
On Mon, Jan 28, 2019 at 7:32 PM Robert Nichols rnicholsNOSPAM@comcast.net wrote:
On 1/28/19 8:03 AM, Ian Malone wrote:
I wouldn't recommend just doing /dev/zero if the CIA, or even a moderately funded newspaper might specifically be after your data,
I would be interested to know if you can name any data recovery service that has ever demonstrated the ability to recover data from a reasonably modern hard disk that has been overwritten once with zeros.
-- Bob Nichols "NOSPAM" is really part of my email address. Do NOT delete it. _______________________________________________ users mailing list -- users@lists.fedoraproject.org To unsubscribe send an email to users-leave@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org
On Tue, 29 Jan 2019 at 03:32, Robert Nichols rnicholsNOSPAM@comcast.net wrote:
On 1/28/19 8:03 AM, Ian Malone wrote:
I wouldn't recommend just doing /dev/zero if the CIA, or even a moderately funded newspaper might specifically be after your data,
I would be interested to know if you can name any data recovery service that has ever demonstrated the ability to recover data from a reasonably modern hard disk that has been overwritten once with zeros.
I can't. This doesn't mean nobody can do it though, and of the possible cases it's the most simple to retrieve the data from, which there is a theoretical possibility of. It's also not a case with that many legitimate uses. Given the additional cost of using at least a single randomised over-write is effectively zero there's basically no reason not to take that option instead. SSD may also simply ignore a zero write and mark blocks as unused instead, though as jdow mentioned the wear-levelling areas can escape an over-write anyway, which is why the ATA secure erase command exists.
-- imalone http://ibmalone.blogspot.co.uk
On 1/27/19 6:47 PM, Wolfgang Pfeiffer wrote:
Yes, something like that is what I suspect: The actual data on disk would be left untouched when the *disk/partition* is encrypted. I had a look through documents explaining luks, and again and again the topic is "disk" encryption, not "data" encryption. So maybe all that happens is that - to use a picture - a high security prison (some sort of crypto layer) is built around the data on disk, while leaving the actual data untouched, and non-encrypted.
It sounds like you're unfamiliar with the implementation, and possibly with filesystems and block devices in general. I'll try to explain, with some simplifications. You suggested this command:
cryptsetup open --type plain -d /dev/urandom /dev/<block-device> to_be_wiped
This creates a mapping in the kernel where blocks on the virtual device named "to_be_wiped" are filtered through an encryption algorithm before writing, or after reading, blocks on the physical device /dev/<block-device>.
Running this command (unless I'm misreading the docs for the "plain" section) doesn't actually write to or modify the content of the block device at all. Only when you write to /dev/mapper/to_be_wiped will encrypted data actually be written to /dev/<block-device>.
(You would normally create a new filesystem on /dev/mapper/to_be_wiped, which would replace a small number of blocks on the disk with new blocks containing encrypted data related to the filesystem. Again, most of the pre-existing data isn't overwritten by that process. The old data would not be readable through /dev/mapper/to_be_wiped, but could be directly through /dev/<block-device>.)
The next step of the process you suggested was:
cat /dev/zero > /dev/mapper/to_be_wiped
This will write zeros to the virtual block device, which will be filtered through the encryption algorithm and the results written to the blocks on the physical device. This will sequentially overwrite the contents of the physical disk.
In other words: It seems the file system is encrypted, not the data: see the already mentioned FAQ: "Create the LUKS container" - at the end of the section: "Done. You can now use the encrypted file system to store data"
The command you referenced above doesn't use LUKS, but that's a minor nit.
Both the filesystem and your data are encrypted when using dm-crypt (whether you use LUKS or not). However, creating a dm-crypt device doesn't modify the pre-existing data. It only writes encrypted data to the disk when data is written to the virtual device.
Does that make sense?
On 1/28/19 2:12 AM, Patrick O'Callaghan wrote:
Another point: several people have mentioned using /dev/urandom. It's important to note that this is a *pseudo-random* generator. It starts from a random seed, but from that generates a completely deterministic pattern. If you have the seed, you have everything. ...
I think that you are confusing two separate concerns. This thread (as confused and meandering as it is) is concerned with clearing the content of a disk. That is completely unrelated to the concerns about generating secure encryption keys, which appears to be what you're alluding to when you raise concerns about predicting the output of the CSPRNG (cryptographically secure pseudorandom number generator).
Even if you could predict the entire sequence of the CSPRNG (which, to be clear, you can't; Linux continues to feed entropy into the state of the CSPRNG used for /dev/urandom) that wouldn't make writing /dev/urandom to the disk any less secure. Either way, the random data will replace the old data, and the drive itself will never read out the data that has been overwritten. As far as reading data from the drive itself is concerned, writing /dev/zero to the disk is the fastest way to clear a disk, and totally effective.
The question of secure erase is largely academic. If you are a military or intelligence organization and the content of a disk might threaten the lives of people in your organization, then you should do something better than writing zeros to your disk. You face one of several problems including:
1: If you have very old spinning magnetic drives, an attacker might be able to use a Spin Polarized Scanning Tunneling Microscope to read residual data after a disk was zeroed.
2: Either HDD or SDD disks might mark a block bad and remap it. Such a block could in theory still be read by a modified controller (or if an attacker could clear the remapping data).
If you are concerned about a well-funded attacker reading your data, then the preferred solution is one or both of:
1: physically destroy the disk
2: always encrypt your disks from the start and never write unencrypted data
In general, though, this thread has entirely too much speculation and is becoming quite detached from reality.
For further reading on SSDs, this paper may be interesting:
https://www.usenix.org/legacy/events/fast11/tech/full_papers/Wei.pdf
This AMA with a data recovery engineer might be, too.
https://www.reddit.com/r/IAmA/comments/2n02lt/iama_data_recovery_engineer_i_...
On Tue, 2019-01-29 at 16:20 -0800, Gordon Messmer wrote:
On 1/28/19 2:12 AM, Patrick O'Callaghan wrote:
Another point: several people have mentioned using /dev/urandom. It's important to note that this is a *pseudo-random* generator. It starts from a random seed, but from that generates a completely deterministic pattern. If you have the seed, you have everything. ...
I think that you are confusing two separate concerns. This thread (as confused and meandering as it is) is concerned with clearing the content of a disk. That is completely unrelated to the concerns about generating secure encryption keys, which appears to be what you're alluding to when you raise concerns about predicting the output of the CSPRNG (cryptographically secure pseudorandom number generator).
I'm actually not confused, I merely made some remarks about PRNGs, though I admit to not knowing that /dev/urandom did occasionally top up its randomness. I guess trying to chip in something in the middle of confusion creates more confusion ...
Even if you could predict the entire sequence of the CSPRNG (which, to be clear, you can't; Linux continues to feed entropy into the state of the CSPRNG used for /dev/urandom) that wouldn't make writing /dev/urandom to the disk any less secure. Either way, the random data will replace the old data, and the drive itself will never read out the data that has been overwritten. As far as reading data from the drive itself is concerned, writing /dev/zero to the disk is the fastest way to clear a disk, and totally effective.
The question of secure erase is largely academic. If you are a military or intelligence organization and the content of a disk might threaten the lives of people in your organization, then you should do something better than writing zeros to your disk. You face one of several problems including:
1: If you have very old spinning magnetic drives, an attacker might be able to use a Spin Polarized Scanning Tunneling Microscope to read residual data after a disk was zeroed.
2: Either HDD or SDD disks might mark a block bad and remap it. Such a block could in theory still be read by a modified controller (or if an attacker could clear the remapping data).
If you are concerned about a well-funded attacker reading your data, then the preferred solution is one or both of:
1: physically destroy the disk
2: always encrypt your disks from the start and never write unencrypted data
In general, though, this thread has entirely too much speculation and is becoming quite detached from reality.
Agreed.
For further reading on SSDs, this paper may be interesting:
https://www.usenix.org/legacy/events/fast11/tech/full_papers/Wei.pdf
This AMA with a data recovery engineer might be, too.
https://www.reddit.com/r/IAmA/comments/2n02lt/iama_data_recovery_engineer_i_...
Good references, thanks.
poc
On 1/30/19 12:48 AM, Gordon Messmer wrote:
Both the filesystem and your data are encrypted when using dm-crypt (whether you use LUKS or not). However, creating a dm-crypt device doesn't modify the pre-existing data. It only writes encrypted data to the disk when data is written to the virtual device.
Does that make sense?
Yes, and the reason for doing this is that encrypting some zeros can be faster than reading from /dev/urandom. I know for sure that /dev/urandom is (or was) not always able to saturate the speed of a disk (or SSD), while dmcrypt with modern hardware (AES instructions in the CPU) is always fast enough.
Regards.
On Tue, Jan 29, 2019 at 03:48:01PM -0800, Gordon Messmer wrote:
On 1/27/19 6:47 PM, Wolfgang Pfeiffer wrote:
[ .. ]
It sounds like you're unfamiliar with the implementation, and possibly with filesystems and block devices in general. I'll try to explain, with some simplifications. You suggested this command:
cryptsetup open --type plain -d /dev/urandom /dev/<block-device> to_be_wiped
This creates a mapping in the kernel where blocks on the virtual device named "to_be_wiped" are filtered through an encryption algorithm before writing, or after reading, blocks on the physical device /dev/<block-device>.
Running this command (unless I'm misreading the docs for the "plain" section) doesn't actually write to or modify the content of the block device at all. Only when you write to /dev/mapper/to_be_wiped will encrypted data actually be written to /dev/<block-device>.
Seems you are right: I've tested it on a USB flash drive, with ext4 on it and a few files. After running the command above "cryptsetup open --type ..." targeting that drive, the files on it were not overwritten but still there: mounting the drive the usual way just showed me exactly that ..
So the warning cryptsetup is giving here:
------------- cryptsetup open --type plain -d /dev/urandom /dev/sdd dmcrypt.test WARNING: Device /dev/sdd already contains a 'ext4' superblock signature.
WARNING! ======== Detected device signature(s) on /dev/sdd. Proceeding further may damage existing data.
[ .. ] -------------
is technically correct (*may* damage ..) - but chances seem to be extremely good the data on the targeted drive are still alive - So the warning might give the user an unjustified sense of security.
(You would normally create a new filesystem on /dev/mapper/to_be_wiped, which would replace a small number of blocks on the disk with new blocks containing encrypted data related to the filesystem. Again, most of the pre-existing data isn't overwritten by that process. The old data would not be readable through /dev/mapper/to_be_wiped, but could be directly through /dev/<block-device>.)
Understood, I think ..
The next step of the process you suggested was:
cat /dev/zero > /dev/mapper/to_be_wiped
This will write zeros to the virtual block device, which will be filtered through the encryption algorithm and the results written to the blocks on the physical device. This will sequentially overwrite the contents of the physical disk.
In other words: It seems the file system is encrypted, not the data: see the already mentioned FAQ: "Create the LUKS container" - at the end of the section: "Done. You can now use the encrypted file system to store data"
The command you referenced above doesn't use LUKS, but that's a minor nit.
Both the filesystem and your data are encrypted when using dm-crypt (whether you use LUKS or not). However, creating a dm-crypt device doesn't modify the pre-existing data. It only writes encrypted data to the disk when data is written to the virtual device.
Provided it works: yes. I had situations where I first ran
cryptsetup open --type plain -d /dev/urandom /dev/<block-device> to_be_wiped then cat /dev/zero > /dev/mapper/to_be_wiped or something similar like dd if=/dev/zero of=/dev/mapper/to_be_wiped bs=1M status=progress
and the data on the targeted drive were still there. No idea whether this happened because of faulty hardware or because of a software bug, or whatever. A sign that something like "cat /dev/zero .. " doesn't write encrypted data to the physical drive might be that the command finishes after only a few seconds (didn't take ten secs here, IIRC)
One last "cat /dev/zero .." worked 'tho after closing "gnome-disks", which previously ran while issuing the cat or dd command. And the reason I tell that is not to open a debate about why cat or dd commands didn't work in that situation. It's just a reminder to be careful if a user is trying to secure their data this way.
Does that make sense?
Definitely: Thank you very much for your time and the effort to explain all this.
Regards, Wolfgang
On Sat, Feb 02, 2019 at 02:48:11PM +0100, Wolfgang Pfeiffer wrote:
So the warning cryptsetup is giving here:
cryptsetup open --type plain -d /dev/urandom /dev/sdd dmcrypt.test WARNING: Device /dev/sdd already contains a 'ext4' superblock signature.
WARNING!
Detected device signature(s) on /dev/sdd. Proceeding further may damage existing data.
[ .. ]
is technically correct (*may* damage ..) - but chances seem to be
should say: "may be technically correct" ...
Sorry. Wolfgang
On 28/01/2019 20:30, Robert Nichols wrote:
On 1/28/19 8:03 AM, Ian Malone wrote:
I wouldn't recommend just doing /dev/zero if the CIA, or even a moderately funded newspaper might specifically be after your data,
I would be interested to know if you can name any data recovery service that has ever demonstrated the ability to recover data from a reasonably modern hard disk that has been overwritten once with zeros.
Wasn't there a prize years ago for any business that could demonstrate the ability to read a drive that was write just once?
I had an issues years ago. I accidentally reformatted the wrong drive but I could recover much data. It was not encrypted though.
Reformatting, doesn't erase any data.
As any raid user knows, rebuilding a raid with multi TB drives takes ages. Why would we expect a random write to work much faster?
Robin
-
Beartooth -
Erik P. Olsen -
fred roller -
Fred Smith -
Gordon Messmer -
Ian Malone -
jdow -
Joe Zeff -
Jonathan Dieter -
Jonathan Ryshpan -
Mike Wright -
Patrick O'Callaghan -
Robert Nichols -
Roberto Ragusa -
Robin Laing -
Samuel Sieb -
sixpack13 -
Tim -
ToddAndMargo -
Wolfgang Pfeiffer