More on my compressing images from SDcards.
I only need the boot info and 1st partition, so I remove the other partitions, but of course there is still 1s out there so it will not compress efficiently. I want to zero out the end of the card so first I use fdisk:
# fdisk -l /dev/sdb
Disk /dev/sdb: 7.4 GiB, 7969177600 bytes, 15564800 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disklabel type: dos Disk identifier: 0x0009e2ad
Device Boot Start End Blocks Id System /dev/sdb1 8192 1007615 499712 83 Linux
Then dd:
# dd if=/dev/zero of=/dev/sdb bs=1M seek=1007620 dd: ‘/dev/sdb’: cannot seek: Invalid argument 0+0 records in 0+0 records out 0 bytes (0 B) copied, 0.000652852 s, 0.0 kB/s
======================
seems you cannot seek on an SDcard. How do I do this? Since the change in partition 1 is trivial, if I have to start again from an SDcard with all 3 paritions, I can do that...
On Thu, 21 Aug 2014, Robert Moskowitz wrote:
More on my compressing images from SDcards.
I only need the boot info and 1st partition, so I remove the other partitions, but of course there is still 1s out there so it will not compress efficiently. I want to zero out the end of the card so first I use fdisk:
# fdisk -l /dev/sdb
Disk /dev/sdb: 7.4 GiB, 7969177600 bytes, 15564800 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disklabel type: dos Disk identifier: 0x0009e2ad
Device Boot Start End Blocks Id System /dev/sdb1 8192 1007615 499712 83 Linux
Then dd:
# dd if=/dev/zero of=/dev/sdb bs=1M seek=1007620 dd: ‘/dev/sdb’: cannot seek: Invalid argument 0+0 records in 0+0 records out 0 bytes (0 B) copied, 0.000652852 s, 0.0 kB/s
======================
seems you cannot seek on an SDcard. How do I do this? Since the change in partition 1 is trivial, if I have to start again from an SDcard with all 3 paritions, I can do that...
it seems like you're trying to seek more than a terabyte into that card.
rday
On 08/21/2014 07:51 AM, Robert Moskowitz wrote:
More on my compressing images from SDcards.
I only need the boot info and 1st partition, so I remove the other partitions, but of course there is still 1s out there so it will not compress efficiently. I want to zero out the end of the card so first I use fdisk:
# fdisk -l /dev/sdb
Disk /dev/sdb: 7.4 GiB, 7969177600 bytes, 15564800 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disklabel type: dos Disk identifier: 0x0009e2ad
Device Boot Start End Blocks Id System /dev/sdb1 8192 1007615 499712 83 Linux
Then dd:
# dd if=/dev/zero of=/dev/sdb bs=1M seek=1007620 dd: ‘/dev/sdb’: cannot seek: Invalid argument 0+0 records in 0+0 records out 0 bytes (0 B) copied, 0.000652852 s, 0.0 kB/s
======================
seems you cannot seek on an SDcard. How do I do this? Since the change in partition 1 is trivial, if I have to start again from an SDcard with all 3 paritions, I can do that...
I'd create a new partition on the remainder of the card, and then dd /dev/zero to that partition. I don't know squat about the bits and pieces of partition structure but that should wipe most of it at least.
On 08/21/2014 09:07 AM, Robert P. J. Day wrote:
On Thu, 21 Aug 2014, Robert Moskowitz wrote:
More on my compressing images from SDcards.
I only need the boot info and 1st partition, so I remove the other partitions, but of course there is still 1s out there so it will not compress efficiently. I want to zero out the end of the card so first I use fdisk:
# fdisk -l /dev/sdb
Disk /dev/sdb: 7.4 GiB, 7969177600 bytes, 15564800 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disklabel type: dos Disk identifier: 0x0009e2ad
Device Boot Start End Blocks Id System /dev/sdb1 8192 1007615 499712 83 Linux
Then dd:
# dd if=/dev/zero of=/dev/sdb bs=1M seek=1007620 dd: ‘/dev/sdb’: cannot seek: Invalid argument 0+0 records in 0+0 records out 0 bytes (0 B) copied, 0.000652852 s, 0.0 kB/s
======================
seems you cannot seek on an SDcard. How do I do this? Since the change in partition 1 is trivial, if I have to start again from an SDcard with all 3 paritions, I can do that...
it seems like you're trying to seek more than a terabyte into that card.
Well, man says about seek:
seek=N skip N obs-sized blocks at start of output
and obs defaults to 512 bytes, and I *THOUGHT* fdisk was reporting # 512 blocks, not bytes.
On Thu, 21 Aug 2014, Robert Moskowitz wrote:
On 08/21/2014 09:07 AM, Robert P. J. Day wrote:
On Thu, 21 Aug 2014, Robert Moskowitz wrote:
More on my compressing images from SDcards.
I only need the boot info and 1st partition, so I remove the other partitions, but of course there is still 1s out there so it will not compress efficiently. I want to zero out the end of the card so first I use fdisk:
# fdisk -l /dev/sdb
Disk /dev/sdb: 7.4 GiB, 7969177600 bytes, 15564800 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disklabel type: dos Disk identifier: 0x0009e2ad
Device Boot Start End Blocks Id System /dev/sdb1 8192 1007615 499712 83 Linux
Then dd:
# dd if=/dev/zero of=/dev/sdb bs=1M seek=1007620 dd: ‘/dev/sdb’: cannot seek: Invalid argument 0+0 records in 0+0 records out 0 bytes (0 B) copied, 0.000652852 s, 0.0 kB/s
======================
seems you cannot seek on an SDcard. How do I do this? Since the change in partition 1 is trivial, if I have to start again from an SDcard with all 3 paritions, I can do that...
it seems like you're trying to seek more than a terabyte into that card.
Well, man says about seek:
seek=N skip N obs-sized blocks at start of outputand obs defaults to 512 bytes, and I *THOUGHT* fdisk was reporting # 512 blocks, not bytes.
if memory serves, using "bs=" overrides both ibs= and obs= ... here's from the man page:
bs=BYTES read and write up to BYTES bytes at a time
notice that says "read *and* write", so i'm simply assuming it is the value used for both ibs and obs.
rday
Once upon a time, Robert Moskowitz rgm@htt-consult.com said:
# dd if=/dev/zero of=/dev/sdb bs=1M seek=1007620
<snip>
Well, man says about seek:
seek=N skip N obs-sized blocks at start of outputand obs defaults to 512 bytes, and I *THOUGHT* fdisk was reporting # 512 blocks, not bytes.
fdisk reports sectors (that's the default in recent years, it can be configured to report in different units). However, look again at the dd command; the block size was set to 1M (=1048576 bytes), and then the seek was set to 1007620 blocks, which is almost 1TB. If you are going to raise the block size, you have to recalculate everything else that is expressed in blocks.
On 08/21/2014 09:50 AM, Chris Adams wrote:
Once upon a time, Robert Moskowitz rgm@htt-consult.com said:
# dd if=/dev/zero of=/dev/sdb bs=1M seek=1007620
<snip> > Well, man says about seek: > > seek=N skip N obs-sized blocks at start of output > > and obs defaults to 512 bytes, and I *THOUGHT* fdisk was reporting # > 512 blocks, not bytes. fdisk reports sectors (that's the default in recent years, it can be configured to report in different units). However, look again at the dd command; the block size was set to 1M (=1048576 bytes), and then the seek was set to 1007620 blocks, which is almost 1TB. If you are going to raise the block size, you have to recalculate everything else that is expressed in blocks.
Ah...
I had cobbled this together from instructions on how to zero out a whole drive. That is were the bs= comes from. So this overrides the 512bytes with a much bigger number, making the seek= wrong. Got it. Will try again.
On Aug 21, 2014, at 6:51 AM, Robert Moskowitz rgm@htt-consult.com wrote:
More on my compressing images from SDcards.
I only need the boot info and 1st partition, so I remove the other partitions, but of course there is still 1s out there so it will not compress efficiently. I want to zero out the end of the card so first I use fdisk:
# fdisk -l /dev/sdb
Disk /dev/sdb: 7.4 GiB, 7969177600 bytes, 15564800 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disklabel type: dos Disk identifier: 0x0009e2ad
Device Boot Start End Blocks Id System /dev/sdb1 8192 1007615 499712 83 Linux
These 487MB aren't zero'd, so whatever is reported as free space by the file system will have stale data on its sectors. Mount sdb1 and use fstrim -v <mountpoint> and see if the command is accepted.
Then dd:
# dd if=/dev/zero of=/dev/sdb bs=1M seek=1007620
It's kinder to your sdcard to create a 2nd partition for the remaining space and trim it. This can be done with either mkfs.btrfs or mkfs.ext4, if the kernel detects it's solid state then both commands issue trim for all of those blocks, and then creates a file system. I'd use btrfs only because it writes out far less metadata.
However, quite a few sdcards don't respond to trim. And if they do, some have non-deterministic read of 'erased/trimmed' pages, in which case even though they're marked as erased, they can still return old data or garbage. And in that case unfortunately zeroing is the only way out.
Chris Murphy
Once upon a time, Chris Murphy lists@colorremedies.com said:
However, quite a few sdcards don't respond to trim.
Do _any_ SD cards support TRIM? I didn't think that was supported by the protocol at all.
On Aug 21, 2014, at 11:00 AM, Chris Adams linux@cmadams.net wrote:
Once upon a time, Chris Murphy lists@colorremedies.com said:
However, quite a few sdcards don't respond to trim.
Do _any_ SD cards support TRIM? I didn't think that was supported by the protocol at all.
It's a good question. fstrim spit back a message to the effect the device doesn't support trim. When I mkfs.btrfs, however, it reports trimming. And then when I dd read LBA's known to have previously contained data, I get zeros. *shrug* I don't actually know what's going on.
And then there's this: https://www.sdcard.org/downloads/formatter_4/
"Using generic formatting utilities may result in less than optimal performance for your memory cards." So that implies they have something like trim, even if not the ATA variety. So I'm curious what this formatter is doing, and I'm also curious if we have an equivalent on linux or if in fact we get it already with normal mkfs; but then if that's true why does fstrim fail to do it.
Chris Murphy
SD Specifications Part 1 Physical Layer Simplified Specification Version 4.10 https://www.sdcard.org/downloads/pls/simplified_specs/part1_410.pdf
page 38
4.3.5 Erase It is desirable to erase many write blocks simultaneously in order to enhance the data throughput. Identification of these write blocks is accomplished with the ERASE_WR_BLK_START (CMD32), ERASE_WR_BLK_END (CMD33) commands. The host should adhere to the following command sequence: ERASE_WR_BLK_START, ERASE_WR_BLK_END and ERASE (CMD38).
So there's something like trim, but trim is an ATA command. But I don't know whether this is what mkfs.btrfs is using, or if it's asking some other library to do it, and it's that library that differentiates and mkfs.btrfs has no idea. And then whether fstrim is using some other library and hence the difference.
Rabbit hole.
Chris Murphy
PNY SDHC card, 16GB - pretty generic. In this case fstrim did work (pfft yeah whatever it didn't the last time I tried it), and so did mkfs.btrfs but they work differently according to strace.
fstrim issues: ioctl(3, FITRIM, 0x7fffbf6b87e0) = 0
The result is write(1, "/mnt/: 13.9 MiB (14598144 bytes)"…, 41/mnt/: 13.9 MiB (14598144 bytes) trimmed
And this is roughly the amount of data deleted during the current mount. It's a 13GB partition, so fstrim is definitely not going to assure you of erasing previously unerased blocks. Just what the fs is aware of having been recently deleted. LBA's with known data in them still had data in them after fstrim, even though they were long ago deleted files from a completely different fs volume.
mkfs.btrfs issues: write(2, "Performing full device TRIM (13."..., 43Performing full device TRIM (13.72GiB) ... ) = 43 ioctl(3, BLKDISCARD, {0, 7fff920ee4a0}) = 0 ioctl(3, BLKDISCARD, {40000000, 7fff920ee4a0}) = 0 ioctl(3, BLKDISCARD, {80000000, 7fff920ee4a0}) = 0 ioctl(3, BLKDISCARD, {c0000000, 7fff920ee4a0}) = 0 ioctl(3, BLKDISCARD, {100000000, 7fff920ee4a0}) = 0 ioctl(3, BLKDISCARD, {140000000, 7fff920ee4a0}) = 0 ioctl(3, BLKDISCARD, {180000000, 7fff920ee4a0}) = 0 ioctl(3, BLKDISCARD, {1c0000000, 7fff920ee4a0}) = 0 ioctl(3, BLKDISCARD, {200000000, 7fff920ee4a0}) = 0 ioctl(3, BLKDISCARD, {240000000, 7fff920ee4a0}) = 0 ioctl(3, BLKDISCARD, {280000000, 7fff920ee4a0}) = 0 ioctl(3, BLKDISCARD, {2c0000000, 7fff920ee4a0}) = 0 ioctl(3, BLKDISCARD, {300000000, 7fff920ee4a0}) = 0 ioctl(3, BLKDISCARD, {340000000, 7fff920ee4a0}) = 0
And this obliterated everything on that partition. A dd read came up with 99.99% zeros, the rest of it was just the btrfs superblocks.
I wasn't successful using scsi kernel tracing to find out what commands are actually sent to the sdcard for either of these. The trace_pipe file only showed the ssd events, not for /dev/mmcblk0 and I'm not sure how to trace it.
But based on available info I'm pretty sure it's not actually getting issued TRIM in the ATA sense, but it's probably getting the SD Card equivalent which seems to be the ERASE_* set of commands.
Chris Murphy
Nice.
http://www.spinics.net/lists/linux-btrfs/msg36918.html
No special tool needed. e.g. I just tested this with Gnome Disks, with the default "don't overwrite with zeros (quick format)" option, using btrfs. All of, and only that chosen partition, was completely erased in about 2 seconds. Every sector returns FF's [1] other than btrfs superblocks.
I did a strace on mkfs.xfs, mkfs.ext4, and mkfs.vfat -F32; but did not test this in Gnome Disks. The strace reveals that both xfs and ext4 use BLKDISCARD, while vfat does not. xfs and ext4 further makes use of:
ioctl(3, BLKDISCARDZEROES, 0) = 0
I'm not sure what this does. Maybe it causes erased blocks to deterministically return zeros instead of FF's?
Anyway, it looks like SD cards are in effect "trimmed" by formatting them ext4, xfs or btrfs, as long as the tool being used is using the fs's mkfs command, and also doesn't pass a nodiscard option.
I didn't test mkfs.ntfs or mkfs.hfs, but like vfat if any of them aren't calling BLKDISCARD it's probably a valid feature request for their upstream. Meanwhile a workaround is to use mkfs.btrfs with no options, and then wipefs -a that same partition, and then format with the preferred filesystem.
Chris Murphy
[1] [root@twenty1 chris]# dd if=/dev/mmcblk0p2 bs=1M skip=313 | hexdump -C 00000000 ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff |................|
-
Chris Adams -
Chris Murphy -
Robert Moskowitz -
Robert P. J. Day -
Stephen Berg (Contractor)