Command line for creating partitions

[Robert Moskowitz]

I am working now more on handcrafting my SD cards for arm testing. 
Gparted did not do a good job, allowing me to make parititions not on 
'cylinder boundaries'.  And the labels it created were not recognized 
when I mounted the drive.  I had to use the disk utility to fix the 
labels.  Anyway, to script it and to put this up on some wikis, I really 
need to do this by command line.

So I have looked at both fdisk and parted.  Neither are for 'simple' 
command lines.  Fdisk takes me back to my DOS days (wonder where MS got 
it from?).

So first I want a command that will delete all partitions on /dev/sdb

then create a partition as ext3, then one as linux-swap, and finally 
ext4.  Of course, I understand how many MB I want each, but I am suppose 
to (or so from the warnings that 'fdisk -l' provided) maintain boundaries.

thanks for any pointers to the best tool(s) for this.  So far my search 
foo has only gone to old fdisk pages.

SOLVED? - Re: Command line for creating partitions

[Chris Murphy]

On Aug 10, 2014, at 5:47 PM, Robert Moskowitz <rgm(a)htt-consult.com&gt; wrote:
>  
>  Thanks for all the help.  And I figured out another way when you want a real clean card
> for dd of an image then xz for a compressed form for distribution.  I did:
>  
>  dd if=/dev/zero of=/dev/sdb bs=1M
>  
>  and everything is gone. 
Three critiques :-)
1. This doesn't remove the backup GPT, if present, which is at the end of the drive in
the last ~34 sectors.
2. It's definitely not reversible, so if you make a mistake (which you shouldn't
anyway of course), maybe one of your boots flips drive designations (which does happen and
is expected), you've just obliterated the wrong drive.
3. It doesn't remove filesystem signatures beyond 1MB, so they could remotely be
trouble down the road.

So I still like wipefs -a for each partition, then for the whole device, better.



>   Built the partition table, the partitions, laid down the files,
> booted up clean, then dd the image, xz compressed it and I now have a ice compressed image
> I can restore to start at a clean point and can distribute to others. 
Oooooh. Well, you will have latent sectors that previously had data on them that will be
vacuumed up with this dd image because it's a sector copy. So even unused sectors that
have latent data will find their data in your dd image and xz will be thwarted making it
as small (and private) as possible. Who knows what data is on those old latent sectors. So
now you can ignore everything else I suggested prior to this.

If SD card reacts to TRIM, which many do, you can:
a.) mkfs.btrfs /dev/sdX
b.) wipefs -a /dev/sdX   

The default mkfs.btrfs behavior is a whole device TRIM, then it writes out a tiny amount
of metadata, much less than other filesystems. Using wipefs removes that btrfs signature,
and now you can build your media. On thing though is that SD card garbage collection might
take a while and depends a lot on its firmware. So in the short term the dd read of the
sdcard might still return latent data, until the firmware actually goes about erasing it.

Another option, is ATA security erase, but I don't know if SD cards support this. I
know the above will work because I've been doing it the past few days for other
testing.

Either way, the result is that there will be essentially no latent sector data, and your
dd image will contain piles of zeros that xz will then compress away.

Chris Murphy

SOLVED? - Re: Command line for creating partitions

[Robert Moskowitz]

On 08/10/2014 05:20 PM, Chris Murphy wrote:
>  On Aug 9, 2014, at 9:07 PM, Robert Moskowitz
> <rgm(a)htt-consult.com&gt; wrote:
>
> > On 08/08/2014 06:11 PM, Chris Murphy wrote:
> >> On Aug 8, 2014, at 2:16 PM, Robert Moskowitz <rgm(a)htt-consult.com&gt; wrote:
> >>
> >>> On 08/08/2014 03:00 PM, Chris Murphy wrote:
> >>>> On Aug 8, 2014, at 4:29 AM, Robert Moskowitz <rgm(a)htt-consult.com&gt;
> wrote:
> >>>>
> >>>>> Unfortuately there is no such command to delete all partitions,
> though you kind of can do it by changing the table type, say from msdos to gpt.
> >>>> I forgot to address this specifically. First, you really should delete
> the filesystem signature before deleting partitions. This makes the filesystem invalid,
> and thus things like libblkid and libparted aren't going to recognize latent (stale)
> filesystems. The tool for this is wipefs part of util-linux. Use it like this for
> example:
> >>>>
> >>>> wipefs -a /dev/sdb[123]
> >>>>
> >>>> That will delete the fs signatures on all file systems found on
> partitions 1 through 3 on disk sdb. The partition table still contains entries of course,
> but the filesystems in them are invalidated.
> >>>>
> >>>> Next, if you want to get rid of all partitions, you can also use wipefs
> on a whole disk.
> >>> wipefs -a /dev/sdb
> >> That will only remove signatures from either an MBR or GPT. It will not remove
> signatures from filesystems. You really should remove filesystem signatures first with
> /dev/sdX[1234…] and then remove the partition map sig with sdX alone.
> > I was very unclear, that is what I meant, that I got the remove sigs for each
> partition, but I am assuming that the command for the partition part was what I gave.  So
> the whole set is:
> >
> > wipefs -a /dev/sdb1
> > wipefs -a /dev/sdb2
> > wipefs -a /dev/sdb3
>  Sure. These can be combined as:
>  wipefs -a /dev/sdb[123]
>
>  There are quite a few examples of this, including making LVM pvs, and Btrfs volumes,
> e.g.
>  mkfs.btrfs -mraid1 -draid1 /dev/sdb[123]
>
>  And also for multiple disk partition table obliteration:
>  wipefs -a /dev/sd[bcd]
>
>  I like wipefs because it tells you what it's removing, so you can restore it, and
> also has an option for backing up that which it's removing so you can easily restore
> it if you got sloppy with the command and bashed the wrong device. Not that this should be
> habit forming, like being able to make a proper gimlet. 

Thanks for all the help.  And I figured out another way when you want a 
real clean card for dd of an image then xz for a compressed form for 
distribution.  I did:

dd if=/dev/zero of=/dev/sdb bs=1M

and everything is gone.  Built the partition table, the partitions, laid 
down the files, booted up clean, then dd the image, xz compressed it and 
I now have a ice compressed image I can restore to start at a clean 
point and can distribute to others.

Making slow, steady progress.  Maybe not as good as other more 
experienced, but hey, for a change I will be providing something. In 
particular Redsleeve for Cubieboards.


> > So much nicer what anaconda does in the kickstart with 'clearpart' !
>  Yes and blivet makes an appearance in F21 proper although I'm not sure yet if this is
> an API or how it'll get leveraged. Another pony trick is system-storage-manager
> package, which takes the form of ssm on the command line; but this tool is post partition
> management, working with LVM and filesystems.
>
>
>  Chris Murphy
>

SOLVED? - Re: Command line for creating partitions

[Chris Murphy]

On Aug 9, 2014, at 9:07 PM, Robert Moskowitz <rgm(a)htt-consult.com&gt; wrote:

>  
>  On 08/08/2014 06:11 PM, Chris Murphy wrote:
> > On Aug 8, 2014, at 2:16 PM, Robert Moskowitz <rgm(a)htt-consult.com&gt; wrote:
> > 
> >> On 08/08/2014 03:00 PM, Chris Murphy wrote:
> >>> On Aug 8, 2014, at 4:29 AM, Robert Moskowitz <rgm(a)htt-consult.com&gt;
> wrote:
> >>> 
> >>>> Unfortuately there is no such command to delete all partitions, though
> you kind of can do it by changing the table type, say from msdos to gpt.
> >>> I forgot to address this specifically. First, you really should delete the
> filesystem signature before deleting partitions. This makes the filesystem invalid, and
> thus things like libblkid and libparted aren't going to recognize latent (stale)
> filesystems. The tool for this is wipefs part of util-linux. Use it like this for
> example:
> >>> 
> >>> wipefs -a /dev/sdb[123]
> >>> 
> >>> That will delete the fs signatures on all file systems found on partitions 1
> through 3 on disk sdb. The partition table still contains entries of course, but the
> filesystems in them are invalidated.
> >>> 
> >>> Next, if you want to get rid of all partitions, you can also use wipefs on a
> whole disk.
> >> wipefs -a /dev/sdb
> > That will only remove signatures from either an MBR or GPT. It will not remove
> signatures from filesystems. You really should remove filesystem signatures first with
> /dev/sdX[1234…] and then remove the partition map sig with sdX alone.
>  
>  I was very unclear, that is what I meant, that I got the remove sigs for each partition,
> but I am assuming that the command for the partition part was what I gave.  So the whole
> set is:
>  
>  wipefs -a /dev/sdb1
>  wipefs -a /dev/sdb2
>  wipefs -a /dev/sdb3 
Sure. These can be combined as:
wipefs -a /dev/sdb[123]

There are quite a few examples of this, including making LVM pvs, and Btrfs volumes, e.g.
mkfs.btrfs -mraid1 -draid1 /dev/sdb[123]

And also for multiple disk partition table obliteration:
wipefs -a /dev/sd[bcd]

I like wipefs because it tells you what it's removing, so you can restore it, and also
has an option for backing up that which it's removing so you can easily restore it if
you got sloppy with the command and bashed the wrong device. Not that this should be habit
forming, like being able to make a proper gimlet.

>  
>  So much nicer what anaconda does in the kickstart with 'clearpart' !   
Yes and blivet makes an appearance in F21 proper although I'm not sure yet if this is
an API or how it'll get leveraged. Another pony trick is system-storage-manager
package, which takes the form of ssm on the command line; but this tool is post partition
management, working with LVM and filesystems.


Chris Murphy

SOLVED? - Re: Command line for creating partitions

[Chris Murphy]

On Aug 8, 2014, at 2:16 PM, Robert Moskowitz <rgm(a)htt-consult.com&gt; wrote:

>  
>  On 08/08/2014 03:00 PM, Chris Murphy wrote:
> > On Aug 8, 2014, at 4:29 AM, Robert Moskowitz <rgm(a)htt-consult.com&gt; wrote:
> > 
> >> Unfortuately there is no such command to delete all partitions, though you kind
> of can do it by changing the table type, say from msdos to gpt.
> > I forgot to address this specifically. First, you really should delete the filesystem
> signature before deleting partitions. This makes the filesystem invalid, and thus things
> like libblkid and libparted aren't going to recognize latent (stale) filesystems. The
> tool for this is wipefs part of util-linux. Use it like this for example:
> > 
> > wipefs -a /dev/sdb[123]
> > 
> > That will delete the fs signatures on all file systems found on partitions 1 through
> 3 on disk sdb. The partition table still contains entries of course, but the filesystems
> in them are invalidated.
> > 
> > Next, if you want to get rid of all partitions, you can also use wipefs on a whole
> disk.
>  
>  wipefs -a /dev/sdb 
That will only remove signatures from either an MBR or GPT. It will not remove signatures
from filesystems. You really should remove filesystem signatures first with
/dev/sdX[1234…] and then remove the partition map sig with sdX alone.


>  No.  I am developing my own notes on how to build up a boot card (and
> eventually drive) for Redsleeve on my Cubieboard2.  At this stage, I tend to build then
> break a number of times, so having commands to build rather than having to use a GUI
> speeds the rebuild time.  But so far no magic for fstab; I am having to put UUIDs in it.
>  
>  I have been given nice scripts for F19 & F20 remixes and F21 alpha.  We will see what
> I will need for Centos7arm development...
>  
>  I will then be putting all that I did on the Redsleeve wiki. 
blkid will get you the UUID and filesystem type so it's a bit easier to build the
fstab; otherwise to automate it with command line instead of GUI, you'd look at
kickstart installs which leverages anaconda and blivet to do all of these things for you,
then does the install per your requirements.

Chris Murphy

SOLVED? - Re: Command line for creating partitions

[Robert Moskowitz]

On 08/08/2014 03:00 PM, Chris Murphy wrote:
>  On Aug 8, 2014, at 4:29 AM, Robert Moskowitz
> <rgm(a)htt-consult.com&gt; wrote:
>
> > Unfortuately there is no such command to delete all partitions, though you kind of
> can do it by changing the table type, say from msdos to gpt.
>  I forgot to address this specifically. First, you really should delete the filesystem
> signature before deleting partitions. This makes the filesystem invalid, and thus things
> like libblkid and libparted aren't going to recognize latent (stale) filesystems. The
> tool for this is wipefs part of util-linux. Use it like this for example:
>
>  wipefs -a /dev/sdb[123]
>
>  That will delete the fs signatures on all file systems found on partitions 1 through 3 on
> disk sdb. The partition table still contains entries of course, but the filesystems in
> them are invalidated.
>
>  Next, if you want to get rid of all partitions, you can also use wipefs on a whole disk.
wipefs -a /dev/sdb

??

thanks for all of this.

>    This removes the signature for the partition table. Most any tool
> will consider it invalid, rather than broken, so it's not going to offer to fix it,
> it'll offer to repartition it: so in the GPT case, it gets a whole new disk identifier
> GUID rather than just restoring the signature. It is possible, btw to restore the
> signature and thus restore the partition table and all of its partitions (since those
> sectors aren't actually erased).
>
>  You could also blow away 34 sectors from the start and end of the drive using dd
> if=/dev/zero. That's a hammer.
>
>  Oh and I mentioned cgdisk (member of the gdisk family) that was wrong, it's
> curse-based. You want to look at sgdisk which is for use in scripts and accepts all
> commands from the CLI. It has a way to delete partitions individually. Note that this does
> not employ wipefs, so the actual filesystem contained within the partition you've
> deleted is still intact; and this also leaves the partition header intact, all it's
> doing is removing a partition table entry. sgdisk also has two zap options: one overwrites
> everything (sectors containing both MBR and GPT structures), the other option overwrites
> only the sectors containing GPT structures.
>
>  So it really depends what you want to achieve, and how arbitrary the source drives are
> going to be.
>
>  If you're writing a program or script you might look at python-blivet which has done
> a ton of work abstracting all of this stuff, if you can do what you need to do in python,
No.  I am developing my own notes on how to build up a boot card (and 
eventually drive) for Redsleeve on my Cubieboard2.  At this stage, I 
tend to build then break a number of times, so having commands to build 
rather than having to use a GUI speeds the rebuild time.  But so far no 
magic for fstab; I am having to put UUIDs in it.

I have been given nice scripts for F19 & F20 remixes and F21 alpha.  We 
will see what I will need for Centos7arm development...

I will then be putting all that I did on the Redsleeve wiki.

>  then you can use sane pthyon code to do things like wipe all
> fs's, delete all partitions, create new GPT, add new partitions, format them. And you
> don't need to know the prose for the 5 different utilities to make that happen.
> python-blivet is actually at the core of the Fedora installer, specifically for
> manipulating storage (it does everything you can imagine in including create, modify,
> destroy LVM objects; btrfs subvolumes; bunch of md raid stuff, etc.)
>
>  http://pkgs.fedoraproject.org/cgit/python-blivet.git/

SOLVED? - Re: Command line for creating partitions

[Robert Moskowitz]

On 08/08/2014 02:24 PM, Chris Murphy wrote:
>  On Aug 8, 2014, at 4:29 AM, Robert Moskowitz
> <rgm(a)htt-consult.com&gt; wrote:
>
> > I am learning how to use parted in command line format. Unfortuately there is no such
> command to delete all partitions, though you kind of can do it by changing the table type,
> say from msdos to gpt.
> >
> > Also learned that the unused 4Mb I am seeing on most SD cards is for a reason.  To
> get on the 4Mb alignment for performance reasons.
>  It's 4KB, not 4MB, and it doesn't apply to SSDs, only HDD's specifically the
> 512e AF variety. a.) It's AF, Advanced Format, which means it has 4096 byte physical
> sectors, and 512e means it's emulating 512 byte sectors. So it appears as having 512
> byte logical sectors, while having 4096 byte physical sectors. So one real sector has 8
> logical sectors, which is where the performance problem can occur, when effectively asking
> the drive to update portions of a sector it causes the drive firmware to read-modify-write
> the sector. 
It is for SDcards that I was dealing with, and my information comes 
from:  http://www.gnu.org/software/parted/manual/parted.html

Now, we will show how to partition a low-end flash device (“low-end”, as 
of 2011/2012). For such devices, you should use 4MiB-aligned partition^. 
This command creates a tiny place-holder partition at the beginning, and 
then uses all remaining space to create the partition you’ll actually use:

$parted -s /dev/sdX -- mklabel msdos \
     mkpart primary fat32 64s 4MiB \
     mkpart primary fat32 4MiB -1s


I figured out that you really don't need the tiny place-holder, just 
leave it unallocated.

Good to know about HDD and SSD as that will be later....

thanks


> > parted /dev/sdb mkpart offset fat32 0 4
>  When I do that, parted complains.
>  "Warning: The resulting partition is not properly aligned for best
> performance."
>
>  What you should do is use 1 and 5, assuming you really want only a 4MB partition. In that
> case the result ends up being:
>
>  Number  Start  End     Size   File system  Name    Flags
>    1      2048s  10239s  8192s               offset  msftdata
>
>  If you want to agree to the widely used convention, use LBA 2048 as the start. If you
> want to do your own thing, you could find out the erase block size for your SSD and align
> on those boundaries, but due to the non-transparency of the flash translation layer,
> there's no guarantee this means you're actually aligned on erase block boundaries:
> but it seems like a good idea, doesn't hurt anything, but might be a waste of effort
> depending on how things are actually implemented in the SSD.
>
>
>
>
>
>
> > parted /dev/sdb mkpart uboot ext3 4 516
> >
> > But it is still acting strange. Print is showing that ext3 partition fat32!
>  That's probably because libparted is finding a latent FAT32 superblock. The first
> thing to understand about parted, is that it's crusty. It used to do a lot of stuff
> that it doesn't do anymore but has the legacy operands of what it used to do. It used
> to format and resize volumes and partitions in one go, and it no longer does this. There
> is no partition type code that tells parted what the filesystem is, so it looks (maybe via
> libblkid, not sure) for the actual partition contents to see what filesystem it is and
> tells you that. But it's totally superfluous information in a partitioning tool. But
> in parted parlance you have to know these weird things because of its legacy. Likewise it
> doesn't directly tell you, or let you directly tell it, what a GPT partition's
> partition type GUID is. You have to tell it a filesystem, and then it sets the type code
> accordingly, which is quite honestly maddening. But whatever you just have to accept it if
> you're going to use parted. Its view of the world is abstracted to the degree
> it'll give you a completely distorted picture, and it's why I don't like it.
>
>  So your first command with fat32, doesn't actually cause the partition to be
> formatted fat32. It merely set the partition type GUID to
> EBD0A0A2-B9E5-4433-87C0-68B6B72699C7 (Microsoft basic data). Due to poor decision making,
> if you're using a Fedora 20 or older version, parted uses that same type code when
> specifying ext3 as the file system. Out of essentially unlimited UUIDs parted devs chose
> to usurp a Microsoft one. It baffles me whether I'm sober or drunk. The Fedora 21
> version, parted will set an ext3 partition to partition type code
> 0FC63DAF-8483-4772-8E79-3D69D8477DE4 (Linux filesystem) instead. And meanwhile gdisk and
> cgdisk have done this correctly for years.
>
>  The parted "File system" column is not a reflection of the partition type GUID,
> it's actually looking for filesystem superblocks to identify what the filesystem is.
>
>
>  Chris Murphy

SOLVED? - Re: Command line for creating partitions

[Chris Murphy]

On Aug 8, 2014, at 4:29 AM, Robert Moskowitz <rgm(a)htt-consult.com&gt; wrote:

>  Unfortuately there is no such command to delete all partitions,
> though you kind of can do it by changing the table type, say from msdos to gpt. 
I forgot to address this specifically. First, you really should delete the filesystem
signature before deleting partitions. This makes the filesystem invalid, and thus things
like libblkid and libparted aren't going to recognize latent (stale) filesystems. The
tool for this is wipefs part of util-linux. Use it like this for example:

wipefs -a /dev/sdb[123]

That will delete the fs signatures on all file systems found on partitions 1 through 3 on
disk sdb. The partition table still contains entries of course, but the filesystems in
them are invalidated.

Next, if you want to get rid of all partitions, you can also use wipefs on a whole disk.
This removes the signature for the partition table. Most any tool will consider it
invalid, rather than broken, so it's not going to offer to fix it, it'll offer to
repartition it: so in the GPT case, it gets a whole new disk identifier GUID rather than
just restoring the signature. It is possible, btw to restore the signature and thus
restore the partition table and all of its partitions (since those sectors aren't
actually erased).

You could also blow away 34 sectors from the start and end of the drive using dd
if=/dev/zero. That's a hammer.

Oh and I mentioned cgdisk (member of the gdisk family) that was wrong, it's
curse-based. You want to look at sgdisk which is for use in scripts and accepts all
commands from the CLI. It has a way to delete partitions individually. Note that this does
not employ wipefs, so the actual filesystem contained within the partition you've
deleted is still intact; and this also leaves the partition header intact, all it's
doing is removing a partition table entry. sgdisk also has two zap options: one overwrites
everything (sectors containing both MBR and GPT structures), the other option overwrites
only the sectors containing GPT structures.

So it really depends what you want to achieve, and how arbitrary the source drives are
going to be.

If you're writing a program or script you might look at python-blivet which has done a
ton of work abstracting all of this stuff, if you can do what you need to do in python,
then you can use sane pthyon code to do things like wipe all fs's, delete all
partitions, create new GPT, add new partitions, format them. And you don't need to
know the prose for the 5 different utilities to make that happen. python-blivet is
actually at the core of the Fedora installer, specifically for manipulating storage (it
does everything you can imagine in including create, modify, destroy LVM objects; btrfs
subvolumes; bunch of md raid stuff, etc.)

http://pkgs.fedoraproject.org/cgit/python-blivet.git/


Chris Murphy

SOLVED? - Re: Command line for creating partitions

[Chris Murphy]

On Aug 8, 2014, at 4:29 AM, Robert Moskowitz <rgm(a)htt-consult.com&gt; wrote:

>  I am learning how to use parted in command line format. Unfortuately
> there is no such command to delete all partitions, though you kind of can do it by
> changing the table type, say from msdos to gpt.
>  
>  Also learned that the unused 4Mb I am seeing on most SD cards is for a reason.  To get on
> the 4Mb alignment for performance reasons. 
It's 4KB, not 4MB, and it doesn't apply to SSDs, only HDD's specifically the
512e AF variety. a.) It's AF, Advanced Format, which means it has 4096 byte physical
sectors, and 512e means it's emulating 512 byte sectors. So it appears as having 512
byte logical sectors, while having 4096 byte physical sectors. So one real sector has 8
logical sectors, which is where the performance problem can occur, when effectively asking
the drive to update portions of a sector it causes the drive firmware to read-modify-write
the sector.


>  
>  parted /dev/sdb mkpart offset fat32 0 4 
When I do that, parted complains.
"Warning: The resulting partition is not properly aligned for best
performance."

What you should do is use 1 and 5, assuming you really want only a 4MB partition. In that
case the result ends up being:

Number  Start  End     Size   File system  Name    Flags
 1      2048s  10239s  8192s               offset  msftdata

If you want to agree to the widely used convention, use LBA 2048 as the start. If you want
to do your own thing, you could find out the erase block size for your SSD and align on
those boundaries, but due to the non-transparency of the flash translation layer,
there's no guarantee this means you're actually aligned on erase block boundaries:
but it seems like a good idea, doesn't hurt anything, but might be a waste of effort
depending on how things are actually implemented in the SSD.






>  parted /dev/sdb mkpart uboot ext3 4 516
>  
>  But it is still acting strange. Print is showing that ext3 partition fat32! 
That's probably because libparted is finding a latent FAT32 superblock. The first
thing to understand about parted, is that it's crusty. It used to do a lot of stuff
that it doesn't do anymore but has the legacy operands of what it used to do. It used
to format and resize volumes and partitions in one go, and it no longer does this. There
is no partition type code that tells parted what the filesystem is, so it looks (maybe via
libblkid, not sure) for the actual partition contents to see what filesystem it is and
tells you that. But it's totally superfluous information in a partitioning tool. But
in parted parlance you have to know these weird things because of its legacy. Likewise it
doesn't directly tell you, or let you directly tell it, what a GPT partition's
partition type GUID is. You have to tell it a filesystem, and then it sets the type code
accordingly, which is quite honestly maddening. But whatever you just have to accept it if
you're going to use parted. Its view of the world is abstracted to the degree
it'll give you a completely distorted picture, and it's why I don't like it.

So your first command with fat32, doesn't actually cause the partition to be formatted
fat32. It merely set the partition type GUID to EBD0A0A2-B9E5-4433-87C0-68B6B72699C7
(Microsoft basic data). Due to poor decision making, if you're using a Fedora 20 or
older version, parted uses that same type code when specifying ext3 as the file system.
Out of essentially unlimited UUIDs parted devs chose to usurp a Microsoft one. It baffles
me whether I'm sober or drunk. The Fedora 21 version, parted will set an ext3
partition to partition type code 0FC63DAF-8483-4772-8E79-3D69D8477DE4 (Linux filesystem)
instead. And meanwhile gdisk and cgdisk have done this correctly for years.

The parted "File system" column is not a reflection of the partition type GUID,
it's actually looking for filesystem superblocks to identify what the filesystem is.


Chris Murphy

SOLVED? - Re: Command line for creating partitions

[Robert Moskowitz]

I am learning how to use parted in command line format. Unfortuately 
there is no such command to delete all partitions, though you kind of 
can do it by changing the table type, say from msdos to gpt.

Also learned that the unused 4Mb I am seeing on most SD cards is for a 
reason.  To get on the 4Mb alignment for performance reasons.  So here 
is some things that I have figured out:

parted /dev/sdb mklabel gpt
parted /dev/sdb mkpart offset fat32 0 4
parted /dev/sdb mkpart uboot ext3 4 516

But it is still acting strange. Print is showing that ext3 partition fat32!

# parted /dev/sdb mkpart uboot ext3 4 516
Information: You may need to update /etc/fstab.

[root@lx120e ~]# parted /dev/sdb print
Model: Generic- Multi-Card (scsi)
Disk /dev/sdb: 7969MB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags:

Number  Start   End     Size    File system  Name    Flags
  1      17.4kB  4000kB  3983kB               offset
  2      4194kB  516MB   512MB   fat32        uboot



On 08/08/2014 01:37 AM, Cameron Simpson wrote:
>  On 07Aug2014 16:29, Robert Moskowitz <rgm(a)htt-consult.com&gt;
> wrote:
> > On 08/07/2014 03:13 PM, Joachim Backes wrote:
> >> have a look at /sbin/cfdisk
> >
> > OK.  One more to study.  I was also told about gdisk.
> >
> > But all of these are command menu programs.  Not one-liners. The nice 
> > 'one-liners' in kckstart files are just commands to anaconda, it 
> > seems.  No such program and 'clearpart' or 'part'. Though I suppose I
>
> > could deal with a program that read a command file of such commands.
>
>  We have used sfdisk for a long time for cloning partition tables from 
>  one drive to another.
>
>  That cloning is done by dumping the table and then reading the dump. 
>  But that means that there is a dump format to sfdisk's output, and you 
>  can just craft the right input.
>
>  Definitely purely command line.
>
>  Cheers,
>  Cameron Simpson <cs(a)zip.com.au&gt;
>
>  Tomkin Coleman (tcolema(a)nv7.uswnvg.com) wrote:
>  | Well, yes, get the bike but BEWARE!!
>  | At first, you will just want to get, like, a 250 Rebel, just for 
>  commuting
>  | and maybe a little fun, but you will wind up buying a 550, (just in 
>  case you
>  | want to tour), and then you will stop paying any attention to the 
>  theatre
>  | company you are working with and the graduate school you are 
>  attending, and
>  | then you will spend all your summers touring and riding around with 
>  your
>  | newfound biking buddies, and then you will find that you have got a 
>  real job
>  | just so you can buy the bike of your dreams and pretty soon you will 
>  find
>  | that your entire life has centered around your monster bike and you 
>  can only
>  | date women who love bikes and hang with buddies who love bikes and 
>  work at a
>  | job that supports your lifestyle and you will be incredibly happy, 
>  but always
>  | very greasy.
>  Shhhh!  The meaning of life is supposed to be a SECRET!   :-)
>  - Bob Larson..DoD#[e-mail]..1994 Suzuki RF600R

[Cameron Simpson]

On 07Aug2014 16:29, Robert Moskowitz <rgm(a)htt-consult.com&gt; wrote:
> On 08/07/2014 03:13 PM, Joachim Backes wrote:
> >have a look at /sbin/cfdisk
>
> OK.  One more to study.  I was also told about gdisk.
>
> But all of these are command menu programs.  Not one-liners.  The nice 
> 'one-liners' in kckstart files are just commands to anaconda, it 
> seems.  No such program and 'clearpart' or 'part'.  Though I suppose I 
> could deal with a program that read a command file of such commands. 
We have used sfdisk for a long time for cloning partition tables from one drive 
to another.

That cloning is done by dumping the table and then reading the dump. But that 
means that there is a dump format to sfdisk's output, and you can just craft 
the right input.

Definitely purely command line.

Cheers,
Cameron Simpson <cs(a)zip.com.au&gt;

Tomkin Coleman (tcolema(a)nv7.uswnvg.com) wrote:
| Well, yes, get the bike but BEWARE!!
| At first, you will just want to get, like, a 250 Rebel, just for commuting
| and maybe a little fun, but you will wind up buying a 550, (just in case you
| want to tour), and then you will stop paying any attention to the theatre
| company you are working with and the graduate school you are attending, and
| then you will spend all your summers touring and riding around with your
| newfound biking buddies, and then you will find that you have got a real job
| just so you can buy the bike of your dreams and pretty soon you will find
| that your entire life has centered around your monster bike and you can only
| date women who love bikes and hang with buddies who love bikes and work at a
| job that supports your lifestyle and you will be incredibly happy, but always
| very greasy.
Shhhh!  The meaning of life is supposed to be a SECRET!   :-)
- Bob Larson..DoD#[e-mail]..1994 Suzuki RF600R

[Michael Hennebry]

On Thu, 7 Aug 2014, Joe Zeff wrote:

>  On 08/07/2014 01:40 PM, Robert Moskowitz wrote:
> > It was fdisk that was complaining.
>
>  Those messages aren't complaints, they're just informing you.  (I'm not 
>  sure any more, but I think that at one time the space between the end of 
>  the partition and the cylinder boundary couldn't be used.)  I suspect 
>  that at this point, the only reason fdisk still tells you that is that 
>  none of the maintainers consider it worth the bother to remove that 
>  code.  Sooner or later, somebody will get tired of seeing it and it will 
>  go away. 
I have had a partition tool quit on me for want of cylinder alignment.
Don't remember which one.

I think that sfdisk will do what OP wants.

-- 
Michael   hennebry(a)web.cs.ndsu.NoDak.edu
"SCSI is NOT magic. There are *fundamental technical
reasons* why it is necessary to sacrifice a young
goat to your SCSI chain now and then."   --   John Woods

[Joe Zeff]

On 08/07/2014 01:40 PM, Robert Moskowitz wrote:
>  It was fdisk that was complaining. 
Those messages aren't complaints, they're just informing you.  (I'm not 
sure any more, but I think that at one time the space between the end of 
the partition and the cylinder boundary couldn't be used.)  I suspect 
that at this point, the only reason fdisk still tells you that is that 
none of the maintainers consider it worth the bother to remove that 
code.  Sooner or later, somebody will get tired of seeing it and it will 
go away.

[Robert Moskowitz]

Chris thanks for this detailed reply!

On 08/07/2014 04:14 PM, Chris Murphy wrote:
>  On Aug 7, 2014, at 12:45 PM, Robert Moskowitz
> <rgm(a)htt-consult.com&gt; wrote:
>
> > I am working now more on handcrafting my SD cards for arm testing. Gparted did not do
> a good job, allowing me to make parititions not on 'cylinder boundaries'.
>  FWIW cylinder boundaries are legacy and irrelevant, for either SSDs (including SD cards)
> or HDDs. You have to go back two epochs to get to CHS being relevant. Everything is LBA
> these days. The boundary you want is simply 2048 for 512 byte (logical) sectors, which is
> a 1MB boundary. That's the easiest and works well for everything, you don't have
> to think about it any further. Any recent parted, fdisk, and gdisk do this. 
It was fdisk that was complaining.

#fdisk -l

Disk /dev/nand: 3984 MB, 3984588800 bytes
255 heads, 63 sectors/track, 484 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0xffffffff


Disk /dev/mmcblk0: 15.6 GB, 15560867840 bytes
4 heads, 16 sectors/track, 474880 cylinders
Units = cylinders of 64 * 512 = 32768 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x000cd80a

         Device Boot      Start         End      Blocks   Id  System
/dev/mmcblk0p1              33       16416      524288   83  Linux
Partition 1 does not end on cylinder boundary.
/dev/mmcblk0p2           16417       49184     1048576   82  Linux swap 
/ Solaris
Partition 2 does not end on cylinder boundary.
/dev/mmcblk0p3           49185      465184    13312000   83  Linux
Partition 3 does not end on cylinder boundary.

> >   And the labels it created were not recognized when I mounted the drive.  I had to
> use the disk utility to fix the labels.
>  Not recognized? That's vague. What does happen? The volume label appears garbled, or
> is blank? What is "disk utility"? There's "Disks" a.k.a.
> gnome-disks. 
With Gparted, I removed the original MSDOS partition on the created and 
formated the above partitions labeling them uboot and rootfs.  I removed 
the drive and remounted it (this wsa on my F20 notebook) and they were 
mounted under /run/media/rgm/ with their UUIDs, not these labels.  I 
started up Gnome-disk-utility and sure enough, no labels, so unmounted, 
I labeled the partitions, and they remounted as I wanted them to be named.

>  Is the SD card being GPT partitioned or MBR partitioned? GPT partitions support partition
> names, MBR partitioning doesn't. So maybe there's a weird conflict somewhere
> between volume labels (a filesystem name), and partition names (a GPT only thing).
>
>
> >   Anyway, to script it and to put this up on some wikis, I really need to do this by
> command line.
> >
> > So I have looked at both fdisk and parted.  Neither are for 'simple' command
> lines.  Fdisk takes me back to my DOS days (wonder where MS got it from?).
> >
> > So first I want a command that will delete all partitions on /dev/sdb
>  fdisk and gdisk have only interactive modes. parted has both command line and interactive
> modes, it sounds like you're only familiar with the interactive mode. But I'm not
> sure if it accepts MB units in command line mode; it does in interactive mode. Possibly
> cfdisk or cgdisk will offer what you want. 
I will look more at parted.

> > Of course, I understand how many MB I want each, but I am suppose to (or so from the
> warnings that 'fdisk -l' provided) maintain boundaries.
>  If you specify all partition start and end values in MB, or in sectors that are 2048
> divisible, it's fine. You can even get away with much less, the only time alignment
> really matters is 4096 sector HDD's, so that's an 8 sector or 4KB alignment.
> It's just that 1MB or 1GB units is ultimately all we care about for partition sizes,
> and they happen to align. 
Further down the road, I will be setting this up on a HD or SSD, so that 
will be a concern before I can say I am 'done'.

[Robert Moskowitz]

On 08/07/2014 03:13 PM, Joachim Backes wrote:
>  On 08/07/2014 08:45 PM, Robert Moskowitz wrote:
> > I am working now more on handcrafting my SD cards for arm testing.
> > Gparted did not do a good job, allowing me to make parititions not on
> > 'cylinder boundaries'.  And the labels it created were not recognized
> > when I mounted the drive.  I had to use the disk utility to fix the
> > labels.  Anyway, to script it and to put this up on some wikis, I really
> > need to do this by command line.
> >
> > So I have looked at both fdisk and parted.  Neither are for 'simple'
> > command lines.  Fdisk takes me back to my DOS days (wonder where MS got
> > it from?).
> >
> > So first I want a command that will delete all partitions on /dev/sdb
> >
> > then create a partition as ext3, then one as linux-swap, and finally
> > ext4.  Of course, I understand how many MB I want each, but I am suppose
> > to (or so from the warnings that 'fdisk -l' provided) maintain boundaries.
> >
> > thanks for any pointers to the best tool(s) for this.  So far my search
> > foo has only gone to old fdisk pages.
> >
>  Hi Robert,
>
>  have a look at /sbin/cfdisk 
OK.  One more to study.  I was also told about gdisk.

But all of these are command menu programs.  Not one-liners.  The nice 
'one-liners' in kckstart files are just commands to anaconda, it seems.  
No such program and 'clearpart' or 'part'.  Though I suppose I could 
deal with a program that read a command file of such commands.

[Chris Murphy]

On Aug 7, 2014, at 12:45 PM, Robert Moskowitz <rgm(a)htt-consult.com&gt; wrote:

>  I am working now more on handcrafting my SD cards for arm testing.
> Gparted did not do a good job, allowing me to make parititions not on 'cylinder
> boundaries'. 
FWIW cylinder boundaries are legacy and irrelevant, for either SSDs (including SD cards)
or HDDs. You have to go back two epochs to get to CHS being relevant. Everything is LBA
these days. The boundary you want is simply 2048 for 512 byte (logical) sectors, which is
a 1MB boundary. That's the easiest and works well for everything, you don't have
to think about it any further. Any recent parted, fdisk, and gdisk do this.


>   And the labels it created were not recognized when I mounted the
> drive.  I had to use the disk utility to fix the labels. 
Not recognized? That's vague. What does happen? The volume label appears garbled, or
is blank? What is "disk utility"? There's "Disks" a.k.a.
gnome-disks.

Is the SD card being GPT partitioned or MBR partitioned? GPT partitions support partition
names, MBR partitioning doesn't. So maybe there's a weird conflict somewhere
between volume labels (a filesystem name), and partition names (a GPT only thing).


>   Anyway, to script it and to put this up on some wikis, I really need
> to do this by command line.
>  
>  So I have looked at both fdisk and parted.  Neither are for 'simple' command
> lines.  Fdisk takes me back to my DOS days (wonder where MS got it from?).
>  
>  So first I want a command that will delete all partitions on /dev/sdb 
fdisk and gdisk have only interactive modes. parted has both command line and interactive
modes, it sounds like you're only familiar with the interactive mode. But I'm not
sure if it accepts MB units in command line mode; it does in interactive mode. Possibly
cfdisk or cgdisk will offer what you want.


>  Of course, I understand how many MB I want each, but I am suppose to
> (or so from the warnings that 'fdisk -l' provided) maintain boundaries. 
If you specify all partition start and end values in MB, or in sectors that are 2048
divisible, it's fine. You can even get away with much less, the only time alignment
really matters is 4096 sector HDD's, so that's an 8 sector or 4KB alignment.
It's just that 1MB or 1GB units is ultimately all we care about for partition sizes,
and they happen to align.



Chris Murphy

[Joachim Backes]

On 08/07/2014 08:45 PM, Robert Moskowitz wrote:
>  I am working now more on handcrafting my SD cards for arm testing. 
>  Gparted did not do a good job, allowing me to make parititions not on 
>  'cylinder boundaries'.  And the labels it created were not recognized 
>  when I mounted the drive.  I had to use the disk utility to fix the 
>  labels.  Anyway, to script it and to put this up on some wikis, I really 
>  need to do this by command line.
>  
>  So I have looked at both fdisk and parted.  Neither are for 'simple' 
>  command lines.  Fdisk takes me back to my DOS days (wonder where MS got 
>  it from?).
>  
>  So first I want a command that will delete all partitions on /dev/sdb
>  
>  then create a partition as ext3, then one as linux-swap, and finally 
>  ext4.  Of course, I understand how many MB I want each, but I am suppose 
>  to (or so from the warnings that 'fdisk -l' provided) maintain boundaries.
>  
>  thanks for any pointers to the best tool(s) for this.  So far my search 
>  foo has only gone to old fdisk pages.
>   
Hi Robert,

have a look at /sbin/cfdisk

It is part of util-linux.

Kind regards

Joachim Backes

-- 

Fedora release 20 (Heisenbug)
Kernel-3.15.8-200.fc20.x86_64


Joachim Backes <joachim.backes(a)rhrk.uni-kl.de&gt;
http://www-user.rhrk.uni-kl.de/~backes

SOLVED? - Re: Command line for creating partitions

[Robert Moskowitz]

thanks.  I will study this a bit.

On 08/11/2014 09:41 PM, Chris Murphy wrote:
>  On Aug 10, 2014, at 5:47 PM, Robert Moskowitz
> <rgm(a)htt-consult.com&gt; wrote:
> > Thanks for all the help.  And I figured out another way when you want a real clean
> card for dd of an image then xz for a compressed form for distribution.  I did:
> >
> > dd if=/dev/zero of=/dev/sdb bs=1M
> >
> > and everything is gone.
>  Three critiques :-)
>  1. This doesn't remove the backup GPT, if present, which is at the end of the drive
> in the last ~34 sectors.
>  2. It's definitely not reversible, so if you make a mistake (which you shouldn't
> anyway of course), maybe one of your boots flips drive designations (which does happen and
> is expected), you've just obliterated the wrong drive.
>  3. It doesn't remove filesystem signatures beyond 1MB, so they could remotely be
> trouble down the road.
>
>  So I still like wipefs -a for each partition, then for the whole device, better.
>
>
>
> >   Built the partition table, the partitions, laid down the files, booted up clean,
> then dd the image, xz compressed it and I now have a ice compressed image I can restore to
> start at a clean point and can distribute to others.
>  Oooooh. Well, you will have latent sectors that previously had data on them that will be
> vacuumed up with this dd image because it's a sector copy. So even unused sectors that
> have latent data will find their data in your dd image and xz will be thwarted making it
> as small (and private) as possible. Who knows what data is on those old latent sectors. So
> now you can ignore everything else I suggested prior to this.
>
>  If SD card reacts to TRIM, which many do, you can:
>  a.) mkfs.btrfs /dev/sdX
>  b.) wipefs -a /dev/sdX
>
>  The default mkfs.btrfs behavior is a whole device TRIM, then it writes out a tiny amount
> of metadata, much less than other filesystems. Using wipefs removes that btrfs signature,
> and now you can build your media. On thing though is that SD card garbage collection might
> take a while and depends a lot on its firmware. So in the short term the dd read of the
> sdcard might still return latent data, until the firmware actually goes about erasing it.
>
>  Another option, is ATA security erase, but I don't know if SD cards support this. I
> know the above will work because I've been doing it the past few days for other
> testing.
>
>  Either way, the result is that there will be essentially no latent sector data, and your
> dd image will contain piles of zeros that xz will then compress away.
>
>  Chris Murphy

SOLVED? - Re: Command line for creating partitions

[Robert Moskowitz]

On 08/08/2014 06:11 PM, Chris Murphy wrote:
>  On Aug 8, 2014, at 2:16 PM, Robert Moskowitz
> <rgm(a)htt-consult.com&gt; wrote:
>
> > On 08/08/2014 03:00 PM, Chris Murphy wrote:
> >> On Aug 8, 2014, at 4:29 AM, Robert Moskowitz <rgm(a)htt-consult.com&gt; wrote:
> >>
> >>> Unfortuately there is no such command to delete all partitions, though you
> kind of can do it by changing the table type, say from msdos to gpt.
> >> I forgot to address this specifically. First, you really should delete the
> filesystem signature before deleting partitions. This makes the filesystem invalid, and
> thus things like libblkid and libparted aren't going to recognize latent (stale)
> filesystems. The tool for this is wipefs part of util-linux. Use it like this for
> example:
> >>
> >> wipefs -a /dev/sdb[123]
> >>
> >> That will delete the fs signatures on all file systems found on partitions 1
> through 3 on disk sdb. The partition table still contains entries of course, but the
> filesystems in them are invalidated.
> >>
> >> Next, if you want to get rid of all partitions, you can also use wipefs on a
> whole disk.
> > wipefs -a /dev/sdb
>  That will only remove signatures from either an MBR or GPT. It will not remove signatures
> from filesystems. You really should remove filesystem signatures first with
> /dev/sdX[1234…] and then remove the partition map sig with sdX alone. 
I was very unclear, that is what I meant, that I got the remove sigs for 
each partition, but I am assuming that the command for the partition 
part was what I gave.  So the whole set is:

wipefs -a /dev/sdb1
wipefs -a /dev/sdb2
wipefs -a /dev/sdb3
wipefs -a /dev/sdb
parted /dev/sdb rm 3
parted /dev/sdb rm 2
parted /dev/sdb rm 1
parted /dev/sdb mklabel msdos (or gpt)

So much nicer what anaconda does in the kickstart with 'clearpart' !  A 
standalone util/script for that is handy for someone working on a USB 
drive of any sort.
> > No.  I am developing my own notes on how to build up a boot card (and eventually
> drive) for Redsleeve on my Cubieboard2.  At this stage, I tend to build then break a
> number of times, so having commands to build rather than having to use a GUI speeds the
> rebuild time.  But so far no magic for fstab; I am having to put UUIDs in it.
> >
> > I have been given nice scripts for F19 & F20 remixes and F21 alpha.  We will see
> what I will need for Centos7arm development...
> >
> > I will then be putting all that I did on the Redsleeve wiki.
>  blkid will get you the UUID and filesystem type so it's a bit easier to build the
> fstab; otherwise to automate it with command line instead of GUI, you'd look at
> kickstart installs which leverages anaconda and blivet to do all of these things for you,
> then does the install per your requirements.
>
>  Chris Murphy
>