df, lvm and 6TB arrays oh my!

[Mark Haney]

Robert Locke wrote:
> On Sat, 2006-02-11 at 11:26 -0500, Mark Haney wrote:
>   
>> I am having a problem with df reporting the wrong disk size on a couple 
>> of 7TB arrays on a Fedora box we are using.  I have an lvm group on each 
>> array of 6.36TB  when I run  df this is what I get:
>>
>> /dev/mapper/Volume02-Volume02lv
>>                       2.4T   33M  2.3T   1% /mnt/arrays/array2
>> /dev/mapper/Volume03-Volume03lv
>>                       2.4T  950G  1.3T  43% /mnt/arrays/array1
>>
>>
>> When I run vg display this is what it reports:
>>
>> pvdisplay  PV Name               /dev/sdd
>> pvdisplay  VG Name               Volume03
>> pvdisplay  PV Size               6.36 TB / not usable 6.00 TB
>> pvdisplay  Allocatable           yes (but full)
>> pvdisplay  PE Size (KByte)       4096
>> pvdisplay  Total PE              1666397
>> pvdisplay  Free PE               0
>> pvdisplay  Allocated PE          1666397
>> pvdisplay  PV UUID               UlaSJI-9Srm-e3AW-LdNW-g2HG-v6ZE-4yWwPw
>> pvdisplay
>> pvdisplay  --- Physical volume ---
>> pvdisplay  PV Name               /dev/sdc
>> pvdisplay  VG Name               Volume02
>> pvdisplay  PV Size               6.36 TB / not usable 6.00 TB
>> pvdisplay  Allocatable           yes (but full)
>> pvdisplay  PE Size (KByte)       4096
>> pvdisplay  Total PE              1666397
>> pvdisplay  Free PE               0
>> pvdisplay  Allocated PE          1666397
>> pvdisplay  PV UUID               mK7WWj-UV0p-N6Qc-zhx6-EK3J-g0CO-g65ljs
>>
>>
>> I've looked around for a lvm tool to display the correct sizes, but 
>> can't find one.  Is there something I'm missing?
>>
>>
>>     
>
> Not sure why you are using LVM if you are going to entirely allocate the
> VG, might as well just use the partition directly....
>
> First problem.  Given the PE Size of 4MiB, you might be hitting a limit
> on how much of the PV can be used.  You probably should have bumped up
> the PE Size when you created the VG....  Take a look at the man page for
> vgcreate (no it can't be changed after the fact), but this may identify
> the "not usable" phrase you are seeing.
>
> pvdisplay has nothing to do with the filesystems, supplying that means
> nothing.
>
> Now lvdisplay will show you the size of the underlying logical volume
> (Virtual partition).  df is showing you the size of the filesystem laid
> on top of that logical volume.  You haven't told us what Fedora version
> you are running, but, yes, I would be concerned about it reporting
> 2TiB....  Showing us lvdisplay output and tune2fs output could probably
> tell someone more....
>
> --Rob
>
>   
Okay here's the output of lvdisplay, as you see it shows the correct 
volume size of 6.36TB:

lvdisplay    Finding all logical volumes
lvdisplay  --- Logical volume ---
lvdisplay  LV Name                /dev/Volume03/Volume03lv
lvdisplay  VG Name                Volume03
lvdisplay  LV UUID                KIP9EC-cmXg-Aa3A-XwQE-Cfuk-c3gq-JFp2az
lvdisplay  LV Write Access        read/write
lvdisplay  LV Status              available
lvdisplay  # open                 1
lvdisplay  LV Size                6.36 TB
lvdisplay  Current LE             1666397
lvdisplay  Segments               1
lvdisplay  Allocation             next free (default)
lvdisplay  Read ahead sectors     0
lvdisplay  Block device           253:2
lvdisplay
lvdisplay  --- Logical volume ---
lvdisplay  LV Name                /dev/Volume02/Volume02lv
lvdisplay  VG Name                Volume02
lvdisplay  LV UUID                F4J6HR-WeND-0RoP-849M-TBJ8-XQl8-62XD1O
lvdisplay  LV Write Access        read/write
lvdisplay  LV Status              available
lvdisplay  # open                 1
lvdisplay  LV Size                6.36 TB
lvdisplay  Current LE             1666397
lvdisplay  Segments               1
lvdisplay  Allocation             next free (default)
lvdisplay  Read ahead sectors     0
lvdisplay  Block device           253:1
lvdisplay


I am not completely familiar with tune2fs so bear with me on that 
output.  Oh, and I'm running this on FC2 unfortunately.  It's a 
production system that was NOT built by me, but I cannot take it out of 
service to upgrade to a newer OS.  I firmly believe this to be a 
limitation of the versions of the e2fsprogs tools.



-- 
Interdum feror cupidine partium magnarum Europae vincendarum

Mark Haney
Sr. Systems Administrator
ERC Broadband
(828) 350-2415