df, lvm and 6TB arrays oh my!

[Robert Locke]

On Sun, 2006-02-12 at 15:23 -0500, Mark Haney wrote:
> Robert Locke wrote:
> > On Sat, 2006-02-11 at 11:26 -0500, Mark Haney wrote:
> >   
> >> I am having a problem with df reporting the wrong disk size on a couple 
> >> of 7TB arrays on a Fedora box we are using.  I have an lvm group on each 
> >> array of 6.36TB  when I run  df this is what I get:
> >>
> >> /dev/mapper/Volume02-Volume02lv
> >>                       2.4T   33M  2.3T   1% /mnt/arrays/array2
> >> /dev/mapper/Volume03-Volume03lv
> >>                       2.4T  950G  1.3T  43% /mnt/arrays/array1
> >>
> >>
> >> When I run vg display this is what it reports:
> >>
> >> pvdisplay  PV Name               /dev/sdd
> >> pvdisplay  VG Name               Volume03
> >> pvdisplay  PV Size               6.36 TB / not usable 6.00 TB
> >> pvdisplay  Allocatable           yes (but full)
> >> pvdisplay  PE Size (KByte)       4096
> >> pvdisplay  Total PE              1666397
> >> pvdisplay  Free PE               0
> >> pvdisplay  Allocated PE          1666397
> >> pvdisplay  PV UUID               UlaSJI-9Srm-e3AW-LdNW-g2HG-v6ZE-4yWwPw
> >> pvdisplay
> >> pvdisplay  --- Physical volume ---
> >> pvdisplay  PV Name               /dev/sdc
> >> pvdisplay  VG Name               Volume02
> >> pvdisplay  PV Size               6.36 TB / not usable 6.00 TB
> >> pvdisplay  Allocatable           yes (but full)
> >> pvdisplay  PE Size (KByte)       4096
> >> pvdisplay  Total PE              1666397
> >> pvdisplay  Free PE               0
> >> pvdisplay  Allocated PE          1666397
> >> pvdisplay  PV UUID               mK7WWj-UV0p-N6Qc-zhx6-EK3J-g0CO-g65ljs
> >>
> >>
> >> I've looked around for a lvm tool to display the correct sizes, but 
> >> can't find one.  Is there something I'm missing?
> >>
> >>
> >>     
> >
> > Not sure why you are using LVM if you are going to entirely allocate the
> > VG, might as well just use the partition directly....
> >
> > First problem.  Given the PE Size of 4MiB, you might be hitting a limit
> > on how much of the PV can be used.  You probably should have bumped up
> > the PE Size when you created the VG....  Take a look at the man page for
> > vgcreate (no it can't be changed after the fact), but this may identify
> > the "not usable" phrase you are seeing.
> >
> > pvdisplay has nothing to do with the filesystems, supplying that means
> > nothing.
> >
> > Now lvdisplay will show you the size of the underlying logical volume
> > (Virtual partition).  df is showing you the size of the filesystem laid
> > on top of that logical volume.  You haven't told us what Fedora version
> > you are running, but, yes, I would be concerned about it reporting
> > 2TiB....  Showing us lvdisplay output and tune2fs output could probably
> > tell someone more....
> >
> > --Rob
> >
> >   
> Okay here's the output of lvdisplay, as you see it shows the correct 
> volume size of 6.36TB:
> 
> lvdisplay    Finding all logical volumes
> lvdisplay  --- Logical volume ---
> lvdisplay  LV Name                /dev/Volume03/Volume03lv
> lvdisplay  VG Name                Volume03
> lvdisplay  LV UUID                KIP9EC-cmXg-Aa3A-XwQE-Cfuk-c3gq-JFp2az
> lvdisplay  LV Write Access        read/write
> lvdisplay  LV Status              available
> lvdisplay  # open                 1
> lvdisplay  LV Size                6.36 TB
> lvdisplay  Current LE             1666397
> lvdisplay  Segments               1
> lvdisplay  Allocation             next free (default)
> lvdisplay  Read ahead sectors     0
> lvdisplay  Block device           253:2
> lvdisplay
> lvdisplay  --- Logical volume ---
> lvdisplay  LV Name                /dev/Volume02/Volume02lv
> lvdisplay  VG Name                Volume02
> lvdisplay  LV UUID                F4J6HR-WeND-0RoP-849M-TBJ8-XQl8-62XD1O
> lvdisplay  LV Write Access        read/write
> lvdisplay  LV Status              available
> lvdisplay  # open                 1
> lvdisplay  LV Size                6.36 TB
> lvdisplay  Current LE             1666397
> lvdisplay  Segments               1
> lvdisplay  Allocation             next free (default)
> lvdisplay  Read ahead sectors     0
> lvdisplay  Block device           253:1
> lvdisplay
> 
> 
> I am not completely familiar with tune2fs so bear with me on that 
> output.  Oh, and I'm running this on FC2 unfortunately.  It's a 
> production system that was NOT built by me, but I cannot take it out of 
> service to upgrade to a newer OS.  I firmly believe this to be a 
> limitation of the versions of the e2fsprogs tools.
> 
> 

OK, so lvdisplay showing the full amount is good, so the underlying
virtual partition is 6.36TB....  Now the tune2fs I was suggesting was
simply the -l option that would dump the ext2/3 superblock.  We should
be able to see the "block count" and the "block size" which when
multiplied should be 6.36TB.  Since you are at least on a 2.6 kernel, I
don't think we have the 2TB filesystem limit, though I don't remember
when that got bumped up to 32....

--Rob