Guest Image File Size

[Aaron Metzger]

virt-install has a checkbox which suggests that you can avoid 
preallocating the entire guest image size -- that the guest image file 
will grow on demand.

Does this work for Xen or is it only supported for KVM and the QEMU COW 
files?

My experience when creating Xen guests as regular files is that I end up 
with a file that is the full size even when I pick the option to not 
preallocate the entire image.


--
Thanks,
Aaron

[Sadique Puthen]

Aaron Metzger wrote:
>  virt-install has a checkbox which suggests that you can avoid 
>  preallocating the entire guest image size -- that the guest image file 
>  will grow on demand.
>
>  Does this work for Xen or is it only supported for KVM and the QEMU 
>  COW files? 
yes, this works fine on xen.

>  My experience when creating Xen guests as regular files is that I end 
>  up with a file that is the full size even when I pick the option to 
>  not preallocate the entire image. 
Basically it if you don't pre-allocate the entire image, virt-manager 
creates a sparse file to store the guest data which only occupies disk 
blocks while you write contents to that sparse file . When checking the 
size of the pre-allocated and sparse images, you should use "du", not 
"ls -lh".

--Sadique

>  -- 
>  Thanks,
>  Aaron
>
>  -- 
>  Fedora-xen mailing list
>  Fedora-xen(a)redhat.com
>  https://www.redhat.com/mailman/listinfo/fedora-xen
>

[Aaron Metzger]

Sadique Puthen wrote:

>  
>  Basically it if you don't pre-allocate the entire image, virt-manager 
>  creates a sparse file to store the guest data which only occupies disk 
>  blocks while you write contents to that sparse file . When checking the 
>  size of the pre-allocated and sparse images, you should use "du", not 
>  "ls -lh".
>   
Thank you for your insight.  You are correct.

I have a guest image stored in a file called "subversion".

"du" shows the actual size being used is smaller than what "ls"
thinks.

du -a subversion
2158768 subversion

ls -ld subversion
-rwxr-xr-x 1 root root 64424509441 2007-11-27 18:16 subversion


I have a follow up question though.  How do you cleanly backup and 
restore these files using the smaller amount of space?  Any simplistic 
program that I try (e.g. "tar") also thinks my file is 60 Gig not 2 Gig 
and creates a real file that uses 60 Gig of actual disk space.

Can anyone share the simple steps with "dd" or other programs that let 
you cleanly store a backup of the guest image which uses the smaller 
amount of space and also correctly restores from that backup in a way 
that preserves the orginal idea of a 60 Gig max guest disk size?

I am sorry if these are just basic Xen or even just basic Unix questions 
that you all already know the answers to as opposed to "fedora-xen" 
questions but I'm sure I'm not the only one who will run into such 
things while trying to virtualize their existing infrastructure using 
Fedora and Xen.  This low traffic list has been a wonderful asset in 
helping with my conversion to Fedora Virtualization.  I appreciate all 
the help I have received.

--
Thanks,
Aaron

[Daniel P. Berrange]

On Sat, Dec 01, 2007 at 12:42:40PM -0500, Aaron Metzger wrote:
>  Sadique Puthen wrote:
>  
>  >
>  >Basically it if you don't pre-allocate the entire image, virt-manager 
>  >creates a sparse file to store the guest data which only occupies disk 
>  >blocks while you write contents to that sparse file . When checking the 
>  >size of the pre-allocated and sparse images, you should use "du", not 
>  >"ls -lh".
>  >
>  
>  Thank you for your insight.  You are correct.
>  
>  I have a guest image stored in a file called "subversion".
>  
>  "du" shows the actual size being used is smaller than what "ls"
> thinks.
>  
>  du -a subversion
>  2158768 subversion
>  
>  ls -ld subversion
>  -rwxr-xr-x 1 root root 64424509441 2007-11-27 18:16 subversion 
You don't need to use  'du' - just use the '-s' flag to ls

$ ls -lsh dan1.img 
16K -rwxr-xr-x 1 root root 2.1G 2007-11-29 17:51 dan1.img

Shows its physical size is 16k, while its logical size is 2.1 GB.

>  I have a follow up question though.  How do you cleanly backup and 
>  restore these files using the smaller amount of space?  Any simplistic 
>  program that I try (e.g. "tar") also thinks my file is 60 Gig not 2 Gig 
>  and creates a real file that uses 60 Gig of actual disk space. 
Use the --sparse option with tar.

       -S, --sparse
              handle sparse files efficiently

>  Can anyone share the simple steps with "dd" or other
> programs that let 
>  you cleanly store a backup of the guest image which uses the smaller 
>  amount of space and also correctly restores from that backup in a way 
>  that preserves the orginal idea of a 60 Gig max guest disk size? 
'cp' will also try to detect sparse files & handle them properly, but
if it gets it wrong you can also use  '--sparse=always'

Dan
-- 
|=- Red Hat, Engineering, Emerging Technologies, Boston.  [phone] -=|
|=-           Perl modules: http://search.cpan.org/~danberr/              -=|
|=-               Projects: http://freshmeat.net/~danielpb/               -=|
|=-  GnuPG: 7D3B9505   F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505  -=|

[Mark Nielsen]

I think you'll get the same sort of thing if you gzip the file. I've 
gzip'd some 20G xen images down to around 2G.

Daniel P. Berrange wrote:
>  On Sat, Dec 01, 2007 at 12:42:40PM -0500, Aaron Metzger wrote:
>    
> > Sadique Puthen wrote:
> >
> >     
> >> Basically it if you don't pre-allocate the entire image, virt-manager 
> >> creates a sparse file to store the guest data which only occupies disk 
> >> blocks while you write contents to that sparse file . When checking the 
> >> size of the pre-allocated and sparse images, you should use "du", not 
> >> "ls -lh".
> >>
> >>       
> > Thank you for your insight.  You are correct.
> >
> > I have a guest image stored in a file called "subversion".
> >
> > "du" shows the actual size being used is smaller than what "ls"
> thinks.
> >
> > du -a subversion
> > 2158768 subversion
> >
> > ls -ld subversion
> > -rwxr-xr-x 1 root root 64424509441 2007-11-27 18:16 subversion
> >     
>
>  You don't need to use  'du' - just use the '-s' flag to ls
>
>  $ ls -lsh dan1.img 
>  16K -rwxr-xr-x 1 root root 2.1G 2007-11-29 17:51 dan1.img
>
>  Shows its physical size is 16k, while its logical size is 2.1 GB.
>
>    
> > I have a follow up question though.  How do you cleanly backup and 
> > restore these files using the smaller amount of space?  Any simplistic 
> > program that I try (e.g. "tar") also thinks my file is 60 Gig not 2 Gig 
> > and creates a real file that uses 60 Gig of actual disk space.
> >     
>
>  Use the --sparse option with tar.
>
>         -S, --sparse
>                handle sparse files efficiently
>
>    
> > Can anyone share the simple steps with "dd" or other programs that let 
> > you cleanly store a backup of the guest image which uses the smaller 
> > amount of space and also correctly restores from that backup in a way 
> > that preserves the orginal idea of a 60 Gig max guest disk size?
> >     
>
>  'cp' will also try to detect sparse files & handle them properly, but
>  if it gets it wrong you can also use  '--sparse=always'
>
>  Dan
>

[Daniel P. Berrange]

On Sat, Dec 01, 2007 at 02:03:06PM -0500, Mark Nielsen wrote:
>  I think you'll get the same sort of thing if you gzip the file.
> I've 
>  gzip'd some 20G xen images down to around 2G. 
AFAIR, this is just because sparse files appear as long sequences of
zeros & thus compress wel - gzip isn't actually optimizing for sparseness.
The trouble is that when you extract the file gunzip will fully allocate
it filling with zeros.

My original 1M file takes 4k

# ls -lhs foo 
4.0K -rw-r--r-- 1 root root 1.0M 2007-12-01 14:18 foo

#gzip  foo
# ls -lhs foo.gz 
8.0K -rw-r--r-- 1 root root 1.1K 2007-12-01 14:18 foo.gz

Now when it uncompress it it takes up the full 1 MB :-(

# gunzip foo.gz 
# ls -lhs foo
1.1M -rw-r--r-- 1 root root 1.0M 2007-12-01 14:18 foo


You *always* want to use tar to preserve sparseness, and then gzip the tar 
file so you get optimal resource usage both on archiving & extracting.

Dan.
-- 
|=- Red Hat, Engineering, Emerging Technologies, Boston.  [phone] -=|
|=-           Perl modules: http://search.cpan.org/~danberr/              -=|
|=-               Projects: http://freshmeat.net/~danielpb/               -=|
|=-  GnuPG: 7D3B9505   F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505  -=|

[John Summerfield]

Daniel P. Berrange wrote:
>  On Sat, Dec 01, 2007 at 02:03:06PM -0500, Mark Nielsen wrote:
> > I think you'll get the same sort of thing if you gzip the file. I've 
> > gzip'd some 20G xen images down to around 2G.
>  
>  AFAIR, this is just because sparse files appear as long sequences of
>  zeros & thus compress wel - gzip isn't actually optimizing for sparseness.
>  The trouble is that when you extract the file gunzip will fully allocate
>  it filling with zeros. 
undo that with "cp --sparse"


-- 

Cheers
John

-- spambait
1aaaaaaa(a)coco.merseine.nu  Z1aaaaaaa(a)coco.merseine.nu
-- Advice
http://webfoot.com/advice/email.top.php
http://www.catb.org/~esr/faqs/smart-questions.html
http://support.microsoft.com/kb/555375

You cannot reply off-list:-)

[David Lutterkort]

On Sat, 2007-12-01 at 14:03 -0500, Mark Nielsen wrote:
>  I think you'll get the same sort of thing if you gzip the file.
> I've 
>  gzip'd some 20G xen images down to around 2G. 
Another nice way to compress disk images is with 'qemu-img convert',
outputting to a compressed qcow.

David

[Markus Armbruster]

Sadique Puthen <sputhenp(a)redhat.com&gt; writes:

>  Aaron Metzger wrote:
> > virt-install has a checkbox which suggests that you can avoid
> > preallocating the entire guest image size -- that the guest image
> > file will grow on demand.
> >
> > Does this work for Xen or is it only supported for KVM and the QEMU
> > COW files?
>
>  yes, this works fine on xen. 
Until you run out of space in the filesystem containing the image
file!  Then the guest can't write any blocks that were previously left
out of the image file, which leads to massive guest filesystem
corruption and data loss.

I use sparse image files all the time.  But only for the throw-away
domains I create for testing.

[...]

[John Summerfield]

Markus Armbruster wrote:
>  Sadique Puthen <sputhenp(a)redhat.com&gt; writes:
>  
> > Aaron Metzger wrote:
> >> virt-install has a checkbox which suggests that you can avoid
> >> preallocating the entire guest image size -- that the guest image
> >> file will grow on demand.
> >>
> >> Does this work for Xen or is it only supported for KVM and the QEMU
> >> COW files?
> > yes, this works fine on xen.
>  
>  Until you run out of space in the filesystem containing the image
>  file!  Then the guest can't write any blocks that were previously left
>  out of the image file, which leads to massive guest filesystem
>  corruption and data loss.
>  
>  I use sparse image files all the time.  But only for the throw-away
>  domains I create for testing. 
I thought I read that ext3 journals save the day. Whether they always 
save it, "man mke2fs" for journal options. Using a device that _is not_ 
sparse, it seems to me, must be safe.


-- 

Cheers
John

-- spambait
1aaaaaaa(a)coco.merseine.nu  Z1aaaaaaa(a)coco.merseine.nu
-- Advice
http://webfoot.com/advice/email.top.php
http://www.catb.org/~esr/faqs/smart-questions.html
http://support.microsoft.com/kb/555375

You cannot reply off-list:-)

[John Summerfield]

Aaron Metzger wrote:
>  virt-install has a checkbox which suggests that you can avoid 
>  preallocating the entire guest image size -- that the guest image file 
>  will grow on demand.
>  
>  Does this work for Xen or is it only supported for KVM and the QEMU COW 
>  files? 
It's a filesystem property:
[root@potoroo ~]# ls -shoog /var/lib/xen/images/*
1.1G -rwxr-xr-x 1 3.1G Nov  7 10:39 /var/lib/xen/images/ClarkConnect.img
2.9G -rwxr-xr-x 1 3.1G Nov 30 18:31 /var/lib/xen/images/DebianEtch.img
4.0G -rwxr-xr-x 1 4.0G Nov 28 17:49 /var/lib/xen/images/F8-1.img
  16K -rwxr-xr-x 1 4.0G Nov 28 17:57 /var/lib/xen/images/F8-2.img
  16K -rwxr-xr-x 1 4.0G Nov 28 17:38 /var/lib/xen/images/F8.img
  16K -rwxr-xr-x 1 4.1G Nov  7 14:06 /var/lib/xen/images/Fedora7.img
  16K -rwxr-xr-x 1 2.1G Nov 12 22:55 /var/lib/xen/images/FreeBSD_70.img
129M -rwxr-xr-x 1 4.0G Nov 18 15:51 /var/lib/xen/images/Gentoo.2.img
2.9G -rwxr-xr-x 1 3.1G Nov 18 17:00 /var/lib/xen/images/Gentoo.img
  16K -rwxr-xr-x 1 3.1G Nov  5 08:45 /var/lib/xen/images/Kubuntu.img
  16K -rwxr-xr-x 1 2.1G Nov  4 10:00 /var/lib/xen/images/NetBSD-31.img
  16K -rwxr-xr-x 1 4.0G Nov 28 17:55 /var/lib/xen/images/f8.img
1.1G -rwxr-xr-x 1 2.1G Nov  7 07:52 /var/lib/xen/images/windows_c.img
[root@potoroo ~]#


>  
>  My experience when creating Xen guests as regular files is that I end up 
>  with a file that is the full size even when I pick the option to not 
>  preallocate the entire image. 
To interpret my listing above,
man ls

-- 

Cheers
John

-- spambait
1aaaaaaa(a)coco.merseine.nu  Z1aaaaaaa(a)coco.merseine.nu
-- Advice
http://webfoot.com/advice/email.top.php
http://www.catb.org/~esr/faqs/smart-questions.html
http://support.microsoft.com/kb/555375

You cannot reply off-list:-)