I have a 2 GB bz2 archive that unzips to over 10 GB (wikipedia dump). Although I have over 50 GB free in /home, / has only about 8 GB free. Thus, as tar uses /tmp, the / filesystem fills up and I cannot continue. How can I specify a tmp directory for tar in my home directory? Note that man tar makes no mention of a tmp option.
Thanks in advance.
Dotan Cohen
On Fri, 2007-08-31 at 23:18 +0300, Dotan Cohen wrote:
I have a 2 GB bz2 archive that unzips to over 10 GB (wikipedia dump). Although I have over 50 GB free in /home, / has only about 8 GB free. Thus, as tar uses /tmp, the / filesystem fills up and I cannot continue. How can I specify a tmp directory for tar in my home directory? Note that man tar makes no mention of a tmp option.
Boot in single user mode, Then as root:
# mkdir /home/tmp # chmod 777 /home/tmp # mv /tmp /tmp-old # ln -s /home/tmp /tmp # cp -a /tmp-old/* /tmp
That creates a /home/tmp directory, allows everyone access, renames the old /tmp to /tmp-old, symlinks /home/tmp to /tmp, then copies everything that was in the old /tmp to the new one. Once that's done, you can reboot and all references to /tmp will now access /home/tmp.
---------------------------------------------------------------------- - Rick Stevens, Principal Engineer rstevens@internap.com - - CDN Systems, Internap, Inc. http://www.internap.com - - - - Blech! ACKth! Ooop! -- Bill the Cat (Outland) - ----------------------------------------------------------------------
On 31/08/2007, Rick Stevens rstevens@internap.com wrote:
On Fri, 2007-08-31 at 23:18 +0300, Dotan Cohen wrote:
I have a 2 GB bz2 archive that unzips to over 10 GB (wikipedia dump). Although I have over 50 GB free in /home, / has only about 8 GB free. Thus, as tar uses /tmp, the / filesystem fills up and I cannot continue. How can I specify a tmp directory for tar in my home directory? Note that man tar makes no mention of a tmp option.
Boot in single user mode, Then as root:
# mkdir /home/tmp # chmod 777 /home/tmp # mv /tmp /tmp-old # ln -s /home/tmp /tmp # cp -a /tmp-old/* /tmpThat creates a /home/tmp directory, allows everyone access, renames the old /tmp to /tmp-old, symlinks /home/tmp to /tmp, then copies everything that was in the old /tmp to the new one. Once that's done, you can reboot and all references to /tmp will now access /home/tmp.
Thanks, Rick, I'll try that. I can then simply erase the symlink? I know that "rm /tmp" will not erase the symlink, rather the content of /home/tmp so how can I remove it afterwards? Thanks.
Also, as a learning experience, is there a way a user without root access could unpack the tar? It's not a problem, but I'd like to learn. Thanks.
Dotan Cohen
On Fri, 2007-08-31 at 23:53 +0300, Dotan Cohen wrote:
On 31/08/2007, Rick Stevens rstevens@internap.com wrote:
On Fri, 2007-08-31 at 23:18 +0300, Dotan Cohen wrote:
I have a 2 GB bz2 archive that unzips to over 10 GB (wikipedia dump). Although I have over 50 GB free in /home, / has only about 8 GB free. Thus, as tar uses /tmp, the / filesystem fills up and I cannot continue. How can I specify a tmp directory for tar in my home directory? Note that man tar makes no mention of a tmp option.
Boot in single user mode, Then as root:
# mkdir /home/tmp # chmod 777 /home/tmp # mv /tmp /tmp-old # ln -s /home/tmp /tmp # cp -a /tmp-old/* /tmpThat creates a /home/tmp directory, allows everyone access, renames the old /tmp to /tmp-old, symlinks /home/tmp to /tmp, then copies everything that was in the old /tmp to the new one. Once that's done, you can reboot and all references to /tmp will now access /home/tmp.
Thanks, Rick, I'll try that. I can then simply erase the symlink? I know that "rm /tmp" will not erase the symlink, rather the content of /home/tmp so how can I remove it afterwards? Thanks.
No, "rm /tmp" will remove the symlink. If you want to revert to the original /tmp, reboot in single user mode again and as root:
# cp -a /tmp/* /tmp-old # rm -f /tmp # mv /tmp-old /tmp
which will copy any changes from /home/tmp (/tmp) to the original /tmp directory, then deletes the symlink and renames /tmp-old to /tmp.
Also, as a learning experience, is there a way a user without root access could unpack the tar? It's not a problem, but I'd like to learn. Thanks.
If you were doing
bunzip2 name-of-tar.bz2 tar xf name-of-tar
or even
bzcat name-of-tar.bz2 | tar xf -
then you're going to use /tmp. Do it all in one command:
tar xJf name-of-tar.bz2
to unzip and untar the thing. Should do it all in the current directory. ---------------------------------------------------------------------- - Rick Stevens, Principal Engineer rstevens@internap.com - - CDN Systems, Internap, Inc. http://www.internap.com - - - - Always remember you're unique, just like everyone else. - ----------------------------------------------------------------------
On Fri, 2007-08-31 at 14:13 -0700, Rick Stevens wrote:
On Fri, 2007-08-31 at 23:53 +0300, Dotan Cohen wrote:
On 31/08/2007, Rick Stevens rstevens@internap.com wrote:
On Fri, 2007-08-31 at 23:18 +0300, Dotan Cohen wrote:
I have a 2 GB bz2 archive that unzips to over 10 GB (wikipedia dump). Although I have over 50 GB free in /home, / has only about 8 GB free. Thus, as tar uses /tmp, the / filesystem fills up and I cannot continue. How can I specify a tmp directory for tar in my home directory? Note that man tar makes no mention of a tmp option.
Boot in single user mode, Then as root:
# mkdir /home/tmp # chmod 777 /home/tmp # mv /tmp /tmp-old # ln -s /home/tmp /tmp # cp -a /tmp-old/* /tmpThat creates a /home/tmp directory, allows everyone access, renames the old /tmp to /tmp-old, symlinks /home/tmp to /tmp, then copies everything that was in the old /tmp to the new one. Once that's done, you can reboot and all references to /tmp will now access /home/tmp.
Thanks, Rick, I'll try that. I can then simply erase the symlink? I know that "rm /tmp" will not erase the symlink, rather the content of /home/tmp so how can I remove it afterwards? Thanks.
No, "rm /tmp" will remove the symlink. If you want to revert to the original /tmp, reboot in single user mode again and as root:
# cp -a /tmp/* /tmp-old # rm -f /tmp # mv /tmp-old /tmp
which will copy any changes from /home/tmp (/tmp) to the original /tmp directory, then deletes the symlink and renames /tmp-old to /tmp.
I should have added that you could then "rm -rf /home/tmp" to recover the space used on /home for the temporary "tmp" directory. That would be after the "mv" command above.
Also, as a learning experience, is there a way a user without root access could unpack the tar? It's not a problem, but I'd like to learn. Thanks.
If you were doing
bunzip2 name-of-tar.bz2 tar xf name-of-tar
or even
bzcat name-of-tar.bz2 | tar xf -
then you're going to use /tmp. Do it all in one command:
tar xJf name-of-tar.bz2
to unzip and untar the thing. Should do it all in the current directory.
---------------------------------------------------------------------- - Rick Stevens, Principal Engineer rstevens@internap.com - - CDN Systems, Internap, Inc. http://www.internap.com - - - - BASIC is the Computer Science version of `Scientific Creationism' - ----------------------------------------------------------------------
On 01/09/2007, Rick Stevens rstevens@internap.com wrote:
If you were doing
bunzip2 name-of-tar.bz2 tar xf name-of-taror even
bzcat name-of-tar.bz2 | tar xf -then you're going to use /tmp. Do it all in one command:
tar xJf name-of-tar.bz2
I was using a graphical KDE tool for the job. It may have been doing the work in different steps. I think that I tried tar from the command line as well, but that was at one in the morning and now I'm not sure. I'll play around with some other suggestion from this thread and report back. Thanks.
Dotan Cohen
Rick Stevens wrote:
On Fri, 2007-08-31 at 23:18 +0300, Dotan Cohen wrote:
I have a 2 GB bz2 archive that unzips to over 10 GB (wikipedia dump). Although I have over 50 GB free in /home, / has only about 8 GB free. Thus, as tar uses /tmp, the / filesystem fills up and I cannot continue. How can I specify a tmp directory for tar in my home directory? Note that man tar makes no mention of a tmp option.
Boot in single user mode, Then as root:
# mkdir /home/tmp # chmod 777 /home/tmp # mv /tmp /tmp-old # ln -s /home/tmp /tmp # cp -a /tmp-old/* /tmp
Doesn't tar only use tmp because it has to uncompress the file first? If that's the case, why not uncompress the file first with bzip2 somewhere on the file system that actually has space, and then run tar to extract the files? In essence he would be renaming his file to archive.tar.bz2, run bunzip2 against it, and then run tar against the archive.tar...
bzip uses whatever the current path is to compress or decompress a file, so /tmp wouldn't play a part here.
That's my theory at least. The last time I had to uncompress a large archive (over 100GiB in size), that's what I did and it worked flawlessly. Worst case, you can specify the -s option to bunzip2 which will cause bzip to use even less memory (and /tmp if he's really low on memory) and still get the file uncompressed.
-- A
On Friday 31 August 2007 16:18:05 Dotan Cohen wrote:
I have a 2 GB bz2 archive that unzips to over 10 GB (wikipedia dump). Although I have over 50 GB free in /home, / has only about 8 GB free. Thus, as tar uses /tmp, the / filesystem fills up and I cannot continue. How can I specify a tmp directory for tar in my home directory? Note that man tar makes no mention of a tmp option.
No, the manual doesn't mention it, but I think the program respects the TMPDIR environment variable. Try setting it.
Dotan Cohen wrote:
I have a 2 GB bz2 archive that unzips to over 10 GB (wikipedia dump). Although I have over 50 GB free in /home, / has only about 8 GB free. Thus, as tar uses /tmp, the / filesystem fills up and I cannot continue. How can I specify a tmp directory for tar in my home directory? Note that man tar makes no mention of a tmp option.
Thanks in advance.
Dotan Cohen
Dumb question - does it help to create a /tmp directory in your home directory and run "export TMP=$home/tmp"? (I don't remember if bzip2 and tar honor TMP.)
Mikkel
On 01/09/2007, Mikkel L. Ellertson mikkel@infinity-ltd.com wrote:
Dotan Cohen wrote:
I have a 2 GB bz2 archive that unzips to over 10 GB (wikipedia dump). Although I have over 50 GB free in /home, / has only about 8 GB free. Thus, as tar uses /tmp, the / filesystem fills up and I cannot continue. How can I specify a tmp directory for tar in my home directory? Note that man tar makes no mention of a tmp option.
Thanks in advance.
Dotan Cohen
Dumb question - does it help to create a /tmp directory in your home directory and run "export TMP=$home/tmp"? (I don't remember if bzip2 and tar honor TMP.)
Mikkel
Thanks, but that did not work either. The / filesystem did not fill up, but I got this error:
ubuntu@ubuntu-laptop:~/Desktop$ tar xjf enwiki-20070802-pages-articles.xml.bz2 tar: This does not look like a tar archive tar: Skipping to next header tar: Error exit delayed from previous errors ubuntu@ubuntu-laptop:~/Desktop$
Maybe the archive is bad?
Dotan Cohen
Dotan Cohen wrote:
On 01/09/2007, Mikkel L. Ellertson mikkel@infinity-ltd.com wrote:
Dotan Cohen wrote:
I have a 2 GB bz2 archive that unzips to over 10 GB (wikipedia dump). Although I have over 50 GB free in /home, / has only about 8 GB free. Thus, as tar uses /tmp, the / filesystem fills up and I cannot continue. How can I specify a tmp directory for tar in my home directory? Note that man tar makes no mention of a tmp option.
Thanks in advance.
Dotan Cohen
Dumb question - does it help to create a /tmp directory in your home directory and run "export TMP=$home/tmp"? (I don't remember if bzip2 and tar honor TMP.)
Mikkel
Thanks, but that did not work either. The / filesystem did not fill up, but I got this error:
ubuntu@ubuntu-laptop:~/Desktop$ tar xjf enwiki-20070802-pages-articles.xml.bz2 tar: This does not look like a tar archive tar: Skipping to next header tar: Error exit delayed from previous errors ubuntu@ubuntu-laptop:~/Desktop$
Maybe the archive is bad?
Maybe it isn't even an archive?
What does "file enwiki-20070802-pages-articles.xml.bz2" return?
What happens if you try "bunzip2 enwiki-20070802-pages-articles.xml.bz2" ?
On 01/09/07, Ed Greshko Ed.Greshko@greshko.com wrote:
Maybe it isn't even an archive?
What does "file enwiki-20070802-pages-articles.xml.bz2" return?
What happens if you try "bunzip2 enwiki-20070802-pages-articles.xml.bz2" ?
You're right.
ubuntu@ubuntu-laptop:~/Desktop$ file enwiki-20070802-pages-articles.xml.tar enwiki-20070802-pages-articles.xml.tar: UTF-8 Unicode English text, with very long lines ubuntu@ubuntu-laptop:~/Desktop$
It's the XML dump, not a tar. I guess that should have been obvious, as it is only one file...
Thanks. Another day, another lesson...
Dotan Cohen
Dotan Cohen wrote:
I have a 2 GB bz2 archive that unzips to over 10 GB (wikipedia dump). Although I have over 50 GB free in /home, / has only about 8 GB free. Thus, as tar uses /tmp, the / filesystem fills up and I cannot continue. How can I specify a tmp directory for tar in my home directory? Note that man tar makes no mention of a tmp option.
Something seems funky here: there's no reason for tar to need a temp space to unpack an archive.
I just checked an strace log from GNU tar unpacking a small bzip2-compressed archive and /tmp was not touched. The decompression and unpacking of the archive should all be streamed through pipes and no temporary files should be needed.
How are you invoking it? Are you using GNU tar?
Surely you're doing this on the command line and not through some front-end like file-roller, right?
At any rate you should be able to use something like this:
$ mkdir $HOME/tmp $ TMPDIR=$HOME/tmp tar xjf archive.tar.bz2
to force tar and anything it runs to use your $HOME for the temp files.
You should also be able to force the unpacking to stream by setting it up manually:
$ bzcat archive.tar.bz2 | tar xf -
That should not require any temp space at all.
If it were me, I would want to understand why the temp files are needed.
<Joe
-
Ashley M. Kirchner -
Dotan Cohen -
Ed Greshko -
Garry Williams -
Joe Smith -
Mikkel -
Rick Stevens