copy and tar performance?

[Kevin Kempter]

Hi List;

Whenever I copy large files (>2G) or create large tar archives I basically 
loose control of my box (2.1Gz Laptop running FC5) until the copy or tar is 
complete. I can still try and work but the mouse movements are delayed 
several seconds, other programs take minutes to respond, etc.

Is there a way I can overcome this so I can still work as normal during these 
operations, maybe a way to limit the amount of cpu or system resources that a 
cp or tar process can have? Other ideas?

Thanks in advance

[Dave Stevens]

Quoting Kevin Kempter :
> Hi List;
>
> Whenever I copy large files (>2G) or create large tar archives I basically 
> loose control of my box (2.1Gz Laptop running FC5) until the copy or tar is 
> complete. I can still try and work but the mouse movements are delayed 
> several seconds, other programs take minutes to respond, etc.
>
> Is there a way I can overcome this so I can still work as normal during these
>
> operations, maybe a way to limit the amount of cpu or system resources that a
>
> cp or tar process can have? Other ideas?
>
> Thanks in advance

man nice
> -- 
> fedora-list mailing list
> [e-mail]
> To unsubscribe: https://www.redhat.com/mailman/listinfo/fedora-list

[Bruno Wolff III]

On Wed, Dec 27, 2006 at 12:32:51 -0700,
  Kevin Kempter  wrote:
> Hi List;
>
> Whenever I copy large files (>2G) or create large tar archives I basically 
> loose control of my box (2.1Gz Laptop running FC5) until the copy or tar is 
> complete. I can still try and work but the mouse movements are delayed 
> several seconds, other programs take minutes to respond, etc.
>
> Is there a way I can overcome this so I can still work as normal during these 
> operations, maybe a way to limit the amount of cpu or system resources that a 
> cp or tar process can have? Other ideas?

Using 'nice' might help solve your problem.
If you are doing the copy graphically, there may be overhead with displaying
the progress (I don't do copies that way, so don't know if in fact that is
the case) which slows things down and doing a command line copy might be
easier on your CPU.

Do you have write caching disabled on your hard drive? If so you might get
better performance using write barriers for safety instead, and turning
caching back on.

Do you have you file systems mounted noatime? If you are copying lots of
small files, updating the last accessed time of those file might be noticiable.

[Les Mikesell]

On Wed, 2006-12-27 at 11:42 -0800, Dave Stevens wrote:
> > Whenever I copy large files (>2G) or create large tar archives I basically 
> > loose control of my box (2.1Gz Laptop running FC5) until the copy or tar is 
> > complete. I can still try and work but the mouse movements are delayed 
> > several seconds, other programs take minutes to respond, etc.
> >
> > Is there a way I can overcome this so I can still work as normal during these
> >
> > operations, maybe a way to limit the amount of cpu or system resources that a
> >
> > cp or tar process can have? Other ideas?
> >
> > Thanks in advance
>
> man nice

I doubt if that will help much.  Cp/tar, etc. do very little work in
the main process - it's all read()/write() system calls.  I'd guess the
problem is that your disk interface is taking all the CPU and
what you need to do is enable dma if it can handle it.  If it is IDE,
it might be worth adding a new card.

[Kevin Kempter]

On Wednesday 27 December 2006 13:17, Les Mikesell wrote:
> On Wed, 2006-12-27 at 11:42 -0800, Dave Stevens wrote:
> > > Whenever I copy large files (>2G) or create large tar archives I
> > > basically loose control of my box (2.1Gz Laptop running FC5) until the
> > > copy or tar is complete. I can still try and work but the mouse
> > > movements are delayed several seconds, other programs take minutes to
> > > respond, etc.
> > >
> > > Is there a way I can overcome this so I can still work as normal during
> > > these
> > >
> > > operations, maybe a way to limit the amount of cpu or system resources
> > > that a
> > >
> > > cp or tar process can have? Other ideas?
> > >
> > > Thanks in advance
> >
> > man nice
>
> I doubt if that will help much.  Cp/tar, etc. do very little work in
> the main process - it's all read()/write() system calls.  I'd guess the
> problem is that your disk interface is taking all the CPU and
> what you need to do is enable dma if it can handle it.  If it is IDE,
> it might be worth adding a new card.
>
> --
>   Les Mikesell
>    [e-mail]

Thanks for the info.  How do I enable DMA on a SATA drive?

[Mogens Kjaer]

Kevin Kempter wrote:
> Thanks for the info.  How do I enable DMA on a SATA drive?

Is your drive /dev/hda or /dev/sda? This could depend
upon settings in your BIOS.

Try running hdparm on /dev/hda or /dev/sda and post
the result.

Mogens

[Jim Cornette]

Mogens Kjaer wrote:
> Kevin Kempter wrote:
> > Thanks for the info.  How do I enable DMA on a SATA drive?
>
> Is your drive /dev/hda or /dev/sda? This could depend
> upon settings in your BIOS.
>
> Try running hdparm on /dev/hda or /dev/sda and post
> the result.
>
> Mogens

Just a note for when the next Fedora release is out, IDE hard drives 
will be referred to as /dev/sdx instead of /dev/hdx devices. If you use 
devices instead of LABEL= to refer to your drives, get ready for editing.

Jim

[Les Mikesell]

Jim Cornette wrote:
> Mogens Kjaer wrote:
> > Kevin Kempter wrote:
> > > Thanks for the info.  How do I enable DMA on a SATA drive?
> >
> > Is your drive /dev/hda or /dev/sda? This could depend
> > upon settings in your BIOS.
> >
> > Try running hdparm on /dev/hda or /dev/sda and post
> > the result.
> >
> > Mogens
>
> Just a note for when the next Fedora release is out, IDE hard drives 
> will be referred to as /dev/sdx instead of /dev/hdx devices. If you use 
> devices instead of LABEL= to refer to your drives, get ready for editing.

And if you use LABEL= to refer to your partitions, be aware that
moving them from one machine to anoher is likely to cause confusion
with duplicates that is difficult to fix.

[Les Mikesell]

Mogens Kjaer wrote:
> Kevin Kempter wrote:
> > Thanks for the info.  How do I enable DMA on a SATA drive?
>
> Is your drive /dev/hda or /dev/sda? This could depend
> upon settings in your BIOS.
>
> Try running hdparm on /dev/hda or /dev/sda and post
> the result.

Also, is it using the controller directly or is this one of
those on-board RAID setups that actually do extra work in
the driver software?

[Kevin Kempter]

On Thursday 28 December 2006 01:18, Mogens Kjaer wrote:
> Kevin Kempter wrote:
> > Thanks for the info.  How do I enable DMA on a SATA drive?
>
> Is your drive /dev/hda or /dev/sda? This could depend
> upon settings in your BIOS.
>
> Try running hdparm on /dev/hda or /dev/sda and post
> the result.
>
> Mogens
>
> --
> Mogens Kjaer, Carlsberg A/S, Computer Department
> Gamle Carlsberg Vej 10, DK-2500 Valby, Denmark
> Phone: [phone], Fax: [phone]
> Email: [e-mail] Homepage: http://www.crc.dk

# hdparm /dev/sda

/dev/sda:
 IO_support   =  0 (default 16-bit)
 readonly     =  0 (off)
 readahead    = 256 (on)
 geometry     = 12161/255/63, sectors = 195371568, start = 0

[Jim Cornette]

Les Mikesell wrote:
> Jim Cornette wrote:
> > Mogens Kjaer wrote:
> > > Kevin Kempter wrote:
> > > > Thanks for the info.  How do I enable DMA on a SATA drive?
> > >
> > > Is your drive /dev/hda or /dev/sda? This could depend
> > > upon settings in your BIOS.
> > >
> > > Try running hdparm on /dev/hda or /dev/sda and post
> > > the result.
> > >
> > > Mogens
> >
> > Just a note for when the next Fedora release is out, IDE hard drives 
> > will be referred to as /dev/sdx instead of /dev/hdx devices. If you 
> > use devices instead of LABEL= to refer to your drives, get ready for 
> > editing.
>
> And if you use LABEL= to refer to your partitions, be aware that
> moving them from one machine to anoher is likely to cause confusion
> with duplicates that is difficult to fix.

Both methods have flaws due to circumstance. I like both methods and 
usually go with labels and specify the names at boot. I like the idea of 
randomizing names in default installations. I put labels as below.
  cat /etc/fstab
LABEL=/80g              /                       ext3    defaults        1 1
LABEL=/boot80g          /boot                   ext3    defaults        1 2
devpts                  /dev/pts                devpts  gid=5,mode=620  0 0
tmpfs                   /dev/shm                tmpfs   defaults        0 0
LABEL=/home80g          /home                   ext3    defaults        1 2
proc                    /proc                   proc    defaults        0 0
sysfs                   /sys                    sysfs   defaults        0 0
LABEL=SWAP-hdb3         swap                    swap    defaults        0 0
/dev/hda5vol/LogVol01   swap                    swap    defaults        0 0

Jim

[Tim]

On Thu, 2006-12-28 at 09:40 -0600, Les Mikesell wrote:
> if you use LABEL= to refer to your partitions, be aware that moving
> them from one machine to anoher is likely to cause confusion with
> duplicates that is difficult to fix.

Yes.  I had to wonder why the installation routine doesn't allow you to
set your own labels.  Sure, it might automatically use /1 instead of /
if it detects a / label already on your system.  But I find that scheme
quite ambiguous.  It'd be useful if I could supply a prefix, at least,
so that I can manually label a drive (with texta, for instance), and all
partition labels include that name.

e.g. I call the drive fred, and it gets labels like /fred, /fredhome,
/fredvar.  Or it might use a serial number, something that's that
written on the drive and can be read when it's not in a computer..

[Tim]

On Wed, 2006-12-27 at 14:04 -0600, Bruno Wolff III wrote:
> If you are doing the copy graphically, there may be overhead with
> displaying the progress (I don't do copies that way, so don't know if
> in fact that is the case) which slows things down and doing a command
> line copy might be easier on your CPU.

Using Nautilus, for instance, does slow down copying immensely.  But not
from the progress bar (in my case, at least), but from what monitors
changes to the file system (where the file is being copied to).  I found
that changing the window showing the destination to show another
directory made a big difference.

[Mogens Kjaer]

Kevin Kempter wrote:
...
> # hdparm /dev/sda
>
> /dev/sda:
>  IO_support   =  0 (default 16-bit)
>  readonly     =  0 (off)
>  readahead    = 256 (on)
>  geometry     = 12161/255/63, sectors = 195371568, start = 0

OK, you're using the drive in SATA mode.

You should run something like this:

dmesg >/tmp/whatever

and check the /tmp/whatever file for what is logged
when the driver is loaded. I get something like this:

...
SCSI subsystem initialized
libata version 2.00 loaded.
ata_piix 0000:00:1f.2: version 2.00
ata_piix 0000:00:1f.2: MAP [ P0 P2 P1 P3 ]
ACPI: PCI Interrupt 0000:00:1f.2[B] -> GSI 19 (level, low) -> IRQ 201
PCI: Setting latency timer of device 0000:00:1f.2 to 64
ata1: SATA max UDMA/133 cmd 0x3818 ctl 0x3832 bmdma 0x34F0 irq 201
ata2: SATA max UDMA/133 cmd 0x3820 ctl 0x3836 bmdma 0x34F8 irq 201
scsi0 : ata_piix
ata1.00: ATA-7, max UDMA/100, 586072368 sectors: LBA48 NCQ (depth 0/32)
ata1.00: ata1: dev 0 multi count 16
ata1.00: configured for UDMA/100
scsi1 : ata_piix
ata2.00: ATA-7, max UDMA/100, 586072368 sectors: LBA48 NCQ (depth 0/32)
ata2.00: ata2: dev 0 multi count 16
ata2.00: configured for UDMA/100
...

This is for a machine with two 300G SATA disks.

Also make a simple read speed test:

# hdparm -t -T /dev/sda

/dev/sda:
  Timing cached reads:   3620 MB in  2.00 seconds = 1810.28 MB/sec
  Timing buffered disk reads:  204 MB in  3.01 seconds =  67.87 MB/sec

It's the number 67.87 which is interesting.

Mogens

[parishd]

On 12/28/06, Tim  wrote:
> Using Nautilus, for instance, does slow down copying immensely.  But not
> from the progress bar (in my case, at least), but from what monitors
> changes to the file system (where the file is being copied to).  I found
> that changing the window showing the destination to show another
> directory made a big difference.

I have observed the same behavior using the gentoo file manager.

Daniel

[James Wilkinson]

Kevin Kempter wrote:
> Whenever I copy large files (>2G) or create large tar archives I basically 
> loose control of my box (2.1Gz Laptop running FC5) until the copy or tar is 
> complete. I can still try and work but the mouse movements are delayed 
> several seconds, other programs take minutes to respond, etc.

Did you turn off DMA when installing Fedora (to make the disk check
work)? Did that make it into your grub configuration? Try
grep dma /etc/grub.conf
as root and see if it returns anything. While you're running as root,
try
hdparm -d /dev/hda
(or whatever your main hard disk is) and check that DMA is on.

What sort of disk subsystem do you have, anyway? Is it IDE, SATA, or
something else? What sort of motherboard do you have?

Hope this helps,

James.