How to verify transmission quality & cables on gigabit network?

[Rob]

Hello,

I have a cluster, master + 8 nodes, connected via
a Netgear Gigabit Ethernet Switch (model JGS516).

The cables are of CAT5 type.
Elsewhere I read that for a gigabit network, one
should use CAT6 type cables, to avoid bad
transmissions. Is that so?

I already have done basic speed checks, that indicate
a throughput of about 0.5 Gbps. Hence the network
appears to operate on half of its spec. speed.

Question 1:
Should I worry about this reduced speed?

Question 2:
How to check the transmission quality?
Should I inspect the output of "/sbin/ifconfig eth0"
for errors/dropped/overruns/collisions etc. in the
RX/TX lines?

Lan card info:
 master: Tigon3 10/100/1000BaseT Ethernet
   tg3: eth0: Link is up at 1000 Mbps, full duplex.
   tg3: eth0: Flow control is on for TX and on for RX.
 nodes: Realtek RTL8169
   eth0: Auto-negotiation Enabled.
   eth0: 1000Mbps Full-duplex operation.

Thanks for advice and help!
Rob.

__________________________________________________
Do You Yahoo!?
Tired of spam?  Yahoo! Mail has the best spam protection around 
http://mail.yahoo.com

[Debbie Deutsch]

Rob wrote:

[SNIP]
> Question 1:
> Should I worry about this reduced speed?

Do you need to be able to transmit at more than 500mbps?  Will it cause
you noticeable delays in accomplishing what you need to do?  If not, why
worry?

I'm not being sarcastic.  I'm just trying to imagine what would need all
that bandwidth on an ongoing basis.
> Question 2:
> How to check the transmission quality?
> Should I inspect the output of "/sbin/ifconfig eth0"
> for errors/dropped/overruns/collisions etc. in the
> RX/TX lines?
>
> Lan card info:
>  master: Tigon3 10/100/1000BaseT Ethernet
>    tg3: eth0: Link is up at 1000 Mbps, full duplex.
>    tg3: eth0: Flow control is on for TX and on for RX.
>  nodes: Realtek RTL8169
>    eth0: Auto-negotiation Enabled.
>    eth0: 1000Mbps Full-duplex operation.
>
> Thanks for advice and help!
> Rob.

The other question I would ask is whether your systems are capable of
driving their gigE interfaces at full capacity (not just bursting, but a
sustained data stream).  It's been a number of years since I looked into
this, but for a long time it was the case that  the networking interface
was not the limiting factor in how fast a computer system could transmit
or receive information.  At the time this was true, I was the product
line manager for some large, enterprise-class gigabit routers/switches.
 So, I needed to understand how much switching our customers really needed.

Finally, if buffers are getting filled faster than they can be drained,
and flow control comes into play, that could reduce your effective
throughput.  It's a symptom of what I was talking about in the paragraph
before this one.  The computers themselves may not be capable of
sourcing or sinking gigabit-speed data flows.

Debbie

[Mike McCarty]

Rob wrote:
> Hello,
>
> I have a cluster, master + 8 nodes, connected via
> a Netgear Gigabit Ethernet Switch (model JGS516).
>
> The cables are of CAT5 type.
> Elsewhere I read that for a gigabit network, one
> should use CAT6 type cables, to avoid bad
> transmissions. Is that so?

Well, there is some debate about this. First, note that
nominally CAT5 is rated for gigabit speeds. Second, note
that gigabit cable does not really carry gigabit rate
transistions, but "only" 250MHz. However, ISTM that
even that is a bit much for simple twisted pair, even if
it is impedance controlled.
> I already have done basic speed checks, that indicate
> a throughput of about 0.5 Gbps. Hence the network
> appears to operate on half of its spec. speed.

That appears quite good to me. 50% occupancy is not
bad.
> Question 1:
> Should I worry about this reduced speed?

No, I don't think so. What have you got filling the pipe?
Also, what protocols are you using? If you have to wait
for an ACK from time to time, then you are going to get
less than full BW. Also, protocols have other overhead,
like headers on messages and CRCs and whatnot.
> Question 2:
> How to check the transmission quality?
> Should I inspect the output of "/sbin/ifconfig eth0"
> for errors/dropped/overruns/collisions etc. in the
> RX/TX lines?

Yep.

# ifconfig eth0
eth0      Link encap:Ethernet  HWaddr 00:0C:76:DF:1B:0D
           inet addr:172.17.205.79  Bcast:172.17.255.255  Mask:255.255.0.0
           inet6 addr: fe80::20c:76ff:fedf:1b0d/64 Scope:Link
           UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
           RX packets:1924127 errors:0 dropped:0 overruns:0 frame:0
           TX packets:1368176 errors:0 dropped:0 overruns:0 carrier:0
           collisions:0 txqueuelen:1000
           RX bytes:2416670249 (2304.7 Mb)  TX bytes:99499187 (94.8 Mb)
           Interrupt:11 Base address:0xc800

# uptime
  23:00:36 up 7 days, 12:38, 11 users,  load average: 0.20, 0.17, 0.17

Admittedly, just me on my machine, so limited data. But it seems
that I never miss a packet.

Mike

[Robin Laing]

Mike McCarty wrote:
> Rob wrote:
> > Hello,
> >
> > I have a cluster, master + 8 nodes, connected via
> > a Netgear Gigabit Ethernet Switch (model JGS516).
> >
> > The cables are of CAT5 type.
> > Elsewhere I read that for a gigabit network, one
> > should use CAT6 type cables, to avoid bad
> > transmissions. Is that so?
>
> Well, there is some debate about this. First, note that
> nominally CAT5 is rated for gigabit speeds. Second, note
> that gigabit cable does not really carry gigabit rate
> transistions, but "only" 250MHz. However, ISTM that
> even that is a bit much for simple twisted pair, even if
> it is impedance controlled.
> > I already have done basic speed checks, that indicate
> > a throughput of about 0.5 Gbps. Hence the network
> > appears to operate on half of its spec. speed.
>
> That appears quite good to me. 50% occupancy is not
> bad.
> > Question 1:
> > Should I worry about this reduced speed?
>
> No, I don't think so. What have you got filling the pipe?
> Also, what protocols are you using? If you have to wait
> for an ACK from time to time, then you are going to get
> less than full BW. Also, protocols have other overhead,
> like headers on messages and CRCs and whatnot.

The last time I did anything with this, I learned that the maximum 
bandwidth for actual data is ~75% of the rated bandwidth.  This 
basically says 750Mb of data should take ~1sec.  And this depends on all 
of the above stated points.  If you really want to learn about network 
bandwidth testing, look at testing network operations for Beowulf clusters.

Install NetPipe and test your network.  At least that is what I used on 
our cluster when we installed it.

[Joel Jaeggli]

On Thu, 27 Apr 2006, Rob wrote:
> Hello,
>
> I have a cluster, master + 8 nodes, connected via
> a Netgear Gigabit Ethernet Switch (model JGS516).
>
> The cables are of CAT5 type.
> Elsewhere I read that for a gigabit network, one
> should use CAT6 type cables, to avoid bad
> transmissions. Is that so?

Not really. The symbol rate for gigabit ethernet is actually the same as 
100Mb/s ethernet, (125mhz) and should run for similar distances over the 
same cable. using "better cable" buys you squat if your bit-error rate is 
currently zero...
> I already have done basic speed checks, that indicate
> a throughput of about 0.5 Gbps. Hence the network
> appears to operate on half of its spec. speed.
>
> Question 1:
> Should I worry about this reduced speed?

Are your cards in 32 bit 33mhz pci slots?

In general that's about what you can expect without careful selection of 
the hardware involved.
> Question 2:
> How to check the transmission quality?

Do you see errors in any significant rate on any of your interfaces? 
Duplex mismatches are rare with gigabit ethernet,
> Should I inspect the output of "/sbin/ifconfig eth0"
> for errors/dropped/overruns/collisions etc. in the
> RX/TX lines?

If your error rates increment rapidly while sending data then yeah you may 
have an issue there.
> Lan card info:
> master: Tigon3 10/100/1000BaseT Ethernet
>   tg3: eth0: Link is up at 1000 Mbps, full duplex.
>   tg3: eth0: Flow control is on for TX and on for RX.
> nodes: Realtek RTL8169
>   eth0: Auto-negotiation Enabled.
>   eth0: 1000Mbps Full-duplex operation.
>
> Thanks for advice and help!
> Rob.
>
> Do You Yahoo!?
> Tired of spam?  Yahoo! Mail has the best spam protection around
> http://mail.yahoo.com

[Don Russell]

Rob wrote:
> Hello,
>
> I have a cluster, master + 8 nodes, connected via
> a Netgear Gigabit Ethernet Switch (model JGS516).
>
> The cables are of CAT5 type.

I assume you mean Cat 5e UTP. If you really mean Cat 5, no, I wouldn't 
expect it to work well in a gigabit environment.
Cat 5e, on the other hand supports gigabit rates.
> Elsewhere I read that for a gigabit network, one
> should use CAT6 type cables, to avoid bad
> transmissions. Is that so?

*Should* is rather subjective, but as you see in many RFCs, *should* 
does not mean *must*. :-)

Cat 5e is adequate for Gigabit rates, pay attention to cable paths and 
termination. i.e avoid running near and parallel to power lines etc. 
When terminating the cables keep the "twist" as intact as you can... 
don't unwind more than actually required to punch the wire-ends down.
> I already have done basic speed checks, that indicate
> a throughput of about 0.5 Gbps. Hence the network
> appears to operate on half of its spec. speed.

Well.... there's more to network speed than the cabling.... are you 
certain your hardware can deliver a sustained rate of a gigabit/second? 
Can the receiving end collect it at that rate?

If you transfer a large file and divide file size by time, that's not a 
true indication because it doesn't take into account the packet 
overhead. It is also an average rate. And, unless the average is the 
actual sustained rated, there are times when the rate was higher than 
the average and of course times when it was lower.
> Question 1:
> Should I worry about this reduced speed?

Are you noticing any negative effects with your applications that need 
such high data rates? Are you sure everything is working in full duplex? 
What protocols are you using to test the throughput? Does every packet 
require an ACK? Or just an ACK every n packets to imply "all OK so far.. 
keep it coming"? In that case, perhaps increase "n".
> Question 2:
> How to check the transmission quality?
> Should I inspect the output of "/sbin/ifconfig eth0"
> for errors/dropped/overruns/collisions etc. in the
> RX/TX lines?

I don't think collisions is what you want here... that's more a measure 
of network congestion, not transmission quality.

Transmission quality could be perfect with a high number of collisions, 
or quality could be crap with no collisions at all... so what is the 
collision count really telling you? Do you have collision count data 
from your pre-gigbit set up? With your new set up I'd expect the 
collision rate to drop because the "in use time" of any given packet 
transmission is shorter, thus making a collisionless likely.

You want to look at errors and retransmissions, that sort of thing.

HTH :-)

[Mike McCarty]

Don Russell wrote:
> Rob wrote:
> > Hello,
> >
> > I have a cluster, master + 8 nodes, connected via
> > a Netgear Gigabit Ethernet Switch (model JGS516).
> >
> > The cables are of CAT5 type.
>
> I assume you mean Cat 5e UTP. If you really mean Cat 5, no, I wouldn't 
> expect it to work well in a gigabit environment.
> Cat 5e, on the other hand supports gigabit rates.

Looking at the specs over on
http://discountcablesusa.com/ethernet-cables.html
I don't see why CAT5 wouldn't work reasonably well for short
runs. For longer runs, it looks like the cross-talk
might get you if the cable really doesn't meet CAT5e
specs.

A quote from that page:
> It might seem that CAT5 and CAT5e are the same. Pretty much they are,
> the CAT5e specification simply included some additional limits over the
> CAT5 specification. The reality is that most CAT5 cable is in fact CAT5e
> cable just not certified as such. Here is a comparison of those extra
> specifications.

And this is my experience.

YMMV

[snip]

Mike