Hello,
I have a cluster, master + 8 nodes, connected via a Netgear Gigabit Ethernet Switch (model JGS516).
The cables are of CAT5 type. Elsewhere I read that for a gigabit network, one should use CAT6 type cables, to avoid bad transmissions. Is that so?
I already have done basic speed checks, that indicate a throughput of about 0.5 Gbps. Hence the network appears to operate on half of its spec. speed.
Question 1: Should I worry about this reduced speed?
Question 2: How to check the transmission quality? Should I inspect the output of "/sbin/ifconfig eth0" for errors/dropped/overruns/collisions etc. in the RX/TX lines?
Lan card info: master: Tigon3 10/100/1000BaseT Ethernet tg3: eth0: Link is up at 1000 Mbps, full duplex. tg3: eth0: Flow control is on for TX and on for RX. nodes: Realtek RTL8169 eth0: Auto-negotiation Enabled. eth0: 1000Mbps Full-duplex operation.
Thanks for advice and help! Rob.
__________________________________________________ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com
Rob wrote:
[SNIP]
Question 1: Should I worry about this reduced speed?
Do you need to be able to transmit at more than 500mbps? Will it cause you noticeable delays in accomplishing what you need to do? If not, why worry?
I'm not being sarcastic. I'm just trying to imagine what would need all that bandwidth on an ongoing basis.
Question 2: How to check the transmission quality? Should I inspect the output of "/sbin/ifconfig eth0" for errors/dropped/overruns/collisions etc. in the RX/TX lines?
Lan card info: master: Tigon3 10/100/1000BaseT Ethernet tg3: eth0: Link is up at 1000 Mbps, full duplex. tg3: eth0: Flow control is on for TX and on for RX. nodes: Realtek RTL8169 eth0: Auto-negotiation Enabled. eth0: 1000Mbps Full-duplex operation.
Thanks for advice and help! Rob.
The other question I would ask is whether your systems are capable of driving their gigE interfaces at full capacity (not just bursting, but a sustained data stream). It's been a number of years since I looked into this, but for a long time it was the case that the networking interface was not the limiting factor in how fast a computer system could transmit or receive information. At the time this was true, I was the product line manager for some large, enterprise-class gigabit routers/switches. So, I needed to understand how much switching our customers really needed.
Finally, if buffers are getting filled faster than they can be drained, and flow control comes into play, that could reduce your effective throughput. It's a symptom of what I was talking about in the paragraph before this one. The computers themselves may not be capable of sourcing or sinking gigabit-speed data flows.
Debbie
Rob wrote:
Hello,
I have a cluster, master + 8 nodes, connected via a Netgear Gigabit Ethernet Switch (model JGS516).
The cables are of CAT5 type. Elsewhere I read that for a gigabit network, one should use CAT6 type cables, to avoid bad transmissions. Is that so?
Well, there is some debate about this. First, note that nominally CAT5 is rated for gigabit speeds. Second, note that gigabit cable does not really carry gigabit rate transistions, but "only" 250MHz. However, ISTM that even that is a bit much for simple twisted pair, even if it is impedance controlled.
I already have done basic speed checks, that indicate a throughput of about 0.5 Gbps. Hence the network appears to operate on half of its spec. speed.
That appears quite good to me. 50% occupancy is not bad.
Question 1: Should I worry about this reduced speed?
No, I don't think so. What have you got filling the pipe? Also, what protocols are you using? If you have to wait for an ACK from time to time, then you are going to get less than full BW. Also, protocols have other overhead, like headers on messages and CRCs and whatnot.
Question 2: How to check the transmission quality? Should I inspect the output of "/sbin/ifconfig eth0" for errors/dropped/overruns/collisions etc. in the RX/TX lines?
Yep.
# ifconfig eth0 eth0 Link encap:Ethernet HWaddr 00:0C:76:DF:1B:0D inet addr:172.17.205.79 Bcast:172.17.255.255 Mask:255.255.0.0 inet6 addr: fe80::20c:76ff:fedf:1b0d/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:1924127 errors:0 dropped:0 overruns:0 frame:0 TX packets:1368176 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:2416670249 (2304.7 Mb) TX bytes:99499187 (94.8 Mb) Interrupt:11 Base address:0xc800
# uptime 23:00:36 up 7 days, 12:38, 11 users, load average: 0.20, 0.17, 0.17
Admittedly, just me on my machine, so limited data. But it seems that I never miss a packet.
Mike
Mike McCarty wrote:
Rob wrote:
Hello,
I have a cluster, master + 8 nodes, connected via a Netgear Gigabit Ethernet Switch (model JGS516).
The cables are of CAT5 type. Elsewhere I read that for a gigabit network, one should use CAT6 type cables, to avoid bad transmissions. Is that so?
Well, there is some debate about this. First, note that nominally CAT5 is rated for gigabit speeds. Second, note that gigabit cable does not really carry gigabit rate transistions, but "only" 250MHz. However, ISTM that even that is a bit much for simple twisted pair, even if it is impedance controlled.
I already have done basic speed checks, that indicate a throughput of about 0.5 Gbps. Hence the network appears to operate on half of its spec. speed.
That appears quite good to me. 50% occupancy is not bad.
Question 1: Should I worry about this reduced speed?
No, I don't think so. What have you got filling the pipe? Also, what protocols are you using? If you have to wait for an ACK from time to time, then you are going to get less than full BW. Also, protocols have other overhead, like headers on messages and CRCs and whatnot.
The last time I did anything with this, I learned that the maximum bandwidth for actual data is ~75% of the rated bandwidth. This basically says 750Mb of data should take ~1sec. And this depends on all of the above stated points. If you really want to learn about network bandwidth testing, look at testing network operations for Beowulf clusters.
Install NetPipe and test your network. At least that is what I used on our cluster when we installed it.
On Thu, 27 Apr 2006, Rob wrote:
Hello,
I have a cluster, master + 8 nodes, connected via a Netgear Gigabit Ethernet Switch (model JGS516).
The cables are of CAT5 type. Elsewhere I read that for a gigabit network, one should use CAT6 type cables, to avoid bad transmissions. Is that so?
Not really. The symbol rate for gigabit ethernet is actually the same as 100Mb/s ethernet, (125mhz) and should run for similar distances over the same cable. using "better cable" buys you squat if your bit-error rate is currently zero...
I already have done basic speed checks, that indicate a throughput of about 0.5 Gbps. Hence the network appears to operate on half of its spec. speed.
Question 1: Should I worry about this reduced speed?
Are your cards in 32 bit 33mhz pci slots?
In general that's about what you can expect without careful selection of the hardware involved.
Question 2: How to check the transmission quality?
Do you see errors in any significant rate on any of your interfaces? Duplex mismatches are rare with gigabit ethernet,
Should I inspect the output of "/sbin/ifconfig eth0" for errors/dropped/overruns/collisions etc. in the RX/TX lines?
If your error rates increment rapidly while sending data then yeah you may have an issue there.
Lan card info: master: Tigon3 10/100/1000BaseT Ethernet tg3: eth0: Link is up at 1000 Mbps, full duplex. tg3: eth0: Flow control is on for TX and on for RX. nodes: Realtek RTL8169 eth0: Auto-negotiation Enabled. eth0: 1000Mbps Full-duplex operation.
Thanks for advice and help! Rob.
Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com
Rob wrote:
Hello,
I have a cluster, master + 8 nodes, connected via a Netgear Gigabit Ethernet Switch (model JGS516).
The cables are of CAT5 type.
I assume you mean Cat 5e UTP. If you really mean Cat 5, no, I wouldn't expect it to work well in a gigabit environment. Cat 5e, on the other hand supports gigabit rates.
Elsewhere I read that for a gigabit network, one should use CAT6 type cables, to avoid bad transmissions. Is that so?
*Should* is rather subjective, but as you see in many RFCs, *should* does not mean *must*. :-)
Cat 5e is adequate for Gigabit rates, pay attention to cable paths and termination. i.e avoid running near and parallel to power lines etc. When terminating the cables keep the "twist" as intact as you can... don't unwind more than actually required to punch the wire-ends down.
I already have done basic speed checks, that indicate a throughput of about 0.5 Gbps. Hence the network appears to operate on half of its spec. speed.
Well.... there's more to network speed than the cabling.... are you certain your hardware can deliver a sustained rate of a gigabit/second? Can the receiving end collect it at that rate?
If you transfer a large file and divide file size by time, that's not a true indication because it doesn't take into account the packet overhead. It is also an average rate. And, unless the average is the actual sustained rated, there are times when the rate was higher than the average and of course times when it was lower.
Question 1: Should I worry about this reduced speed?
Are you noticing any negative effects with your applications that need such high data rates? Are you sure everything is working in full duplex? What protocols are you using to test the throughput? Does every packet require an ACK? Or just an ACK every n packets to imply "all OK so far.. keep it coming"? In that case, perhaps increase "n".
Question 2: How to check the transmission quality? Should I inspect the output of "/sbin/ifconfig eth0" for errors/dropped/overruns/collisions etc. in the RX/TX lines?
I don't think collisions is what you want here... that's more a measure of network congestion, not transmission quality.
Transmission quality could be perfect with a high number of collisions, or quality could be crap with no collisions at all... so what is the collision count really telling you? Do you have collision count data from your pre-gigbit set up? With your new set up I'd expect the collision rate to drop because the "in use time" of any given packet transmission is shorter, thus making a collisionless likely.
You want to look at errors and retransmissions, that sort of thing.
HTH :-)
Don Russell wrote:
Rob wrote:
Hello,
I have a cluster, master + 8 nodes, connected via a Netgear Gigabit Ethernet Switch (model JGS516).
The cables are of CAT5 type.
I assume you mean Cat 5e UTP. If you really mean Cat 5, no, I wouldn't expect it to work well in a gigabit environment. Cat 5e, on the other hand supports gigabit rates.
Looking at the specs over on http://discountcablesusa.com/ethernet-cables.html I don't see why CAT5 wouldn't work reasonably well for short runs. For longer runs, it looks like the cross-talk might get you if the cable really doesn't meet CAT5e specs.
A quote from that page:
It might seem that CAT5 and CAT5e are the same. Pretty much they are, the CAT5e specification simply included some additional limits over the CAT5 specification. The reality is that most CAT5 cable is in fact CAT5e cable just not certified as such. Here is a comparison of those extra specifications.
And this is my experience.
YMMV
[snip]
Mike