Is there an application to log system usage that will enable me to keep track of usage by individual computers on our LAN?
Our internet usage is limited by our ISP [Wildblue] and when our 30 day usage goes too high I tend to panic, and I receive notifications. We are presently at 84%, within less than 3 gB of maximum and it can take a couple of weeks of little or no usage to reduce that.
I think the present problem was caused by a new iPad added lately. It tends to get left on continuously and the user didn't realize it was connecting and doing whatever while he thought it was idle. By the time we identified the probable culprit we burned a lot of b.w.
DD-WRT in the router logs some usage but not enough to be of much use and their mailing list/forum is nearly useless for problem solving.
So I am looking for a scheme for tracking usage by mac or device address. I check usage daily but when I see a jump upward I need to know what caused it, I always begin to think through what I have done before panicking the whole family. Any suggestion appreciated.
Bob
--
On Wednesday, March 30, 2011 12:31:31 pm Bob Goodwin wrote:
Is there an application to log system usage that will enable me to keep track of usage by individual computers on our LAN?
[snip]
So I am looking for a scheme for tracking usage by mac or device address. I check usage daily but when I see a jump upward I need to know what caused it, I always begin to think through what I have done before panicking the whole family. Any suggestion appreciated.
ntop is one possibility, and it's present in the Fedora repositories. You need a netflow source; nprobe can do that, and ntop can use built-in interfaces, or can take netflow data from your switch, if that switch is capable of netflow export. You need either what's known as a 'SPAN' port on your switch, or you need a hub on a common connection, possibly the WAN port itself, to be able to sniff all the traffic in lieu of netflow data export.
For the DD-WRT side of things, since you mention that you use that, please see: http://netflowninjas.lancope.com/blog/2009/07/turn-your-linksys-into-a-netfl...
and
http://www.dd-wrt.com/wiki/index.php/Useful_tools_for_the_WiFi_Network
On Wed, Mar 30, 2011 at 12:31:31 -0400, Bob Goodwin bobgoodwin@wildblue.net wrote:
DD-WRT in the router logs some usage but not enough to be of much use and their mailing list/forum is nearly useless for problem solving.
Openwrt may be a better option. There is a package management system and it is relatively easy to do custom builds if you want something outside of the available packages or built differently.
So I am looking for a scheme for tracking usage by mac or device address. I check usage daily but when I see a jump upward I need to know what caused it, I always begin to think through what I have done before panicking the whole family. Any suggestion appreciated.
If you run the router as an actual router rather than a bridge you should be able to get a pretty good handle on this using traffic shapping. The Linux Advanced Routing and Traffic Control documentation is a bit dated, but should give you some ideas of what is possible and how to implement policy. The doucmentation is at lartc.org.
On 30/03/11 13:39, Bruno Wolff III wrote:
On Wed, Mar 30, 2011 at 12:31:31 -0400, Bob Goodwinbobgoodwin@wildblue.net wrote:
DD-WRT in the router logs some usage but not enough to be of much use and their mailing list/forum is nearly useless for problem solving.Openwrt may be a better option. There is a package management system and it is relatively easy to do custom builds if you want something outside of the available packages or built differently.
So I am looking for a scheme for tracking usage by mac or device address. I check usage daily but when I see a jump upward I need to know what caused it, I always begin to think through what I have done before panicking the whole family. Any suggestion appreciated.If you run the router as an actual router rather than a bridge you should be able to get a pretty good handle on this using traffic shapping. The Linux Advanced Routing and Traffic Control documentation is a bit dated, but should give you some ideas of what is possible and how to implement policy. The doucmentation is at lartc.org.
Bruno and Lamar ~
You have overwhelmed me with information. This is a complex topic!
I need a bit more help to get my thought processes working. I have a "modem" and a router [Linksys E3000 w/dd-wrt] between which I could install a an old Linksys 10/100 ethernet hub and run a line [~50 ft] to this computer to process the data.
I was hoping to find an application that would process that data into something I could interpret. I tried that a couple of years ago but was unable to get anything I could deal with ...
Netflow says their application is not intended for home use? It's not clear to me if that has to be installed in a computer/router or if it's something I can install here in this computer or if it might already be installed in some routers out of the box?
It would be nice if someone could say try this approach. If there is an advantage to using openwrt instead for collecting this data I have another router I can probably install it on, a Netgear WNDR3300/dd-wrt. I can follow instructions but I can't write an application.
Bob
On Wed, Mar 30, 2011 at 14:40:16 -0400, Bob Goodwin bobgoodwin@wildblue.net wrote:
I need a bit more help to get my thought processes working. I have a "modem" and a router [Linksys E3000 w/dd-wrt] between which I could install a an old Linksys 10/100 ethernet hub and run a line [~50 ft] to this computer to process the data.
I was suggesting using the current router to do traffic shaping to keep you data usage under control on a daily basis. Though you can also track aggregate totals as well. OpenWRT is more flexible than ddwrt, though you can probably do this with ddwrt as well. If you have the device set up as a bridge (which is likely) that makes it harder to distinguish which traffic is going where, though it should be possible.
I was hoping to find an application that would process that data into something I could interpret. I tried that a couple of years ago but was unable to get anything I could deal with ...
You should be able to query and reset counters with a script. Just make sure to check them often enough that a power outage won't mess things up too badly.
Netflow says their application is not intended for home use? It's not clear to me if that has to be installed in a computer/router or if it's something I can install here in this computer or if it might already be installed in some routers out of the box?
Consumer routers generally don't have a lot of memory and may prevent you from doing that. I am not familiar with that partciular package and don't know if it is likely to fit or not.
It would be nice if someone could say try this approach. If there is an advantage to using openwrt instead for collecting this data I have another router I can probably install it on, a Netgear WNDR3300/dd-wrt. I can follow instructions but I can't write an application.
Maybe start by seeing what you can do with ddwrt to see if that approach is practical for you at all. Writing scripts to pull counter data periodically shouldn't be too bad. The main thing is making sure the traffic is being properly counted. Doing shaping/policing has the advantage of proactively preventing you from using up your quota before you can react.
On Wednesday, March 30, 2011 02:40:16 pm Bob Goodwin wrote:
Netflow says their application is not intended for home use? It's not clear to me if that has to be installed in a computer/router or if it's something I can install here in this computer or if it might already be installed in some routers out of the box?
Sorry for overwhelming with info; here's the simpler version.
Netflow data export is a way the router can keep track of 'flows' of data (think of a flow as a connection; it isn't really, but it's still a good analogy) and export data on those flows passing through it to a 'collector.' DD-WRT apparently has some support for netflow data export (NDE for short) in this manner. One of the links I sent was a page that listed a few things about that, and possibly more links to how to set that up in DD-WRT.
Once you have NDE set up to export (but before you actually turn the export on) you need to set up the collector; this is the ntop package that is included in Fedora. It is a web-based application; there are other flow collectors, but the key thing is that the box running the collector needs to have its firewall opened for the export from the router, and the router needs to know to export the flow data to that IP address.
Once you have ntop collecting the flows, you can get all kinds of statistics on the top talkers, total bandwidth, connections used, IP addresses contacted, just to start.
The setup isn't the easiest in the world; but, then again you have DD-WRT set up, so you've apparently got at least part of the skillset needed. Just tackle it with patience, and you can make that work.
A hub and doing the collection with a sniffer and ntop will also work, but hubs have their own problems, and unless you'd just rather do it that way, having the router do NDE is the simplest way of getting the information you want.
I'm doing this, using CentOS and ntop, with several Cisco routers of various types (a couple of 12000 series, a 7609, a 7206, a 7507, and a 7401) and it works pretty well. On CentOS 4 ntop isn't exceptionally stable; not a whole lot better on CentOS 5, but I would expect that the latest and greatest running on F14 might be the ticket.
But my setup isn't the typical home setup, either, so your mileage may vary.
What would be the 'cat's meow' would be ntop or similar integrated into the DD-WRT or other similar router interface, then it's all 'appliance based' and easy.
On Wednesday, March 30, 2011 02:52:04 pm Bruno Wolff III wrote:
Maybe start by seeing what you can do with ddwrt to see if that approach is practical for you at all. Writing scripts to pull counter data periodically shouldn't be too bad.
If the device has any SNMP functionality, the Fedora package of MRTG works fine and will give basic statistics.
On 30/03/11 15:10, Lamar Owen wrote:
On Wednesday, March 30, 2011 02:40:16 pm Bob Goodwin wrote:
Netflow says their application is not intended for home use? It's not clear to me if that has to be installed in a computer/router or if it's something I can install here in this computer or if it might already be installed in some routers out of the box?Sorry for overwhelming with info; here's the simpler version.
Netflow data export is a way the router can keep track of 'flows' of data (think of a flow as a connection; it isn't really, but it's still a good analogy) and export data on those flows passing through it to a 'collector.' DD-WRT apparently has some support for netflow data export (NDE for short) in this manner. One of the links I sent was a page that listed a few things about that, and possibly more links to how to set that up in DD-WRT.
Once you have NDE set up to export (but before you actually turn the export on) you need to set up the collector; this is the ntop package that is included in Fedora. It is a web-based application; there are other flow collectors, but the key thing is that the box running the collector needs to have its firewall opened for the export from the router, and the router needs to know to export the flow data to that IP address.
Once you have ntop collecting the flows, you can get all kinds of statistics on the top talkers, total bandwidth, connections used, IP addresses contacted, just to start.
The setup isn't the easiest in the world; but, then again you have DD-WRT set up, so you've apparently got at least part of the skillset needed. Just tackle it with patience, and you can make that work.
A hub and doing the collection with a sniffer and ntop will also work, but hubs have their own problems, and unless you'd just rather do it that way, having the router do NDE is the simplest way of getting the information you want.
I'm doing this, using CentOS and ntop, with several Cisco routers of various types (a couple of 12000 series, a 7609, a 7206, a 7507, and a 7401) and it works pretty well. On CentOS 4 ntop isn't exceptionally stable; not a whole lot better on CentOS 5, but I would expect that the latest and greatest running on F14 might be the ticket.
But my setup isn't the typical home setup, either, so your mileage may vary.
What would be the 'cat's meow' would be ntop or similar integrated into the DD-WRT or other similar router interface, then it's all 'appliance based' and easy.
Well I'm still overwhelmed but I installed ntop and it turns out that dd-wrt has a function called Rflow, and another MACupd which I also enabled, and I am getting some pretty impressive displays.
It looks like it will do what I want if I can just master it's operation. I will have fun with this! It is serious business though, I've got to get usage under control or they throttle user speed and threaten worse!
I'll be back with questions once I know what to ask.
Thanks all for the excellent help and advice.
Bob
-
Bob Goodwin -
Bruno Wolff III -
Lamar Owen