This all started with a power out crash thanks to my Electrical Company. After power came back up I could not access the server via the network.
When I first got to the console I could not log in as root nor could if do an "su" as my user log in. Using Webmin via localhost I found the password hosed but I was able to reset the password via Webmin and after a reboot I could log in as root.
I then found I could not ping my static IP from inside or outside. Using the GUI at System/Administration/Network and checking the Interface settings they were correct. If I hit the DNS tab or the Hosts tab I found bogus information. I changed it back to the correct info and did a save followed by a Stop/Start on the interface and gave control back to Network Manager.
After a reboot the bogus info had returned! I could only ping from the inside the bogus IP address and of course could not ping it from the outside! So I manually went to /etc/hosts and corrected the bogus info that had a note "Added by Network Manager".
I also check /etc/resolv.conf and corrected the bogus DNS info.
I checked /etc/sysconfig/networking/devices/ifcfg-eth0 and the info there was correct.
Reboot and all the bogus info is back, all with the "Added by Network Manager" caveat!
Beat the bushes as well as I can I can find no info as to where Network Manage conf files might be named and or located!
Anyone have any idea where they are or am I just going to have to kill Network Manager?
This one is weird!
On 09/25/2010 06:50 AM, Mike Dwiggins wrote:
This all started with a power out crash thanks to my Electrical Company.
you trust you electrical supplier and it their fault that you do not use an ups?
do you have a full backup that you can use to wipe drive, run a full drive test, then restore your system?
you can use rpm to run a check on all you have installed, but it will not test configurations.
This one is weird!
not really. your drive got trashed.
On 9/25/2010 6:02 AM, g wrote:
On 09/25/2010 06:50 AM, Mike Dwiggins wrote:
This all started with a power out crash thanks to my Electrical Company.
you trust you electrical supplier and it their fault that you do not use an ups?
do you have a full backup that you can use to wipe drive, run a full drive test, then restore your system?
you can use rpm to run a check on all you have installed, but it will not test configurations.
This one is weird!
not really. your drive got trashed.
Well g, if I could afford a UPS that had a longer run time than two hours I would have it!.
I have checked fairly well and everything works as should be from localhost and I can find no other problem other than the IP configuration problem.
I am leaning towards a hack or a corruption of Network Manager.
I
On Saturday, September 25, 2010 10:33:05 am Mike Dwiggins did opine:
On 9/25/2010 6:02 AM, g wrote:
On 09/25/2010 06:50 AM, Mike Dwiggins wrote:
This all started with a power out crash thanks to my Electrical
Company.
you trust you electrical supplier and it their fault that you do not use an ups?
do you have a full backup that you can use to wipe drive, run a full drive test, then restore your system?
you can use rpm to run a check on all you have installed, but it will not test configurations.
This one is weird!
not really. your drive got trashed.
Well g, if I could afford a UPS that had a longer run time than two hours I would have it!.
Frankly, expecting a UPS to carry for 2 hours is an unreal expectation.
I did have one that could do that at one time, an old 200+ pound NCR branded thing I had installed a quartet of 14AH wet L-A batteries in, making arrangements to handle the overflow tubes by directing then into a pint jar half full of soda. It worked well but floated the wet batteries at too high a voltage and ran them dry at 6 month intervals, so that was about $120 2x a year to keep it running, even after I found the float setting and turned it down 2 volts to 52 volts. So for several years and quite a few battery packs, I have used a pair of 1500wa rated UPS's that I can get batteries for at a lot less at 2 to 3 year intervals. With the load here, I have about 5 minutes to do a graceful shutdown else it drops the line in less than 10. The modern ext3 file system tolerates that well and I have not had to fsck a drive at other than the scheduled intervals in many years now unless the drive itself was on its way out.
If you want a 2 hour holdup time, be aware that it tends to cost real money both at purchase time and at maintenance time. It is not going to be $350 in pocket change, but more likely in the 10 grand range for commercial rack mounted stuff.
I have checked fairly well and everything works as should be from localhost and I can find no other problem other than the IP configuration problem.
I am leaning towards a hack or a corruption of Network Manager.
I
On 9/25/10 7:52 AM, Gene Heskett wrote:
On Saturday, September 25, 2010 10:33:05 am Mike Dwiggins did opine:
On 9/25/2010 6:02 AM, g wrote:
On 09/25/2010 06:50 AM, Mike Dwiggins wrote:
This all started with a power out crash thanks to my ElectricalCompany.
you trust you electrical supplier and it their fault that you do not use an ups?
do you have a full backup that you can use to wipe drive, run a full drive test, then restore your system?
you can use rpm to run a check on all you have installed, but it will not test configurations.
This one is weird!
not really. your drive got trashed.
Well g, if I could afford a UPS that had a longer run time than two hours I would have it!.
Frankly, expecting a UPS to carry for 2 hours is an unreal expectation.
Not even commercial installations demand that type of hold time. Those things take sealed L-A batteries weighing in at about 1/2 ton each (I work with a facility that has six of these.)If you want a 2 hour holdup time, be aware that it tends to cost real money
both at purchase time and at maintenance time. It is not going to be $350 in pocket change, but more likely in the 10 grand range for commercial rack mounted stuff.
Those things come in at about USD 50K each as well. They would hold your home PC setup for hours and hours. A Honda self start generator is much less expensive.
James McKenzie
On 09/25/2010 02:21 PM, Mike Dwiggins wrote: <snip>
Well g, if I could afford a UPS that had a longer run time than two hours I would have it!.
an ups should not be considered as a means to keep a system up and running until power is restored. that is not a battery system, unless you have an auto start gas powered generator to kick in and supply mains.
it should be thought of as a means to allow a proper shutdown if power is not restored within a short period.
an ups that will supply just 5 or 10 minutes of up time is better than none.
the name ups, uninterrupted power supply. not un-interruptible power supply, as in continuous.
a good ups, runs from a floated battery. that is, it is powered by battery and ac mains supply a low level charging.
this way, when mains are lost, you are already on battery power, so there is no 'glitch' of sine wave.
once mains are lost, a signal notifies system and a proper shutdown can be started.
I am leaning towards a hack or a corruption of Network Manager.
yes. jb's points are much to be considered.
here again, 'rpm -V|--verify *' should be run.
much luck on your recovery.
On 9/25/2010 8:06 AM, g wrote:
On 09/25/2010 02:21 PM, Mike Dwiggins wrote:
<snip>
Well g, if I could afford a UPS that had a longer run time than two hours I would have it!.
an ups should not be considered as a means to keep a system up and running until power is restored. that is not a battery system, unless you have an auto start gas powered generator to kick in and supply mains.
it should be thought of as a means to allow a proper shutdown if power is not restored within a short period.
an ups that will supply just 5 or 10 minutes of up time is better than none.
the name ups, uninterrupted power supply. not un-interruptible power supply, as in continuous.
a good ups, runs from a floated battery. that is, it is powered by battery and ac mains supply a low level charging.
this way, when mains are lost, you are already on battery power, so there is no 'glitch' of sine wave.
once mains are lost, a signal notifies system and a proper shutdown can be started.
I am leaning towards a hack or a corruption of Network Manager.
yes. jb's points are much to be considered.
here again, 'rpm -V|--verify *' should be run.
much luck on your recovery.
Excellent description. That being said the reason I have such a big honking APC brand UPS is that my company's Logistics department overbought on a major contract! Rather than paying Inventory Tax on the extras the Boss sold them to several of us below our Dealer cost! I have approx. two hours run time but, this outage lasted over three and 1/2 hours while no one was home!
According to my logs I did get a controlled shutdown on all three of my servers, the two Red Hat ES 4 severs have no problem only my Fedora 13 server. Thus I am leaning more towards a hack that was just waiting for ANY reboot to pounce!
Especially after I got this:
# rpm -V| --verify * bash: --verify: command not found rpm: no arguments given for verify
Something is not right!
On 09/25/2010 12:55 PM, Mike Dwiggins wrote:
Especially after I got this:
# rpm -V| --verify * bash: --verify: command not found rpm: no arguments given for verify
Something is not right!
That command is not correct ... its piping output of 'rpm -V' into a command '--verify' which it correctly tells you does not exist.
the options -V and --verify are equivalent - and wildcard '*" wont work.
Try instead:
rpm -V NetworkManager
or whatever list of things you want to verify. If you want to verify all install packages try:
rpm -V -a
gene/
On 9/25/2010 10:18 AM, Genes MailLists wrote:
On 09/25/2010 12:55 PM, Mike Dwiggins wrote:
Especially after I got this:
# rpm -V| --verify * bash: --verify: command not found rpm: no arguments given for verify
Something is not right!
That command is not correct ... its piping output of 'rpm -V' into a command '--verify' which it correctly tells you does not exist.
the options -V and --verify are equivalent - and wildcard '*" wont work.
Try instead: rpm -V NetworkManageror whatever list of things you want to verify. If you want to verify all install packages try:
rpm -V -agene/
Thanks Gene, Clear cut case of miss-reading a Man page on my part!
Mike Dwiggins <mike <at> azdwiggins.com> writes:
This all started with a power out crash thanks to my Electrical Company. After power came back up I could not access the server via the network.
When I first got to the console I could not log in as root nor could if do an "su" as my user log in. Using Webmin via localhost I found the password hosed but I was able to reset the password via Webmin and after a reboot I could log in as root.
You do not get your root passwd hosed just because your machine went down or some unrelated software package malfunctions ... You have to consider that you have been hacked, I guess. Normally you should take your machine offline until you understand what is the damage.
I then found I could not ping my static IP from inside or outside. Using the GUI at System/Administration/Network and checking the Interface settings they were correct. If I hit the DNS tab or the Hosts tab I found bogus information.
Well, where do you get that info from ? Are you auto-configured by dhclient ? System/Administration/Network/Devices - interface - Edit General Controlled by NetworkManager ? Automatically obtain IP address settings with DHCP ? Automatically obtain DNS info from provider ?
Also, check: $ ps aux |grep -i dhc jb 6982 0.0 0.0 4360 708 pts/3 S+ 15:21 0:00 grep -i dhc root 14415 0.0 0.0 2984 676 ? S 06:13 0:00 /sbin/dhclient -d -4 -sf /usr/libexec/nm-dhcp-client.action -pf /var/run/dhclient-eth0.pid -lf /var/lib/dhclient/dhclient-5fb06bd0-0bb0-7ffb-45f1-d6edd65f3e03-eth0.lease -cf /var/run/nm-dhclient-eth0.conf eth0
That's response on my system.
Look at what kind of info you got last time: # less /var/lib/dhclient/dhclient-5fb06bd0-0bb0-7ffb-45f1-d6edd65f3e03-eth0.lease
Look at your own config settings: # less /var/run/nm-dhclient-eth0.conf That's perhaps from: # # ls -al /etc/dhclient-* -rw-r--r--. 1 root root 40 Feb 21 2010 /etc/dhclient-eth0.conf -rw-r--r--. 1 root root 40 Feb 21 2010 /etc/dhclient-wlan0.conf
I changed it back to the correct info and did a save followed by a Stop/Start on the interface and gave control back to Network Manager.
After a reboot the bogus info had returned!
As above... If you are, then on reboot you were auto-configured again (and overwritten).
I could only ping from the inside the bogus IP address and of course could not ping it from the outside! So I manually went to /etc/hosts and corrected the bogus info that had a note "Added by Network Manager".
As above... If you are, it will be overwritten by NetworkManager via dhclient.
I also check /etc/resolv.conf and corrected the bogus DNS info.
As above... If you are, it will be overwritten by NetworkManager via dhclient.
I checked /etc/sysconfig/networking/devices/ifcfg-eth0 and the info there was correct.
Reboot and all the bogus info is back, all with the "Added by Network Manager" caveat!
As above ...
Beat the bushes as well as I can I can find no info as to where Network Manage conf files might be named and or located!
$ man NetworkManager
$ man NetworkManager.conf ... ifcfg-rh plugin is used on the Fedora and Red Hat Enterprise Linux distributions to read and write configuration from the standard /etc/sysconfig/network-scripts/ifcfg-* files. ... $ cat /etc/NetworkManager/NetworkManager.conf [main] plugins=ifcfg-rh $ ls -al /etc/sysconfig/network-scripts/ifcfg-* $ cat /etc/sysconfig/network-scripts/ifcfg-eth0
Anyone have any idea where they are or am I just going to have to kill Network Manager?
This one is weird!
Tell us how your system is configured.
JB
On 9/25/2010 6:38 AM, JB wrote:
some unrelated software package malfunctions ... You have to consider that you have been hacked, I guess. Normally you should take your machine offline until you understand what is the damage.
I am only online long enough to test the ping
Well, where do you get that info from ?
System/Administration/Network/
Are you auto-configured by dhclient ?
Not supposed to be eth0 is set to Static IP
Controlled by NetworkManager ?
Yes
Automatically obtain IP address settings with DHCP ?
Again it is not set to
Automatically obtain DNS info from provider ?
No
Also, check: $ ps aux |grep -i dhc jb 6982 0.0 0.0 4360 708 pts/3 S+ 15:21 0:00 grep -i dhc root 14415 0.0 0.0 2984 676 ? S 06:13 0:00 /sbin/dhclient -d -4 -sf /usr/libexec/nm-dhcp-client.action -pf /var/run/dhclient-eth0.pid -lf /var/lib/dhclient/dhclient-5fb06bd0-0bb0-7ffb-45f1-d6edd65f3e03-eth0.lease -cf /var/run/nm-dhclient-eth0.conf eth0
That's response on my system.
On mine
# ps aux|grep -i dhc root 1047 0.0 0.1 2828 1192 ? S 08:10 0:00 /sbin/dhclient -d -4 -sf /usr/libexec/nm-dhcp-client.action -pf /var/run/dhclient-eth0.pid -lf /var/lib/dhclient/dhclient-15087fb0-92c7-40fe-ad3e-373bf0997205-eth0.lease -cf /var/run/nm-dhclient-eth0.conf eth0 root 2349 0.0 0.0 4360 736 pts/1 S+ 08:26 0:00 grep -i dhc #
Look at what kind of info you got last time: # less /var/lib/dhclient/dhclient-5fb06bd0-0bb0-7ffb-45f1-d6edd65f3e03-eth0.lease
Look at your own config settings: # less /var/run/nm-dhclient-eth0.conf That's perhaps from: # # ls -al /etc/dhclient-* -rw-r--r--. 1 root root 40 Feb 21 2010 /etc/dhclient-eth0.conf -rw-r--r--. 1 root root 40 Feb 21 2010 /etc/dhclient-wlan0.conf
on mine
# ls -al /etc/dhclient-* ls: cannot access /etc/dhclient-*: No such file or directory #
/etc/sysconfig/network-scripts/ifcfg-eth0 is as follows
# Intel Corporation 82540EM Gigabit Ethernet Controller DEVICE=eth0 BOOTPROTO=none DNS1=68.2.16.30 GATEWAY=x.x.x.1 HWADDR=00:C0:9F:20:FF:BA IPADDR=x.x.x.12 NETMASK=255.255.255.240 ONBOOT=yes DNS2=68.1.203.30 TYPE=Ethernet NM_CONTROLLED=yes IPV6INIT=no USERCTL=no PREFIX=28 DEFROUTE=yes IPV4_FAILURE_FATAL=yes NAME="System eth0" UUID=5fb06bd0-0bb0-7ffb-45f1-d6edd65f3e03
At his point I am thinking about pulling the data for my Bind and Web pages and doing a scorched earth recovery.
If this was as I am beginning to think a hack just waiting for a reboot to pounce< I am not sure if my back-up is clean!
Mike Dwiggins <mike <at> azdwiggins.com> writes:
On 9/25/2010 6:38 AM, JB wrote:
some unrelated software package malfunctions ... You have to consider that you have been hacked, I guess. Normally you should take your machine offline until you understand what is the damage.
I am only online long enough to test the ping
Well, where do you get that info from ?
System/Administration/Network/
Are you auto-configured by dhclient ?
Not supposed to be eth0 is set to Static IP
Not quite. But read on.
Controlled by NetworkManager ?
Yes
Just a propos. Enable (check off) the "Activate device when computer starts".
Automatically obtain IP address settings with DHCP ?
Again it is not set to
OK. If you select for "Statically set IP addresses", then the "Automatically obtain IP address settings with DHCP" is turned off.
Automatically obtain DNS info from provider ?
No
Not quite. But read on.
Also, check: $ ps aux |grep -i dhc jb 6982 0.0 0.0 4360 708 pts/3 S+ 15:21 0:00 grep -i dhc root 14415 0.0 0.0 2984 676 ? S 06:13 0:00 /sbin/dhclient -d -4 -sf /usr/libexec/nm-dhcp-client.action -pf /var/run/dhclient-eth0. pid -lf /var/lib/dhclient/dhclient-5fb06bd0-0bb0-7ffb-45f1-d6edd65f3e03-eth0.lease -cf /var/run/nm-dhclient-eth0.conf eth0
That's response on my system.
On mine
# ps aux|grep -i dhc
root 1047 0.0 0.1 2828 1192 ? S 08:10 0:00 /sbin/dhclient -d -4 -sf /usr/libexec/nm-dhcp-client.action -pf /var/run/dhclient-eth0.pid -lf /var/lib/dhclient/dhclient-15087fb0-92c7-40fe-ad3e-373bf0997205-eth0.lease -cf /var/run/nm-dhclient-eth0.conf eth0 root 2349 0.0 0.0 4360 736 pts/1 S+ 08:26 0:00 grep -i dhc #
Here we go !
The entry 'ps aux |grep -i dhc' that there is a dhclient run under control of NetworkManager. When it runs, it obtains all default and user requested data from DHCP and DNS servers and modifies those pesky system files.
Important: Normally, when you configure your interface as you described (static IP, DNS), the NetworkManager is run, but without NetworkManager-controlled dhclient. I just checked that that on my other machine :-)
So, something got screwed up in the past, either during configuration thru System/Administration/Network/ utility or panel's NetworkManager Applet utility. FYI, I had bad experience with the second one some months ago, submitted report and they did these and other fixes to it.
Let's try to clean up some of this stuff.
Let's save that dhcp-lease file for interrogation later on (it probably contains lease data that relates to invalid IP addresses, etc; that's what screwed up your IP data in various system files): # mv /var/lib/dhclient/dhclient-15087fb0-92c7-40fe-ad3e-373bf0997205-eth0.lease /var/lib/dhclient/dhclient-15087fb0-92c7-40fe-ad3e-373bf0997205-eth0.lease- crash
and create an empty file instead: # touch /var/lib/dhclient/dhclient-15087fb0-92c7-40fe-ad3e-373bf0997205- eth0.lease
Later on, you should examine that saved file for IP addresses, etc; check them with some DNS-type entries (dig, nslookup) on the Internet; you may want to talk to your ISP about them, time of the presumed attack (system downtime), check with your utilities provider about a time of presumed downtime in electricity supply, etc.
Let's kill that dhclient that should not run. # killall /sbin/dhclient Confirm that it is gone: # ps aux|grep -i dhc
After that you should restart your desktop (GNOME, etc), but best would be to verify the entire startup sequence and reboot your machine back to the desktop. After that verify again as above that dhclient is gone and that lease file is still empty as it should be.
Look at what kind of info you got last time: # less
/var/lib/dhclient/dhclient-5fb06bd0-0bb0-7ffb-45f1-d6edd65f3e03-eth0.lease
Look at your own config settings: # less /var/run/nm-dhclient-eth0.conf That's perhaps from: # # ls -al /etc/dhclient-* -rw-r--r--. 1 root root 40 Feb 21 2010 /etc/dhclient-eth0.conf -rw-r--r--. 1 root root 40 Feb 21 2010 /etc/dhclient-wlan0.conf
on mine
# ls -al /etc/dhclient-* ls: cannot access /etc/dhclient-*: No such file or directory #
/etc/sysconfig/network-scripts/ifcfg-eth0 is as follows
# Intel Corporation 82540EM Gigabit Ethernet Controller DEVICE=eth0 BOOTPROTO=none DNS1=68.2.16.30 GATEWAY=x.x.x.1 HWADDR=00:C0:9F:20:FF:BA IPADDR=x.x.x.12 NETMASK=255.255.255.240 ONBOOT=yes DNS2=68.1.203.30 TYPE=Ethernet NM_CONTROLLED=yes IPV6INIT=no USERCTL=no PREFIX=28 DEFROUTE=yes IPV4_FAILURE_FATAL=yes NAME="System eth0" UUID=5fb06bd0-0bb0-7ffb-45f1-d6edd65f3e03
At his point I am thinking about pulling the data for my Bind and Web pages and doing a scorched earth recovery.
If this was as I am beginning to think a hack just waiting for a reboot to pounce< I am not sure if my back-up is clean!
I understand that you have 2 more servers, so I assume that you can take this one (Fedora) offline without any impact on business you run. Regardless, do some investigation - you may learn something ...
Do run these security programs. First install them: # yum install chkrootkit rkhunter Note that some of them are interactive when run, so stand by.
Run this one and see output for any warnings: # chkrootkit Run next one and see output for any warnings: # rkhunter
Now, regarding what to do next. Because of that root password hosing I am inclined to believe that your system has been compromised. This to me would be enough to reinstall the system.
Regardless of any suspicion about shutdown process being compromised by an attacker, you should test your Fedora under stress condition to verify that it works correctly with controlled shutdown when AC fails and UPS jumps in.
Good luck and let us know the results.
JB
-
g -
Gene Heskett -
Genes MailLists -
James McKenzie -
JB -
mike