Hi,
I've got a Linux server at my parent's house. It had some hardware troubles and I shipped them a new computer and asked them to move the hard drive over. That worked out great, except now Fedora 8 won't boot. I think I know why (I think I need to rebuild my initrd), but how can I do that from remote? Unfortunately, due to time zone issues, I can't talk with my parents directly, just via e-mail. This has led to some really slow reboot cycles... They put in a live cd, configure the box for remote login and e-mail me. Later (maybe the next day), I login make a change and reboot. The system doesn't come back up so I have to e-mail them to tell me the error and have them put in the live cd again. This has repeated several times. One reboot takes days! It's getting very painful to try and recover this computer! So, I want to improve my ability to debug problems in the future... Any suggestions on better ways to debug a remote server?
Is there a way to redirect the Linux console over the network like you can to a serial port? My google searches keep leading me to "ethercons" for FreeBSD which claims it is based on a Linux driver that I can't find any information about. Is there something else that I haven't found? This would help with interacting with the console to read error messages.
Is there a way to have the kernel reboot the system automatically if it can't mount the root filesystem? I'm not 100% sure where the system is hanging, but I think nash gives up when it can't find the filesystem and just hangs. Can I specify that the kernel reboot on panic on the command line? My setting in /etc/sysctl.conf doesn't get run because the root filesystem isn't mounted yet!
Is there something I can use to trigger remote reboots? Like a wake-on-lan packet that reboots the system when hung?
Is there a linux distro (preferably Fedora or one of its variants) that I can boot a rescue session from using PXE? I'd like to build a setup with grub, pxelinux, etc... (if possible) where it boots into the main OS and on reboot, it will switch back-and-forth to a rescue distro. I just bought a Linksys WRT54GL (only $50 on amazon!) and I plan on using openvpn to setup a tunnel between this server and me and use the DHCP/TFTP server to serve up PXE. Maybe with PXE I could serve one config for booting the real hard drive and then another config when I want the rescue CD. This rescue distro would need to bring up an ssh server with a password preset or a public key so I could login remotely (if there is no way to redirect the console to the network). I've considered, if no other solution is available, about rolling a cusom Fedora spin with all the tools and configuration I need for this.
Thanks, Ryan
Ryan Wilson wrote, On 10/30/2008 04:16 PM:
Hi,
I've got a Linux server at my parent's house. It had some hardware troubles and I shipped them a new computer and asked them to move the hard drive over. That worked out great, except now Fedora 8 won't boot. I think I know why (I think I need to rebuild my initrd), but how can I do that from remote?
<SNIP>
Is there a way to have the kernel reboot the system automatically if it can't mount the root filesystem? I'm not 100% sure where the system is hanging, but I think nash gives up when it can't find the filesystem and just hangs. Can I specify that the kernel reboot on panic on the command line? My setting in /etc/sysctl.conf doesn't get run because the root filesystem isn't mounted yet!
http://www.google.com/search?hl=en&q=kernel+reboot+on+panic&btnG=Goo... http://www.cyberciti.biz/tips/reboot-linux-box-after-a-kernel-panic.html says you can "add panic=10 to the kernel command line, it will have exactly the same effect." as setting kernel.panic = 10 in /etc/sysctl.conf
Is there something I can use to trigger remote reboots? Like a wake-on-lan packet that reboots the system when hung?
Is there a linux distro (preferably Fedora or one of its variants) that I can boot a rescue session from using PXE? I'd like to build a setup with grub, pxelinux, etc... (if possible) where it boots into the main OS and on reboot, it will switch back-and-forth to a rescue distro.
<SNIP>
https://www.redhat.com/archives/fedora-test-list/2008-September/msg00353.htm... https://bugzilla.redhat.com/show_bug.cgi?id=159287 suggests that a rescue mode image (initrd) might be setup as a grub target. Perhaps you should add to that bug, with some ideas from your experience.
And if you are adventuresome you might try this idea from the same thread: https://www.redhat.com/archives/fedora-test-list/2008-September/msg00380.htm...
Still seems a bit scary having a machine up on the net with the non protected rescue environment. I hope you get it fixed quickly.
Of course with your situation, it MIGHT be easier to have them express the hard drive to you and work on it locally. :)
Thanks for your suggestions! I'll definitely add the panic reboot boot option. I think that will help me out alot. And I'm going to mull over that bugzilla bug and see if I have something to add.
See more below.
On Thu, Oct 30, 2008 at 9:51 PM, Todd Denniston < Todd.Denniston@ssa.crane.navy.mil> wrote: <snip>
Still seems a bit scary having a machine up on the net with the non protected rescue environment. I hope you get it fixed quickly.
Fortunately, the box isn't connected directly to the Internet, it sits behind a NAT/firewall which provides some minimal protections. At least enough so that someone won't be able to ssh in directly!
Of course with your situation, it MIGHT be easier to have them express the hard drive to you and work on it locally. :)
Now that's thinking out of the box. I'm always looking for a technical
solution (I'm a geek. There HAS to be a geeky way to fix it!). But sometimes the non-technical solution is the shortest path. Thanks for a good alternative. This might actually be faster...
Ryan
Todd Denniston wrote:
Is there a linux distro (preferably Fedora or one of its variants) that I can boot a rescue session from using PXE? I'd like to build a setup with grub, pxelinux, etc... (if possible) where it boots into the main OS and on reboot, it will switch back-and-forth to a rescue distro.
<SNIP>
https://www.redhat.com/archives/fedora-test-list/2008-September/msg00353.htm...
https://bugzilla.redhat.com/show_bug.cgi?id=159287 suggests that a rescue mode image (initrd) might be setup as a grub target. Perhaps you should add to that bug, with some ideas from your experience.
And if you are adventuresome you might try this idea from the same thread: https://www.redhat.com/archives/fedora-test-list/2008-September/msg00380.htm...
Still seems a bit scary having a machine up on the net with the non protected rescue environment. I hope you get it fixed quickly.
Of course with your situation, it MIGHT be easier to have them express the hard drive to you and work on it locally. :)
I've always thought it would be nice to be able to use the boot options of "linux rescue vnc vncconnect=host_or_IP" with an install CD or bootable USB drive and end up with at least a framebuffer containing a root xterm in the waiting remote vncviewer. But, those options don't work together.
-
Les Mikesell -
Ryan Wilson -
Todd Denniston