xen failure - xenstrore with 2.6.15-1.2009.4.2_FC5hypervisor #1 SMP

[Naoki]

Should have added the xen version sorry : xen-3.0.1-4
It's all latest rawhide.

On Thu, 2006-03-16 at 16:31 +0900, Naoki wrote:
> I can't get xend to start, have not been able to for a couple of weeks,
> if not more actually. Using 2.6.15-1.2009.4.2_FC5hypervisor #1 SMP
> 
> Log says this :
> 
> [2006-03-16 16:15:47 xend] INFO (SrvDaemon:285) Xend Daemon started
> [2006-03-16 16:15:47 xend] INFO (SrvDaemon:289) Xend changeset:
> unavailable .
> [2006-03-16 16:15:47 xend] ERROR (SrvDaemon:299) Exception starting xend
> ((111, 'Connection refused'))
> Traceback (most recent call last):
>   File "/usr/lib/python2.4/site-packages/xen/xend/server/SrvDaemon.py",
> line 293, in run
>     servers = SrvServer.create()
>   File "/usr/lib/python2.4/site-packages/xen/xend/server/SrvServer.py",
> line 106, in create
>     root.putChild('xend', SrvRoot())
>   File "/usr/lib/python2.4/site-packages/xen/xend/server/SrvRoot.py",
> line 40, in __init__
>     self.get(name)
>   File "/usr/lib/python2.4/site-packages/xen/web/SrvDir.py", line 82, in
> get
>     val = val.getobj()
>   File "/usr/lib/python2.4/site-packages/xen/web/SrvDir.py", line 52, in
> getobj
>     self.obj = klassobj()
>   File
> "/usr/lib/python2.4/site-packages/xen/xend/server/SrvDomainDir.py", line
> 39, in __init__
>     self.xd = XendDomain.instance()
>   File "/usr/lib/python2.4/site-packages/xen/xend/XendDomain.py", line
> 550, in instance
>     inst.init()
>   File "/usr/lib/python2.4/site-packages/xen/xend/XendDomain.py", line
> 70, in init
>     xstransact.Mkdir(VMROOT)
>   File
> "/usr/lib/python2.4/site-packages/xen/xend/xenstore/xstransact.py", line
> 317, in Mkdir
>     complete(path, lambda t: t.mkdir(*args))
>   File
> "/usr/lib/python2.4/site-packages/xen/xend/xenstore/xstransact.py", line
> 323, in complete
>     t = xstransact(path)
>   File
> "/usr/lib/python2.4/site-packages/xen/xend/xenstore/xstransact.py", line
> 20, in __init__
>     self.transaction = xshandle().transaction_start()
>   File "/usr/lib/python2.4/site-packages/xen/xend/xenstore/xsutil.py",
> line 18, in xshandle
>     xs_handle = xen.lowlevel.xs.xs()
> RuntimeError: (111, 'Connection refused')
> 
> I traced the binary and I see this :
> [pid  2954] socket(PF_FILE, SOCK_STREAM, 0) = 4
> [pid  2954] connect(4, {sa_family=AF_FILE,
> path="/var/run/xenstored/socket"}, 110) = -1 ECONNREFUSED (Connection
> refused)
> [pid  2954] close(4)  
> [pid  2943] socket(PF_FILE, SOCK_STREAM, 0 <unfinished ...>
> [pid  2954] write(2, "Failed to contact xenstore (Conn"..., 65
> <unfinished ...>
> 
> The socket exists but nothing has a handle on it though :
> 
> $ ls -l /var/run/xenstored/socket
> srw------- 1 root root 0 Mar 16 16:18 /var/run/xenstored/socket
> 
> It suggests xenstore is my problem but in the same trace I see it
> executing ok :
> 
> [pid  2946] execve("/usr/sbin/xenstored", ["xenstored",
> "--pid-file=/var/run/xenstore.pid"...], [/* 20 vars */]) = 0
> 
> So I ran it manually and I get this :
> 
> # FATAL: Failed to initialize dom0 state: No such device
> full talloc report on 'null_context' (total 56 bytes in 3 blocks)
>     struct domain                  contains     56 bytes in   2 blocks
> (ref 0)
>         /local/domain/0                contains     16 bytes in   1
> blocks (ref 0)
> 
> Another trace shows the failure probably being here :
> 
> open("/proc/xen/xsd_kva", O_RDWR)       = 10
> mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_SHARED, 10, 0) = -1 ENODEV
> (No such device)
> close(10)    
> 
> # ls -l /proc/xen/xsd_kva
> -r-------- 1 root root 0 Mar 16 16:31 /proc/xen/xsd_kva
> 
> 
> That's about as far as my debugging skills can go, anybody seen/know
> this?