New tcl-8.5 and 8Kingdoms

[Hans de Goede]

Hi all,

I just tried building 8Kingdoms with the new tcl in rawhide, it builds fine, 
but it does not work.

Any of the actions (moving a unit for example) which normally invoke a tcl 
script underneath now no longer work.

I'm quite good in fixing C bugs but this problem lies either in the interfacing 
with tcl or in the tcl code itself and I'm helpless there. So I hope that 
someone can try running 8Kingdoms with tcl 8.5, and write a fix.

Note I've also send a request upstream asking for help, I'll report back here 
as soon as I get response.

If I don't have it fixed by F-9 then I will have to remove 8Kingdoms from 
Fedora's repositories (better then shipping a broken version).

Thanks & Regards,

Hans

[Michael Thomas]

Hans de Goede wrote:
> Hi all,
>
> I just tried building 8Kingdoms with the new tcl in rawhide, it builds
> fine, but it does not work.
>
> Any of the actions (moving a unit for example) which normally invoke a
> tcl script underneath now no longer work.
>
> I'm quite good in fixing C bugs but this problem lies either in the
> interfacing with tcl or in the tcl code itself and I'm helpless there.
> So I hope that someone can try running 8Kingdoms with tcl 8.5, and write
> a fix.
>
> Note I've also send a request upstream asking for help, I'll report back
> here as soon as I get response.
>
> If I don't have it fixed by F-9 then I will have to remove 8Kingdoms
> from Fedora's repositories (better then shipping a broken version).

Did you have to make any changes to 8Kingdoms to get it to build with
Tcl 8.5?  If so please pass along any patches.  I'll try building and
running it myself to see if I can track down the problem.

--Wart

[Hans de Goede]

Michael Thomas wrote:
> Hans de Goede wrote:
> > Hi all,
> >
> > I just tried building 8Kingdoms with the new tcl in rawhide, it builds
> > fine, but it does not work.
> >
> > Any of the actions (moving a unit for example) which normally invoke a
> > tcl script underneath now no longer work.
> >
> > I'm quite good in fixing C bugs but this problem lies either in the
> > interfacing with tcl or in the tcl code itself and I'm helpless there.
> > So I hope that someone can try running 8Kingdoms with tcl 8.5, and write
> > a fix.
> >
> > Note I've also send a request upstream asking for help, I'll report back
> > here as soon as I get response.
> >
> > If I don't have it fixed by F-9 then I will have to remove 8Kingdoms
> > from Fedora's repositories (better then shipping a broken version).
>
> Did you have to make any changes to 8Kingdoms to get it to build with
> Tcl 8.5?  If so please pass along any patches.  I'll try building and
> running it myself to see if I can track down the problem.

Nope,

No changes, I also did a diff between the buildlogs, no new compiler warnings 
or anything like that.

I did build it with gcc-4.3, as I was working on it not building with gcc-4.3 too.

I've just committed my latest 8Kingdoms work to cvs, so you can get it from 
there, including a patch for the crash you reported (tested using tcl 8.4).

Thanks,

Hans

[Marcela Mašláňová]

I had also look at it and ask upstream about help.

Hans de Goede wrote:
> Michael Thomas wrote:
> > Hans de Goede wrote:
> > > Hi all,
> > >
> > > I just tried building 8Kingdoms with the new tcl in rawhide, it builds
> > > fine, but it does not work.
> > >
> > > Any of the actions (moving a unit for example) which normally invoke a
> > > tcl script underneath now no longer work.
> > >
> > > I'm quite good in fixing C bugs but this problem lies either in the
> > > interfacing with tcl or in the tcl code itself and I'm helpless there.
> > > So I hope that someone can try running 8Kingdoms with tcl 8.5, and 
> > > write
> > > a fix.
> > >
> > > Note I've also send a request upstream asking for help, I'll report 
> > > back
> > > here as soon as I get response.
> > >
> > > If I don't have it fixed by F-9 then I will have to remove 8Kingdoms
> > > from Fedora's repositories (better then shipping a broken version).
> >
> > Did you have to make any changes to 8Kingdoms to get it to build with
> > Tcl 8.5?  If so please pass along any patches.  I'll try building and
> > running it myself to see if I can track down the problem.
>
> Nope,
>
> No changes, I also did a diff between the buildlogs, no new compiler 
> warnings or anything like that.
>
> I did build it with gcc-4.3, as I was working on it not building with 
> gcc-4.3 too.
>
> I've just committed my latest 8Kingdoms work to cvs, so you can get it 
> from there, including a patch for the crash you reported (tested using 
> tcl 8.4).
>
> Thanks,
>
> Hans

[Hans de Goede]

Marcela Maslanova wrote:
> I had also look at it and ask upstream about help.

Thanks,

I assume you mean tcl upstream? I'm already in contact about this with 
8Kingdoms.

I've had a reply from them that the 8Kingdoms project is sort of 
sleeping (boh authors graduated from university and are working now, we 
all know whats that like), but they would try to take a look at the tcl 
8.5 issue.

Regards,

Hans
> Hans de Goede wrote:
> > Michael Thomas wrote:
> > > Hans de Goede wrote:
> > > > Hi all,
> > > >
> > > > I just tried building 8Kingdoms with the new tcl in rawhide, it builds
> > > > fine, but it does not work.
> > > >
> > > > Any of the actions (moving a unit for example) which normally invoke a
> > > > tcl script underneath now no longer work.
> > > >
> > > > I'm quite good in fixing C bugs but this problem lies either in the
> > > > interfacing with tcl or in the tcl code itself and I'm helpless there.
> > > > So I hope that someone can try running 8Kingdoms with tcl 8.5, and 
> > > > write
> > > > a fix.
> > > >
> > > > Note I've also send a request upstream asking for help, I'll report 
> > > > back
> > > > here as soon as I get response.
> > > >
> > > > If I don't have it fixed by F-9 then I will have to remove 8Kingdoms
> > > > from Fedora's repositories (better then shipping a broken version).
> > >
> > > Did you have to make any changes to 8Kingdoms to get it to build with
> > > Tcl 8.5?  If so please pass along any patches.  I'll try building and
> > > running it myself to see if I can track down the problem.
> >
> > Nope,
> >
> > No changes, I also did a diff between the buildlogs, no new compiler 
> > warnings or anything like that.
> >
> > I did build it with gcc-4.3, as I was working on it not building with 
> > gcc-4.3 too.
> >
> > I've just committed my latest 8Kingdoms work to cvs, so you can get it 
> > from there, including a patch for the crash you reported (tested using 
> > tcl 8.4).
> >
> > Thanks,
> >
> > Hans

[Michael Thomas]

Hans de Goede wrote:
> Michael Thomas wrote:
> > Did you have to make any changes to 8Kingdoms to get it to build with
> > Tcl 8.5?  If so please pass along any patches.  I'll try building and
> > running it myself to see if I can track down the problem.
>
> Nope,
>
> No changes, I also did a diff between the buildlogs, no new compiler
> warnings or anything like that.
>
> I did build it with gcc-4.3, as I was working on it not building with
> gcc-4.3 too.
>
> I've just committed my latest 8Kingdoms work to cvs, so you can get it
> from there, including a patch for the crash you reported (tested using
> tcl 8.4).

It looks like it's a result of the stack checking code in the Tcl
library itself.  Turning off the stack checking code by compiling Tcl
with -DTCL_NO_STACK_CHECK=1 makes the problem in 8Kingdoms go away.

With the stack checking turned on, the Tcl scripts in 8Kingdoms fail
with the error "out of stack space (infinite loop?)".  This error is
generated by the stack checking code in tclBasic.c line 3439.

I don't understand enough of this stack checking stuff to debug it any
further, unfortunately.

--Mike

[Hans de Goede]

Wart wrote:
> Hans de Goede wrote:
> > Michael Thomas wrote:
> > > Did you have to make any changes to 8Kingdoms to get it to build with
> > > Tcl 8.5?  If so please pass along any patches.  I'll try building and
> > > running it myself to see if I can track down the problem.
> >
> > Nope,
> >
> > No changes, I also did a diff between the buildlogs, no new compiler
> > warnings or anything like that.
> >
> > I did build it with gcc-4.3, as I was working on it not building with
> > gcc-4.3 too.
> >
> > I've just committed my latest 8Kingdoms work to cvs, so you can get it
> > from there, including a patch for the crash you reported (tested using
> > tcl 8.4).
>
> It looks like it's a result of the stack checking code in the Tcl
> library itself.  Turning off the stack checking code by compiling Tcl
> with -DTCL_NO_STACK_CHECK=1 makes the problem in 8Kingdoms go away.
>
> With the stack checking turned on, the Tcl scripts in 8Kingdoms fail
> with the error "out of stack space (infinite loop?)".  This error is
> generated by the stack checking code in tclBasic.c line 3439.
>
> I don't understand enough of this stack checking stuff to debug it any
> further, unfortunately.

Well that looks like C-code to me, so I'll dive into it when I find the time.

Regards,

Hans

[Hans de Goede]

Wart wrote:
> Hans de Goede wrote:
> > Michael Thomas wrote:
> > > Did you have to make any changes to 8Kingdoms to get it to build with
> > > Tcl 8.5?  If so please pass along any patches.  I'll try building and
> > > running it myself to see if I can track down the problem.
> >
> > Nope,
> >
> > No changes, I also did a diff between the buildlogs, no new compiler
> > warnings or anything like that.
> >
> > I did build it with gcc-4.3, as I was working on it not building with
> > gcc-4.3 too.
> >
> > I've just committed my latest 8Kingdoms work to cvs, so you can get it
> > from there, including a patch for the crash you reported (tested using
> > tcl 8.4).
>
> It looks like it's a result of the stack checking code in the Tcl
> library itself.  Turning off the stack checking code by compiling Tcl
> with -DTCL_NO_STACK_CHECK=1 makes the problem in 8Kingdoms go away.
>
> With the stack checking turned on, the Tcl scripts in 8Kingdoms fail
> with the error "out of stack space (infinite loop?)".  This error is
> generated by the stack checking code in tclBasic.c line 3439.
>
> I don't understand enough of this stack checking stuff to debug it any
> further, unfortunately.

This:

http://www.nabble.com/--tcl-Bugs-1815573---Stack-space-check-fails-in-Linux-...

Is very interesting in this regard, it may be a good idea to completely disable 
stack checking for now, but I'm a bit quick with judging this atm. I'll take a 
further look as time permits.

Right now I first have to go fill in some Tax Forms :(

Regards,

Hans

[Hans de Goede]

Wart wrote:
> Hans de Goede wrote:
> > Michael Thomas wrote:
> > > Did you have to make any changes to 8Kingdoms to get it to build with
> > > Tcl 8.5?  If so please pass along any patches.  I'll try building and
> > > running it myself to see if I can track down the problem.
> >
> > Nope,
> >
> > No changes, I also did a diff between the buildlogs, no new compiler
> > warnings or anything like that.
> >
> > I did build it with gcc-4.3, as I was working on it not building with
> > gcc-4.3 too.
> >
> > I've just committed my latest 8Kingdoms work to cvs, so you can get it
> > from there, including a patch for the crash you reported (tested using
> > tcl 8.4).
>
> It looks like it's a result of the stack checking code in the Tcl
> library itself.  Turning off the stack checking code by compiling Tcl
> with -DTCL_NO_STACK_CHECK=1 makes the problem in 8Kingdoms go away.
>
> With the stack checking turned on, the Tcl scripts in 8Kingdoms fail
> with the error "out of stack space (infinite loop?)".  This error is
> generated by the stack checking code in tclBasic.c line 3439.
>
> I don't understand enough of this stack checking stuff to debug it any
> further, unfortunately.

And here is another program which is known to brake with stack checking:
http://wiki.tcl.tk/20153

Quoting the author:
"The stack checking is somewhat of a hack IMO anyway"
"Unfortunately as long as the broken stack checking remains in Tcl, this must 
require compiling Tcl with -DTCL_NO_STACK_CHECK."

So maybe it would be best to just disable it on Fedora?

Regards,

Hans

[Hans de Goede]

Wart wrote:
> Hans de Goede wrote:
> > Michael Thomas wrote:
> > > Did you have to make any changes to 8Kingdoms to get it to build with
> > > Tcl 8.5?  If so please pass along any patches.  I'll try building and
> > > running it myself to see if I can track down the problem.
> >
> > Nope,
> >
> > No changes, I also did a diff between the buildlogs, no new compiler
> > warnings or anything like that.
> >
> > I did build it with gcc-4.3, as I was working on it not building with
> > gcc-4.3 too.
> >
> > I've just committed my latest 8Kingdoms work to cvs, so you can get it
> > from there, including a patch for the crash you reported (tested using
> > tcl 8.4).
>
> It looks like it's a result of the stack checking code in the Tcl
> library itself.  Turning off the stack checking code by compiling Tcl
> with -DTCL_NO_STACK_CHECK=1 makes the problem in 8Kingdoms go away.
>
> With the stack checking turned on, the Tcl scripts in 8Kingdoms fail
> with the error "out of stack space (infinite loop?)".  This error is
> generated by the stack checking code in tclBasic.c line 3439.
>
> I don't understand enough of this stack checking stuff to debug it any
> further, unfortunately.

Okay,

I've investigated this a bit more and the conclusion is simple, disable the 
completely broken stack checking please.

The stack checking code works by comparing a pointer returned from malloc with 
that of an address from a variable in the current stack frame, assuming that if 
   if stackaddress < heapaddress (on architectures where the stack grows down) 
that the stack has grown into the heap.

I don't know what dinosaur wrote that code, but with things like address 
randomization, threads (and thus multiple stacks in one address space) etc, the 
assumptions of the stack checking don't hold foot. Also an any protected mode 
os, the os should disallow the stack to grow into already allocated memory and 
the check thus is useless. This will only work in things like dos / embedded 
systems without much of an OS.

Also someone please educate upstream about the dumbness of this code.

Problem solved, just disable the check, _really_

Regards,

Hans

[Marcela Mašláňová]

I've rebuilt tcl without stack checking. Please try rebuild 8Kingdoms.

[Michael Thomas]

Marcela Maslanova wrote:
> I've rebuilt tcl without stack checking. Please try rebuild 8Kingdoms.

I don't think 8Kingdoms needs to be rebuilt to get the fix.  It should 
get the fix automatically since it's dynamically linked with the tcl 
shared library.

--Wart

[Hans de Goede]

Michael Thomas wrote:
> Marcela Maslanova wrote:
> > I've rebuilt tcl without stack checking. Please try rebuild 8Kingdoms.
>
> I don't think 8Kingdoms needs to be rebuilt to get the fix.  It should 
> get the fix automatically since it's dynamically linked with the tcl 
> shared library.

TCL upstream has been in contact with me and I have to rectify my statement 
about the badness of the code, it still doesn't work with 8Kingdoms, but my 
initial assesment was wrong, to be precise I take back:

"The stack checking code works by comparing a pointer returned from malloc with 
that of an address from a variable in the current stack frame, assuming that if 
  if stackaddress < heapaddress (on architectures where the stack grows down) 
that the stack has grown into the heap."

I read the TCL stack checking code as "(localIntPtr) < (iPtr)", where it 
clearly reads: "(localIntPtr) < (iPtr)->stackBound", completely my bad! I also 
take the dinosaur back, I assumed (by my misreading) the code was trying to 
determine wether the stack had grown into the heap, which would only work on 
really old platforms hence the dinosaur reference. I'll be spending some time 
next properly debugging this and trying to find out whats really going amiss.

Here are some first clue's I've added a simple printf to TclInterpReady(), 
which on 8Kingdoms startup prints (lots of times):
stackbound: 0x7fffb4db2244, localint: 0x7fffb57a7564

But then I get:
stackbound: 0x7fffb4db2244, localint: 0x43c04a14
out of stack space (infinite loop?)

Notice how the actualstack is Tera bytes away from the determined stackbound 
(this is a 64 bit system, so yes those are real/correct addresses)

Regards,

Hans

[Michael Thomas]

Hans de Goede wrote:
> Michael Thomas wrote:
> > Marcela Maslanova wrote:
> > > I've rebuilt tcl without stack checking. Please try rebuild 8Kingdoms.
> >
> > I don't think 8Kingdoms needs to be rebuilt to get the fix.  It should 
> > get the fix automatically since it's dynamically linked with the tcl 
> > shared library.
>
> TCL upstream has been in contact with me and I have to rectify my 
> statement about the badness of the code, it still doesn't work with 
> 8Kingdoms, but my initial assesment was wrong, to be precise I take back:
>
> "The stack checking code works by comparing a pointer returned from 
> malloc with that of an address from a variable in the current stack 
> frame, assuming that if  if stackaddress < heapaddress (on architectures 
> where the stack grows down) that the stack has grown into the heap."
>
> I read the TCL stack checking code as "(localIntPtr) < (iPtr)", where it 
> clearly reads: "(localIntPtr) < (iPtr)->stackBound", completely my bad! 
> I also take the dinosaur back, I assumed (by my misreading) the code was 
> trying to determine wether the stack had grown into the heap, which 
> would only work on really old platforms hence the dinosaur reference. 
> I'll be spending some time next properly debugging this and trying to 
> find out whats really going amiss.
>
> Here are some first clue's I've added a simple printf to 
> TclInterpReady(), which on 8Kingdoms startup prints (lots of times):
> stackbound: 0x7fffb4db2244, localint: 0x7fffb57a7564
>
> But then I get:
> stackbound: 0x7fffb4db2244, localint: 0x43c04a14
> out of stack space (infinite loop?)
>
> Notice how the actualstack is Tera bytes away from the determined 
> stackbound (this is a 64 bit system, so yes those are real/correct 
> addresses)

Could this be some problem with interaction with threads?  I noticed the 
'out of stack space' error first occur as soon as 8Kingdoms launched a 
new thread.

--Mike

[Hans de Goede]

Michael Thomas wrote:
> Hans de Goede wrote:
> > Michael Thomas wrote:
> > > Marcela Maslanova wrote:
> > > > I've rebuilt tcl without stack checking. Please try rebuild 8Kingdoms.
> > >
> > > I don't think 8Kingdoms needs to be rebuilt to get the fix.  It 
> > > should get the fix automatically since it's dynamically linked with 
> > > the tcl shared library.
> >
> > TCL upstream has been in contact with me and I have to rectify my 
> > statement about the badness of the code, it still doesn't work with 
> > 8Kingdoms, but my initial assesment was wrong, to be precise I take back:
> >
> > "The stack checking code works by comparing a pointer returned from 
> > malloc with that of an address from a variable in the current stack 
> > frame, assuming that if  if stackaddress < heapaddress (on 
> > architectures where the stack grows down) that the stack has grown 
> > into the heap."
> >
> > I read the TCL stack checking code as "(localIntPtr) < (iPtr)", where 
> > it clearly reads: "(localIntPtr) < (iPtr)->stackBound", completely my 
> > bad! I also take the dinosaur back, I assumed (by my misreading) the 
> > code was trying to determine wether the stack had grown into the heap, 
> > which would only work on really old platforms hence the dinosaur 
> > reference. I'll be spending some time next properly debugging this and 
> > trying to find out whats really going amiss.
> >
> > Here are some first clue's I've added a simple printf to 
> > TclInterpReady(), which on 8Kingdoms startup prints (lots of times):
> > stackbound: 0x7fffb4db2244, localint: 0x7fffb57a7564
> >
> > But then I get:
> > stackbound: 0x7fffb4db2244, localint: 0x43c04a14
> > out of stack space (infinite loop?)
> >
> > Notice how the actualstack is Tera bytes away from the determined 
> > stackbound (this is a 64 bit system, so yes those are real/correct 
> > addresses)
>
> Could this be some problem with interaction with threads?  I noticed the 
> 'out of stack space' error first occur as soon as 8Kingdoms launched a 
> new thread.

Yes, it most likely is, still busy investigating ...

Regards,

Hans

[Michael Thomas]

Michael Thomas wrote:
> Hans de Goede wrote:
> > Hi all,
> >
> > I just tried building 8Kingdoms with the new tcl in rawhide, it builds
> > fine, but it does not work.
> >
> > Any of the actions (moving a unit for example) which normally invoke a
> > tcl script underneath now no longer work.
> >
> > I'm quite good in fixing C bugs but this problem lies either in the
> > interfacing with tcl or in the tcl code itself and I'm helpless there.
> > So I hope that someone can try running 8Kingdoms with tcl 8.5, and write
> > a fix.
> >
> > Note I've also send a request upstream asking for help, I'll report back
> > here as soon as I get response.
> >
> > If I don't have it fixed by F-9 then I will have to remove 8Kingdoms
> > from Fedora's repositories (better then shipping a broken version).
>
> Did you have to make any changes to 8Kingdoms to get it to build with
> Tcl 8.5?  If so please pass along any patches.  I'll try building and
> running it myself to see if I can track down the problem.

I got it built and installed, but X on my Rawhide box is busted at the
moment.  I'll take another look once it's working again.

--Wart