Switching to ncurses

[Bernardo Innocenti]

Are we still planning to completely replace termcap with
ncurses in Fedora?

While I welcome reducing the number of dupe packages, I'm
worried by the impact of this change on the size and
performance of many core programs.

libncurses is much bigger than libtermcap, and also has
oddly large .bss and .data sections:

 bender:/[1/0]# size /lib64/libncurses.so.5 /lib64/libtermcap.so.2
   text    data     bss     dec     hex filename
 319006   56608    3592  379206   5c946 /lib64/libncurses.so.5
  10483     788     112   11383    2c77 /lib64/libtermcap.so.2

this is going to impact very negatively on the RSS of several
critical programs such as bash and python.

I'm also worried that the overall time required spent for a
fork may increase considerably.

But maybe ncurses can easily be fixed to avoid global data and
buffers?

-- 
   // Bernardo Innocenti - Develer S.r.l., R&D dept.
 \X/  http://www.develer.com/

[Ralf Ertzinger]

Hi.

On Fri, 01 Dec 2006 10:13:55 +0100, Bernardo Innocenti wrote:

>  I'm also worried that the overall time required spent for a
>  fork may increase considerably. 
Why would that be? The library is shared, and Linux has been doing
COW for ages.

[Jakub Jelinek]

On Fri, Dec 01, 2006 at 10:13:55AM +0100, Bernardo Innocenti wrote:
>  Are we still planning to completely replace termcap with
>  ncurses in Fedora?
>  
>  While I welcome reducing the number of dupe packages, I'm
>  worried by the impact of this change on the size and
>  performance of many core programs.
>  
>  libncurses is much bigger than libtermcap, and also has
>  oddly large .bss and .data sections:
>  
>  bender:/[1/0]# size /lib64/libncurses.so.5 /lib64/libtermcap.so.2
>    text    data     bss     dec     hex filename
>  319006   56608    3592  379206   5c946 /lib64/libncurses.so.5
>   10483     788     112   11383    2c77 /lib64/libtermcap.so.2
>  
>  this is going to impact very negatively on the RSS of several
>  critical programs such as bash and python. 
The really big writable section (at least in FC6 ncurses; I know
many libraries are much bigger, but bash and similar programs are
really performance critical, slowing down all configure scripts, libtool
etc. by say 50% is not acceptable I guess) is .data.rel.ro:

  [15] .ctors            PROGBITS        000000346904e088 04e088 000010 00  WA  0   0  8
  [16] .dtors            PROGBITS        000000346904e098 04e098 000010 00  WA  0   0  8
  [17] .jcr              PROGBITS        000000346904e0a8 04e0a8 000008 00  WA  0   0  8
  [18] .data.rel.ro      PROGBITS        000000346904e0c0 04e0c0 00c4e8 00  WA  0   0 32
  [19] .dynamic          DYNAMIC         000000346905a5a8 05a5a8 0001a0 10  WA  3   0  8
  [20] .got              PROGBITS        000000346905a748 05a748 0001f0 08  WA  0   0  8
  [21] .got.plt          PROGBITS        000000346905a938 05a938 000940 08  WA  0   0  8
  [22] .data             PROGBITS        000000346905b280 05b280 000b40 00  WA  0   0 32
  [23] .bss              NOBITS          000000346905bdc0 05bdc0 000e08 00  WA  0   0 32

There is a bunch of quite big objects:

   121: 0000003469050fc0  7960 OBJECT  GLOBAL DEFAULT   18 _nc_cap_hash_table
   129: 0000003468e244b0    18 FUNC    GLOBAL DEFAULT   10 bkgd
   176: 000000346904e0e0   360 OBJECT  GLOBAL DEFAULT   18 boolcodes
   189: 0000003468e2aa50    18 FUNC    GLOBAL DEFAULT   10 scrollok
   196: 0000003468e24490    18 FUNC    GLOBAL DEFAULT   10 bkgdset
   244: 0000003468e21c50    18 FUNC    GLOBAL DEFAULT   10 clearok
   249: 0000003469058160  3320 OBJECT  GLOBAL DEFAULT   18 strnames
   261: 0000003469057d60   360 OBJECT  GLOBAL DEFAULT   18 boolfnames
   262: 000000346904e260   320 OBJECT  GLOBAL DEFAULT   18 numcodes
   294: 0000003469057220  2480 OBJECT  GLOBAL DEFAULT   18 _nc_key_names
   311: 000000346904f0a0  7960 OBJECT  GLOBAL DEFAULT   18 _nc_info_hash_table
   341: 0000003469052ee0  1080 OBJECT  GLOBAL DEFAULT   18 _nc_capalias_table
   344: 0000003468e243a0    18 FUNC    GLOBAL DEFAULT   10 echochar
   345: 0000003469058020   320 OBJECT  GLOBAL DEFAULT   18 numfnames
   378: 0000003468e24370    18 FUNC    GLOBAL DEFAULT   10 addch
   403: 0000003469058e60  3320 OBJECT  GLOBAL DEFAULT   18 strfnames
   422: 0000003469057be0   360 OBJECT  GLOBAL DEFAULT   18 boolnames
   440: 0000003468e25af0    18 FUNC    GLOBAL DEFAULT   10 leaveok
   467: 0000003468e23f50    18 FUNC    GLOBAL DEFAULT   10 insch
   488: 000000346904e3a0  3320 OBJECT  GLOBAL DEFAULT   18 strcodes
   515: 0000003468e3ad40    18 FUNC    GLOBAL DEFAULT   10 notimeout
   525: 0000003469057ee0   320 OBJECT  GLOBAL DEFAULT   18 numnames
   547: 0000003469053320   168 OBJECT  GLOBAL DEFAULT   18 _nc_infoalias_table
   551: 0000003468e2d5b0    18 FUNC    GLOBAL DEFAULT   10 syncok

and 3850 relative relocations.

Several of these are used only by code which parses the terminfo sources,
does that really have to be in a library that is used by all programs?
Is there anything but tic (or a couple of other ncurses utilities)
which parses the terminfo sources?

I think if it makes sense to remove termcap and replace it only with ncurses
(I'm not convinced), then the first step needs to be change ncurses so that
it is not a kitchen sink library.

	Jakub

[Arjan van de Ven]

On Fri, 2006-12-01 at 04:45 -0500, Jakub Jelinek wrote:
>  On Fri, Dec 01, 2006 at 10:13:55AM +0100, Bernardo Innocenti wrote:
>  > Are we still planning to completely replace termcap with
>  > ncurses in Fedora?
>  > 
>  > While I welcome reducing the number of dupe packages, I'm
>  > worried by the impact of this change on the size and
>  > performance of many core programs.
>  > 
>  > libncurses is much bigger than libtermcap, and also has
>  > oddly large .bss and .data sections:
>  > 
>  > bender:/[1/0]# size /lib64/libncurses.so.5 /lib64/libtermcap.so.2
>  >   text    data     bss     dec     hex filename
>  > 319006   56608    3592  379206   5c946 /lib64/libncurses.so.5
>  >  10483     788     112   11383    2c77 /lib64/libtermcap.so.2
>  > 
>  > this is going to impact very negatively on the RSS of several
>  > critical programs such as bash and python.
>  
>  The really big writable section 
how much of that would be missing const ?

[Miroslav Lichvar]

On Fri, Dec 01, 2006 at 10:13:55AM +0100, Bernardo Innocenti wrote:
>  While I welcome reducing the number of dupe packages, I'm
>  worried by the impact of this change on the size and
>  performance of many core programs. 
It shouldn't be that bad.

>  libncurses is much bigger than libtermcap, and also has
>  oddly large .bss and .data sections:
>  
>  bender:/[1/0]# size /lib64/libncurses.so.5 /lib64/libtermcap.so.2
>    text    data     bss     dec     hex filename
>  319006   56608    3592  379206   5c946 /lib64/libncurses.so.5
>   10483     788     112   11383    2c77 /lib64/libtermcap.so.2
>  
>  this is going to impact very negatively on the RSS of several
>  critical programs such as bash and python. 
$ ps -o rss,comm
  RSS COMMAND
 1428 bash_termcap
 1480 bash_curses

Not sure it proves anything.

>  I'm also worried that the overall time required spent for a
>  fork may increase considerably. 
Note that termcap has everything in one big file. A simple program
doing just tgetent() and tgetstr() is significantly faster with
ncurses.

$ time ( a=0;while [ $a -lt 1000 ]; do a=$[$a + 1] && ./testtermcap <
/dev/null &> /dev/null; done )

real	0m2.078s
user	0m1.137s
sys	0m0.936s

$ time ( a=0;while [ $a -lt 1000 ]; do a=$[$a + 1] && ./testncurses <
/dev/null &> /dev/null; done )

real	0m1.464s
user	0m0.496s
sys	0m0.962s

-- 
Miroslav Lichvar

[Bernardo Innocenti]

Miroslav Lichvar wrote:

> > libncurses is much bigger than libtermcap, and also has
> > oddly large .bss and .data sections:
> >
> > bender:/[1/0]# size /lib64/libncurses.so.5 /lib64/libtermcap.so.2
> >   text    data     bss     dec     hex filename
> > 319006   56608    3592  379206   5c946 /lib64/libncurses.so.5
> >  10483     788     112   11383    2c77 /lib64/libtermcap.so.2
> >
> > this is going to impact very negatively on the RSS of several
> > critical programs such as bash and python.
>  
>  $ ps -o rss,comm
>    RSS COMMAND
>   1428 bash_termcap
>   1480 bash_curses
>  
>  Not sure it proves anything. 
We see almost exactly the +50KB size difference reported
by "size".

The fact that bash also carries an additional 1400KB of bloat
does not make the ncurses problem more appealing : - )


>  Note that termcap has everything in one big file. A simple program
>  doing just tgetent() and tgetstr() is significantly faster with
>  ncurses. 
I too dislike big binary files, but I would at least hope it's being
loaded with mmap() and/or has at least an index to seek to the correct
terminal as an O(1) operation.


>  $ time ( a=0;while [ $a -lt 1000 ]; do a=$[$a + 1] &&
> ./testtermcap <
>  /dev/null &> /dev/null; done )
>  
>  real	0m2.078s
>  user	0m1.137s
>  sys	0m0.936s
>  
>  $ time ( a=0;while [ $a -lt 1000 ]; do a=$[$a + 1] && ./testncurses <
>  /dev/null &> /dev/null; done )
>  
>  real	0m1.464s
>  user	0m0.496s
>  sys	0m0.962s 
Hmmm!  So termcap also has algoritmic disadvantages that outweight
its smaller size relative to ncurses.

This makes me prefer ncurses over termcap, but fixing its abnormal
data+bss size is still very desiderable.

-- 
   // Bernardo Innocenti - Develer S.r.l., R&D dept.
 \X/  http://www.develer.com/

[Bernardo Innocenti]

Ralf Ertzinger wrote:

>  On Fri, 01 Dec 2006 10:13:55 +0100, Bernardo Innocenti wrote:
>  
> > I'm also worried that the overall time required spent for a
> > fork may increase considerably.
>  
>  Why would that be? The library is shared, and Linux has been doing
>  COW for ages. 
COW only saves the memcpy(), you still have to do some per-page
book keeping at fork time, I suppose (*).

And if those pages are actually being written by the forked process,
it's going to cost a page fault and a memcpy().  That's why I wrote
"overall time spent for a fork".


(*) on second thought, it may actually be a per-vma thing, which would
make the nr. of pages irrelevant.  I shall study the Linux VM one day...

-- 
   // Bernardo Innocenti - Develer S.r.l., R&D dept.
 \X/  http://www.develer.com/

[Jakub Jelinek]

On Fri, Dec 01, 2006 at 11:00:54AM +0100, Arjan van de Ven wrote:
>  On Fri, 2006-12-01 at 04:45 -0500, Jakub Jelinek wrote:
>  > On Fri, Dec 01, 2006 at 10:13:55AM +0100, Bernardo Innocenti wrote:
>  > > Are we still planning to completely replace termcap with
>  > > ncurses in Fedora?
>  > > 
>  > > While I welcome reducing the number of dupe packages, I'm
>  > > worried by the impact of this change on the size and
>  > > performance of many core programs.
>  > > 
>  > > libncurses is much bigger than libtermcap, and also has
>  > > oddly large .bss and .data sections:
>  > > 
>  > > bender:/[1/0]# size /lib64/libncurses.so.5 /lib64/libtermcap.so.2
>  > >   text    data     bss     dec     hex filename
>  > > 319006   56608    3592  379206   5c946 /lib64/libncurses.so.5
>  > >  10483     788     112   11383    2c77 /lib64/libtermcap.so.2
>  > > 
>  > > this is going to impact very negatively on the RSS of several
>  > > critical programs such as bash and python.
>  > 
>  > The really big writable section
>  
>  how much of that would be missing const ? 
The 2 biggest objects (8KB each) are const, pregenerated array
of:
const struct name_table_entry * const _nc_{info,cap}_hash_table[] =
{
        0,
        0,
        0,
        0,
        0,
        0,
        _nc_info_table + 465,
        0,
        0,
        0,
        0,
        0,
        _nc_info_table + 261,
...
};

I think the easiest would be just nuke the terminfo source
parsing routines from libncurses*.so, _nc_read_entry_source nor
_nc_parse_entry aren't even prototyped in any installed ncurses
headers.  It can be IMHO moved to libtic.a which will be linked
into ncurses utilities that need it.

If that is not possible, there are other options, e.g. switching
to a more compact and relocation friendly representation.
In those 2 big (sparse) tables all entries are either
0 or _nc_{info,cap}_table + small const offset >= 0 < 512.
Representing it as array of short int where -1 would mean
the current 0 and 0 .. 511 would mean _nc_info_table + that_value
would be trivial, save 2x6KB of memory and save hundreds of relocations.
Guess other large .data.rel.ro objects could be treated similarly,
again with a < 10 line change in code using the tables and some changes
in the generator.

	Jakub

[Chris Adams]

Once upon a time, Bernardo Innocenti <bernie(a)develer.com&gt; said:
>  Hmmm!  So termcap also has algoritmic disadvantages that outweight
>  its smaller size relative to ncurses. 
Remember though that /bin/sh (aka bash) has many uses that are
non-interactive and don't ever look up a termcap/terminfo entry
(probably many more such uses than interactive).  The lookup slowdown
probably does not outweigh any dynamic linking slowdown or memory
overhead.

-- 
Chris Adams <cmadams(a)hiwaay.net&gt;
Systems and Network Administrator - HiWAAY Internet Services
I don't speak for anybody but myself - that's enough trouble.

[Miroslav Lichvar]

On Fri, Dec 01, 2006 at 05:32:12AM -0500, Jakub Jelinek wrote:
>  > > > libncurses is much bigger than libtermcap, and also
> has
>  > > > oddly large .bss and .data sections:
>  > > > 
>  > > > bender:/[1/0]# size /lib64/libncurses.so.5 /lib64/libtermcap.so.2
>  > > >   text    data     bss     dec     hex filename
>  > > > 319006   56608    3592  379206   5c946 /lib64/libncurses.so.5
>  > > >  10483     788     112   11383    2c77 /lib64/libtermcap.so.2 
[...]
>  
>  I think the easiest would be just nuke the terminfo source
>  parsing routines from libncurses*.so, _nc_read_entry_source nor
>  _nc_parse_entry aren't even prototyped in any installed ncurses
>  headers.  It can be IMHO moved to libtic.a which will be linked
>  into ncurses utilities that need it.
>  
>  If that is not possible, there are other options, e.g. switching
>  to a more compact and relocation friendly representation. 
[...]

Thanks for the suggestions. ncurses-5.5-27.20061209.fc7 should be much
better.

$ size /lib64/libncurses.so.5
   text	   data	    bss	    dec	    hex	filename
 261273	  20504	   3144	 284921	  458f9	/lib64/libncurses.so.5

Linker statistics show that total startup time in dynamic loader for
bash with libncurses is about 10% longer than with libtermcap.

Is it good enough to replace libtermcap?

-- 
Miroslav Lichvar

[Bernardo Innocenti]

Bernardo Innocenti wrote:

>   bender:/[1/0]# size /lib64/libncurses.so.5 /lib64/libtermcap.so.2
>     text    data     bss     dec     hex filename
>   319006   56608    3592  379206   5c946 /lib64/libncurses.so.5
>    10483     788     112   11383    2c77 /lib64/libtermcap.so.2 
It's somewhat better now:

bender:/# size /lib64/libncurses.so.5 /lib64/libtinfo.so.5 /lib64/libtermcap.so.2
   text    data     bss     dec     hex filename
 122768    2296     520  125584   1ea90 /lib64/libncurses.so.5
 110682   14048    2064  126794   1ef4a /lib64/libtinfo.so.5
  10483     788     112   11383    2c77 /lib64/libtermcap.so.2

And now bash links against libtinfo only.

-- 
   // Bernardo Innocenti - Develer S.r.l., R&D dept.
 \X/  http://www.develer.com/