pygtk2 and its numpy dependency

[Adam Jackson]

pygtk2 implements a function called gtk.gdk.get_pixels_array(), which
returns the pixel contents of a GDK pixbuf as a numpy array.  Fine and
dandy, but this means it links against numpy (7 megs) which is itself
linked against atlas (12 megs).  Kind of a lot for a single function,
especially on a live image.

Especially for a function that's basically unused!  gnome-applet-music
uses it to implement a poor-man's Porter-Duff blend, and that's the only
caller currently packaged in Fedora, at least according to package deps.
I have a patch (attached) that fixes that [1], which means we could
compile our pygtk2 without numpy support and not break anything in
Fedora proper.

However, google codesearch does turn up what look like a few other users
of that function, some of which we may actually want to ship someday.
So we've got options:

a) remove the explicit Requires: numpy from pygtk2, require apps that
actually want this function to Require it themselves

b) fake the numpy data type ABI in pygtk2 itself by cult-and-pasting it
from numpy

c) declare that get_pixels_array() just doesn't work in Fedora
(historically true, back in like FC3)

d) leave things as they are

I lean towards a).  I think b) is icky but doable, since that ABI is
effectively unbreakable anyway (inherited from the older python-numeric
module).  The other two are way lame.

Thoughts?

[1] - Readers are invited to count the wtf's in the code being replaced,
as well as in its callers.  Don't treat it as a drinking game though.

- ajax

[Jon Ciesla]

Adam Jackson wrote:
>  pygtk2 implements a function called gtk.gdk.get_pixels_array(),
> which
>  returns the pixel contents of a GDK pixbuf as a numpy array.  Fine and
>  dandy, but this means it links against numpy (7 megs) which is itself
>  linked against atlas (12 megs).  Kind of a lot for a single function,
>  especially on a live image.
>
>  Especially for a function that's basically unused!  gnome-applet-music
>  uses it to implement a poor-man's Porter-Duff blend, and that's the only
>  caller currently packaged in Fedora, at least according to package deps.
>  I have a patch (attached) that fixes that [1], which means we could
>  compile our pygtk2 without numpy support and not break anything in
>  Fedora proper.
>
>  However, google codesearch does turn up what look like a few other users
>  of that function, some of which we may actually want to ship someday.
>  So we've got options:
>
>  a) remove the explicit Requires: numpy from pygtk2, require apps that
>  actually want this function to Require it themselves
>
>  b) fake the numpy data type ABI in pygtk2 itself by cult-and-pasting it
>  from numpy
>
>  c) declare that get_pixels_array() just doesn't work in Fedora
>  (historically true, back in like FC3)
>
>  d) leave things as they are
>
>  I lean towards a).  I think b) is icky but doable, since that ABI is
>  effectively unbreakable anyway (inherited from the older python-numeric
>  module).  The other two are way lame.
>
>  Thoughts?
>
>  [1] - Readers are invited to count the wtf's in the code being replaced,
>  as well as in its callers.  Don't treat it as a drinking game though.
>
>  - ajax
>     
Since pygtk2 does actually use numpy, isn't d) the best (albeit most 
annoying) option?

Full disclosure:  Numpy maintainer.

-- 
in your fear, seek only peace
in your fear, seek only love

-d. bowie

[Adam Jackson]

On Mon, 2009-08-10 at 14:40 -0500, Jon Ciesla wrote:
>  Adam Jackson wrote:
>  > a) remove the explicit Requires: numpy from pygtk2, require apps that
>  > actually want this function to Require it themselves
>  >
>  > b) fake the numpy data type ABI in pygtk2 itself by cult-and-pasting it
>  > from numpy
>  >
>  > c) declare that get_pixels_array() just doesn't work in Fedora
>  > (historically true, back in like FC3)
>  >
>  > d) leave things as they are
>  >
>  Since pygtk2 does actually use numpy, isn't d) the best (albeit most 
>  annoying) option? 
Internally?  Or just to implement that one entrypoint?  I believe it to
be the latter.

- ajax

[Jon Ciesla]

Adam Jackson wrote:
>  On Mon, 2009-08-10 at 14:40 -0500, Jon Ciesla wrote:
>    
> > Adam Jackson wrote:
> >     
> >> a) remove the explicit Requires: numpy from pygtk2, require apps that
> >> actually want this function to Require it themselves
> >>
> >> b) fake the numpy data type ABI in pygtk2 itself by cult-and-pasting it
> >> from numpy
> >>
> >> c) declare that get_pixels_array() just doesn't work in Fedora
> >> (historically true, back in like FC3)
> >>
> >> d) leave things as they are
> >>
> >>       
> > Since pygtk2 does actually use numpy, isn't d) the best (albeit most 
> > annoying) option?
> >     
>
>  Internally?  Or just to implement that one entrypoint?  I believe it to
>  be the latter.
>
>  - ajax
>     
I'm not sure, but what practical difference would that make?

-- 
in your fear, seek only peace
in your fear, seek only love

-d. bowie

[Adam Jackson]

On Tue, 2009-08-11 at 07:12 -0500, Jon Ciesla wrote:
>  Adam Jackson wrote:
>  > On Mon, 2009-08-10 at 14:40 -0500, Jon Ciesla wrote:
>  >> Since pygtk2 does actually use numpy, isn't d) the best (albeit most 
>  >> annoying) option?
>  >
>  > Internally?  Or just to implement that one entrypoint?  I believe it to
>  > be the latter. 
Having double-checked: it's just to implement get_pixels_array().

>  I'm not sure, but what practical difference would that make?
Well, it's a soft dep...

If built with numpy support, get_pixels_array() starts off with:

    if (!have_numpy())
        return NULL;

The important bit of have_numpy() looks like:

    static int import_done = 0;
    if (!import_done()) {
        import_done = 1;
        import_array1(1);
        if (PyErr_Occurred()) {
            /* throw */
        }
    }

import_array1() is a small wrapper around _import_array(), which starts
off with:

    PyObject *numpy = PyImport_ImportModule("numpy.core.multiarray");
    if (numpy == NULL) return -1;

And it turns out this is all static inlines that get built straight into
pygtk2 itself.  In other words, if this bit of pygtk2 were written in
python, it'd look like:

class gdk:
    def get_pixels_array(self):
        import numpy.core.multiarray
        # do a bunch of stuff

and if that python module weren't available it'd just throw an
exception.

So: python apps that call get_pixels_array() can Requires: numpy
themselves, and then that entrypoint will work.  Python apps that
_don't_, need not, and then numpy and its deps go missing, but it
doesn't matter because you never call get_pixels_array() so the
exception never happens.

So I think my b) suggestion (of replicating the ABI in pygtk2) is
actually redundant, it's what's already happening.  pygtk2 already knows
the object layout of numpy arrays thanks to #includes of doom, it just
doesn't try to create them unless the rest of the numpy exists.

The question is only whether to keep the 'Requires: numpy' in pygtk2 or
to push it out to apps that use get_pixels_array().  And I think the
latter sounds just fine to me.

- ajax

[Jon Ciesla]

Adam Jackson wrote:
>  On Tue, 2009-08-11 at 07:12 -0500, Jon Ciesla wrote:
>    
> > Adam Jackson wrote:
> >     
> >> On Mon, 2009-08-10 at 14:40 -0500, Jon Ciesla wrote:
> >>       
> >>> Since pygtk2 does actually use numpy, isn't d) the best (albeit most 
> >>> annoying) option?
> >>>         
> >> Internally?  Or just to implement that one entrypoint?  I believe it to
> >> be the latter.
> >>       
>
>  Having double-checked: it's just to implement get_pixels_array().
>
>    
> > I'm not sure, but what practical difference would that make?
> >     
>
>  Well, it's a soft dep...
>
>  If built with numpy support, get_pixels_array() starts off with:
>
>      if (!have_numpy())
>          return NULL;
>
>  The important bit of have_numpy() looks like:
>
>      static int import_done = 0;
>      if (!import_done()) {
>          import_done = 1;
>          import_array1(1);
>          if (PyErr_Occurred()) {
>              /* throw */
>          }
>      }
>
>  import_array1() is a small wrapper around _import_array(), which starts
>  off with:
>
>      PyObject *numpy = PyImport_ImportModule("numpy.core.multiarray");
>      if (numpy == NULL) return -1;
>
>  And it turns out this is all static inlines that get built straight into
>  pygtk2 itself.  In other words, if this bit of pygtk2 were written in
>  python, it'd look like:
>
>  class gdk:
>      def get_pixels_array(self):
>          import numpy.core.multiarray
>          # do a bunch of stuff
>
>  and if that python module weren't available it'd just throw an
>  exception.
>
>  So: python apps that call get_pixels_array() can Requires: numpy
>  themselves, and then that entrypoint will work.  Python apps that
>  _don't_, need not, and then numpy and its deps go missing, but it
>  doesn't matter because you never call get_pixels_array() so the
>  exception never happens.
>
>  So I think my b) suggestion (of replicating the ABI in pygtk2) is
>  actually redundant, it's what's already happening.  pygtk2 already knows
>  the object layout of numpy arrays thanks to #includes of doom, it just
>  doesn't try to create them unless the rest of the numpy exists.
>
>  The question is only whether to keep the 'Requires: numpy' in pygtk2 or
>  to push it out to apps that use get_pixels_array().  And I think the
>  latter sounds just fine to me.
>
>  - ajax
>     
That's fine with me, assuming there's a way to determine that list.

-- 
in your fear, seek only peace
in your fear, seek only love

-d. bowie

[Adam Jackson]

On Tue, 2009-08-11 at 10:05 -0500, Jon Ciesla wrote:
>  Adam Jackson wrote:
>  > The question is only whether to keep the 'Requires: numpy' in pygtk2 or
>  > to push it out to apps that use get_pixels_array().  And I think the
>  > latter sounds just fine to me.
>  >
>  That's fine with me, assuming there's a way to determine that list. 
To a first approximation:

find . -name \*.py | xargs grep '\<get_pixels_array\>'

It appears to be a remarkably uncommon function name.  This won't catch
anyone calling it from C code instead, but anyone doing that is a
cretin.

- ajax

[Pádraig Brady]

Jon Ciesla wrote:
>  Since pygtk2 does actually use numpy, isn't d) the best (albeit
> most
>  annoying) option?
>  
>  Full disclosure:  Numpy maintainer. 
I vote for a) (removing the dependency).
One loses the point of packages unless one actively
minimizes dependencies between them.

cheers,
Pádraig.

[Peter Robinson]

Hi,

>  pygtk2 implements a function called gtk.gdk.get_pixels_array(),
> which
>  returns the pixel contents of a GDK pixbuf as a numpy array.  Fine and
>  dandy, but this means it links against numpy (7 megs) which is itself
>  linked against atlas (12 megs).  Kind of a lot for a single function,
>  especially on a live image.
>
>  Especially for a function that's basically unused!  gnome-applet-music
>  uses it to implement a poor-man's Porter-Duff blend, and that's the only
>  caller currently packaged in Fedora, at least according to package deps.
>  I have a patch (attached) that fixes that [1], which means we could
>  compile our pygtk2 without numpy support and not break anything in
>  Fedora proper.
>
>  However, google codesearch does turn up what look like a few other users
>  of that function, some of which we may actually want to ship someday.
>  So we've got options:
>
>  a) remove the explicit Requires: numpy from pygtk2, require apps that
>  actually want this function to Require it themselves
>
>  b) fake the numpy data type ABI in pygtk2 itself by cult-and-pasting it
>  from numpy
>
>  c) declare that get_pixels_array() just doesn't work in Fedora
>  (historically true, back in like FC3)
>
>  d) leave things as they are
>
>  I lean towards a).  I think b) is icky but doable, since that ABI is
>  effectively unbreakable anyway (inherited from the older python-numeric
>  module).  The other two are way lame.
>
>  Thoughts? 
I know sugar use numpy and pygtk. They had forked pygtk packages with
patches so that they could use the functionality in older releases.
I'm not sure whether including adding numpy to the requires would work
or not as I don't know the codebase.

Peter

>  [1] - Readers are invited to count the wtf's in the code being
> replaced,
>  as well as in its callers.  Don't treat it as a drinking game though.

[Jonathan Underwood]

2009/8/10 Adam Jackson <ajax(a)redhat.com&gt;:
>  a) remove the explicit Requires: numpy from pygtk2, require apps
> that
>  actually want this function to Require it themselves
>
>  b) fake the numpy data type ABI in pygtk2 itself by cult-and-pasting it
>  from numpy
>
>  c) declare that get_pixels_array() just doesn't work in Fedora
>  (historically true, back in like FC3)
>
>  d) leave things as they are
>
>  I lean towards a).  I think b) is icky but doable, since that ABI is
>  effectively unbreakable anyway (inherited from the older python-numeric
>  module).  The other two are way lame.
>
>  Thoughts? 
A possible (e) occured to me:

e) Place the numpy.linalg stuff in a subpackage of numpy, which
requires atlas and friends. Keep only the core numpy functionality in
numpy itself, with other subpackages for the numpy.foo stuff. Then
pygtk2 could just require the basic numpy package which would be
smaller in size and have a lot less deps.

I've no idea how hard this would be, and what might break, though. So
it may be a stupid idea.

Jonathan.

[Jon Ciesla]

Jonathan Underwood wrote:
>  2009/8/10 Adam Jackson <ajax(a)redhat.com&gt;:
>    
> > a) remove the explicit Requires: numpy from pygtk2, require apps that
> > actually want this function to Require it themselves
> >
> > b) fake the numpy data type ABI in pygtk2 itself by cult-and-pasting it
> > from numpy
> >
> > c) declare that get_pixels_array() just doesn't work in Fedora
> > (historically true, back in like FC3)
> >
> > d) leave things as they are
> >
> > I lean towards a).  I think b) is icky but doable, since that ABI is
> > effectively unbreakable anyway (inherited from the older python-numeric
> > module).  The other two are way lame.
> >
> > Thoughts?
> >     
>
>  A possible (e) occured to me:
>
>  e) Place the numpy.linalg stuff in a subpackage of numpy, which
>  requires atlas and friends. Keep only the core numpy functionality in
>  numpy itself, with other subpackages for the numpy.foo stuff. Then
>  pygtk2 could just require the basic numpy package which would be
>  smaller in size and have a lot less deps.
>
>  I've no idea how hard this would be, and what might break, though. So
>  it may be a stupid idea.
>
>  Jonathan.
>
>     
We actually tried some of this when f2py was split.  Nightmare.  Best 
left alone IMHO.

-- 
in your fear, seek only peace
in your fear, seek only love

-d. bowie

[Toshio Kuratomi]

On 08/10/2009 12:36 PM, Adam Jackson wrote:
>  pygtk2 implements a function called gtk.gdk.get_pixels_array(),
> which
>  returns the pixel contents of a GDK pixbuf as a numpy array.  Fine and
>  dandy, but this means it links against numpy (7 megs) which is itself
>  linked against atlas (12 megs).  Kind of a lot for a single function,
>  especially on a live image.
>  
>  Especially for a function that's basically unused!  gnome-applet-music
>  uses it to implement a poor-man's Porter-Duff blend, and that's the only
>  caller currently packaged in Fedora, at least according to package deps.
>  I have a patch (attached) that fixes that [1], which means we could
>  compile our pygtk2 without numpy support and not break anything in
>  Fedora proper.
>   
Package deps are of minimal use in detecting this.  We have to detect
when this function is used by an application which isn't part of package
deps.  gnome-games and specto are two packages that make use of this
function.

Of note, both Debian and Ubuntu have had this problem as well.  Ubuntu
chose to go the route that you're suggesting, remove the Requires[1]_.
Debian tried that and chose to revert the workaround, going back to a
strict requirement on numpy[2]_.  The Ubuntu bug report also mentions
that porting to the new buffer[3]_ interface in python2.6 and above may
be a long term solution.

Has someone filed this with upstream pygtk2 yet?

.. _[1]: https://bugs.launchpad.net/ubuntu/+source/python-numpy/+bug/309215
.. _[2]: http://osdir.com/ml/debian-bugs-closed/2009-04/msg01088.html
.. _[3]: http://www.python.org/dev/peps/pep-3118/
         http://numpy.scipy.org/#buffer_protocol
         http://docs.scipy.org/doc/numpy/reference/arrays.interface.html

-Toshio

-Toshio

[Adam Jackson]

On Tue, 2009-08-11 at 12:05 -0700, Toshio Kuratomi wrote:
>  On 08/10/2009 12:36 PM, Adam Jackson wrote:
>  > pygtk2 implements a function called gtk.gdk.get_pixels_array(), which
>  > returns the pixel contents of a GDK pixbuf as a numpy array.  Fine and
>  > dandy, but this means it links against numpy (7 megs) which is itself
>  > linked against atlas (12 megs).  Kind of a lot for a single function,
>  > especially on a live image.
>  > 
>  > Especially for a function that's basically unused!  gnome-applet-music
>  > uses it to implement a poor-man's Porter-Duff blend, and that's the only
>  > caller currently packaged in Fedora, at least according to package deps.
>  > I have a patch (attached) that fixes that [1], which means we could
>  > compile our pygtk2 without numpy support and not break anything in
>  > Fedora proper.
>  > 
>  Package deps are of minimal use in detecting this.  We have to detect
>  when this function is used by an application which isn't part of package
>  deps.  gnome-games and specto are two packages that make use of this
>  function. 
I did check this by installing all the packages in the distro with a
pygtk2 dep and then grepping their installed .py files.  Not sure how
gnome-games escaped...

- ajax