Redo the interaction with the lookaside cache

Mathieu Bridon bochecha at fedoraproject.org
Wed May 6 11:52:56 UTC 2015


This patch series moves the code which talks to the remote lookaside cache,
rewriting parts of it. This makes the code more maintainable, and provides a
nicer, easier to reuse and extend API.

At the same time, some features are added which could allow some downstreams
(fedpkg for example, but also probably centpkg) to remove some of their code
completely, without losing any functionality.

Speaking of downstream users of the pyrpkg API, this patch series does not
break the API, downstreams don't actually require any code change to keep
working as they currently do.

All of this is thoroughly tested, the newly introduced modules are fully
covered by unit tests. As a result, global test coverage goes from 26% to 36%.

In order to make this more maintainable and better tested, the code now does
not call the 'curl' command as a subprocess any more. Instead, everything uses
pycurl, which was already used in pyrpkg and as such no new runtime dependency
is introduced.

Some code paths are marked as deprecated, and will emit messages when they are
executed, so that downstream users of the pyrpkg API can learn about the
deprecations, and port their code. That code could then be removed from pyrpkg
eventually, but this patch series does not do that.

The new APIs are entirely documented. Try this for example:

 >>> from pyrpkg.lookaside import CGILookasideCache
 >>> help(CGILookasideCache)

As a cherry on the cake, the new modules (and their unit tests) are fully
compatible with Python 3, so we'll have less porting work when the time comes.
Of course, the code still is compatible with Python 2.6, so things should
still work on EL6.

Finally, these changes are going to help immensely when we try to move away
from MD5 checksums for the lookaside cache in Fedora.

All in all, this is a lot of changes:

 src/pyrpkg/__init__.py  | 263 +++++++-------------
 src/pyrpkg/cli.py       |   8 +-
 src/pyrpkg/errors.py    |  49 ++++
 src/pyrpkg/lookaside.py | 299 ++++++++++++++++++++++
 src/pyrpkg/sources.py   |  14 +-
 src/pyrpkg/utils.py     |  63 +++++
 test/test_lookaside.py  | 591 ++++++++++++++++++++++++++++++++++++++++++++
 test/test_utils.py      | 159 ++++++++++++
 8 files changed, 1258 insertions(+), 188 deletions(-)

I've made a copr where I've pushed packages built with these changes, and a
couple of packager-friends have been using them for a few days already. They
are heavy-users of pyrpkg (through fedpkg, rhpkg and rdopkg) and all the
problems they could find are fixed in this patch series.



More information about the rel-eng mailing list