koji: add_external_rpm() unique constraint violation

[John Morris]

Here's the post-mortem for an exception in koji-hub (EPEL6 1.8.0-2 RPM).

Earlier, I did something weird, and built a kernel RPM with some pieces
disabled, including the 'kernel-headers' subpackage.  Builds following
the repo regeneration failed, since the incomplete kernel package build
overrode other kernel packages, including subpackages; 'kernel-headers'
is a dependency of build root packages.  The incomplete kernel package
was removed to the trashcan and repos regenerated.

The next build of another kernel package resulted in the exception
pasted below.

*** My best guess at the problem:

Kernel packages have both 'i686' and 'noarch' builds that run
simultaneously.

The i686 build inserted the 'kernel-headers' row into the 'rpminfo'
table, along with a related row in 'buildroot_listing'.

At the same time, the noarch build attempted to do the same thing, but
the i686 build won the race.  The new row broke uniqueness constraints,
(NVRAs must be unique within an external repo), and hence the exception.

If these two builds' add_external_rpm() calls hadn't overlapped, the
second would have created its 'buildroot_listing' row referencing the
'rpminfo' row that the first created.

*** Estimate of problem gravity:

This overlap is probably rarely seen because (1) most packages have
either 'noarch' builds or arch-specific builds, but not both; (2) this
happened to be the first build after the repos were regenerated after
addition of a new package; and (3) even if the first two conditions are
met, it is unlikely that the two builds happened to run
add_external_rpm() at exactly the same time.

The exception stops the build, but no other harm is done.  Rebuilding
will not repeat this exception, since the row is created and the race
condition does not exist.

    John


> 2013-12-20 13:43:56,876 [WARNING] m=host.setBuildRootList u=infra0.zultron.com p=25442 \
>            r=192.168.7.29:47734 koji.xmlrpc: Traceback (most recent call last):
>   File "/usr/share/koji-hub/kojixmlrpc.py", line 212, in _wrap_handler
>     response = handler(environ)
>   File "/usr/share/koji-hub/kojixmlrpc.py", line 255, in handle_rpc
>     return self._dispatch(method, params)
>   File "/usr/share/koji-hub/kojixmlrpc.py", line 292, in _dispatch
>     ret = koji.util.call_with_argcheck(func, params, opts)
>   File "/usr/lib/python2.6/site-packages/koji/util.py", line 139, in call_with_argcheck
>     return func(*args, **kwargs)
>   File "/usr/share/koji-hub/kojihub.py", line 10237, in setBuildRootList
>     return br.setList(rpmlist)
>   File "/usr/share/koji-hub/kojihub.py", line 9371, in setList
>     self._setList(rpmlist,update=False)
>   File "/usr/share/koji-hub/kojihub.py", line 9353, in _setList
>     data = add_external_rpm(an_rpm, location, strict=False)
>   File "/usr/share/koji-hub/kojihub.py", line 4590, in add_external_rpm
>     _dml(q, rpminfo)
>   File "/usr/share/koji-hub/kojihub.py", line 3985, in _dml
>     c.execute(operation, values)
>   File "/usr/lib/python2.6/site-packages/koji/db.py", line 105, in execute
>     ret = self.cursor.execute(operation, parameters)
>   File "/usr/lib64/python2.6/site-packages/pgdb.py", line 174, in execute
>     self.executemany(operation, (params,))
>   File "/usr/lib64/python2.6/site-packages/pgdb.py", line 195, in executemany
>     raise DatabaseError, "error '%s' in '%s'" % ( msg, sql )
> DatabaseError: error 'ERROR:  duplicate key value violates unique constraint "rpminfo_unique_nvra"
> ' in 'INSERT INTO rpminfo (id, build_id, buildroot_id,
>             name, version, release, epoch, arch,
>             external_repo_id,
>             payloadhash, size, buildtime)
>     VALUES (10579, NULL, NULL,
>             'kernel-headers', '3.11.10', '200.fc19', NULL, 'i686',
>             21,
>             '1e6b65e55f166cb40dff21842941be2b', 3165695, 1386020554)
>     '