Mash and rpmdb

[Panu Matilainen]

On 11/15/2012 04:30 PM, Thomas wrote:
> When I run  /usr/lib/rpm/rpmdb_stat -CA :
>
> Default locking region information:
> 24857 Last allocated locker ID
> 0x7fffffff Current maximum unused locker ID
> 5 Number of lock modes
> 1000 Maximum number of locks possible
> 1000 Maximum number of lockers possible
> 1000 Maximum number of lock objects possible
> 160 Number of lock object partitions
> 0 Number of current locks
> 20 Maximum number of locks at any one time
> 5 Maximum number of locks in any one bucket
> 0 Maximum number of locks stolen by for an empty partition
> 0 Maximum number of locks stolen for any one partition
> 999 Number of current lockers
> 1000 Maximum number of lockers at any one time
> 0 Number of current lock objects
> 5 Maximum number of lock objects at any one time
> 1 Maximum number of lock objects in any one bucket
> 0 Maximum number of objects stolen by for an empty partition
> 0 Maximum number of objects stolen for any one partition
> 90021 Total number of locks requested
> 90021 Total number of locks released
> 0 Total number of locks upgraded
> 13509 Total number of locks downgraded
> 18 Lock requests not available due to conflicts, for which we waited
> 0 Lock requests not available due to conflicts, for which we did not wait
> 0 Number of deadlocks
> 0 Lock timeout value
> 0 Number of locks that have timed out
> 0 Transaction timeout value
> 0 Number of transactions that have timed out
> 752KB The size of the lock region
> 46 The number of partition locks that required waiting (0%)
> 20 The maximum number of times any partition lock was waited for (0%)
> 0 The number of object queue operations that required waiting (0%)
> 65 The number of locker allocations that required waiting (0%)
> 0 The number of region locks that required waiting (0%)
> 1 Maximum hash bucket length
> =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
>
> It seems the Number of current lockers is the issue.
> So the db may no be corrupted but just out of lockers.

Yup, running out of lockers is not corruption, its just out of 
resources. It does prevent further rpmdb opens unless dealt with though...

> Any idea on why ?

Well, something is leaving open rpmdb / iterator handles around. To find 
what that something is, 'fuser -uv /var/lib/rpm/*' should give clues.

One possibility is something (maybe mash) hitting this: 
http://rpm.org/ticket/820. While the dangling iterators issue is 
entirely avoidable with careful programming, older rpm versions (such as 
the one in RHEL 6) isn't doing a very good job of managing its 
resources. IIRC yum's API has or at least had some corners where it was 
all too easy to trigger this issue which arguably is a bug in rpm.

Like Bill noted, an easy workaround should be running as non-root, as 
unprivileged rpmdb accesses uses a private "locker room" which is wiped 
out from existance after use so stale locks from unclosed / dangling 
iterators dont get to pile up.

	- Panu -