F27 System Wide Change: RPM 4.14

[Jaroslav Reznik]

= System Wide Change: RPM 4.14 =
https://fedoraproject.org/wiki/Changes/RPM-4.14

Change owner(s):
* Igor Gnatenko <ignatenkobrain(a)fedoraproject.org&gt;
* Florian Festi <ffesti(a)redhat.com&gt;
* Panu Matilainen <pmatilai(a)redhat.com&gt;

Update RPM to the upcoming 4.14 release.

== Detailed Description ==
RPM 4.14 contains several improvements that needs to get released and
integrated in Fedora:
* Major macro engine bug fix + sanity work:
** Macro scope simplification + enforcing
** Macro arguments expanded
** Nested lua macro scoping fixes
** Improved error reporting
* Major header/package/signature rewrite:
** Unified code path for all header read/import
** Major hardening work on header parsing
** Unified code path for all header/package signature checking
** Signature checking before header imports
** Support for multiple signatures per package
** Support for configurable signature policies
* Major debuginfo rewrite (covered by two other changes and already
applied in F27)
* Signal handling rewrite:
** Custom signal handlers while rpmdb open
** Signals blocked throughout write transactions
* SSD conservation mode
* Improved support for reproducible builds
* RPMCALLBACK_ELEM_PROGRESS now carries index of header
* Support for OpenSSL as a one of crypto libraries used for
digests/signatures (already part of F27)
* Support for rich dependencies coming out from dependency generators
* %include can contain paths with whitespaces
* Dependency generator for pkg-config files doesn't check dependencies
in .pc recursively, but rather print top-level ones (if pkgconf is
used)
* Header digests use SHA256 by default
* Improvements in Python dependency generator
* Improvements and stabilization of "ndb"
* Support for "with" rich-operator:
** Specifying version range dependencies
** Specifying packages which provide special ability

== Scope ==
* Proposal owners:
Rebase RPM

* Other developers:
Test new release, report issues and bugs, fix bugs in packaging (if it
is not bug in RPM, should be detected during Mass Rebuild)

* Release engineering: [1] (a check of an impact with Release
Engineering is needed)

* List of deliverables:
Change affects whole distribution rather than deliverables

* Policies and guidelines:
FPC should look (and possibly approve) "with" rich dependency in
Packaging Guidelines

* Trademark approval: N/A (not needed for this Change)

[1] https://pagure.io/releng/issue/6875

Thanks,
Jaroslav

Using ndb in RPM (was: Re: F27 System Wide Change: RPM 4.14)

[Richard W.M. Jones]

On Fri, Jul 07, 2017 at 04:15:53PM +0200, Jaroslav Reznik wrote:
>  = System Wide Change: RPM 4.14 =
>  https://fedoraproject.org/wiki/Changes/RPM-4.14 
The web page has:

   Improvements and stabilization of "ndb" (New RPM DB Format database
   format)
...
   Changing database from bdb to ndb requires patching of some programs
   who read/expect /var/lib/rpm/Packages and other files directly
   without using librpm

which were not included in this email.

The problem is that "NDB" is a custom homebrew database invented in
the RPM codebase.  I agree that because of licensing problems we need
to move away from Berkeley DB, but why not switch to the obvious,
bulletproof choice - Sqlite?

Look at the all new nbd code here:

  https://github.com/rpm-software-management/rpm/tree/master/lib/backend/ndb

Inventing a new database including a new disk format and all new code
is bound to cause problems with database corruption, incorrect
database synchronization, incorrect handling of parallel queries, and
a rich source of other bugs for a long time.

Not to mention ‘ndb requires patching of some programs who read/expect
/var/lib/rpm/Packages and other files directly without using librpm’ -
this would still require changes, but they are much smaller if you're
using sqlite.

It was a bad upstream decision, but it's not too late to avoid it by
sticking with BDB for now and using sqlite in the long run.

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-top is 'top' for virtual machines.  Tiny program with many
powerful monitoring features, net stats, disk stats, logging, etc.
http://people.redhat.com/~rjones/virt-top

Using ndb in RPM (was: Re: F27 System Wide Change: RPM 4.14)

[Jos Vos]

Hi,

On Mon, Jul 10, 2017 at 09:14:07AM +0100, Richard W.M. Jones wrote:

>  The web page has:
>  
>     Improvements and stabilization of "ndb" (New RPM DB Format database
>     format)
>
>  [...]
>  
>  The problem is that "NDB" is a custom homebrew database invented in
>  the RPM codebase.  I agree that because of licensing problems we need
>  to move away from Berkeley DB, but why not switch to the obvious,
>  bulletproof choice - Sqlite? 
SQLite is is totally different piece of cake (RDBMS vs. key/value).

Why not use LMDB?  It's the replacement for BerkeleyDB in OpenLDAP.

http://www.lmdb.tech/doc/

It's extremely fast...

Cheers,

-- 
--    Jos Vos <jos(a)xos.nl&gt;
--    X/OS Experts in Open Systems BV   |   Office: [phone]
--    Amsterdam, The Netherlands        |   Mobile: [phone]

Using ndb in RPM (was: Re: F27 System Wide Change: RPM 4.14)

[Neal Gompa]

On Mon, Jul 10, 2017 at 4:14 AM, Richard W.M. Jones <rjones(a)redhat.com&gt; wrote:
>  On Fri, Jul 07, 2017 at 04:15:53PM +0200, Jaroslav Reznik wrote:
> > = System Wide Change: RPM 4.14 =
> > https://fedoraproject.org/wiki/Changes/RPM-4.14
>
>  The web page has:
>
>     Improvements and stabilization of "ndb" (New RPM DB Format database
>     format)
>  ...
>     Changing database from bdb to ndb requires patching of some programs
>     who read/expect /var/lib/rpm/Packages and other files directly
>     without using librpm
>
>  which were not included in this email.
>
>  The problem is that "NDB" is a custom homebrew database invented in
>  the RPM codebase.  I agree that because of licensing problems we need
>  to move away from Berkeley DB, but why not switch to the obvious,
>  bulletproof choice - Sqlite?
>
>  Look at the all new nbd code here:
>
>    https://github.com/rpm-software-management/rpm/tree/master/lib/backend/ndb
>
>  Inventing a new database including a new disk format and all new code
>  is bound to cause problems with database corruption, incorrect
>  database synchronization, incorrect handling of parallel queries, and
>  a rich source of other bugs for a long time.
>
>  Not to mention ‘ndb requires patching of some programs who read/expect
>  /var/lib/rpm/Packages and other files directly without using librpm’ -
>  this would still require changes, but they are much smaller if you're
>  using sqlite.
>
>  It was a bad upstream decision, but it's not too late to avoid it by
>  sticking with BDB for now and using sqlite in the long run.
>  
A long time ago, we did actually have a SQLite backend. It was removed
in RPM 4.9.0[1][2]. I'm not sure *why* it was removed.

[1]: http://rpm.org/wiki/Releases/4.9.0
[2]:
https://github.com/rpm-software-management/rpm/commit/e2c217b4b76118e6dab...

On Mon, Jul 10, 2017 at 5:28 AM, Jos Vos <jos(a)xos.nl&gt; wrote:
>  Hi,
>
>  On Mon, Jul 10, 2017 at 09:14:07AM +0100, Richard W.M. Jones wrote:
>
> > The web page has:
> >
> >    Improvements and stabilization of "ndb" (New RPM DB Format database
> >    format)
> >
> > [...]
> >
> > The problem is that "NDB" is a custom homebrew database invented in
> > the RPM codebase.  I agree that because of licensing problems we need
> > to move away from Berkeley DB, but why not switch to the obvious,
> > bulletproof choice - Sqlite?
>
>  SQLite is is totally different piece of cake (RDBMS vs. key/value).
>
>  Why not use LMDB?  It's the replacement for BerkeleyDB in OpenLDAP.
>
>  http://www.lmdb.tech/doc/
>
>  It's extremely fast...
>  
LMDB has been brought up in the past[3] as well as recently[4].

[3]: https://bugzilla.redhat.com/show_bug.cgi?id=1086784
[4]: https://github.com/rpm-software-management/rpm/issues/128


-- 
真実はいつも一つ！/ Always, there's only one truth!

Using ndb in RPM (was: Re: F27 System Wide Change: RPM 4.14)

[Richard W.M. Jones]

On Mon, Jul 10, 2017 at 11:28:13AM +0200, Jos Vos wrote:
>  Hi,
>  
>  On Mon, Jul 10, 2017 at 09:14:07AM +0100, Richard W.M. Jones wrote:
>  
>  > The web page has:
>  > 
>  >    Improvements and stabilization of "ndb" (New RPM DB Format database
>  >    format)
>  >
>  > [...]
>  > 
>  > The problem is that "NDB" is a custom homebrew database invented in
>  > the RPM codebase.  I agree that because of licensing problems we need
>  > to move away from Berkeley DB, but why not switch to the obvious,
>  > bulletproof choice - Sqlite?
>  
>  SQLite is is totally different piece of cake (RDBMS vs. key/value).
>  
>  Why not use LMDB?  It's the replacement for BerkeleyDB in OpenLDAP.
>  
>  http://www.lmdb.tech/doc/
>  
>  It's extremely fast... 
TBH I'm not fussed about what specific technology is used as long as
it's not some homebrew thing, is battle-tested, and is reasonably
widely available with as few external dependencies as possible.

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
libguestfs lets you edit virtual machines.  Supports shell scripting,
bindings from many languages.  http://libguestfs.org

Using ndb in RPM

[Alek Paunov]

On 2017-07-10 12:28, Jos Vos wrote:
>  Hi,
>  
>  On Mon, Jul 10, 2017 at 09:14:07AM +0100, Richard W.M. Jones wrote:
>  
> > The web page has:
> >
> >     Improvements and stabilization of "ndb" (New RPM DB Format database
> >     format)
> >
> > [...]
> >
> > The problem is that "NDB" is a custom homebrew database invented in
> > the RPM codebase.  I agree that because of licensing problems we need
> > to move away from Berkeley DB, but why not switch to the obvious,
> > bulletproof choice - Sqlite? 
Totally agree with Rich. If the problem with sqlite is the lack of SQL 
developers, I am highly motivated to work on all SQL queries needed for 
such backend, leveraging the whole bunch of fantastic features in recent 
sqlite.

>  
>  SQLite is is totally different piece of cake (RDBMS vs. key/value). 
If you say:
create table col1(key blob primary key, value blob) without rowid;/

you physically get no more/no less simple key/value store (given 30 more 
lines of trivial code for implementing your favorite key/value lib 
operations).

But you are right - sqlite is different piece of cake: sqlite has 
powerful query language. Sqlite based backend gives the future advantage 
that big chunk of the rpm code could be elegantly and declarative 
offloaded to the query engine (with efficiency benefits as bonus).

Kind regards,
Alek

Using ndb in RPM

[Jos Vos]

On Mon, Jul 10, 2017 at 02:44:31PM +0300, Alek Paunov wrote:

>  If you say:
>  create table col1(key blob primary key, value blob) without rowid;/
>  
>  you physically get no more/no less simple key/value store (given 30 more 
>  lines of trivial code for implementing your favorite key/value lib 
>  operations).
>  
>  But you are right - sqlite is different piece of cake: sqlite has 
>  powerful query language. Sqlite based backend gives the future advantage 
>  that big chunk of the rpm code could be elegantly and declarative 
>  offloaded to the query engine (with efficiency benefits as bonus). 
I did performance tests to compare SQLite with LMDB for my own case here
(using key/value-like things) and the performance difference is *huge*.

-- 
--    Jos Vos <jos(a)xos.nl&gt;
--    X/OS Experts in Open Systems BV   |   Office: [phone]
--    Amsterdam, The Netherlands        |   Mobile: [phone]

Using ndb in RPM

[Pierre-Yves Chibon]

On Mon, Jul 10, 2017 at 01:51:57PM +0200, Jos Vos wrote:
>  On Mon, Jul 10, 2017 at 02:44:31PM +0300, Alek Paunov wrote:
>  
>  > If you say:
>  > create table col1(key blob primary key, value blob) without rowid;/
>  > 
>  > you physically get no more/no less simple key/value store (given 30 more 
>  > lines of trivial code for implementing your favorite key/value lib 
>  > operations).
>  > 
>  > But you are right - sqlite is different piece of cake: sqlite has 
>  > powerful query language. Sqlite based backend gives the future advantage 
>  > that big chunk of the rpm code could be elegantly and declarative 
>  > offloaded to the query engine (with efficiency benefits as bonus).
>  
>  I did performance tests to compare SQLite with LMDB for my own case here
>  (using key/value-like things) and the performance difference is *huge*. 
How which side?

Did you blog about your test and findings?


Pierre

Using ndb in RPM

[Pierre-Yves Chibon]

On Mon, Jul 10, 2017 at 02:27:38PM +0200, Pierre-Yves Chibon wrote:
>  On Mon, Jul 10, 2017 at 01:51:57PM +0200, Jos Vos wrote:
>  > On Mon, Jul 10, 2017 at 02:44:31PM +0300, Alek Paunov wrote:
>  > 
>  > > If you say:
>  > > create table col1(key blob primary key, value blob) without rowid;/
>  > > 
>  > > you physically get no more/no less simple key/value store (given 30 more 
>  > > lines of trivial code for implementing your favorite key/value lib 
>  > > operations).
>  > > 
>  > > But you are right - sqlite is different piece of cake: sqlite has 
>  > > powerful query language. Sqlite based backend gives the future advantage 
>  > > that big chunk of the rpm code could be elegantly and declarative 
>  > > offloaded to the query engine (with efficiency benefits as bonus).
>  > 
>  > I did performance tests to compare SQLite with LMDB for my own case here
>  > (using key/value-like things) and the performance difference is *huge*.
>  
>  How which side? 
Oups, that was meant to be: On which side?

> Did you blog about your test and findings?
> 
> 
> Pierre
> _______________________________________________
> devel mailing list -- devel(a)lists.fedoraproject.org
> To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org

Using ndb in RPM

[Jos Vos]

On Mon, Jul 10, 2017 at 02:30:59PM +0200, Pierre-Yves Chibon wrote:

>  Oups, that was meant to be: On which side? 
LDBM was many factors faster than SQLite.  No concrete figures,
it was a year ago or so when we decided to go for LMDB (still
a product in development...).  I had attended some talk from
Howard Chu (the author) about LMDB and knew quite soon it was
the way to go.

-- 
--    Jos Vos <jos(a)xos.nl&gt;
--    X/OS Experts in Open Systems BV   |   Office: [phone]
--    Amsterdam, The Netherlands        |   Mobile: [phone]

Using ndb in RPM

[Florian Weimer]

* Jos Vos:

>  Hi,
>
>  On Mon, Jul 10, 2017 at 09:14:07AM +0100, Richard W.M. Jones wrote:
>
> > The web page has:
> > 
> >    Improvements and stabilization of "ndb" (New RPM DB Format database
> >    format)
> >
> > [...]
> > 
> > The problem is that "NDB" is a custom homebrew database invented in
> > the RPM codebase.  I agree that because of licensing problems we need
> > to move away from Berkeley DB, but why not switch to the obvious,
> > bulletproof choice - Sqlite?
>
>  SQLite is is totally different piece of cake (RDBMS vs. key/value). 
But it's already part of the base system anyway.

>  Why not use LMDB?  It's the replacement for BerkeleyDB in
> OpenLDAP.
>
>  http://www.lmdb.tech/doc/ 
This was discussed before:

  https://bugzilla.redhat.com/show_bug.cgi?id=1086784

I don't know if LMDB upstream addressed the limitations mentioned in
the bug.

Using ndb in RPM

[Przemek Klosowski]

On 07/10/2017 09:16 AM, Jos Vos wrote:
>  On Mon, Jul 10, 2017 at 02:30:59PM +0200, Pierre-Yves Chibon wrote:
>
> > Oups, that was meant to be: On which side?
>  LDBM was many factors faster than SQLite.  No concrete figures,
>  it was a year ago or so when we decided to go for LMDB (still
>  a product in development...).  I had attended some talk from
>  Howard Chu (the author) about LMDB and knew quite soon it was
>  the way to go.
>  
Could you possibly summarize what you remember about those tests? What 
scenario did you look at? Was the difference more like seconds vs hours, 
or microseconds vs milliseconds?

I don't doubt your conclusions, but I keep being impressed with 
sqlite---one of my little projects grew to 500MB of data, and it is 
chugging along snappily, to my pleasant surprise. Caveat: I am no 
database guru, just a casual user.

Using ndb in RPM

[Leonid Podolny]

Correct me if I'm wrong, but the database here is several thousands rows in total,
several MBs in size. Every database engine should be fine. We probably care much more
about things like ease of development, stability, proper locking and such.

>  On Jul 10, 2017, at 9:16 AM, Jos Vos <jos(a)xos.nl&gt; wrote:
>  
>  On Mon, Jul 10, 2017 at 02:30:59PM +0200, Pierre-Yves Chibon wrote:
>  
> > Oups, that was meant to be: On which side?
>  
>  LDBM was many factors faster than SQLite.  No concrete figures,
>  it was a year ago or so when we decided to go for LMDB (still
>  a product in development...).  I had attended some talk from
>  Howard Chu (the author) about LMDB and knew quite soon it was
>  the way to go.
>  
>  -- 
>  --    Jos Vos <jos(a)xos.nl&gt;
>  --    X/OS Experts in Open Systems BV   |   Office: [phone]
>  --    Amsterdam, The Netherlands        |   Mobile: [phone]
>  _______________________________________________
>  devel mailing list -- devel(a)lists.fedoraproject.org
>  To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org

Using ndb in RPM

[Chris Adams]

Once upon a time, Leonid Podolny <leonid(a)podolny.net&gt; said:
>  Correct me if I'm wrong, but the database here is several
> thousands rows in total, several MBs in size. Every database engine should be fine. We
> probably care much more about things like ease of development, stability, proper locking
> and such. 
On my fairly normal desktop system, /var/lib/rpm is 128MB.  There's a
lot of information stored in there: each RPM has a bunch of metadata,
scripts, lists of provides and requires, and a list of all files and
directories (each entry with its own metadata information).

One requirement that tends to set RPM apart from some of the candidate
database libraries is non-root read-only access.  I'm not sure this is
an absolute requirement, but I expect it would cause significant upset
with system administrators if "rpm -q foo" changed to require root
access.

-- 
Chris Adams <linux(a)cmadams.net&gt;

Using ndb in RPM

[Björn 'besser82' Esser]

Am 11.07.2017 um 05:49 schrieb Chris Adams:
>  Once upon a time, Leonid Podolny <leonid(a)podolny.net&gt; said:
> > Correct me if I'm wrong, but the database here is several thousands rows in
> total, several MBs in size. Every database engine should be fine. We probably care much
> more about things like ease of development, stability, proper locking and such.
>  On my fairly normal desktop system, /var/lib/rpm is 128MB.  There's a
>  lot of information stored in there: each RPM has a bunch of metadata,
>  scripts, lists of provides and requires, and a list of all files and
>  directories (each entry with its own metadata information).
>
>  One requirement that tends to set RPM apart from some of the candidate
>  database libraries is non-root read-only access.  I'm not sure this is
>  an absolute requirement, but I expect it would cause significant upset
>  with system administrators if "rpm -q foo" changed to require root
>  access.
>  
Well, the RPM db on my desktop system is about 193 MBytes, from which 
the most amount (170 MBytes) comes from the 'Packages' db… I'd still 
consider that several MBytes given today's state of technology and 
SQLite should be able to deal with databases of that size with 
reasonable time consumption when it comes to queries.

There are in fact some technical limits in LMDB, but wouldn't it be more 
feasable to put man power into fixing those limits, than spending time 
on some homebrew solution?

Just my 2 cents…

Cheers,
   Björn

Using ndb in RPM

[Florian Weimer]

* Björn 'besser82' Esser:

>  Well, the RPM db on my desktop system is about 193 MBytes, from which
>
>  the most amount (170 MBytes) comes from the 'Packages' db… I'd still 
>  consider that several MBytes given today's state of technology and 
>  SQLite should be able to deal with databases of that size with 
>  reasonable time consumption when it comes to queries. 
Obviously, that depends on how the database is defined and if the
queries are half-decent (so that not everything has to fall back to
full table scans).

And it's not just about processing time.  With small VMs and
containers, it's also about not putting everything into RAM.

>  There are in fact some technical limits in LMDB, but wouldn't it
> be more 
>  feasable to put man power into fixing those limits, than spending time 
>  on some homebrew solution? 
I think SQLite would be the better choice, after comparing the
upstream reactions (not features) when discussing RPM's special
requirements with them (key sizes, read-only access without locking).

Using ndb in RPM

[Neal Gompa]

On Tue, Jul 11, 2017 at 6:36 AM, Florian Weimer <fw(a)deneb.enyo.de&gt; wrote:
>  * Björn 'besser82' Esser:
>
> > Well, the RPM db on my desktop system is about 193 MBytes, from which
> > the most amount (170 MBytes) comes from the 'Packages' db… I'd still
> > consider that several MBytes given today's state of technology and
> > SQLite should be able to deal with databases of that size with
> > reasonable time consumption when it comes to queries.
>
>  Obviously, that depends on how the database is defined and if the
>  queries are half-decent (so that not everything has to fall back to
>  full table scans).
>
>  And it's not just about processing time.  With small VMs and
>  containers, it's also about not putting everything into RAM.
>
> > There are in fact some technical limits in LMDB, but wouldn't it be more
> > feasable to put man power into fixing those limits, than spending time
> > on some homebrew solution?
>
>  I think SQLite would be the better choice, after comparing the
>  upstream reactions (not features) when discussing RPM's special
>  requirements with them (key sizes, read-only access without locking). 
And we do use SQLite today in DNF with the yumdb, as well as the new
SWDB coming soon(TM). I'm not sure why the SQLite backend was removed
in rpm 4.9.0, but maybe it should be revisited for rpm 4.14.



-- 
真実はいつも一つ！/ Always, there's only one truth!

Using ndb in RPM

[Michael Schroeder]

On Tue, Jul 11, 2017 at 06:41:05AM -0400, Neal Gompa wrote:
>  And we do use SQLite today in DNF with the yumdb, as well as the new
>  SWDB coming soon(TM). I'm not sure why the SQLite backend was removed
>  in rpm 4.9.0, but maybe it should be revisited for rpm 4.14. 
AFAIR it was removed because it was unbearable slow, nobody used it,
and nobody wanted to maintain it.


Here's a bit of technical input for discussions about rpm's database:

1) Background: how rpm uses the database

An Rpm package contains a header, which is basically a key->value
store. This header is put in the database as a binary blob (you
need the blob form for verification purposes). Rpm's database
layer maintains a couple of indices for parts of the header
like dependencies or the filelist, so queries are fast.
rpm has just two operations that modify the database: "add a
new header" and "remove an old header".

2) Problems with Berkeley DB

  - license issues
  - way too complex and buggy
  - bad locking scheme (stale locks)
  
3) What about ndb?

  - written as experiment how an ideal database would look like
  - currently optional to gather experience with it

  - main "packages" database written with data safty in mind, i.e.
    checksumming, extra tests to make sure data is not overwritten,
    simplistic design (1300 lines of code).
  - index databases are basically mmaped hashes
  - the database detects if the index is out of sync and auto-rebuilds
  - reader/writer locking scheme (multiple readers, one writer)

4) Does it make sense to use a self-made database?

There are pros and cons. On the pro side is that the code is pretty small
and tailored for rpm's need (speed & locking). With Berkeley DB, the
database code was a "black box", you couldn't do much which bugzilla
reports about a corrupt rpm database (and there were lots of those).
With the "database" being part of rpm the rpm maintainers can
actively fix issues.
The con side is, of course, that there's a bit more code to maintain.


So, suggesting different databases is fine and all, but they have
to be integrated and well tested. We re-added support for multiple
database just for that, so that we can test things and decide what
to do.

Cheers,
  Michael.

-- 
Michael Schroeder                                   mls(a)suse.de
SUSE LINUX GmbH,           GF Jeff Hawn, HRB 16746 AG Nuernberg
main(_){while(_=~getchar())putchar(~_-1/(~(_|32)/13*2-11)*13);}

Using ndb in RPM

[Stephen John Smoogen]

On 11 July 2017 at 07:52, Michael Schroeder <mls(a)suse.de&gt; wrote:
>  On Tue, Jul 11, 2017 at 06:41:05AM -0400, Neal Gompa wrote:
> > And we do use SQLite today in DNF with the yumdb, as well as the new
> > SWDB coming soon(TM). I'm not sure why the SQLite backend was removed
> > in rpm 4.9.0, but maybe it should be revisited for rpm 4.14.
>
>  AFAIR it was removed because it was unbearable slow, nobody used it,
>  and nobody wanted to maintain it.
>  
1. It was very slow.. and developers complained a lot about how long
it took to get various things done.
2. Because it was slow people would think it was deadlocked and lots
of debugging would be done to show it was just slow.
3. When it did deadlock it was very hard to debug because a lot of the
problems seemed to be system dependent. Move the db over to some other
system and no deadlock. Move it back, deadlock, rebuild the system
maybe it would deadlock maybe it wouldn't.
4. When it was put in there were a lot of people who were happy to
help debug etc, however most of them were not the level of debugging
actually needed as the problems were rarely the queries themselves as
much as the locking/rollback/multiple user code
5. Rollback did not always rollback.

I believe there are about 4 or 5 other reasons that were being
complained about during that time.

>  Here's a bit of technical input for discussions about rpm's
> database:
>
>  1) Background: how rpm uses the database
>
>  2) Problems with Berkeley DB
>  3) What about ndb?
>  4) Does it make sense to use a self-made database?
>  
5) This looks to be a cross operating system development versus a RH
NIH development.


>  --
>  Michael Schroeder                                   mls(a)suse.de 

-- 
Stephen J Smoogen.

Using ndb in RPM

[stan]

On Tue, 11 Jul 2017 13:52:32 +0200
Michael Schroeder <mls(a)suse.de&gt; wrote:

>  So, suggesting different databases is fine and all, but they have
>  to be integrated and well tested. We re-added support for multiple
>  database just for that, so that we can test things and decide what
>  to do. 
Does this mean it would be possible to add support for a real
database as an option?  Like postgresql or mariahdb or mysql?  That
would give access to views, triggers, and stored procedures.  Then when
an rpm update occurred, the views of what it depends on and what
depends on it could be updated by a stored procedure triggered by the
update.  This would also simplify, and speed, dnf since it could just
run a sql query to find all dependencies and their versions (I think).
And real databases allow granular access to be defined for users per
table.  Heavy weight solution, but some benefits.

Just curious.

Using ndb in RPM

[Jos Vos]

Hi,

On Mon, Jul 10, 2017 at 12:13:58PM -0400, Przemek Klosowski wrote:

>  Could you possibly summarize what you remember about those tests?
> What 
>  scenario did you look at? Was the difference more like seconds vs hours, 
>  or microseconds vs milliseconds? 
I found some of my old mail back.  In my tests a full processing cycle
for reading and processing (JSON deserializing in my case) data,
measured on a large amount of operations, was 4 times faster in LMDB
than in SQLite.

Note that this includes application processing (at this moment
I can't recall what that test did, it was 1,5 year ago...), so the
pure difference in DB processing time might even be much larger.

In general, I agree that SQLite is a fast DB when comparing, for
example, with ProgreSQL.  I did tests for that too in the past and
the difference is huge (and the feature set difference is huge too ;-)).

BTW, I'm pretty sure Howard Chu, the author of LMDB, would be happy to
discuss things with the Fedora/RPM community, if someone is interested.

-- 
--    Jos Vos <jos(a)xos.nl&gt;
--    X/OS Experts in Open Systems BV   |   Office: [phone]
--    Amsterdam, The Netherlands        |   Mobile: [phone]

Using ndb in RPM

[Florian Weimer]

* Stephen John Smoogen:

>  On 11 July 2017 at 07:52, Michael Schroeder <mls(a)suse.de&gt;
> wrote:
> > On Tue, Jul 11, 2017 at 06:41:05AM -0400, Neal Gompa wrote:
> >> And we do use SQLite today in DNF with the yumdb, as well as the new
> >> SWDB coming soon(TM). I'm not sure why the SQLite backend was removed
> >> in rpm 4.9.0, but maybe it should be revisited for rpm 4.14.
> >
> > AFAIR it was removed because it was unbearable slow, nobody used it,
> > and nobody wanted to maintain it.
> >
>
>  1. It was very slow.. and developers complained a lot about how long
>  it took to get various things done. 
It looks like the backend was ported from SQLite 2 code initially.
The result set processing code indeed loks quite inefficient (result
set cells are are allocated individually, memcpy is used to copy an
int, and so on).  But I'm still surprised this was observable to
users.

Using ndb in RPM

[Stephen John Smoogen]

On 11 July 2017 at 16:48, Florian Weimer <fw(a)deneb.enyo.de&gt; wrote:
>  * Stephen John Smoogen:
>
> > On 11 July 2017 at 07:52, Michael Schroeder <mls(a)suse.de&gt; wrote:
> >> On Tue, Jul 11, 2017 at 06:41:05AM -0400, Neal Gompa wrote:
> >>> And we do use SQLite today in DNF with the yumdb, as well as the new
> >>> SWDB coming soon(TM). I'm not sure why the SQLite backend was removed
> >>> in rpm 4.9.0, but maybe it should be revisited for rpm 4.14.
> >>
> >> AFAIR it was removed because it was unbearable slow, nobody used it,
> >> and nobody wanted to maintain it.
> >>
> >
> > 1. It was very slow.. and developers complained a lot about how long
> > it took to get various things done.
>
>  It looks like the backend was ported from SQLite 2 code initially.
>  The result set processing code indeed loks quite inefficient (result
>  set cells are are allocated individually, memcpy is used to copy an
>  int, and so on).  But I'm still surprised this was observable to
>  users. 
Never doubt the amount a developer will complain when either an
install or build takes longer than it did before. Especially when they
have to do dozens or hundreds of builds. And also never doubt how much
complaining if they find that rpm/yum is now using up more memory.
[There were memory issues in yum which I think went back to this.] Or
if someone uses that speed and memory usage to explain why apt/.deb is
superior to yum/.rpm. Those kinds of complaints end up getting
features yanked.

I do believe the extra memory did cause OOM on certain scenarios which
was another reason for it to be removed.

However it has been a while so I could be remembering 10 other bugs.



-- 
Stephen J Smoogen.

Using ndb in RPM

[Florian Weimer]

* Stephen John Smoogen:

>  On 11 July 2017 at 16:48, Florian Weimer <fw(a)deneb.enyo.de&gt;
> wrote:
> > * Stephen John Smoogen:
> >
> >> On 11 July 2017 at 07:52, Michael Schroeder <mls(a)suse.de&gt; wrote:
> >>> On Tue, Jul 11, 2017 at 06:41:05AM -0400, Neal Gompa wrote:
> >>>> And we do use SQLite today in DNF with the yumdb, as well as the new
> >>>> SWDB coming soon(TM). I'm not sure why the SQLite backend was
> removed
> >>>> in rpm 4.9.0, but maybe it should be revisited for rpm 4.14.
> >>>
> >>> AFAIR it was removed because it was unbearable slow, nobody used it,
> >>> and nobody wanted to maintain it.
> >>>
> >>
> >> 1. It was very slow.. and developers complained a lot about how long
> >> it took to get various things done.
> >
> > It looks like the backend was ported from SQLite 2 code initially.
> > The result set processing code indeed loks quite inefficient (result
> > set cells are are allocated individually, memcpy is used to copy an
> > int, and so on).  But I'm still surprised this was observable to
> > users.
>
>  Never doubt the amount a developer will complain when either an
>  install or build takes longer than it did before. Especially when they
>  have to do dozens or hundreds of builds. 
In mock?  Yes, that (and installation in general) look rather slow
because *every key lookup* appears to be wrapped in a
getcwd/chroot/chdir/chroot sequence.  It looks like this was
implemented to fix this bug:

  <https://bugzilla.redhat.com/show_bug.cgi?id=159424>

Unfortunately, the bug doesn't say why the fix took this particular
form (and why the database wasn't always opened outside of the chroot
instead).

Using ndb in RPM

[Richard W.M. Jones]

On Tue, Jul 11, 2017 at 06:53:40AM -0700, stan wrote:
>  On Tue, 11 Jul 2017 13:52:32 +0200
>  Michael Schroeder <mls(a)suse.de&gt; wrote:
>  
>  > So, suggesting different databases is fine and all, but they have
>  > to be integrated and well tested. We re-added support for multiple
>  > database just for that, so that we can test things and decide what
>  > to do.
>  
>  Does this mean it would be possible to add support for a real
>  database as an option?  Like postgresql or mariahdb or mysql?  That
>  would give access to views, triggers, and stored procedures.  Then when
>  an rpm update occurred, the views of what it depends on and what
>  depends on it could be updated by a stored procedure triggered by the
>  update.  This would also simplify, and speed, dnf since it could just
>  run a sql query to find all dependencies and their versions (I think).
>  And real databases allow granular access to be defined for users per
>  table.  Heavy weight solution, but some benefits. 
You'd be able to do most of this if sqlite was used as the default
backend.

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
Fedora Windows cross-compiler. Compile Windows programs, test, and
build Windows installers. Over 100 libraries supported.
http://fedoraproject.org/wiki/MinGW

Using ndb in RPM

[Michael Schroeder]

On Thu, Jul 13, 2017 at 09:11:42AM +0100, Richard W.M. Jones wrote:
>  On Tue, Jul 11, 2017 at 06:53:40AM -0700, stan wrote:
>  > On Tue, 11 Jul 2017 13:52:32 +0200
>  > Michael Schroeder <mls(a)suse.de&gt; wrote:
>  > 
>  > > So, suggesting different databases is fine and all, but they have
>  > > to be integrated and well tested. We re-added support for multiple
>  > > database just for that, so that we can test things and decide what
>  > > to do.
>  > 
>  > Does this mean it would be possible to add support for a real
>  > database as an option?  Like postgresql or mariahdb or mysql?  That
>  > would give access to views, triggers, and stored procedures.  Then when
>  > an rpm update occurred, the views of what it depends on and what
>  > depends on it could be updated by a stored procedure triggered by the
>  > update.  This would also simplify, and speed, dnf since it could just
>  > run a sql query to find all dependencies and their versions (I think).
>  > And real databases allow granular access to be defined for users per
>  > table.  Heavy weight solution, but some benefits.
>  
>  You'd be able to do most of this if sqlite was used as the default
>  backend. 
That wouldn't speed up dnf in any way, though. Dnf uses a sat solver
for solving, it works by translating all dependencies into package
rules and then running the sat algorithm on the rules.

This translation needs to be done for all installed packages, so it
always has to read them all. A Query mechanism is thus not needed
at all.

Dnf itself only reads installed packages that are new, it uses a
cache for the ones it already knows about (the @System.solv file
for the curious).

Cheers,
  Michael.

-- 
Michael Schroeder                                   mls(a)suse.de
SUSE LINUX GmbH,           GF Jeff Hawn, HRB 16746 AG Nuernberg
main(_){while(_=~getchar())putchar(~_-1/(~(_|32)/13*2-11)*13);}