[389-devel] Re: Please review new Replication Diff Tool

[Mark Reynolds]

On 04/12/2017 06:58 PM, William Brown wrote:
>  On Wed, 2017-04-12 at 17:02 -0400, Mark Reynolds wrote:
> > Hello,
> >
> > This is a beta version of a replication diff tool written in python. 
> >
> > Design page (this needs updating - I hope to get that done tonight)
> >
> > http://www.port389.org/docs/389ds/design/repl-diff-tool-design.html
> >
> >
> > Current usage:
> >
> >   -v, --verbose         Verbose output
> >   -o FILE, --outfile=FILE
> >                         The output file
> >   -D BINDDN, --binddn=BINDDN
> >                         The Bind DN (REQUIRED)
> >   -w BINDPW, --bindpw=BINDPW
> >                         The Bind password (REQUIRED)
> >   -h MHOST, --master_host=MHOST
> >                         The Master host (default localhost)
> >   -p MPORT, --master_port=MPORT
> >                         The Master port (default 389)
> >   -H RHOST, --replica_host=RHOST
> >                         The Replica host (REQUIRED)
> >   -P RPORT, --replica_port=RPORT
> >                         The Replica port (REQUIRED)
> >   -b SUFFIX, --basedn=SUFFIX
> >                         Replicated suffix (REQUIRED)
> >   -l LAG, --lagtime=LAG
> >                         The amount of time to ignore inconsistencies
> > (default
> >                         300 seconds)
> >   -Z CERTDIR, --certdir=CERTDIR
> >                         The certificate database directory for startTLS
> >                         connections
> >   -i IGNORE, --ignore=IGNORE
> >                         Comma separated list of attributes to ignore
> >   -M MLDIF, --mldif=MLDIF
> >                         Master LDIF file (offline mode)
> >   -R RLDIF, --rldif=RLDIF
> >                         Replica LDIF file (offline mode)
> >
> > |Examples: python repl-diff.py -D "cn=directory manager" -w PASSWORD -h
> > localhost -p 389 -H remotehost -P 5555 -b "dc=example,dc=com" ||python
> repl-diff.py -D "cn=directory manager" -w PASSWORD -h localhost
> > -p 389 -H remotehost -P 5555 -b "dc=example,dc=com" -Z
> > /etc/dirsrv/slapd-localhost|
> > |python repl-diff.py -M /tmp/master.ldif -R /tmp/replica.ldif |
> >
> >
> > How long the tool takes to run depends on the number of entries per
> > database.  See performance numbers below 
> >
> > Entries per Replica     Time
> >
> > ---------------------------------
> > 100k                    40 seconds
> > 500k                    3m 30secs
> > 1 million               7m 30secs
> > 2 million               14 minutes
> > 10 million              ~70 minutes
> >
> >
> >
> > I'd be very interested in feedback, RFE's, and bugs.
>  Hey mate, 
>
>  The tool looks great, awesome work on this. Really impressive that you
>  got it to 70 minutes for 10 million entries. 
And in reality its 20 million entries
(master + replica)
>  How responsive is the server during this process? We aren't going to
>  cause some odd resource exhaustion?  
Not really  :)  We don't have to rely on
server side sorting, and it's
just a paged result search - so it breaks up the load (slightly).  But,
it's still expensive because it is returning all the entries, but I
didn't see any extreme CPU usage.
>  import optparse
>
>  With python, optparse is deprecated. Can we use argparse instead? It's
>  nearly identical. Lots of examples of this in dsctl.  
Easy, no problem.
>  With connect to replicas, some sites may only have ldaps (provided by a
>  load balancer). So our scripts should really be taking an LDAPurl, a
>  certdir, and a starttls flag. Because ldaps://localhost + certdir is a
>  valid option, but if we force call start_tls_s(), we break it.  
Good idea
>  As well,
>  someone may use ldapi:// etc. It also saves on port options and more
>  flags to the cli because we can do ldap://localhost:30389 etc. 
>
>  Hope that helps, I'll be happy to review again later!
>
>  For now, I think our strategy with this should be to add it to
>  389-ds-base, and later we can move this into lib389 when we can. How
>  does that sound?  
I think it should always be a standalone tool, but we can tie it
in with
lib389 (move the main guts out of the tool and into lib389)
>  _______________________________________________
>  389-devel mailing list -- 389-devel(a)lists.fedoraproject.org
>  To unsubscribe send an email to 389-devel-leave(a)lists.fedoraproject.org