[389-users] bulk initialization with MMR

[Gary Windham]

Thanks, Luke!  Since my original post, I did some more research and  
arrived at a solution very close to yours.  I did turn the servers  
into MMR, and am using the "ldifsort" and "ldifdiff" utilities that  
come with the perl-ldap module to generate "delta" LDIF files that are  
loaded via ldapmodify.  For our "real time" updates we have PeopleSoft  
sending SOAP messages via Integration Broker to some web services  
which perform the appropriate LDAP operations.

I appreciate your response...it serves as a good "sanity check". :)

--Gary

--
Gary Windham
Senior Enterprise Systems Architect
The University of Arizona, UITS
+1 520 626 5981

On Jul 1, 2009, at 3:44 PM, Luke Bigum wrote:

> Hi Gary,
>
> I used to work for a university that does something similar to what  
> you are trying to do. I'll explain their setup and it might give you  
> a few ideas. They have a custom user management database that's the  
> authoritative source of computer account information, a series of  
> FDS servers are used for identification and authentication. A Perl  
> script is used to turn the database contents into LDIF format as it  
> would be used to populate an empty database (like one of your  
> ldif2db batch extracts). They then take a dump of the LDAP directory  
> into LDIF format and compare the database LDIF to directory LDIF and  
> come up with a delta LDIF file. This delta LDIF is then run on the  
> directory server to bring it in line with the database contents.
>
> This update process runs every couple minutes, so the delta never  
> really gets that big and password changes / new users only take a  
> few minutes to propagate around the university. They would never  
> need to batch import the entire database contents unless there was a  
> catastrophe.
>
> So, for your scenario, you might consider scrapping the nightly bulk  
> initialisations, turn your servers into MMR and look at doing more  
> frequent updates with delta files to provide faster synchronisation  
> between your data sources.
>
> If you actually need to do real real-time updates, you can do that  
> with the same setup above, you just need to fire off a specific LDAP  
> update to your load balanced LDAP from Peoplesoft.
>
> Luke Bigum
> Systems Administrator
> (p) 1300 661 668
> (f)  1300 661 540
> (e)  lbigum at iseek.com.au
> http://www.iseek.com.au
> Level 1, 100 Ipswich Road Woolloongabba QLD 4102
>
>
>
> This e-mail and any files transmitted with it may contain  
> confidential and privileged material for the sole use of the  
> intended recipient. Any review, use, distribution or disclosure by  
> others is strictly prohibited. If you are not the intended recipient  
> (or authorised to receive for the recipient), please contact the  
> sender by reply e-mail and delete all copies of this message.
>
>
> -----Original Message-----
> From: fedora-directory-users-bounces at redhat.com [mailto:fedora-directory-users-bounces at redhat.com 
> ] On Behalf Of Rich Megginson
> Sent: Wednesday 1 July 2009 1:21 AM
> To: General discussion list for the 389 Directory server project.
> Subject: Re: [389-users] bulk initialization with MMR
>
> Gary Windham wrote:
>> We have a setup where we are running 2 servers behind a load balancer
>> (for HA purposes), where each of these servers is bulk-initialized
>> daily (via ldif2db.pl) with a large set of data fed to us via batch
>> extracts from various administrative systems.  Up till now, there has
>> been no need to configure replication between these 2 servers, as all
>> of the data is read-only.  However, we now have a requirement to
>> update some of the directory data in a "real-time" fashion (e.g.,  
>> when
>> particular events fire in our PeopleSoft system we want to update the
>> directory)--hence, the need for MMR.  The batch extracts will still  
>> be
>> our "checkpoints", so we will want to load them in once-per-day, as  
>> we
>> do now.
> How does the data get from peoplesoft to the directory server?
>>
>> So, the question is: what would be the "recommended" approach for a
>> scenario like this?  How do we (can we?) make MMR coexist peacefully
>> with frequent bulk initializations?
> In general, it's not a good idea to do a bulk load daily.
>>
>> TIA,
>> --Gary
>>
>> --
>> Gary Windham
>> Senior Enterprise Systems Architect
>> The University of Arizona, UITS
>> +1 520 626 5981
>>
>> --
>> 389 users mailing list
>> 389-users at redhat.com
>> https://www.redhat.com/mailman/listinfo/fedora-directory-users
>
>
> --
> 389 users mailing list
> 389-users at redhat.com
> https://www.redhat.com/mailman/listinfo/fedora-directory-users
>