Our current production Registry replication solution uses HornetQ to propagate replication messages to downstream clients. This solution has the following steps:
ED-LDAP, mail, Google, and AD replication all work this way.
HornetQ has historically worked well for us, but it also has its problems including strange locking, configuration directives changing, buffering that could cause priority records to not be replicated efficiently, and poor logging. When developing ED-MW, the question was raised if dealing with HornetQ going forward was worth the effort.
Our group has expertise in LDAP from both client and server side perspectives, and the replication technology that we use for our master and slave (producer and consumer) LDAP machines has proven to be reliable and fast. This technology, syncrepl, allows a client to perform a sync search against an LDAP. When records are changed in the LDAP, those changes will be reflected in an accesslog, and those entries will be sent to the client that requested the sync search. The client can then determine if it cares about the record that has changed and take action accordingly.
The benefits of such a solution include better throughput (don't have to deal with transforms), dealing with LDAP records directly instead of something text-based (XML), improved logging, and better ability to determine the state of a repl client, including if it is up-to-date or behind.
JMS has the concept of priority for messages, which allows higher priority messages to be delivered before lower priority messages.
LDAP syncrepl has no conception of priority; messages are sent in the order they are written. Though we will be able to process more records per second than HornetQ which will hopefully result in less wait times for updates, we still would like some changes to make it through faster than others. These are typically account related things like passwords.
Instead of message priority, we are going to use “channels”. The idea is that most changes can go through the standard channel at what we would previously refer to as normal priority. Account changes or changes that we need to replicate quickly can go through another channel named “accounts” or similar.
To implement the idea of channels, each client repl will perform a syncrepl search for the configured channels. We will start with 2 channels, but this can be extended to any number of channels.
It should be noted that things in the account channel will only be processed faster if there are generally more records sent through the standard channel.
Each repl client needs the following:
(&(objectClass=auditWriteObject)(reqMod=channel:= 2))" # specify channel (&(objectClass=auditWriteObject))" # all channels
Locking will be used to ensure that certain channels can take precedence when processing records.
CREATE TABLE "VTREGISTRY"."VTREPLICATION" ( "UID" NUMBER(19,0) NOT NULL ENABLE, "TYPE" VARCHAR2(20) NOT NULL ENABLE, "CHANNEL" NUMBER(2,0) NOT NULL ENABLE, "CREATED_DATE" TIMESTAMP NOT NULL ENABLE, );
vtReplPerson, vtReplGroup, vtReplService, vtReplEntitlement, vtReplAddress, vtReplMail, vtReplHealthCheck
Corresponding objectClasses for clients to request specific attributes.
TODO: add schema here