Reltio Connect

Hi,

How can I filter streaming events, so that only events having at least 1 crosswalk from particular source systems would be streamed, while other events would be excluded.  

Can this be done and if possible provide example.

Context: After going live with one source system in a co-existence integration pattern, I want to start capturing content from additional source(s), but only create the streaming events if any of the related crosswalk for the related Entity or Relationship is from the first source system. This is intended to permit continued streaming events for the original system and only those from other systems impacting entities from the first system while minimizing impact of streaming events from historical one-time loads.

I understand that alternatively, I can ignore the messages I'm not interested in, but would prefer not having to include this logic in process accessing the event queue and also avoid performance implications from excessive logging of events.

Thanks,

Mark

Hi Mark,

Yes is possible to define filters for outbound queues.
It is defined at the queue (destination) level, so you can have different queues with different filter conditions if you wanted to.
This gives you flexibility in your destination setup and configuration.  For example, I could have two queues and set up one to receive only entityTypes of 'Account', while another queue can be used for 'Location'.

What you're looking for is the queue 'object filter'.  

If you wanted to add some source based logic, you could use something like not equals(sources,'configuration/sources/MySource')

Given that we have a whole range of filtering options available there are probably many different ways to solve for a given requirements.
Since you can filter on any element of the entity payload (such as: "createdBy", "createdTime", "updatedBy", "updatedTime") there may be slicker ways to implement what you are looking for.

Pro Tip:
Consider creating different service accounts (can be one-offs) for these different data loads you want to suppress events for. Then use createdBy / updatedBy in your logic instead. 

Example:
(not equals(createdBy,'idl-sourceX') and not equals(updatedBy,'idl-sourceX'))​
Where 'idl-sourceX' is the Reltio account username that you are using to load the data that you do not want to generate events for.

It starts to get a little more complex if you're merging these records automatically during the load (new record could merge with an existing record you care to receive merge events for) ... I have used this technique successfully and it is currently the method I use for solving for this particular requirement.

As always, please test your filters and queues in a lower environment first before requesting the change to your PROD queues - you can start with simple GET entities w/ filter requests.

Hope this helps !

------------------------------
Jacques Lateo
Solutions Architect
Reltio
------------------------------

@mmmmm ​

Jacques,

Appreciate the response. Can you go into a bit more detail around implications with automatic merge in place?

Thanks,

Mark

Hi Mark,

The implication is that if you have automatic merging enabled, and your existing solution already consumes from "ENTITY_CHANGED" events, then you may be filtering out events you actually want to allow through.

It really all depends on a number of variables, so please keep the following in mind when designing the filter:

• What does the existing configuration and setup of streaming (MDM outbound events) look like ?
• What Reltio event types does the existing solution consume from ?
• How would altering the existing configuration and setup of streaming affect the existing solution ?
• Review the appropriate / proposed filtering strategy and test thoroughly before promoting to PROD