Reltio Connect

Hi Everyone,

We are currently using message streaming to Google Cloud Pub/Sub in order to pass all updates and merges to our 4 other systems and keep them all in sync.  Because of our integration setup, we require a snapshot rather than a delta, and we cannot require that our integration makes any API calls to Reltio to get any additional information about the change.  Because the "ENTITIES_MERGED" message gives us only the URIs involved, we must use the ENTITY_CHANGED message instead to figure out that a merge has occurred by checking for the presence of an attributes.accountID value with with OV=False that has been contributed by a crosswalk which does not have a deleteDate on it.

So, we need to include a Snapshot with all attributes, and we need OV=True for all of those attributes and OV=False for a few of the attributes, plus we need the crosswalks.

The problem is that our messages are growing quite large now, and this is eating up our storage in Kafka/Splunk.  So we need to try to shrink the message size.

I know there aren't any easy solutions at this point.  Here are maybe a few options.
 
For regular updates, we don't need any OV=false attribute values at all, but we currently have to use the ENTITY_CHANGED messages to check for merges, since the ENTITIES_MERGED message currently only includes the winning and losing URIs and nothing else.
 
Probably the best option would be if Reltio could include the actual resulting entity in the ENTITIES_MERGED event message.  Ideally, I would love to be able to include the OV=True values for all attributes we specify, but also include the OV=False attribute values for the accountID attribute only.  Also, we do need to include the crosswalks, but we probably do not need the singleAttributeUpdateDates sections of those crosswalks right now.  So if we could suppress only that section of the crosswalks, that would help reduce the size too.  Lastly, if a crosswalk has a deleteDate on it, do not include it in the event messaging.
 
Or if we could have that level of control even for the ENTITY_CHANGED event messages, that would work for us too.
1.    The ability to control for which attributes we want to include the non-OV values.
2.    The ability to leave out the singleAttributeUpdateDates from the crosswalks.
3.    The ability to leave out the crosswalks which have a deleteDate.
 
It would be great if we could have different settings for each type of event message, but at least having the ability to have 1 list of OV-only attributes and another list of OV plus non-OV attributes for all events would help so much with getting the message size down.

So, does anyone know of any way to implement this level of differentiation in streaming?  I believe it is still not possible, but I thought I would ask here in case anyone else had a similar issue that they resolved.

One last option that I could think of is to add a cleanser that could take all of the OV=True and OV=False values for a certain attribute (accountID for us) and put it into another attribute where they would all be OV=True.  That might allow us to stop sending the OV=False values in our streaming.

Thank you in advance for any assistance!

Hi Trisha,

Without understanding all the intricacies of your integration, is it not possible to read the merge record and then do a 'get by id' of the winning record?  Its possible that thinking the data has to come from the queue is limiting for you.

Hi Everyone,

We are currently using message streaming to Google Cloud Pub/Sub in order to pass all updates and merges to our 4 other systems and keep them all in sync.  Because of our integration setup, we require a snapshot rather than a delta, and we cannot require that our integration makes any API calls to Reltio to get any additional information about the change.  Because the "ENTITIES_MERGED" message gives us only the URIs involved, we must use the ENTITY_CHANGED message instead to figure out that a merge has occurred by checking for the presence of an attributes.accountID value with with OV=False that has been contributed by a crosswalk which does not have a deleteDate on it.

So, we need to include a Snapshot with all attributes, and we need OV=True for all of those attributes and OV=False for a few of the attributes, plus we need the crosswalks.

The problem is that our messages are growing quite large now, and this is eating up our storage in Kafka/Splunk.  So we need to try to shrink the message size.

I know there aren't any easy solutions at this point.  Here are maybe a few options.
 
For regular updates, we don't need any OV=false attribute values at all, but we currently have to use the ENTITY_CHANGED messages to check for merges, since the ENTITIES_MERGED message currently only includes the winning and losing URIs and nothing else.
 
Probably the best option would be if Reltio could include the actual resulting entity in the ENTITIES_MERGED event message.  Ideally, I would love to be able to include the OV=True values for all attributes we specify, but also include the OV=False attribute values for the accountID attribute only.  Also, we do need to include the crosswalks, but we probably do not need the singleAttributeUpdateDates sections of those crosswalks right now.  So if we could suppress only that section of the crosswalks, that would help reduce the size too.  Lastly, if a crosswalk has a deleteDate on it, do not include it in the event messaging.
 
Or if we could have that level of control even for the ENTITY_CHANGED event messages, that would work for us too.
1.    The ability to control for which attributes we want to include the non-OV values.
2.    The ability to leave out the singleAttributeUpdateDates from the crosswalks.
3.    The ability to leave out the crosswalks which have a deleteDate.
 
It would be great if we could have different settings for each type of event message, but at least having the ability to have 1 list of OV-only attributes and another list of OV plus non-OV attributes for all events would help so much with getting the message size down.

So, does anyone know of any way to implement this level of differentiation in streaming?  I believe it is still not possible, but I thought I would ask here in case anyone else had a similar issue that they resolved.

One last option that I could think of is to add a cleanser that could take all of the OV=True and OV=False values for a certain attribute (accountID for us) and put it into another attribute where they would all be OV=True.  That might allow us to stop sending the OV=False values in our streaming.

Thank you in advance for any assistance!

Hi Gino,

Thank you for your reply.  Yes, I agree that requiring the data to all come from the queue is limiting.  We have plans in the future to implement something with more back and forth communication rather than relying completely on the queue to already have everything, but that will have to wait for several months since our integration team is already booked up with other projects until then.  So I was hoping there would be something that we could implement now to make the message sizes smaller in the meantime while still having access to the same information.

Kind regards,

Hi Trisha,

Without understanding all the intricacies of your integration, is it not possible to read the merge record and then do a 'get by id' of the winning record?  Its possible that thinking the data has to come from the queue is limiting for you.

Hi Everyone,

We are currently using message streaming to Google Cloud Pub/Sub in order to pass all updates and merges to our 4 other systems and keep them all in sync.  Because of our integration setup, we require a snapshot rather than a delta, and we cannot require that our integration makes any API calls to Reltio to get any additional information about the change.  Because the "ENTITIES_MERGED" message gives us only the URIs involved, we must use the ENTITY_CHANGED message instead to figure out that a merge has occurred by checking for the presence of an attributes.accountID value with with OV=False that has been contributed by a crosswalk which does not have a deleteDate on it.

So, we need to include a Snapshot with all attributes, and we need OV=True for all of those attributes and OV=False for a few of the attributes, plus we need the crosswalks.

The problem is that our messages are growing quite large now, and this is eating up our storage in Kafka/Splunk.  So we need to try to shrink the message size.

I know there aren't any easy solutions at this point.  Here are maybe a few options.
 
For regular updates, we don't need any OV=false attribute values at all, but we currently have to use the ENTITY_CHANGED messages to check for merges, since the ENTITIES_MERGED message currently only includes the winning and losing URIs and nothing else.
 
Probably the best option would be if Reltio could include the actual resulting entity in the ENTITIES_MERGED event message.  Ideally, I would love to be able to include the OV=True values for all attributes we specify, but also include the OV=False attribute values for the accountID attribute only.  Also, we do need to include the crosswalks, but we probably do not need the singleAttributeUpdateDates sections of those crosswalks right now.  So if we could suppress only that section of the crosswalks, that would help reduce the size too.  Lastly, if a crosswalk has a deleteDate on it, do not include it in the event messaging.
 
Or if we could have that level of control even for the ENTITY_CHANGED event messages, that would work for us too.
1.    The ability to control for which attributes we want to include the non-OV values.
2.    The ability to leave out the singleAttributeUpdateDates from the crosswalks.
3.    The ability to leave out the crosswalks which have a deleteDate.
 
It would be great if we could have different settings for each type of event message, but at least having the ability to have 1 list of OV-only attributes and another list of OV plus non-OV attributes for all events would help so much with getting the message size down.

So, does anyone know of any way to implement this level of differentiation in streaming?  I believe it is still not possible, but I thought I would ask here in case anyone else had a similar issue that they resolved.

One last option that I could think of is to add a cleanser that could take all of the OV=True and OV=False values for a certain attribute (accountID for us) and put it into another attribute where they would all be OV=True.  That might allow us to stop sending the OV=False values in our streaming.

Thank you in advance for any assistance!