Reltio Connect

I know that this is a challenge for customers right now because there isn't really a FIFO, a first in, first out necessarily available mechanism in the context of something like SQA. Which means that as you pick those messages up, you may receive them out of sequence.

This on the outbound side for Reltio, sounds like a very similar problem to the one that we were just talking about the solution to on the inbound side. And really requires that, again, that you're going to buffer for a short period of time until you understand everything you think you need to know before you send that information wherever they need to go next beyond Reltio.

Make sure you watch the Reltio Community Webinar

On the FIFO, so it's not just about SQS because the events themselves on the platform level are generated asynchronously. And that's due to Reltio's scalable nature of the platform because events are being processed by multiple nodes in parallel at the same time.

But you can use Kafka specifically. Once you stream from SQS to Kafka, you can create a producer that will aggregate. And you can put storage key, use the timestamp that every event has as a storage key to sort aggregated values. And you can aggregate everything for the last five seconds, for example. In this way, you sort them in Kafka. And as a result, get it sorted. 

I know that this is a challenge for customers right now because there isn't really a FIFO, a first in, first out necessarily available mechanism in the context of something like SQA. Which means that as you pick those messages up, you may receive them out of sequence.

This on the outbound side for Reltio, sounds like a very similar problem to the one that we were just talking about the solution to on the inbound side. And really requires that, again, that you're going to buffer for a short period of time until you understand everything you think you need to know before you send that information wherever they need to go next beyond Reltio.

Make sure you watch the Reltio Community Webinar

And I oversimplified my response there, but you're right. Reltio is an asynchronous platform. There is no guarantee that any event is going to complete in advance of any other, given the distributed nature of the way those components work internally.

But that's a great solution also, that you propose there to leverage those timestamps, which will be down to the millisecond level in terms of their detail.

On the FIFO, so it's not just about SQS because the events themselves on the platform level are generated asynchronously. And that's due to Reltio's scalable nature of the platform because events are being processed by multiple nodes in parallel at the same time.

But you can use Kafka specifically. Once you stream from SQS to Kafka, you can create a producer that will aggregate. And you can put storage key, use the timestamp that every event has as a storage key to sort aggregated values. And you can aggregate everything for the last five seconds, for example. In this way, you sort them in Kafka. And as a result, get it sorted. 

I know that this is a challenge for customers right now because there isn't really a FIFO, a first in, first out necessarily available mechanism in the context of something like SQA. Which means that as you pick those messages up, you may receive them out of sequence.

This on the outbound side for Reltio, sounds like a very similar problem to the one that we were just talking about the solution to on the inbound side. And really requires that, again, that you're going to buffer for a short period of time until you understand everything you think you need to know before you send that information wherever they need to go next beyond Reltio.

Make sure you watch the Reltio Community Webinar

"We are struggling with that. When a merge happens with an entity, Reltio throws out 15 different events. The merge could happen between three or four organizations and in 15 events. Pushing them all the way to the system back is just too much traffic. We want to aggregate and then pass one single message, giving them a view of what happened, what exactly happened."

And I oversimplified my response there, but you're right. Reltio is an asynchronous platform. There is no guarantee that any event is going to complete in advance of any other, given the distributed nature of the way those components work internally.

But that's a great solution also, that you propose there to leverage those timestamps, which will be down to the millisecond level in terms of their detail.

On the FIFO, so it's not just about SQS because the events themselves on the platform level are generated asynchronously. And that's due to Reltio's scalable nature of the platform because events are being processed by multiple nodes in parallel at the same time.

But you can use Kafka specifically. Once you stream from SQS to Kafka, you can create a producer that will aggregate. And you can put storage key, use the timestamp that every event has as a storage key to sort aggregated values. And you can aggregate everything for the last five seconds, for example. In this way, you sort them in Kafka. And as a result, get it sorted. 

I know that this is a challenge for customers right now because there isn't really a FIFO, a first in, first out necessarily available mechanism in the context of something like SQA. Which means that as you pick those messages up, you may receive them out of sequence.

This on the outbound side for Reltio, sounds like a very similar problem to the one that we were just talking about the solution to on the inbound side. And really requires that, again, that you're going to buffer for a short period of time until you understand everything you think you need to know before you send that information wherever they need to go next beyond Reltio.

Make sure you watch the Reltio Community Webinar

Kafka has this capability and SQS does not have this capability today. We will have to use Kafka for that.

And this is very, very common question from our customers and partners. And sometimes, there is a perception that using FIFO, first in, first out is a solution here. But you need to realize the platform will input the first in not guaranteed that it's first.

You have to sort by timestamp. And Kafka is the best solution right now. There is best practice I found in things like Stack Overflow, specifically for Kafka, how to create the producers that aggregates and source by timestamp. So that's already known use case for Kafka. And I would suggest you use that, yes.

"We are struggling with that. When a merge happens with an entity, Reltio throws out 15 different events. The merge could happen between three or four organizations and in 15 events. Pushing them all the way to the system back is just too much traffic. We want to aggregate and then pass one single message, giving them a view of what happened, what exactly happened."

And I oversimplified my response there, but you're right. Reltio is an asynchronous platform. There is no guarantee that any event is going to complete in advance of any other, given the distributed nature of the way those components work internally.

But that's a great solution also, that you propose there to leverage those timestamps, which will be down to the millisecond level in terms of their detail.

On the FIFO, so it's not just about SQS because the events themselves on the platform level are generated asynchronously. And that's due to Reltio's scalable nature of the platform because events are being processed by multiple nodes in parallel at the same time.

But you can use Kafka specifically. Once you stream from SQS to Kafka, you can create a producer that will aggregate. And you can put storage key, use the timestamp that every event has as a storage key to sort aggregated values. And you can aggregate everything for the last five seconds, for example. In this way, you sort them in Kafka. And as a result, get it sorted. 

I know that this is a challenge for customers right now because there isn't really a FIFO, a first in, first out necessarily available mechanism in the context of something like SQA. Which means that as you pick those messages up, you may receive them out of sequence.

This on the outbound side for Reltio, sounds like a very similar problem to the one that we were just talking about the solution to on the inbound side. And really requires that, again, that you're going to buffer for a short period of time until you understand everything you think you need to know before you send that information wherever they need to go next beyond Reltio.

Make sure you watch the Reltio Community Webinar