Reltio Connect

Previously we use to run Consistency 3 times a week, per Reltio this is no longer needed unless you see issues. 
Re-Index and Match Rebuild - previously we ran any time we did a new data load (new source),  or at minimum 1xweek since we do a daily load of existing sources.   Per Reltio these are also not needed anymore unless issue is seen. 

I am wondering what others have done,  because not running them and waiting for issue usually means our Business spots the issue and reports which causes lack of trust that data seen and process in Reltio is working.    

The jobs were sometimes taking 24-48 hours to run which also has an impact,  so wondering what others are doing?

Thank you!
Angie


------------------------------
Angela Wawrzaszek
Manager, MDM governance and Application Enablement
------------------------------

Angela,
   You are correct that running consistency checks are not needed for normal tenant operations.  What the consistency check is doing is comparing our primary storage with our secondary storage and looking for any version differences.  If it finds any differences, it updates the second storage.  The thing is that this process of building out the secondary storage is also done automatically, and we automatically do that double checking to be absolutely sure there are no differences.  We do this behind the scenes in the core platform.  

Today, a consistency check or a reindex may be useful only if you are disabling some of the secondary processing during an initial data load.  For ongoing operations, we would recommend that you let the normal Reltio processing handle this.  Otherwise, you are actually doing that check twice and incurring additional overhead in the process.

One reason why you may have seen differences in storages previously though is that when you load data into Reltio, a bunch of background asyncronous processes run.  This updating of the secondary storage is one of them.  (Match is the other really big one.)  These processes will still be going on after all of the load APIs are completed.  You can check the status of these though in your Reltio Console.

Monitoring Queues

Reltio		remove preview
Monitoring Queues The Queue size indicates the number of events in the crud and match queues. The queue performance can be visualized from the trend chart. If the queue size remains the same or decreases as events are added to the queue, then the queue is processing the events. View this on Reltio >

 

Above is a link to our docs on this part of the console, but the basic gist is that you want to wait for both the CRUD and Match queues to be empty before running your extract to ensure that all processing is complete.  (The same is true when running consistency checks and reindexes!)  Only when these queues are empty has all of the background processing been completed.  I hope that helps and saves you some time on your data processing.

------------------------------
Mike Frasca
------------------------------

Original Message:
Sent: 05-11-2021 12:54
From: Angela Wawrzaszek
Subject: How often do you run Reltio Jobs? Consistency, Re-index, match re-build?

Previously we use to run Consistency 3 times a week, per Reltio this is no longer needed unless you see issues. 
Re-Index and Match Rebuild - previously we ran any time we did a new data load (new source),  or at minimum 1xweek since we do a daily load of existing sources.   Per Reltio these are also not needed anymore unless issue is seen. 

I am wondering what others have done,  because not running them and waiting for issue usually means our Business spots the issue and reports which causes lack of trust that data seen and process in Reltio is working.    

The jobs were sometimes taking 24-48 hours to run which also has an impact,  so wondering what others are doing?

Thank you!
Angie


------------------------------
Angela Wawrzaszek
Manager, MDM governance and Application Enablement
------------------------------

In my experience with Reltio, we have always reindexed and run consistency checks after any of the following 3 events:

1. Observing any unusual behavior in the UI where the UI seems out of sync with the db
2. Performing very large (> 1MIL entities) data loads
3. Performing a configuration change

I am very interested to know if changes to the Reltio platform have eliminated the need for this.

I will also say that even though these jobs do sometimes take 24-48 hours to run, we have never had any negative impacts. We do try to run them over a weekend whenever possible. 

------------------------------
Walt Feldman
Digital Data Lead
Subway
------------------------------

Original Message:
Sent: 05-11-2021 17:05
From: Mike Frasca
Subject: How often do you run Reltio Jobs? Consistency, Re-index, match re-build?

Angela,
   You are correct that running consistency checks are not needed for normal tenant operations.  What the consistency check is doing is comparing our primary storage with our secondary storage and looking for any version differences.  If it finds any differences, it updates the second storage.  The thing is that this process of building out the secondary storage is also done automatically, and we automatically do that double checking to be absolutely sure there are no differences.  We do this behind the scenes in the core platform.  

Today, a consistency check or a reindex may be useful only if you are disabling some of the secondary processing during an initial data load.  For ongoing operations, we would recommend that you let the normal Reltio processing handle this.  Otherwise, you are actually doing that check twice and incurring additional overhead in the process.

One reason why you may have seen differences in storages previously though is that when you load data into Reltio, a bunch of background asyncronous processes run.  This updating of the secondary storage is one of them.  (Match is the other really big one.)  These processes will still be going on after all of the load APIs are completed.  You can check the status of these though in your Reltio Console.

Monitoring Queues

Reltio		remove preview
Monitoring Queues The Queue size indicates the number of events in the crud and match queues. The queue performance can be visualized from the trend chart. If the queue size remains the same or decreases as events are added to the queue, then the queue is processing the events. View this on Reltio >

 

Above is a link to our docs on this part of the console, but the basic gist is that you want to wait for both the CRUD and Match queues to be empty before running your extract to ensure that all processing is complete.  (The same is true when running consistency checks and reindexes!)  Only when these queues are empty has all of the background processing been completed.  I hope that helps and saves you some time on your data processing.

------------------------------
Mike Frasca

Original Message:
Sent: 05-11-2021 12:54
From: Angela Wawrzaszek
Subject: How often do you run Reltio Jobs? Consistency, Re-index, match re-build?

Previously we use to run Consistency 3 times a week, per Reltio this is no longer needed unless you see issues. 
Re-Index and Match Rebuild - previously we ran any time we did a new data load (new source),  or at minimum 1xweek since we do a daily load of existing sources.   Per Reltio these are also not needed anymore unless issue is seen. 

I am wondering what others have done,  because not running them and waiting for issue usually means our Business spots the issue and reports which causes lack of trust that data seen and process in Reltio is working.    

The jobs were sometimes taking 24-48 hours to run which also has an impact,  so wondering what others are doing?

Thank you!
Angie


------------------------------
Angela Wawrzaszek
Manager, MDM governance and Application Enablement
------------------------------

As per my experience, it is not required to run periodically. I have not seen any issues till now.

In your case:

Business spots the issue and reports – If you are performing large size data load (around 1M) or any configuration changes then you may run these jobs to avoid issues. Try to run these jobs over a weekend.

​

------------------------------
Atul Ghorpade
------------------------------

Original Message:
Sent: 05-11-2021 12:54
From: Angela Wawrzaszek
Subject: How often do you run Reltio Jobs? Consistency, Re-index, match re-build?

Previously we use to run Consistency 3 times a week, per Reltio this is no longer needed unless you see issues. 
Re-Index and Match Rebuild - previously we ran any time we did a new data load (new source),  or at minimum 1xweek since we do a daily load of existing sources.   Per Reltio these are also not needed anymore unless issue is seen. 

I am wondering what others have done,  because not running them and waiting for issue usually means our Business spots the issue and reports which causes lack of trust that data seen and process in Reltio is working.    

The jobs were sometimes taking 24-48 hours to run which also has an impact,  so wondering what others are doing?

Thank you!
Angie


------------------------------
Angela Wawrzaszek
Manager, MDM governance and Application Enablement
------------------------------