Reltio Integration Hub

Hi All,

We are trying to load 232k data into an entity via Reltio Integration Hub. 
We are fetching the data from Snowflake in a batch of 10k.
From the batch of 10k we are passing 2k in a batch(Max upload size to Reltio) to a child recipe where the "Create bulk into Reltio" is happening. The call to the child recipe is an asynchronous call.
 We have set the concurrency of the child recipe to 5.

From the above approach we have been able to load only 119K records and the recipe threw a timed out error.

Is there any different way/approach/design to achieve the complete load of 232k records without the recipe getting timed out error?
Any quick help would be really appreciated as we are in the middle of deployment.

------------------------------
Aditya Gupta
Freshgravity
------------------------------

Hi @Gaurav Gera @Saurabh Agarwal Can you help/provide any suggestion on my above query?

------------------------------
Aditya Gupta
Freshgravity
------------------------------

Original Message:
Sent: 11-21-2024 04:19
From: Aditya Gupta
Subject: Recipe is getting Timed out error after 90 mins while loading 232k records into Reltio

Hi All,

We are trying to load 232k data into an entity via Reltio Integration Hub. 
We are fetching the data from Snowflake in a batch of 10k.
From the batch of 10k we are passing 2k in a batch(Max upload size to Reltio) to a child recipe where the "Create bulk into Reltio" is happening. The call to the child recipe is an asynchronous call.
 We have set the concurrency of the child recipe to 5.

From the above approach we have been able to load only 119K records and the recipe threw a timed out error.

Is there any different way/approach/design to achieve the complete load of 232k records without the recipe getting timed out error?
Any quick help would be really appreciated as we are in the middle of deployment.

------------------------------
Aditya Gupta
Freshgravity
------------------------------

@Aditya Gupta @Ashish Rawat
Why not attempt all necessary transformations in the Snowflake layer, then create a csv/excel/json in any cloud storage? You can load it using the default scheduler of Reltio DataLoader or use an RIH recipe to call the DataLoader API to push data into Reltio. Once complete, you will receive an email. This process would typically use just 10 RIH tasks. Why not consider this approach? I have implemented this for many integrations and it works wonderfully. If there are any faulty records, you can easily retrieve the csv of the failed records from the Reltio DataLoader UI. By using the DataLoader job, load speed processing will increase, leading to faster task completion. 
If you need real-time integration, consider using the near real-time Reltio entity API from RIH for improved performance.

------------------------------
Vineeth Choudary Malepati
Integration specialist
Eaton
------------------------------

Original Message:
Sent: 11-21-2024 05:17
From: Aditya Gupta
Subject: Recipe is getting Timed out error after 90 mins while loading 232k records into Reltio

Hi @Gaurav Gera @Saurabh Agarwal Can you help/provide any suggestion on my above query?

------------------------------
Aditya Gupta
Freshgravity

Original Message:
Sent: 11-21-2024 04:19
From: Aditya Gupta
Subject: Recipe is getting Timed out error after 90 mins while loading 232k records into Reltio

Hi All,

We are trying to load 232k data into an entity via Reltio Integration Hub. 
We are fetching the data from Snowflake in a batch of 10k.
From the batch of 10k we are passing 2k in a batch(Max upload size to Reltio) to a child recipe where the "Create bulk into Reltio" is happening. The call to the child recipe is an asynchronous call.
 We have set the concurrency of the child recipe to 5.

From the above approach we have been able to load only 119K records and the recipe threw a timed out error.

Is there any different way/approach/design to achieve the complete load of 232k records without the recipe getting timed out error?
Any quick help would be really appreciated as we are in the middle of deployment.

------------------------------
Aditya Gupta
Freshgravity
------------------------------

..

..