Reltio Integration Hub

We have a recipe to load our reference data. Most contain a small number of rows. However,, some have more. Due to the newly enforced limitation of  50k rows per file, we are needing to split into smaller files. I can get one to run with 26k but cannot get any to run with closer to the 50k. Also, from time to time, the smaller file (20k) will not complete. See error below. It suggests that the recipe has timed out. 

Anyone have suggestions as to how to get around this? I could split into much smaller files but if I cannot get 20k (with only 12 columns) to run, not sure what to do. We want to do a monthly refresh of around 500k rows for around 12k lookup types. I don't want to have to create 50 files with 10k each unless I have to. 

------------------------------
Doug Yates
Humana
KY
------------------------------

There is a time limit for recipes and they will run out after 90 minutes.

Two things I suggest:

1) Review the recipe to see if bulk actions can be used and make the recipe more efficient and require fewer RIH tasks and time.

2) If 1) above has been considered and you still are having issues then you can proceed with this approach that will extend the timeout time.  

The assumption here is that you are performing something in a loop and that no individual task exceeds the 90 minute threshold.

Inside that loop if you add a wait task of any amount of time (ie. 0.1 is good) that will reset the timeout.   This can extend your recipe infinitely...so use with caution and keep an eye on it so you don't have runaway rih task consumption.

There is another caveat to using this wait task.  You will not be able to execute this in Test or Repeat Job mode (only Start).  If you do that then the wait task will hang and never come back.  This can make development and unit testing challenging and one way I have find around this for my testing is to create a recipe as a function and call it from another recipe function (that i execute in test mode) - but call it asynchronously.

------------------------------
Michelle Popovits
Technical Consultant
Reltio
ON
------------------------------