Reltio Connect

Hello Reltio Community,

I have a business requirement to delete any Interactions that are older than 90 days. Normally I would look to the Reltio API solution to delete the interactions, but we need to delete about 50 - 60k interactions per day. The Interaction Search endpoint is limited to 10k results (would also have to page through the offset) and can only delete 150 URIs at a time. Due to this we have been using a Qubole notebook and the Data Science API for bulk deletions: Data Delete API. The Qubole environment is configured and managed by the Reltio team.

However Qubole notebooks are getting migrated to Databricks towards the end of November. The Databricks solution will not work for our case due to HIPPA concerns. I'm now trying to brainstorm other ideas to do bulk interaction deletes. Some ideas we considered:

- GBQ Connector: this was first suggested but our shop does not have GBQ and is not willing to go down this route.

- RIH Recipe: looked at Reltio in app actions running on a schedule, but seems like it would have the same limitations as the API (limited interaction search results and deletions per API call)

- Custom Retention Policy: I'm not as familiar with this functionality but seems Reltio has default options with possibility of custom retention policies, Quota and Limits

- Use Data Science API outside of Qubole: not sure how feasible this is to set up locally within our own data science platform. Would need to determine how to enable/configure the Reltio Data Science SDK and script outside of the Qubole environment.

Does anyone else do bulk deletions, with interactions or entities? Any other ideas to successfully run bulk deletions on a daily schedule?

Thanks everyone!
Nate

------------------------------
Nathan Richter
Blue Cross Blue Shield North Carolina
------------------------------

Hi Nathan,

Either you can use RIH or a custom solution (like a Java or Python code) to bulk delete the interactions.

The APIs have a limit on the number of objects it returns at a time, but you can paginate and collect all the URIs before firing the Delete API.

You may also want to look at a custom JAVA utility to bulk delete entities, relations or interactions at this link. Its a open source utility, and so you can refer the code and customize it to suit your use case.

Hope that helps!

------------------------------
Thanks,
Snehil Kamal
Solution Architect
Reltio
Bangalore
------------------------------

Original Message:
Sent: 06-28-2022 09:29
From: Nathan Richter
Subject: Bulk Delete Interactions

Hello Reltio Community,

I have a business requirement to delete any Interactions that are older than 90 days. Normally I would look to the Reltio API solution to delete the interactions, but we need to delete about 50 - 60k interactions per day. The Interaction Search endpoint is limited to 10k results (would also have to page through the offset) and can only delete 150 URIs at a time. Due to this we have been using a Qubole notebook and the Data Science API for bulk deletions: Data Delete API. The Qubole environment is configured and managed by the Reltio team.

However Qubole notebooks are getting migrated to Databricks towards the end of November. The Databricks solution will not work for our case due to HIPPA concerns. I'm now trying to brainstorm other ideas to do bulk interaction deletes. Some ideas we considered:

- GBQ Connector: this was first suggested but our shop does not have GBQ and is not willing to go down this route.

- RIH Recipe: looked at Reltio in app actions running on a schedule, but seems like it would have the same limitations as the API (limited interaction search results and deletions per API call)

- Custom Retention Policy: I'm not as familiar with this functionality but seems Reltio has default options with possibility of custom retention policies, Quota and Limits

- Use Data Science API outside of Qubole: not sure how feasible this is to set up locally within our own data science platform. Would need to determine how to enable/configure the Reltio Data Science SDK and script outside of the Qubole environment.

Does anyone else do bulk deletions, with interactions or entities? Any other ideas to successfully run bulk deletions on a daily schedule?

Thanks everyone!
Nate

------------------------------
Nathan Richter
Blue Cross Blue Shield North Carolina
------------------------------

Hi Snehil,

Thanks for this information! I did look at RIH and using the API, but even when paginating it will be inefficient trying to delete 100 interactions at a time when we have to go through roughly 60k a day. We even have cases where more than 100k would have to be deleted.

I did not see this Java utility though, which looks promising! I'll have to dig further with this.

Another option we are starting to look into is the Snowflake connector. It looks like it might be possible to set up our interactions within a Snowflake data warehouse, then use a stored procedure to help with the bulk deletes.

Thanks again, appreciate the response!
Nate

------------------------------
Nathan Richter
Blue Cross Blue Shield North Carolina
------------------------------

Original Message:
Sent: 07-22-2022 06:16
From: Snehil Kamal
Subject: Bulk Delete Interactions

Hi Nathan,

Either you can use RIH or a custom solution (like a Java or Python code) to bulk delete the interactions.

The APIs have a limit on the number of objects it returns at a time, but you can paginate and collect all the URIs before firing the Delete API.

You may also want to look at a custom JAVA utility to bulk delete entities, relations or interactions at this link. Its a open source utility, and so you can refer the code and customize it to suit your use case.

Hope that helps!

------------------------------
Thanks,
Snehil Kamal
Solution Architect
Reltio
Bangalore

Original Message:
Sent: 06-28-2022 09:29
From: Nathan Richter
Subject: Bulk Delete Interactions

Hello Reltio Community,

I have a business requirement to delete any Interactions that are older than 90 days. Normally I would look to the Reltio API solution to delete the interactions, but we need to delete about 50 - 60k interactions per day. The Interaction Search endpoint is limited to 10k results (would also have to page through the offset) and can only delete 150 URIs at a time. Due to this we have been using a Qubole notebook and the Data Science API for bulk deletions: Data Delete API. The Qubole environment is configured and managed by the Reltio team.

However Qubole notebooks are getting migrated to Databricks towards the end of November. The Databricks solution will not work for our case due to HIPPA concerns. I'm now trying to brainstorm other ideas to do bulk interaction deletes. Some ideas we considered:

- GBQ Connector: this was first suggested but our shop does not have GBQ and is not willing to go down this route.

- RIH Recipe: looked at Reltio in app actions running on a schedule, but seems like it would have the same limitations as the API (limited interaction search results and deletions per API call)

- Custom Retention Policy: I'm not as familiar with this functionality but seems Reltio has default options with possibility of custom retention policies, Quota and Limits

- Use Data Science API outside of Qubole: not sure how feasible this is to set up locally within our own data science platform. Would need to determine how to enable/configure the Reltio Data Science SDK and script outside of the Qubole environment.

Does anyone else do bulk deletions, with interactions or entities? Any other ideas to successfully run bulk deletions on a daily schedule?

Thanks everyone!
Nate

------------------------------
Nathan Richter
Blue Cross Blue Shield North Carolina
------------------------------