Hi Rushyendar,
If I follow you correctly, your bullet, ''Leverage the Snowflake connector in Reltio Integration Hub (RIH) to set up near real-time updates" under the section "Data Ingestion (Snowflake to Reltio)", indicates possibly a misunderstanding about the Reltio Snowflake Connector. The Reltio Snowflake Connector is unidirectional only, delivering data FROM Reltio TO Snowflake. It does not support updates from Snowflake back into Reltio.
Let me know if I've misinterpreted your steps.
Curt
------------------------------
Curt Pearlman
Reltio
------------------------------
Original Message:
Sent: 01-23-2025 21:04
From: Rushyendar Akula
Subject: Guidance on Data Ingestion and Consumption Between Snowflake and Reltio MDM
Hello Reltio Community,
I am currently working on a project where we need to set up data ingestion and consumption between our Snowflake database and Reltio MDM, and I'd like to get your feedback on the approach we're considering.
Data Ingestion (Snowflake to Reltio):
Initial Load:
- Use an ETL tool to extract data from Snowflake and create files.
- Load these files into Reltio using the Data Loader tool.
Incremental Load:
- Leverage the Snowflake connector in Reltio Integration Hub (RIH) to set up near real-time updates.
- Snowflake updates happen in batch mode, with data changes often in the range of a few hundred thousand records per batch.
Data Consumption (Reltio to Snowflake):
For sending data back to Snowflake from Reltio, here's my understanding:
Initial Export:
- Similar to the initial data load, we can use an ETL tool to export data from Reltio and send it to Snowflake.
Incremental Updates:
- The Snowflake connector in Reltio Integration Hub (RIH) can be used to send incremental updates directly to Snowflake.
Questions for the Community:
Data Ingestion:
- Is our proposed approach for data ingestion correct?
- Some data architects are questioning the need to create files for incremental loads when real-time integration via the Snowflake connector is available. Are there better approaches or best practices we should consider?
Data Consumption:
- Can the Snowflake connector in Reltio Integration Hub handle sending both incremental updates and large initial exports effectively?
- Are there alternative approaches for exporting data to Snowflake that are more efficient or commonly used?
I'd really appreciate any insights, validation, or suggestions you can share to improve our understanding and implementation of this process.
Looking forward to learning from your experiences!
------------------------------
Rushy A
------------------------------