Reltio Connect

 View Only
Expand all | Collapse all

Guidance on Data Ingestion and Consumption Between Snowflake and Reltio MDM

  • 1.  Guidance on Data Ingestion and Consumption Between Snowflake and Reltio MDM

    Posted 25 days ago

    Hello Reltio Community,

    I am currently working on a project where we need to set up data ingestion and consumption between our Snowflake database and Reltio MDM, and I'd like to get your feedback on the approach we're considering.

    Data Ingestion (Snowflake to Reltio):

    1. Initial Load:

      • Use an ETL tool to extract data from Snowflake and create files.
      • Load these files into Reltio using the Data Loader tool.
    2. Incremental Load:

      • Leverage the Snowflake connector in Reltio Integration Hub (RIH) to set up near real-time updates.
      • Snowflake updates happen in batch mode, with data changes often in the range of a few hundred thousand records per batch.

    Data Consumption (Reltio to Snowflake):

    For sending data back to Snowflake from Reltio, here's my understanding:

    1. Initial Export:

      • Similar to the initial data load, we can use an ETL tool to export data from Reltio and send it to Snowflake.
    2. Incremental Updates:

      • The Snowflake connector in Reltio Integration Hub (RIH) can be used to send incremental updates directly to Snowflake.

    Questions for the Community:

    1. Data Ingestion:

      • Is our proposed approach for data ingestion correct?
      • Some data architects are questioning the need to create files for incremental loads when real-time integration via the Snowflake connector is available. Are there better approaches or best practices we should consider?
    2. Data Consumption:

      • Can the Snowflake connector in Reltio Integration Hub handle sending both incremental updates and large initial exports effectively?
      • Are there alternative approaches for exporting data to Snowflake that are more efficient or commonly used?

    I'd really appreciate any insights, validation, or suggestions you can share to improve our understanding and implementation of this process.

    Looking forward to learning from your experiences!



    ------------------------------
    Rushy A
    ------------------------------


  • 2.  RE: Guidance on Data Ingestion and Consumption Between Snowflake and Reltio MDM

    Reltio Employee
    Posted 25 days ago

    Hi Rushyendar, 

    If I follow you correctly, your bullet, ''Leverage the Snowflake connector in Reltio Integration Hub (RIH) to set up near real-time updates" under the section "Data Ingestion (Snowflake to Reltio)", indicates possibly a misunderstanding about the Reltio Snowflake Connector.  The Reltio Snowflake Connector is unidirectional only, delivering data FROM Reltio TO Snowflake. It does not support updates from Snowflake back into Reltio. 

    Let me know if I've misinterpreted your  steps. 

    Curt



    ------------------------------
    Curt Pearlman
    Reltio
    ------------------------------