Reltio Connect

 View Only

  • 1.  What are the feature available in Reltio if we need to load 300 Million records as initial data load?

    Reltio Partner
    Posted 02-16-2023 10:17

    What are the feature available in Reltio if we need to load 300 Million records as initial data load? Normal load via data loader may take huge time.



    ------------------------------
    Abhijit Auddy
    Cognizant
    ------------------------------


  • 2.  RE: What are the feature available in Reltio if we need to load 300 Million records as initial data load?

    Reltio Employee
    Posted 02-16-2023 13:01

    Abhijit,

    There are no specific features that are activated when a data load crosses a specific threshold/size, but it should be noted that the Console Data Loader is specifically designed for scalability. In other words, you will not see a linear increase in the length of time required for a job to execute as the number of records/entities in the job increases. The Data Loader will analyze the job size and complexity during pre-processing and allocate resources accordingly. That said, we do often recommend as a best practice that giant data loads be "chunked" into smaller batches, just as a way to reduce the possibility of a point of failure impacting the entire load. In your case, for instance, it might make sense to break the 300M record job into, say, 6 jobs at 50M records apiece.



    ------------------------------
    Best regards,

    Jack Hain
    Senior Product Manager
    Reltio, Inc.
    ------------------------------

    Original Message


  • 3.  RE: What are the feature available in Reltio if we need to load 300 Million records as initial data load?

    Reltio Employee
    Posted 02-17-2023 11:26

    Hi Abhijit,

    As Jack says, the data loader is built for scalability.  A co-worker recently loaded 900M records with it.  In his case, he broke it into 90 jobs of 10M each and the loading took less than a two days.  If you choose not to use data loader, the ROCS utility is available too.  But don't think you can't do this in data loader.

    One caveat when using large input files will be to put the data in cloud storage vs. upload the data from your desktop.  That is probably obvious, but I want to put that out there.



    ------------------------------
    Gino Fortunato
    Senior Solution Engineer
    Reltio
    ------------------------------

    Original Message