What are the feature available in Reltio if we need to load 300 Million records as initial data load? Normal load via data loader may take huge time.
There are no specific features that are activated when a data load crosses a specific threshold/size, but it should be noted that the Console Data Loader is specifically designed for scalability. In other words, you will not see a linear increase in the length of time required for a job to execute as the number of records/entities in the job increases. The Data Loader will analyze the job size and complexity during pre-processing and allocate resources accordingly. That said, we do often recommend as a best practice that giant data loads be "chunked" into smaller batches, just as a way to reduce the possibility of a point of failure impacting the entire load. In your case, for instance, it might make sense to break the 300M record job into, say, 6 jobs at 50M records apiece.
As Jack says, the data loader is built for scalability. A co-worker recently loaded 900M records with it. In his case, he broke it into 90 jobs of 10M each and the loading took less than a two days. If you choose not to use data loader, the ROCS utility is available too. But don't think you can't do this in data loader.
One caveat when using large input files will be to put the data in cloud storage vs. upload the data from your desktop. That is probably obvious, but I want to put that out there.
100 Marine Pkwy #275, Redwood City, CA 94065
+1 (855) 360-DATA+1 (855) 360-3282
© 2022 Reltio. All Rights Reserved
Site by eConverse MediaAlso of Interest: Master Data Management (MDM) Reltio Community Reltio Integration Hub Community, Data Management Community Matching, Merge, Survivorship Reltio Workflow Master Data Management (MDM)