Reltio Connect

 View Only
  • 1.  Best practise for Larger file size

    Posted 05-15-2023 07:31

    Hi Team,

    This is regarding knowing best practise or ways to load source file with 150 columns with over 500k records.
    Was going through process which could give best outcome based on the load we pass to data loader.

    I tried to load it using dataloader application with file in local system and limitation we have is "application is not allowing to accept with file with above specification".

    Based on your experience, could anyone suggest an optimal way to achieve this.



    ------------------------------
    Chetan P
    Senior MDM Specialist
    Freshworks
    Chennai
    ------------------------------


  • 2.  RE: Best practise for Larger file size

    Reltio Employee
    Posted 05-15-2023 12:04

    Check this post out: Load Large entity Data into Reltio  and Best Practices for Loading Data into Reltio

    When dealing with larger file sizes in Reltio and trying to load a source file with 150 columns and over 500k records, there are a few best practices you can follow to achieve optimal results:

    1. Use Bulk API: Reltio provides a Bulk API for loading large datasets efficiently. This API allows you to send multiple records in a single request, reducing the overhead of individual API calls. It is recommended to use the Bulk API for loading large files.
    2. Split the File: If the application is not allowing you to load the entire file at once, you can split the file into smaller chunks. Splitting the file can be done based on record count or file size. This way, you can load the smaller files individually using the data loader.
    3. File Compression: Compressing the file before loading can help reduce its size and make it more manageable. You can use standard compression formats like ZIP or GZIP. Reltio supports loading compressed files directly.
    4. Incremental Loading: If your data allows it, consider loading the data incrementally instead of loading the entire dataset at once. This approach involves dividing the dataset into smaller batches and loading them in stages. It can help distribute the load and optimize the loading process.
    5. Data Validation and Preparation: Before loading the data, ensure that the file is properly formatted and validated. Cleaning up the data and removing any unnecessary columns or rows can improve the loading performance. Additionally, consider pre-processing the file to remove any duplicates or inconsistencies.

    It's worth noting that the exact approach may vary depending on your specific requirements, infrastructure, and the capabilities of your Reltio instance.



    ------------------------------
    Christopher Detzel
    Reltio
    ------------------------------



  • 3.  RE: Best practise for Larger file size

    Posted 05-16-2023 02:00

    Thanks you so much Chris. This helps !!



    ------------------------------
    Chetan P
    Senior MDM Specialist
    Freshworks
    Chennai
    ------------------------------