Reltio Connect

 View Only

  • 1.  Best practice to get all entities of a type

    Posted 11-03-2021 11:42
    Hi!

    We have an entity type named Asset. We have over 17k Assets, and I want to get all of them.
    Im using the api to get all the Assets, but when the offset hits 10 000, I get an error.

    GET {{URL}}reltio/api/{{tenantID}}/entities?filter=(equals(type, 'configuration/entityTypes/Asset'))&offset=10000&max=200

    {
        "severity": "Error",
        "errorMessage": "Internal application error.",
        "errorCode": 2001,
        "errorDetailMessage": "Internal application exception caught during executing operation Result window is too large, offset + max should be less or equal to 10000. Please view application log for details",
        "innerErrorData": {
            "exception": "com.reltio.metadata.errors.CommonException:Code: INTERNAL_SERVER_ERROR; Message parameters: [Result window is too large, offset + max should be less or equal to 10000]. Internal application error.. Internal application exception caught during executing operation Result window is too large, offset + max should be less or equal to 10000. Please view application log for details at com.reltio.search.CommonSearchAdapter.getSearchRequestBuilder, line 205"
        }
    }

    What is the best practice to get all the entities of a type?


  • 2.  RE: Best practice to get all entities of a type

    Reltio Employee
    Posted 11-04-2021 09:53
    Hi Minh,

    It seems like you are using the wrong API.  There is a bulk export API described here that is more appropriate for GETs of more than a couple thousand records.  I would past the help document, but its got several parts to describe all the features, so I'll just paste the link:

    https://docs.reltio.com/exportapi/exportapi.html

    Hope that helps,


  • 3.  RE: Best practice to get all entities of a type

    Posted 11-04-2021 11:10
    Hi Gino,

    The export api is sending a link of the json result to the registered user mail. I need the json result in the response.
    Do you know how I can achive that?


  • 4.  RE: Best practice to get all entities of a type

    Reltio Employee
    Posted 11-04-2021 11:30
    what are you doing that you need real time request for > 10k records?


  • 5.  RE: Best practice to get all entities of a type

    Posted 11-04-2021 11:43
    We have a datahub that need all the records. I need all the records in real time so i can map and customize it for the datahub.


  • 6.  RE: Best practice to get all entities of a type

    Reltio Employee
    Posted 11-04-2021 12:05
    For data movement of this type, you should be looking at ETL tools that can move the data in batch...

    If you really think you need to do this in real time, you should consult with your customer success manager for alternatives.


  • 7.  RE: Best practice to get all entities of a type

    Reltio Employee
    Posted 11-04-2021 12:05
    For data movement of this type, you should be looking at ETL tools that can move the data in batch...


  • 8.  RE: Best practice to get all entities of a type

    Founding Member
    Posted 11-04-2021 20:26
    Why not feed the entities into the datahub via the SQS queue?


  • 9.  RE: Best practice to get all entities of a type

    Reltio Employee
    Posted 11-05-2021 09:28
    Hi Minh,
    How often do you need to export all 17,000 records?   Is it really only the first time after all the Assets have been loaded into Reltio?  Perhaps you could manually trigger a batch process to act on the export API result file.   And then subsequent changes to the assets in the live PROD environment (assuming less than 1000 are being updated at any given time) could be handled through the streaming event architecture (SQS) that Walt mentioned.   
    You could always re-use the initial batch process whenever a full 17k record pull was needed.
    This is what our customers typically do.


Related Content