Reltio Connect

 View Only

  • 1.  SQS Message Limits

    Founding Member
    Posted 07-19-2023 06:36

    Hi All,

    As we know the maximum limit of a message size in SQS is 256KB, in case if there are entities which is greater than 256KB, is there a way to identify it ? 



    ------------------------------
    Gowthamprabhu Sivaraj
    AstraZeneca
    ------------------------------


  • 2.  RE: SQS Message Limits

    Reltio Partner
    Posted 07-19-2023 09:26

    Hi Gowthamprabhu,

    Not sure if you are referring to increase the size limit but you can check this documentation for increasing the size of SQS:

    Add an external queue configuration | Reltio

    Let us know if this helps



    ------------------------------
    Sheshadri V
    Fresh Gravity
    ------------------------------

    Original Message


  • 3.  RE: SQS Message Limits

    Founding Member
    Posted 07-19-2023 09:40

    Hi Sheshadri,

    Thank you. 

    I should have given more context, is there a way to identity the list of entities which are greater than 256KB in size ?



    ------------------------------
    Gowthamprabhu Sivaraj
    AstraZeneca
    ------------------------------

    Original Message


  • 4.  RE: SQS Message Limits

    Reltio Partner
    Posted 07-19-2023 10:25

    Hi Gowthamprabhu,

    Is there any consuming system if yes, you can try the size function and segregate the messages, and i don't think there is direct way from reltio to achieve this.

    Here is sample code in python:

    This gives in Kb

    Not sure if this is helpful in your case.



    ------------------------------
    Sheshadri V
    Fresh Gravity
    ------------------------------

    Original Message


  • 5.  RE: SQS Message Limits

    Posted 07-20-2023 10:00

    Gowthamprabhu.

    Without largeObjectsSupport set to "true", the events were being dropped by Reltio and never written to the event queue - there was no indication that this was occurring until we tracked down some inconsistent data.

    With the settings for Large Object Support, the json payload will be: 

    {"type":"ENTITY_CHANGED","exceededQueueSizeLimit":true}

    Then in code consuming the events handling of this payload can make necessary api calls to capture required attributed needed for event processing.

    When finallyrealizing we were losing messages we were able to work with support for identification of recent occurrences of the problem.

    Hope this helps.



    ------------------------------
    Mark Burlock
    Dodge Data & Analytics
    Hamilton NJ
    ------------------------------

    Original Message


Related Content