Reltio Connect

 View Only
  • 1.  Copying Subsets of Data between Tenants - Roadblock

    Posted 11-02-2021 15:08

    I have been asked to look into the feasibility of using the Reltio UI export/import feature as a method to get data from our Production tenant for all entities/rels from a search result (ie. 100 parties plus all related entities and relationships), download, and import it to our Development tenant.  The idea is that this would be useful for introducing good data to our DEV tenant (which is sadly lacking in this area and is also small so normal full data loads would not be feasible) .  I haven't been able to do this as I've been running into roadblocks.  So I thought I would throw it out to you...the community...to see your thoughts on if the road I'm trying to go down is feasible or not.


    As mentioned above the basic use case is to:

    1.  In Prod tenant, go to advanced search, create a result of up to 100 parties and at the top of the screen I click the little down arrow and export the data (JSON).
    2. Go to Dev tenant, use the load data icon on the search screen or the data loader in the hub console and attempt import there, create mapping…look at the fields to map...then I get stuck there.

    The main issues I'm finding so far are:

    • Importing the file (JSON) has a ton of fields….and looking at more than 500+ fields staring back at me in the data loader mapper is very intimidating (and this is just an Individual party entity).  I remember using the mapper in the basic reltio course way back when...but that's about it.  The thing that really bothers me about this...is that both tenants have the same definition...so why would I go through the pain of trying to map all those fields again.  The value of the data loader seems to be in bringing data from a different type of data source and then it makes sense that you provide the mapping.  I'm looking at data from the same type...just different tenants.
    • Once I get the party entities mapped...I probably would have to go and map the contract entities, then the various relationships.  I'm thinking I might possibly go insane at the thought of it.  Then I think again - maybe it's not so bad...maybe just fear of the unknown.
    • Ultimately, the thing I'm struggling the most with is my own lack of experience and the fear that I might go through a ton of work trying to make this work only to realize it's not really worth it and there were better options.  That's my main roadblock.  I'm hoping that someone in this community can give me some perspective as to whether this is an appropriate road to go down or not and if it is...provide some pointers to help alleviate the sense of dread I get whenever I look at that mapping page.  There may be a better way than what I'm thinking of - I sure hope there is.



    Other Options


    API for Individual Crosswalks - I already know how to load individual party crosswalks using the api via postman (get from prod tenant and load to dev tenant) and there's no mapping required of me for that.  However, I would need to copy and paste for every single crosswalk and execute the corresponding JSON API calls.  This may take a while but is pretty simple.  However, I've only done this for Party entities...I would need to figure out how to do this to get the corresponding relationships and eventually piece all the pieces together if I wanted to get everything related to a specific party (Party, Contract, Relationships).  I probably would need to do the same for the Reltio UI version...but I haven't quite gotten that far yet.  This approach would always require effort for every little bit of data...and so not likely useful for bringing in a few hundred parties and all relationships and related entities.  This I would likely go mad with this one as well for anything more than a few parties.


    Leverage Existing Process - We potentially have a third option that doesn't require us to use the Reltio API or the UI.  

    1. We already have a data warehouse filled with our exported reltio api and that gets refreshed daily.  
    2. We also have a load process built from an ETL PDI application that takes data in landing tables and loads them up to Reltio.  
    3. So My thought is just to build a bridge between reltio data from our data warehouse - based on input criteria and load those landing tables in our DEV environment and run a dev version of our load process to load to our DEV tenant.  Once this bridge is built then it can be reused with minimal effort for any amount of data we would want as often as we want.  Initial development effort would be needed but would pay off quickly.
    4. I really like this idea - however, I feel like I'm using it to avoid figuring out how to make the Reltio only solution work...so I don't feel good about going down this road until I've fully understood the feasibility of the original option (or other alternatives not yet thought of).


    ------------------------------
    Michelle Popovits
    Empire Life
    ------------------------------


  • 2.  RE: Copying Subsets of Data between Tenants - Roadblock

    Founding Member
    Posted 11-03-2021 17:00
    Hi Michele,

    Could you perform a filtered export out of prod as a CSV file, and then load it into your dev tenant with Data Loader?

    Regards,
    Walt

    ------------------------------
    Walt Feldman
    Solutions Architect
    Tenerity
    ------------------------------



  • 3.  RE: Copying Subsets of Data between Tenants - Roadblock

    Posted 11-04-2021 08:22
    Hi Walt,
    I'm thinking that exporting as a csv or  a json file would pretty much result in the same.....many hundreds of columns that would need to be manually mapped for multiple entity types.  That's the part that I find challenging especially since both tenants are of the same configuration...I keep thinking it shouldn't be this complicated.  It was presented to me that this approach would be an easy way to get data from one environment to another...and I'm not really seeing what's easy about it.  I keep thinking that I'm missing something somehow with all of this.

    ------------------------------
    Michelle Popovits
    Empire Life
    ------------------------------



  • 4.  RE: Copying Subsets of Data between Tenants - Roadblock

    Founding Member
    Posted 11-04-2021 09:04
    How many attributes do you manage per profile? Can you do a filtered export to just export out the attributes you want to populate in your dev tenant? I will also assume that many of the hundreds you are looking at are non-OV, and I assume you only want to load OVs into dev.

    ------------------------------
    Walt Feldman
    Solutions Architect
    Tenerity
    ------------------------------



  • 5.  RE: Copying Subsets of Data between Tenants - Roadblock

    Posted 11-04-2021 09:14
    Hi Walt,

    Generally it's not a huge amount and likely as you said - probably mostly from crosswalks and also postal address bits and pieces (otherwise how could I get over 500 count showing up in the mapping ui).    I'm not sure if bringing just the OV values would be as useful....especially if things like match/merge and survivor-ship rules are to be worked on and tested.

    ------------------------------
    Michelle Popovits
    Empire Life
    ------------------------------



  • 6.  RE: Copying Subsets of Data between Tenants - Roadblock

    Founding Member
    Posted 11-04-2021 09:18
    I recommend you perform an OV-only export and review the resulting data.

    ------------------------------
    Walt Feldman
    Solutions Architect
    Tenerity
    ------------------------------