In this blog, we will discuss 12 common questions that will help users understand the functions of the data loader application in the Reltio Identity 360 product. The data loader application comes bundled with Identity 360. Data loader is not a classic ETL tool, but it helps Identity 360 users to get started by loading file data from different sources and locations into Reltio Identity 360 hub.
Q1. Where can I find the data loader application?
Data loader is currently available directly on top of the search page screen once you log in. If you click the “cloud” icon as shown in the below picture, it will take you directly into the data loader:
You might need to provide your login credentials once more to access the data loader.
Alternatively, the data loader application is available in the Reltio console. If you click the chocolate bar icon on the top right corner of the Identity 360 screen, you will see the Reltio Identity 360 console option.
From the console screen, you will be able to see and select the data loader.
Q2. What is the functionality of the data loader?
Data loader helps to define and configure mappings and execute basic transformations so that data from input files can be loaded into the existing data model of Identity 360 tenant. The supported file formats are:
- CSV b. JSON c. Excel (.xlsx)
Q3. Do I need to place the input files in any specific location?
Once you get into the data loader - you will get the “Load Data” option under the job definition tab (on the left side of the screen), as shown in the picture below. Once you click “Load Data,'' you will have the option to load either “Entities” (person profile) or relationships.
Once you choose one of these options, it will show from which locations the input file can be read. As shown in the below picture, it currently supports:
- Local b. SFTP c. major public cloud providers (AWS, Azure, and GCP)
Q4. What are the steps to load records in the tenant ?
Please follow the below steps to load data into Reltio tenant:
- Select a file that you want to load with the correct delimiter type: comma, semicolon, single-pipe, and double-pipe are the available options.
2. Click on continue and create a new mapping (or use an existing one) to map columns in the file to attributes already present in the individual entity type.
3. Now we want to map crosswalks and attributes. For crosswalk - add a new crosswalk type (crosswalk types are a representation of the source from which the data is originally being loaded (e.g., Salesforce CRM, Microsoft Dynamics, SAP, etc.) and select an attribute from the incoming file to be the primary key of that crosswalk.
4. Similarly, go to the “Attributes” tab under mapping and map columns from incoming file to attributes of the I360 data model like First Name, Last Name, city, state, zip, email, phone, etc.
5. Once you are done with the mapping, click continue, and it will ask you for your preference regarding the records getting updated. If we are loading data for the first time - we can choose full. For subsequent loads, the significance of each option is explained later in this blog.
6. Once you click continue, the review section shows the overall picture of the data load, including the total number of records and the count of mapped/unmapped columns/attributes:
7. Once you click on “Load Data,” the job gets processed. Once the job is completed, the overall status will be visible in the job status tab.
8. Next, it’s time to load relationships between individuals. The process is similar - we have to define the relationship type that we are loading and crosswalk for the start object and end object.
Watch the quick Youtube video to understand the functionalities of the data loader:
Q5. What are the update options for records which are getting reloaded into the tenant?
There are two options available while you are trying to reload profile data that already exists:
- Full update: For all attributes, current file values will replace existing tenant values. So if the current file has some missing profile attributes, then those attributes will be null after the current file is loaded.
- Partial Update: Only the attributes which have a value in the current file will be loaded. If the attribute already has a value populated, then the new values will replace the existing values. Otherwise, the new attribute value from the current file gets added to the existing record. If the current file has missing attributes, then those attributes will remain as is in the tenant.
Q6. Is there any feature difference for data loaders between I360 Free, I360 Premium, or Reltio Enterprise Platform?
Feature-wise, the data loader application is the same for all of the above products.
Q7. Can I schedule a data loader job?
During the “define” phase of the load job configuration (stage 3), we can enable “recurring job definition,” which schedules data loader jobs. This is possible if the source file location is AWS, Azure, GCP, or SFTP.
Q8. What if I have multiple addresses in my input file for a single individual?
Address is a nested attribute in the I360 data model. As a result, while configuring the mapping, we can use the “+ADD” link/button (as shown in the picture below) to provide an additional instance of the same nested attribute so that all the incoming addresses can be mapped.
Q9. What are the transformations supported in the I360 data loader?
In I360 free version data loader - address (North America), email, and phone cleansing transformations are built-in and supported out of the box. You can’t add any custom transformations as of now in this product.
Q10. Can we load interaction data in I360 using the data loader?
Currently, the data loader only supports loading entities and relationships. Loading interaction data is not supported at this time.
Q11. What approach should I follow in the data loader if I have separate files for related data, such as phone numbers, email addresses, or identifiers for individual data?
Assuming that you have multiple files containing related data for the same source - the same crosswalk will be used for the records. In such a case for the first file, let's say containing id and address, you can select the full data load on the “define” step of the data loader. When the remaining files are loaded, say id and phone or containing id and email, you need to select Partial data load on the define step. This makes sure the new data for phone, email, etc., are progressively added to the profile.
The other scenario is the additional data like phone, email, etc., are coming from different sources, and you want to capture those sources as well. In such a case, you need to have the id and source (crosswalk) from the first file (containing id and address) as part of the new file (containing phone number and crosswalk of the new source). The crosswalk value from the first file will be for searching the record, and the new crosswalk values, along with phone numbers attributes, will be added to the profile. You can achieve this while creating the mapping for the additional data. When you load the file containing the phone numbers/emails, you need to map the old crosswalk value under the crosswalks tab and mark it as a contributor provider. You will also need to map the new crosswalk and mark it as a data provider. This will make sure the email or phone number is appended to the profile.
Q12.What happens if I load more than 10,000 records into my instance?
Reltio Identity 360 is licensed for free based on a soft limit of 10,000 consolidated records. Your Identity 360 instance will still keep working just fine if you load more than 10,000 records, but you will be in violation of the license terms for the free version. There are many legitimate cases where you may start with more than 10,000 records. For example, you may load 50,000 or more records across various sources into your Identity 360 instance at the outset, with the goal of consolidating down to 10,000 or less. In this case, even though you are starting with more, your goal is to remain at or below the 10k limit, so you are good. If your use case is such that you will end up with more than 10,000, you can simply upgrade (button directly in the UI) at a very low cost to manage more data.
If you have additional questions, please post them in the Q&A section of the Identity 360 Community, and a Reltio expert will circle back to ensure that no question is unanswered.