First, I need to thank my colleagues, particularly @Rakesh Yapamakala who helped me figure this out! This blog post could not have been done without him! Thank you, Rakesh!
In this blog post, I'm excited to share how to programmatically retrieve the error file for a Data Loader job in Reltio. Recently, I was working with a user to import data into Reltio using the Data Loader. As many of you know, Data Loader offers an intuitive UI to identify and map files to the Reltio data model before invoking the Reltio API for data loading.
Upon completing a Data Loader job, users can view the job's status and, if there are errors, download the error file to review what went wrong. This process works seamlessly when done manually. However, the challenge arises when the job is executed programmatically, as Reltio operates entirely through API calls.
I will guide you through the process of checking the response from the Data Loader API to identify any errors. This involves a two-step process using the Data Loader API errors endpoint. I'll explain how to retrieve the job ID and error file name, and how to use these to download the error file programmatically.
Let's dive into the details and make this task straightforward for your automation needs.
When a data loader job is completed, the user loading the file can then see the status of the completed job. If there are errors, the user can click on the name of the error file and then download it to see what the errors were.
That works great if there is a user involved. But what if the job was called programmatically? As you know, EVERYTHING in Reltio is triggered by an API call. How do we check the response from the Data Loader and see what errors happened (if any).
This is done with the data loader API errors endpoint. Getting the file is a two-step process. First: you need to know the job id and the name of the errors file. You get this information from the API: GET
https://361-dataloader.reltio.com/dataloader/api/{{tenantID}}/project/jobs?inProgress=false&page=1&size=20
inProgress=false is necessary to get just the completed jobs. The page and size parameters let you page through the data loader until you find the one you want. This is useful if you only want to check for errors periodically.
If you know the name of the job, you can use GET
https://361-dataloader.reltio.com/dataloader/api/{{tenantID}}/project/jobs?filter=like(name,Individual)&inProgress=false&page=1&size=20
to filter for the same.
Typically, you will know the name of the job (after all, you called it!) but remember that Reltio will add the word 'job' and a timestamp to what you called the job to prevent duplicate job names. So you'll be better off searching with the 'like' operator. In the example above, all jobs with the word 'Individual' in the name of the job are returned.
To call the error file API, you need the id of the job. There are several id's in the response from the first job. The one you'll need is at the top between the name and updatedDate property of the job object and the name of the error file, which is also in the return of the first API.. You can then GET
https://361-dataloader.reltio.com/dataloader/api/{{tenantID}}/project/job/{{JobID}}/errors?errorFile={{fileName}}
What this looks like in a 'real' payload is:
https://361-dataloader.reltio.com/dataloader/api/{{tenantID}}/project/job/0cf1ccaf-dfee-4179-94d5-f4421eb42ab4/errors?errorFile=testcase_samples_csv_dataload_1_6_Errors.csv
The response is the CSV file contents.
In conclusion, retrieving the error file for a Data Loader job programmatically in Reltio is a straightforward process once you understand the necessary API calls and parameters. By leveraging the Data Loader API errors endpoint, you can efficiently monitor and manage your data loading operations without manual intervention. This approach ensures that you can promptly address any issues that arise, maintaining the integrity and accuracy of your data.
Once again, a special thanks to Rakesh YL for his invaluable assistance in figuring out this process. With this guide, I hope you feel confident in implementing these steps to streamline your data loading workflows. Happy data loading!