Where to Find Errors?
To see IDR job’s errors:
- Go to the IDR Dashboard in Platform. (Tools > Enhancements > Identity Resolution)
- Click on the pipeline of concern
- Choose the run of concern if it is not the last run
- Error details should be visible in the middle of the page below the error diagram.
Error Sections of a Pipeline
ETL
This is the main job in the matching process. Many top-level errors such as invalid user inputs, or missing permissions will surface here. Below is the image you’ll see in Identity Resolution when encountering one of these errors. Some common errors from this pipeline area are PipelineRunFailure, CivisJobFailure, MissingServicePermissions, EmptySourcesError, LockedLimitException, and KeyError.
Matching
This is where the matching algorithm is processed. Data quality errors can sometimes surface here. Below is the image you’ll see in Identity Resolution when encountering one of these errors.
Common Errors
Below is a list of common errors that you may see during runs of an Identity Resolution Pipeline.
The errors are organized into a table of Error Type, the Error Message that you will see in the IDR UI or your job’s logs, a short Explanation of the error, and the Recommended Action for you to take.
| Error Type | Error Message | Explanation | Recommended Action |
| PipelineRunFailure | Pipeline [XX], Platform Job [XX], Run [XX] Failed. Error: [ERROR_DETAILS] Please reach out to support via the Help widget. | This is a generic error indicating that something went wrong while running your pipeline. There should be a more specific error in the ERROR_DETAILS or in the IDR UI for that pipeline. |
Visit the IDR App in Platform and view the error following the steps above |
| CivisJobFailure | None | This is a generic error that something went wrong while running your pipeline. There should be a more specific error in the IDR UI for that pipeline. |
Visit the IDR App in Platform and view the error following the steps above. |
| EMRClusterFailed | EMR cluster failed! [ERROR_DETAILS] | This is an error indicating something went wrong on the resources running your match job. This is likely an internal issue and not a user error. | Contact support |
| OversizedClustersException | Our matching algorithm skipped processing on extremely large cluster(s) with resolved id(s)/size(s): [CLUSTERS]. Large clusters can indicate quality issues with your source data. Please contact support for more details. |
One of the resulting clusters from your matching run was skipped due to it being too large (> 4,000 records). The rest of your job should have ran smoothly and you can find the results of your job minus the skipped cluster(s) in the outputs. We skip large clusters because the matching algorithm would not be able to parse through the cluster within the 12 hour time limit we have set for all matching jobs. |
When this error occurs there is usually an issue with data quality. For example, we've seen in the past 15,000 duplicate records for a single person in a client's data. You should leverage the Expanded Cluster Query to find the corresponding PII for the resolved id and backtrack from there. You can find the Expanded Cluster Query on the Run details page in the IDR App just below the error. |
| MissingServicePermissions | The user running this job does not have view permissions for the IDR service. Please ask Support to set up this permission. | Your user has not been given the correct permissions by our team to run this job. | Contact support |
| KeyError / ValueError / AssertionError / AttributeError | Pipeline run failed due to an error that occurred while parsing and validating your input: [INPUT_DETAILS] | One of the job’s provided inputs was invalid. | Validate your job’s inputs and check for typos or inaccuracies. |
| EmptySourcesError | We've detected that all of your sources have 0 records in them. Please make sure your sources have data in them. | The provided data sources did not have any records. | Validate your job’s data sources. |
|
LockedLimitException* *Will only start after Dec. 1, 2020 |
Your organization ([YOUR_ORG]) has exceeded its monthly Identity Resolution [runs | records] limit. Starting in December 2020, Your pipelines will be locked from running once this 'limit is exceeded. | You have surpassed the limit for the amount of runs or records your organization can use. | Contact Support to see what your options are for increasing your limit. |
Uncommon Errors
| Error Type | Error Message | Explanation | Recommended Action |
| CancelPipelineRunFailure | Failed to cancel run [RUN_ID] for pipeline [PIPELINE_ID]; HTTP Status code was [STATUS]; Error was [ERROR_DETAILS] | Someone or some process attempted to cancel your pipeline and was not able to. | N/A |
| CreatePipelineRunFailure | Failed to create run for pipeline [PIPELINE_ID]; HTTP Status code was [STATUS]; Error was [ERROR_DETAILS] | Something went wrong while trying to create a run of your pipeline. | Try running your pipeline again and if the error persists contact support. |
| EMRTimeout | EMR cluster [CLUSTER_ID] timed out waiting for [STATE] state after [TIMEOUT] hours. | The matching job has taken too long and timed-out (the limit is 12 hours). This could be due to large data sources, or too many resulting records from the matching process. | Contact Support |
| ValueError | A CivisJobFailure exception was raised when unloading the input Redshift table to S3: "[ERROR_DETAILS]" If the exception was not due to network or Civis platform issues (e.g., a 5xx error), please check that your input database + schema + tablename as well as the field mapping are all correct. The SQL query used to unload your input table: [SQL] Database: [DB_NAME] The parsed field mapping: [FIELD_MAPPING] | Something went wrong while moving your data between our services. If this error includes a 5XX network error, it may be due to a lapse in our network and could possibly be resolved by running again. | Try again when a network error is specified, contact support otherwise or if the error persists. |
Comments
0 comments
Article is closed for comments.