DHIS2 Integration with M-DIVE (April 2021)

This documentation walks through the usage of the DHIS2 API integration with M-DIVE. This integration is a prototype designed to help users transmit data from a DHIS2 instance directly into M-DIVE as part of the PMI Quarterly Report. Feedback on ways to improve this script and associated documentation are greatly appreciated. 

DHIS2-Table Prior to API import

Step 1. Filtering Malaria Data Elements only

The use of the DHIS2- data dictionary tool to identify every country's specific malaria metadata can be accessed by using the DHIS2 login credentials (Username and Password) as provided by the National Malaria Control Programme (NMCP) or Ministry of Health department responsible for HMIS. These credentials can then be used to connect with R-shiny app interface to DHIS2 to download all the Malaria Data Elements and its metadata. 

Step 2. Creating DHIS Analytic Tables in DHIS2 (HMIS)

The downloaded NMCP data elements from the data dictionary tool will be subjected to review and mapping with PMI data elements for validation. After this validation by the NMCP and PMI country team, a Malaria specific table will then be created within DHIS2 using the same login credentials. The creation of this desired table can be done using the DHIS2 Table environment window as shown below.

Take note that the pivot table to be generated within DHIS2  must include three key attributes. Data, Periods and Organisational Unit  as shown below.

Data: This is where you will select all the relevant malaria data elements to the right side as shown above.

Period: This is where you will select the 3 months period to the right side as shown above.

Above is Organisation Unit:  This is where you will select levels and you will see a drop down that allows you to click-on District ( or LGA). Since that is the lowest level of HMIS data to be collected for PMI reporting.

Once you have utilized Data, Periods and Organisational Unit, then you will click Update to generate your pivot or analytics table as highlighted below:

Step 3. Saving the Table for M-DIVE API import

Once the standard table has been created, it will be saved using the favorite icon on top. This table will now be permanently available for re-use in any future PMI Quarterly report. However it must be updated to capture the Month period every quarter. Below is a sample of the standard table saved.

Step 4. Creating the API Call URL for MDIVE

The saved favorite will be opened every quarter in order to create the API url  that will be used to connect within the MDIVE notebook script. The API url can be generated by going to Download[1], THEN moving down to Advanced:[2], THEN to JSON:[3]  ( under the Data value set ). CLICK on the sub-section called “JSON” as highlighted in the diagram below:

After clicking JSON, the API URL window will open with several lines of data values below it; then we are ready for MDIVE API import.  The API url window will be displayed as shown below in red highlight - copy only the URL.

The several lines of data values to be imported will appear below the API url address as shown above for the API import.

This completes all actions needed in DHIS in order to perform the API import in the MDIVE environment. Remember to use the PMI-MDIVEs specific log-in credentials and follow the next set of instruction below on “How to use the API import”

How to use the API import 

Step 5. Adding a Custom Credential to M-DIVE

Before using the API import, the user must have created or know the name of the “DHIS2 analytic-table” desired to be transmitted from the DHIS2 instance to M-DIVE (see above documentation on how to create the DHIS2 analytic-table with malaria data elements using the Data Dictionary Tool).Once this is known/identified, the user must add their DHIS2 login information as a Custom Credential on M-DIVE. The login information is stored in this way, as a credential object, for security reasons. Only the user who added the custom credential has access to it by default.

To create a new stored credential, click on the “Admin” tab in the upper right-hand corner of M-DIVE, select “Credentials”, and then click “Create Credential” to create a new credential object.  For more information on creating credentials in M-DIVE, see additional documentation here. 

After navigating to a “Create Credential” creation screen, users should enter the following information for the credential object:

• Name: this field will be the name of the credential in M-DIVE, and it can take on any value. A descriptive value (e.g. “Ghana DHIS2 Credential”, “Benin DHIS2”, etc.) is encouraged to help users distinguish their different credentials, but this will not affect the credential’s use.
• Credential Type: users should select ‘Custom’ from the dropdown menu
• Username: users should enter their DHIS2 username as if they were logging in to the DHIS2 instance
• Password: users should enter their DHIS2 password as if they were logging into the DHIS2 instance 
• Description: this field is optional, users can use it to help distinguish different credentials 
• Remote Host: Leave this dropdown selector blank 

Step 6. Accessing the DHIS2 API import

Within M-DIVE, using the bar at the top of the page, navigate to “Code” and then choose “More Script Templates” from the menu that appears. 

Using the pane which appears to the right, search for “DHIS2 API” in the search box. 

Next, click on the template titled “DHIS2 API Extract”. This will create a new script using the template. The page should have a URL like https://platform.civisanalytics.com/spa/#/scripts/custom/XXXXXXX. Here users will enter the relevant information for the DHIS2 API import.

This page will look something like this: 

The default name “DHIS2 API Extract #XXXXX” may be re-named for easier discovery of the script.

Step 7. Using the API import 

The API import from DHIS2 to M-DIVE expects the user to input information into two required fields and two optional fields, which control what data should be accessed, where data should be stored, and what the user intends this data to contain. 

DHIS2 API Call: This parameter is the raw API call (as a URL) which should be used to access the data. The script will attempt to use this to extract data from the specified DHIS2 system as a json or a csv, depending on the format specified in the URL. 

DHIS2 Credential: This parameter expects a Custom Credential, a type of M-DIVE credential object, which has been added to this user’s account. This credential should hold the username and password used to access the DHIS2 instance in question (see “Adding a Custom Credential to M-DIVE” above for more information on adding a credential). Select the name of the credential to use from the drop-down menu.  Please note that these credentials will need to be kept up to date; if your credentials in DHIS2 change, please update your credentials stored in Platform as well.

Destination Table Name: This parameter allows the user to specify a location in M-DIVE’s SQL database where a copy of the imported data can be made available for their personal use. If a table already exists at the location specified, it will be dropped and replaced with the new data.  Suggested naming: schema_name.[data type]_[start quarter]_[end quarter], e.g. `laos_staging.hmis_2020q1_2020q2`.  

Comments: This parameter is an open text box that allows the user to make a note about what information they intend for this data submission to contain.

Once these fields have been populated with the desired values, click “Run now” in the top right corner of the page to run the script. Note that the script will take a few minutes or more to finish, depending on the size of the dataset. 

How the API import works

The import script uses the API call specified in the DHIS2 API Call argument to extract the data from DHIS2. It stores this data in M-DIVE as a file, and it uploads the data to a table in M-DIVE’s SQL database. 

Based on the group membership of the person running the script (typically their country), this script also adds the file to a project in M-DIVE which holds all DHIS2 imports and grants the country team access on the table created. 

The script in M-DIVE will store the files imported, together with the associated API calls against DHIS2, as outputs. These are accessible by clicking on “Run History” on the right side of the page. 

Next, click the arrow to expand details about a run, and these items will be accessible under the “Outputs” section of each run for which they were ingested.

The files, stored as .csv’s, are persisted in M-DIVE. They are available as the “Run History >  Outputs” of the script and in the project [country name] Data Imports. For PMI HQ users, files will be available in the PMI HQ Data Imports project. The API call that was used to access that data is also stored in the script output for visibility and reproducibility. Data copied from DHIS2 using this script will automatically be made available to the M-DIVE Civis Team for potential inclusion in PMI’s Malaria Quarterly Report.

Quarterly Import from DHIS2

M-DIVE users can use the same script to import DHIS2 data into M-DIVE on an automated schedule. The credential and the parameters in the script are stored securely in M-DIVE and do not need to be modified unless the DHIS2 credentials or other configurations change.