Importing Cut Turf Data from NGPVAN

The NGPVAN: Import Saved Lists script template allows users to import data from cut turf in NGPVAN into Platform. With the two modes of this script template, users are able to 1) use a VAN folder ID to import the IDs and metadata of the folder’s contents (map turfs) and 2) use the metadata from part 1 to import the VAN IDs within a given turf. This allows users to audit the turf cutting process and report on progress once canvassing has begun.

To import cut turf data, we recommend using the following workflow which will work with VAN folders having less than 250 lists/turfs (simply copy and paste the YAML code below into the configuration view of a new workflow). Note that the workflow will fail with VAN folders having more than 250 lists/tufts. Essentially, this workflow has two parts and will create NGPVAN: Import Saved Lists jobs in each part. If you copy and paste this YAML, the following jobs will be created for you:

1. Use List Metadata Import mode in the NGPVAN: Import Saved Lists template (which takes a VAN folder ID as a parameter) to import the names, descriptions, counts, and map turf IDs (which happen to be list_ids) for all of the turf saved in the folder. 
2. Use Saved List Import mode in the NGPVAN: Import Saved Lists template to get a list of the VAN IDs within each turf (from each list with an ID in list_ids).

version: '2.0'
workflow:
  input:
    - ngpvan_credential:
    - ngpvan_mode: "1"
    - cluster:
    - cluster_credential:
    - metadata_table:
    - turf_table:
    - table_setting: append
    - folder_id:
  tasks:
    get_folder_metadata:
      action: civis.scripts.custom
      input:
        name: Import Folder Metadata
        from_template_id: 36552
        hidden: true
        arguments:
          CLUSTER:
            database: <% int($.cluster) %>
            credential: <% int($.cluster_credential) %>
          MODE: list_metadata_import
          OUTPUT_TABLE: <% $.metadata_table %>
          OUTPUT_TABLE_SETTING: <% $.table_setting %>
          NGPVAN_MODE: <% $.ngpvan_mode %>
          NGPVAN: <% $.ngpvan_credential %>
          FOLDER_ID: <% $.folder_id %>
      on-success:
        - import_saved_lists
    import_saved_lists:
      action: civis.scripts.custom
      concurrency: 20
      with-items: result in <% task(get_folder_metadata).result.outputs.where($.name = 'list_data').single().value %>
      input:
        name: Import Saved Lists
        from_template_id: 36552
        hidden: true
        arguments:
          CLUSTER:
            database: <% int($.cluster) %>
            credential: <% int($.cluster_credential) %>
          MODE: list_import
          OUTPUT_TABLE: <% $.turf_table %>
          OUTPUT_TABLE_SETTING: <% $.table_setting %>
          NGPVAN_MODE: <% $.ngpvan_mode %>
          NGPVAN: <% $.ngpvan_credential %>
          LIST_ID: <% $.result['savedListId'] %>

The parameters for this workflow are:

• ngpvan_credential: NGPVAN credential ID - can be found under https://platform.civisanalytics.com/spa/#/credentials; see Adding an NGPVAN Credential for instructions on how to create an NGPVAN credential
• ngpvan_mode: “0” for MyVoters, “1” for MyCampaign
• cluster: name of the destination cluster/database for imported data
• cluster_credential: database credential ID - can be found under https://platform.civisanalytics.com/spa/#/credentials
• metadata_table: output table to store turf metadata
• turf_table: output table to store turf VAN IDs
• table_setting: option for existing table rows - either “append” or “drop”
• folder_id: VAN folder ID for folder holding turfs

Please note that one of the steps in the NGPVAN hierarchy (Folder → Map Regions → Map Turfs → VAN IDs) is not available. In the current VAN API there is not a way to connect Map Regions to both folders and Map Turfs. That said, the naming conventions of Map Turfs will enable you to know which Map Region a turf belongs to.