Import from Socrata

Overview

The Import from Socrata template allows you to connect to Socrata and pull open data to Civis.

In order to use the import from Socrata script, you will need:

• To access private data, authentication is required using Socrata’s HTTP Basic Authentication (this template doesn’t support OAuth 2.0). A Civis Platform custom credential should be set up with either your Socrata username and password pair, or a Socrata API key and secret pair.
• Though not strictly required, providing an application token is highly recommended to give you a higher throttling limit. A Civis Platform custom credential should be set up where the username is the Socrata app token and the password is the secret token.

Features

• Fetch data by providing a SODA dataset ID.
• Support filtering data by providing filter SOQL queries and also filter strings.

Getting started

Parameters

• SCRIPT MODE (required): There are four standard modes that the script can run against. They are:
  • run: Fetch dataset and write them to your destination database.
  • test connection: Test connectivity. Checks if authenticated calls can be made to Socrata.
  •  
• SODA API APP TOKEN (optional): Socrata SODA application token. Calls that don’t provide an app token or username and password will be subject to throttling limits.
• SODA AUTH USER CREDENTIAL (optional): Socrata username and password or an API key and secret
•  
• SODA DATASET ID (required): Socrata dataset unique identifier e.g 4ca8-mxuh.
• SODA API DOMAIN (required): Url domain for where to pull the dataset e.g https://data.lacity.org.
• SODA FILTER QUERY (optional): Simple optional filters to add when making calls. This can be as simple as using the column’s field as parameter and content to filter with as value (More info). You can also provide a query similar to an SQL query called Socrata Query Language(SoQL) for filtering (More info).
• DESTINATION DATABASE (required): Destination database to write data to.
• DESTINATION SCHEMA AND TABLE (required): Destination schema and table name.
• ACTION IF TABLE EXIST (optional): The behavior if a table with the requested name already exists.  If the table does not exist, it will be created for you. Options are:

• Fail: The sync job fails if the table exists. This is the default.
• Append: Records will be appended to the existing table
• Truncate: Records on the table will be wiped out while maintaining the current columns.
• Drop: Removes the entire table including column information.

• MAX ERRORS (optional): The maximum number of rows with errors to ignore before failing.
• LOG LEVEL (optional): Log level for output (can be DEBUG (most output), INFO (some output), WARNING (least output))