Script Parameters

Creating and inserting parameters into your code offers an easy and convenient way to make your script flexible and reusable: 

• Parameter “arguments” can be set for each script run to customize the result.
• If published as a template, parameters allow users of the template to configure it to their particular needs. 

Parameterization is available for every script type.

Important Terms

• Parameters: Parameters define runtime variables. Parameters are defined by their name and type, along with optional configuration options (see “Configuration” below for details).
  • In the API, parameters are referenced using the “params” attribute, which is an array of objects defining each parameter.
  • In order for parameters to be useful, the script’s author needs to reference them in the script’s code.
• Arguments: Arguments define the values for your parameters to use at runtime. These values can be set by the runner of the script, who may or may not be the author.
  • In the API, arguments are referenced using the “arguments” attribute, which is an object of key-value pairs. The keys should match parameters’ names, and the corresponding values are the runner-supplied arguments for that run.
  • For a given parameter name, the argument (aka “value”) must match the Parameter’s configuration (type, allowed values, etc).

Creating a Parameter

UI

To create a parameter via the UI, click on Set Parameters on the top-right section of the script page. 

Configure your parameter, then Add it to your parameters list. Once you’re finished adding parameters, Save your changes. 

API

To create a parameter via the API, use the “params” array. For example, to create a string parameter named “foo”:

import civis
client = civis.APIClient()
client.scripts.patch_python3(job_id, params=[
 {
   "name": "foo",
   "type": "string",
 }
])

Configuration

Parameter Names

Parameter names must start and end with a number or English letter. Only numbers, letters, and underscores are allowed. Parameters may NOT be named PATH, HOME, HOSTNAME, PWD, ON, CIVIS_API_KEY, DATABASE, RUNNER_USER, or RUNNER_USER_ID.

Parameter Types

Valid parameter types are the following: string, multi_line_string, integer, float, bool, file, database, credential_redshift, credential_aws, and credential_custom. Valid arguments for each of the parameter types are listed below.

Argument Types

• String - Any string value. Any "numbers" used will just be treated as the string representation of that number.
• Multi-line String - Any string value. Presents the argument for the parameter as a multi-line text box.
• Integer - Any integer value. Float values are not allowed.
• Float - Any float value.
• Boolean - Either true or false.
• File - Any integer value. Use the ID of the file in the Civis platform. Must be readable by the runner of the job. The file must not be expired when adding it as an argument and the job will fail if the file is expired at runtime.
• Database - A dictionary containing the keys 'database' and 'credential'. The values for these keys are the ID of the database and the ID of the database credential respectively.
• Redshift, AWS, or Custom Credential - Any integer value. Use the ID of the credential in Civis Platform, which can be found via the Credentials endpoint or UI.

Note that credentials of other types (e.g., “Google”, “Salesforce Client”) can’t be passed in as parameter arguments. To use such credentials as script parameter arguments, you will need to create “Custom” credentials.

Default Values

When creating a parameter, you can specify a default value for the parameter. This is the value for the parameter that will be used if no argument is supplied. Defaults cannot be set on required parameters since, by definition, an argument must be supplied for them. Defaults also cannot be set for fixed parameters, since fixed parameters use a constant value.

If a parameter is 1) optional, 2) has no default or fixed value, and 3) no argument is supplied for it, the following will happen:

• For SQL Scripts, an argument will still be set since valid SQL must be produced. Strings will be the empty string. Integers and Floats will be 0. Booleans will be false. Credential values (.id, .username, .password, .access_key_id, and .secret_access_key) will be the empty string.
• For Python, R, and Container Scripts, there will be no environment variable created.

Fixed Parameters

If you would like to define a parameter that holds a predetermined value which is not viewable or editable by custom script users, you can use a fixed parameter. 

Fixed parameters are most useful for affixing credentials and databases to your script. The values for fixed parameters are placed in environment variables in the same way arguments are. You can configure a fixed parameter using the “Fixed Value” field in the UI.

Notes: 

• Fixed files must be readable by the author of a template.
• When a script is templatized, fixed parameters will not be visible to the template or to custom scripts created from it.

API Example

To set a fixed parameter via the API, set the value attribute on the parameter to the value you would like to use.

The example code below shows how to create fixed credentials using a Python script. Note that when you create a database credential, json.dumps() is required.  

import civis
client = civis.APIClient()
params = []
params.append({'name': 'NUM_DOGS', 'type': 'integer'})  # a normal integer cred
params.append({'name': 'PRIVATE_CUSTOM_CRED', 'type': 'credential_custom', 'value': my_secret_custom_cred_id})  # a fixed custom credential
params.append({'name': 'PRIVATE_POSTGIS_DB', 'type': 'database', 'value': json.dumps({'database': my_secret_database_id, 'credential': my_secret_cred_id})})  # a fixed database credential
# now params is [
#    {'name': 'NUM_DOGS', 'type': 'integer'},
#    {'name': 'PRIVATE_CUSTOM_CRED', 'type': 'credential_custom', 'value': 6142},
#    {'name': 'PRIVATE_POSTGIS_DB', 'type': 'database', 'value': '{"database": 989, "credential": 1504}'}
# ]
client.scripts.patch_python3(job_id, params=params)

Allowed Values (Custom Drop-Down Parameters)

These are parameters that appear in the UI as dropdown lists of pre-selected choices. For example, you could create a custom drop-down parameter for 'Month' and have it appear as a list of the twelve months.

To create a custom drop-down parameter, click “Add Allowed Values” in the new parameter form. “Value Name” is the value that will be used in your code, while “Display Name” is the string that will show as an option in the drop-down.

You can also create allowed values via the API, example below: 

import civis
client = civis.APIClient()
client.scripts.patch_python3(job_id, params=[
 {
   "name": "month",
   "required": False,
   "type": "integer",
   "allowedValues": [
     { "label": "January", "value": "1" },
     { "label": "February", "value": "2" },
     { "label": "March", "value": "3" },
     { "label": "April", "value": "4" },
     { "label": "May", "value": "5" },
     { "label": "June", "value": "6" },
     { "label": "July", "value": "7" },
     { "label": "August", "value": "8" },
     { "label": "September", "value": "9" },
     { "label": "October", "value": "10" },
     { "label": "November", "value": "11" },
     { "label": "December", "value": "12" }
   ]
 }
])

On the script page, the parameter appears as follows: 

In this example, “January”, “February”, “March”, “April”, “May”, “June”, “August”, “September”, “October”, “November”, and “December” are the only valid (allowed) arguments for the parameter “Month”.

Required Resource Parameters

The resources allocated to a script can be set through parameters named REQUIRED_CPU, REQUIRED_MEMORY or REQUIRED_DISK_SPACE. If any of these parameters are present and arguments are supplied, the given values will be used to allocate resources. Otherwise, resources will be set based on the values in the script's requiredResources attribute. 

Note that parameters must be of the correct types (Integer for REQUIRED_CPU and REQUIRED_MEMORY and Float for REQUIRED_DISK_SPACE) in order to have an effect. In the Platform UI, these fields are accessible under Script Settings.

Editing Parameters

Once your parameter is created, you can edit:

• Input Display Name (‘label’)
• Input Helper Text (‘description’)
• Required (‘required’)
• Default Value (‘default’)
• Fixed Value (‘value’)
• Allowed Values (‘allowedValues’) 

You cannot edit: 

• Parameter Type (‘type’) or
• Parameter Name (‘name’) 

Type and Name are not allowed to be changed as these could break the functioning of your script. If you need to make changes to these fields we suggest deleting the existing parameter and recreating it as intended. You can rearrange how the parameters display on the script page by dragging and dropping parameters in the table in the “Set Parameters” side pane or by re-arranging the params list in an API request. 

Adding a Parameter to Your Code

Once your parameter is created, you'll need to reference it in your code.  Here are the ways to use parameters within the different script types. The variables available to the script vary based on the parameter type.

SQL Scripts

Variables are accessible to SQL code by wrapping them in double curly braces, e.g. SELECT COUNT(*) FROM some_schema.some_tablename WHERE id = {{foo}}. 

Once parameters are created for a SQL script, you'll see a “Generated Code” section below the editor that shows you the code as it will be executed.  This allows you to ensure that parameter arguments are applied as expected before running the script.

The following table shows the environment variables made available for a parameter "foo" of each type. Note: the parameter type to be used via the API is the name listed but lowercased, unless noted by parentheses.

Python, R, and Container Scripts

For these script types, parameter names are turned into environment variables. All environment variables will always be strings, however their types will be validated and you will be able to cast them to the parameter type. Environment variables are made available under both the name you enter, and the name in all uppercase letters.

The following environment variables are made available for a parameter "foo". Note: the valid parameter type to be used through the API is the name listed but lowercased unless noted by parentheses.

Historical Parameter Values

The History pane shows the argument values used on previous runs. To reuse them, click Copy Arguments to load them into the current run.

For additional run details (including git branch, docker image/tag, resource allocation, and code) use the API. The job inputs endpoint works across all job types; individual job types also have dedicated endpoints with more structured responses (e.g., python script inputs).