Container Scripts

Civis allows you to run any code that you'd like on a dedicated server from within Civis.  We use Docker to create and maintain the environment in which your code will run.  This code can be parameterized and published as a template so that other users can run your code but enter their own inputs where necessary.  This allows your programmers to put their custom code directly in the hands of end-users.  Your code can also be run on a schedule, or connected to other Civis jobs like Imports or Models as part of an automated workflow.

Getting Started

From the Code menu on the top navigation pane, select Scripts, and click New Script in the top-right corner of the page. Find Container underneath Python.

Setting Up the Script

Click the script title to rename your script.

If you haven't already, connect your Civis account to your GitHub account.  This will allow you to pull code from any repository that your GitHub account has access to from within Civis.

In the GITHUB REPOSITORY URL, select the GitHub repository that your code lives in.  Select the appropriate branch in the GITHUB REPOSITORY REFERENCE field.

In the COMMAND field, enter the necessary command to run your code (in bash).  Code installed from GitHub is installed in the /app directory by default.  To run a script called "myscript.py", your bash command should include either

cd /app
python myscript.py

or

python /app/myscript.py

Additionally, there are /data and /tmp directories available to your script to store data during the execution if necessary.

You can use linux install tools to install any additional packages that are requirements for your script.  For example, if you need to use the python module SQLAlchemy in your script "myscript.py", you could use the following syntax:

cd /app
pip install SQLAlchemy
python myscript.py

If your code is parameterized you can reference those parameters as environment variables either in your script, or by passing them in as arguments from your command.  To reference them in your command, using the syntax "$foo" for a parameter named "foo", and if myscript.py takes two input arguments called "input1" ad "input2", and you had parameters called "foo" and "BAR" that you want users to pass into those inputs, the syntax would be the following:

cd /app
pip install SQLAlchemy
python myscript.opy --input1 $foo --input2 $BAR

The DOCKER IMAGE NAME field is the place to enter the name of the Docker image (note that this field is case-sensitive) that you want to use to create the environment in which your code will run.  Civis has several public images available.  To grab a particular image version, enter the version tag name in the DOCKER IMAGE TAG field.  You can leave this blank, the job will use the tag "latest".  

If you have your own Docker image that you'd like to access via a Container Script in Civis, see this page for more information: Using a Custom Docker Image with Civis Platform.

Finally, enter the processing capacity (CPU) you want available for your job and the amount of memory (MEMORY) you'd like.  These computing resources are dedicated to your organization, but are shared between different users within your organization.  This means that if two people attempt to run scripts that request 60% of the maximum available memory, the script that is submitted second will not run until the first one completes.  The maximum resources available may differ per organization, so please contact support@civisanalytics.com to determine what resources are available to you.

To run the script, click Run Now.  By clicking on the Run History pane, you will be able to see the log outputs for the job.  Civis will display all outputs written to both stdout and stderr in the log.

Environment Variables

Scripts run in Civis automatically inherit several environment variables at run time. The relevant variables provided by Civis are:

Other environment variables will become available based on the parameters you add to your script.

Outputs 

Depending on what your script does, you may have outputs that you'll want to associate with the script.  For instance, if your code generates a file that you want made available in Civis, you can use the Files endpoint to upload it to Civis and Run Outputs to associate the file with the script's run.

Variables in Notification Emails

You can embed values produced by your script in the script's success notification email. To leverage this feature, upload any values or tables you want to include in your email as a JSONValue run output named "email_outputs".

Open the Notify pane (paper airplane icon) and include {{variable}} in the Email Body text box where you want to input the corresponding value. If you are embedding a Markdown table, you need to include line breaks before and after the {{table}}. 

See here for a Python code example that provides query_to_variable and query_to_markdown_table functions, and demonstrates how to upload tables and variables to a JSON run output.