The Export to S3 template exports data from Civis Platform to AWS S3.
The data to be exported can be specified by one of the following:
- A Civis Platform table
- Data from a Civis Platform SQL query
- A Civis file
The output of this template is a single file to be located at the S3 bucket and key that the user specifies. If the data to be exported is either a Civis Platform table or data from a Civis Platform SQL query, then the output is a CSV file. If the data to be exported is a Civis file, then the resulting file on S3 is simply an exact copy of the Civis file.
This template requires an AWS S3 credential on Civis Platform that has your AWS access key ID and AWS secret access key, with the appropriate “write” permissions for the specified S3 bucket and key. Please see here for creating an AWS S3 credential.
Getting Started
You can create your script using this link, or in Platform you can navigate to Data → Imports → More Imports.
Fill out the following parameters (required parameters are marked with a red asterisk in the template).
Parameters
- AWS Key Pair Credential (optional): A valid AWS credential on Civis Platform, with “write” permissions for the specified S3 location.
- AWS Amazon Role Name Credential (optional): A valid Civis Custom Credential with your Amazon role name and “write” permission on the specified S3 location. See this documentation for how to create one.
- S3 Bucket (required): S3 bucket to export data to.
- S3 Key (required): S3 key for the exported data. It is recommended to provide a key with a reasonable filename and its file extension. For instance, if your exported data is going to be a gzipped CSV file, then the file extension “.csv.gz” for a key like “this/is/my/key/data.csv.gz” is recommended.
- S3 Region (optional): AWS region of the destination bucket. The default is “us-east-1”.
- Schema and Tablename (optional): The “schema.tablename” of a Civis Platform table to export to S3. If provided, both parameters “Civis File ID” and “SQL Query” cannot be used.
- SQL Query (optional): A SQL query for data from Civis Platform to export to S3. If provided, both parameters “Civis File ID” and “Schema and Tablename” cannot be used.
- Civis File ID (optional): ID for the Civis file object to export to S3. If provided, both parameters “Schema and Tablename” and “SQL Query” cannot be used.
- Database (optional): The Civis Platform database to export data from. This is required if either “Schema and Tablename” or “SQL Query” is used.
- Database Credential (required if database is provided): The Civis Platform database credential to use with the selected database to export data from. This parameter is required when either “Schema and Tablename” or “SQL Query” is used.
- Include Header (optional): Whether or not to include the header in the CSV file to export. This parameter is only used when either “Schema and Tablename” or “SQL Query” is used.
- Delimiter (optional): The delimiter to use in the CSV file to export. The options are “comma” (default), “tab”, and “pipe”. This parameter is only used when either “Schema and Tablename” or “SQL Query” is used.
- Compression (optional): Which file compression to apply to the CSV file to export. The options are “none” (default) and “gzip”. This parameter is only used when either “Schema and Tablename” or “SQL Query” is used.
- Log Level (optional): Which level of logging to use. The options are “INFO” (default), “DEBUG” (more verbose), and “WARNING” (less verbose).
Comments
0 comments
Please sign in to leave a comment.