IAM Permissions for CSV Imports and Exports from/to S3

Overview

When using an AWS S3 Credential on Civis Platform with Imports/Exports from/to S3, there are a minimum set of IAM permissions that the AWS Access Keys must have.

There are two types of actions or permissions listed below; s3 bucket actions and s3 object actions. S3 bucket actions must be granted on the S3 bucket, not on a path below the bucket. S3 object actions can be granted on the bucket or a specific prefix (i.e., "bucket/prefix/*"). Do not forget to include the "*" wildcard on S3 object actions.

The "*" wildcard can also be used with actions to grant several at a time. For example, "s3:Get*" grants access to all S3 actions that begin with "Get". For more information on S3 and IAM please see AWS documentation.

Troubleshooting Issues

If you encounter an AWS S3 error during the course of your CSV import or export, it may be the result of a missing IAM permission or an existing IAM permission that does not have the required level of access. Please reach out to the system administrator that provisions your IAM access for assistance.

For additional troubleshooting, it is often helpful to use the AWS S3 CLI to validate the access provided by your AWS Access Keys for your S3 bucket and path. This way you can test specific actions, such as getting an object, putting an object, listing your bucket, etc. For more information on the available CLI commands see the AWS S3 CLI reference.

Specific CLI examples include:

• List your bucket - aws s3 ls s3://my-bucket
• List a path in your bucket - aws s3 ls s3://my-bucket/my/path/
• Get an object - aws s3 cp s3://my-bucket/path/file_exists.csv /local/path/new_file.csv
• Put an object - aws s3 cp /local/path/file_exists.csv s3://my-bucket/path/new_file.csv
• Copy an object - aws s3 cp s3://my-bucket/path/file_exists.csv s3://my-bucket/path/file_copy.csv
• Delete an object - aws s3 rm s3://my-bucket/path/delete_file.csv

CSV Imports from S3

The following IAM permissions are required.

Bucket Actions:

• ListBucket
• GetBucketLocation

Object Actions:

• GetObject

When importing multiple CSVs to Redshift, a manifest file must be created in order to import the files. After the import, the manifest is removed. As a result, the following Object actions are also required:

• DeleteObject
• PutObject

CSV Exports to S3

The following IAM permissions are required.

Bucket Actions:

• ListBucket
• GetBucketLocation

Object Actions:

• PutObject

When exporting data from Redshift, the default behavior returns a single CSV file. This requires combining the various S3 objects unloaded by Redshift. As a result, the following Object actions are also required.

For combining S3 objects:

• DeleteObject
• GetObject

For multipart uploads:

• AbortMultipartUpload
• ListMultipartUploadParts