File Import

Civis allows you to import a CSV to the schema and table of your choosing in Redshift. 

Getting Started

1. On the top navigation menu, click Data, and then under Imports, select "File."

Selecting a Source

Upload a file by either dragging-and-dropping the file from your computer onto the page or clicking on the Upload File button to choose your file through the file explorer.

Selecting a Destination

Once your file has been successfully uploaded, specify the schema and table name you would like to import the data into.

Import Options

After uploading the file to your import, advanced options will appear below the sync.  You can read full documentation on these advanced settings here.  These options include seeing a file preview, specifying the maximum number of error rows, or truncating the destination table if it already exists. It's important to note that by default the "If Table Exists" setting will be set to "Fail," so if you're importing into a file that already exists you'll want to change this setting appropriately.

In the Table Preview section, you are able to preview what your data look like after the import. Please note that the data types presented in the preview are Civis' best guess based on a sample of data in the import file. You can click on the column header to override the data types.

You can also edit column names in the Table Preview section. Column names should only contain letters, numbers (cannot start with a number), underscores, and dashes. Civis will automatically notify you if any column names need to be adjusted to ensure a successful import.

Running the Import

Once you've completed the setup of your file import, click Run Now in the top-right corner of the page to start importing your file into Redshift.

File Restrictions

The following guidelines minimize the time it takes to load files and reduce the risk of data loss due to load errors. Civis can accommodate other file formats, but they may require additional effort and time to load.

Files should be a flat file- with no nested hierarchy (e.g., XML, JSON, etc.).

 Valid

“1”|“John Doe”|“5/20/2000”

“2”|“Jane Doe”|“6/10/2004”

Invalid

People

     “1”|“John Doe”|“5/20/2000”

     “2”|“Jane Doe”|“6/10/2004”

Birthdays

     “1”|“5/20/2000”

     “2”|“6/10/2004”

Platform accepts the following file formats:

1. Delimited file with all fields enclosed with double quotes
   1. Any double quote appearing within a field should be enclosed in additional double quotes or escaped with a backslash
   2. Examples:
      • Valid: “1”|“John Doe”|“5/20/2000”|“This is some ““plain”” text”
      • Valid: “1”|“John Doe”|“5/20/2000”|“This is some \“plain\” text”
      • Invalid: “1”|“John Doe”|“5/20/2000”|“This is some “plain” text” 
2. Delimited file with all special characters escaped with a backslash
   1. Special characters - delimiter, single quotes, double quotes, newline character (\n), carriage returns (\r)
   2. Examples:
      • Valid: 1|John Doe|5/20/2000|This is \| someone\’s \“plain\” \\n\\r text
      • Invalid: 1|John Doe|5/20/2000|This is | someone’s “plain” \n\r text
   3. If your file is escaped with a backslash, please select Yes for Backslash Escape Parameter under Advanced Settings.

Delimiters: pipe, comma, tab (in order of preference)

Compression:

1. gzip with no archives (e.g., tar)
2. zip containing only one file
3. bzip2

Password Protected Files: Not supported

Encoding: several encodings are supported assuming a valid BOM (byte order marker) at the beginning of the file

Character Sets: Valid UTF-8 characters up to 4-bytes

Recurring File Delivery:

For recurring deliveries of the same file, the format and structure should not change. This includes adding/removing columns, changing column order, changing data types of columns, and any other formatting options that might affect the output.

If changes are required to the file, the destination table will first need to be updated before the new format can be loaded.