Civis Platform Release Notes 2020

Week ending in 12/25/2020

Updates

• We removed the column tooltips from the Table Details pane to improve readability.
• We added additional warnings in database sync jobs when the primary key used in the sync is not of a sequential data type.   For reference, documentation on primary key setting is available here.
• Shortcut Assume Role: Under the profile menu in Platform, we’ve added shortcut options to your most recently assumed roles from Admin Center in Platform. This significantly reduces the number of clicks needed to switch roles in Platform and retains the user's page while switching roles.
• Expanded Privileges when Assuming Role: Now when assuming a robot’s role you can create and edit credentials and generate API keys. Previously, this required users to login as the robot account rather than assuming its role, which can be painful because of shared passwords. Note that this requires the user to be marked as a robot in Admin Center.
• We improved the retry behavior for CASS/NCOA jobs to improve stability

Bug Fixes

• We fixed a database sync bug so that users with access to multiple Snowflake databases do not have issues running database syncs.
• We fixed links to CASS/NCOA jobs to point to the Enhancements page
• We fixed a CASS/NCOA permissions issue that caused some jobs with different users for author and “running as” to fail

Week ending in 12/18/2020

Updates

• We updated various dependencies to improve the long-term reliability of Identity Resolution and Civis Data Match, and we improved the matching algorithm’s use of frequency information related to first and last names.
• Documentation on Platform roles now available here.

Bug Fixes

• We fixed an export to SQL Server bug so that integer data types are properly mapped to SQL server.
• We fixed a couple Google Sheet export bugs such that:
  • When using the “Create Document” export option and the destination spreadsheet name is left empty, exports will not fail and still be able to save results.
  • When the destination spreadsheet is updated, export will save results to the updated spreadsheet.

Week ending in 12/11/2020

Updates

• IDR usage limit enforcement has been initialized.  Clients will receive email notifications as they approach and exceed usage milestones. For more information read IDR Usage Policy.

• The “Services” dropdown (located underneath the top-right initials menu) has been renamed to “Remote Hosts”.

Bug Fixes

• We fixed a SQL script but so that a runner’s access to input and output files are validated appropriately and said runner can access these files with proper access.
• We resolved an issue that caused duplicate “Failed to get outputs. Request failed with status code 503” error messages on Platform workloads. 

Week ending in 12/4/2020

No updates

Week ending in 11/27/2020

Bug Fixes

• We fixed Google Sheet Exports so that users can export SQL queries into google sheets up to 5 million cells

Week ending in 11/20/2020

Bug Fixes

• We fixed a CASS/NCOA bug so that users can access the output files quickly.
• We fixed a Google Sheet Export bug so that when users create or update the sync job of a Google Sheet Export, the proper credential is used.

Week ending in 11/13/2020

No updates due to Platform code freeze.

Week ending in 11/6/2020

Updates

• Documentation on how to interpret maximum memory & cpu usage now available here.
• Service reports can now be archived in Platform and via the API. To archive a service report in Platform, navigate to the report’s standard action menu and select “Archive”.

Week ending in 10/31/2020

Bug Fixes

• We fixed the “Add Repository” button in the add git repo side pane.

Week ending in 10/24/2020

Updates

• Updated the “Monthly Compute Hours” field on Platform Usage Overview to account for hours used on multiple instance types.

Bug Fixes

• We fixed a Redshift import bug when a system table returned 0 results.
• We resolved a Salesforce objects bulk API import bug which was creating columns that were too wide for Redshift's varchar column width.

Week ending in 10/17/2020

Updates

• Logging for maximum CPU used and maximum memory used is now available for python, R, and container scripts as well as notebooks and services.

Bug Fixes

• We fixed a Google Sheets Export bug so that the export will not fail when there are extra whitespaces after a semicolon.
• We fixed an expiration setting bug so that users can patch an unexpired file and set a NULL expires_at date.

Week ending in 10/9/2020

Updates

• We simplified the database import experience by removing deprecated setting Chunking Method from Database Import
• We updated our Civis Python API client to v1.15.0.  This update improves retry behavior that should help when network connections are unreliable and removes Python 3.5 support.  More information here.
• We updated the Civis Python Notebook, Python Script, and Container Script Docker Images
  • base datascience-python version → v6.3.0
  • civis-jupyter-python3 →  v2.3.0

Bug Fixes

• Resolved a bug causing the Import Multi from Civis script to fail because of invalid UTF-8 characters

Week ending in 10/2/2020

Updates

• Users can now view enhanced error messages for failed pipelines in the Identity Resolution tool. New troubleshooting documentation on Common IDR Errors can be found in Zendesk.

Bug Fixes

• Resolved a bug showing incorrect load errors on CSV file imports.
• Resolved a bug on database syncs so that users can now make constant updates to their source tables while incrementally syncing to Platform.
• Resolved a bug on CSV File file imports so that users will get correct error messages when a Redshift copy fails.
• Resolved a bug where a workflow execution task failed though the failed icon did not appear.

Week ending in 9/25/2020

Updates

• Users can now view enhanced error messages for failed pipelines in the Identity Resolution tool. New troubleshooting documentation on Common IDR Errors can be found in Zendesk.

Bug Fixes

• Resolved a bug showing incorrect load errors on CSV file imports.
• Resolved a bug causing the Import Multi from Civis script to fail because of invalid UTF-8 characters.

Week ending in 9/18/2020

Bug Fixes

• Resolved a bug so that users can view the full table name in Table Details, even if the table name is very long
• Resolved a bug so that database sync will not fail if the update_column contains null values

Week ending in 9/11/2020

Updates

• To improve client support and issue resolution on database syncs, we standardized our implementation logic on how we import and export data across databases.  

Bug Fixes

• Resolved a bug that automatically redirected users to a workflow execution page after navigating to a new pane in Platform.

Week ending in 9/4/2020

Updates

• Identity Resolution customers can now view their monthly pipeline usage via our new IDR Usage Dashboard found under Account in the upper right hand corner of the IDR homepage.
• New Script Template Detail Page allows users to manage template sharing and monitor Custom Scripts created from a template all in one place.
• Custom Scripts created from a template now have a clean new look.
• Custom Scripts in runner context now have resource fields under Settings (CPU, Memory and Disk Space). Previously, Custom Scripts inherited the resource settings from the backing script, but now have added flexibility to allow for resource optimization.
• We updated our workflows documentation to provide examples of passing parameters into scripts within workflows. More details available here.

Bug Fixes

• We fixed a bug preventing users from deploying services with max resources.

Week ending in 8/28/2020

Updates

• Users can hide executed queries in the query pane so that they can quickly sift through query history to locate relevant executed queries
• Security updates

Bug Fixes

• A Google Sheets Export job is not intended for executing an arbitrary number of SQL statements to prepare data for export.  Users now receive validation errors when their Google Sheet Export jobs contain more than one SQL statement.

Week ending in 8/21/2020

Updates

• We updated the show workflows pane so that users can now open any listed workflows on a new tab.
• Users can now sync tables whose names are reserved words onto their database.
• Database syncs now support SLL encrypted connections for Oracle. More details available here.

Week ending in 8/14/2020

Updates

• The R environments on Civis Platform will be upgraded to R 4.0.0 on 2020-08-10. It contains a breaking change to a default option when reading data into memory: following the upgrade, the stringsAsFactors option will be set to FALSE instead of TRUE in data.frame, read.table, and read.csv/read_civis. For more info, see this page.
• Workflows now support the action std.fail, which can be used to manually fail a task in a workflow. More details available here.
• Platform Usage Overview is now available to view and monitor your organization’s compute resources. More details available here.

Bug Fixes

• Fixed a bug that caused Workflow’s UI to be blank when using a conditional fail.
• Resolved a bug being raised in the POST /workflows endpoint that led to a “(500) An unexpected condition was encountered” error message.
• Removed double ‘add’ button on sharing pane.

Week ending in 8/7/2020

Updates

• Documentation on Encrypting Database Connections with SSL in Database Syncs now available here.

Bug Fixes

• Resolved a data type mapping error for INT and DATETIME data types when importing from Oracle databases.
• JDBC remote host urls can no longer start with whitespace. The extra whitespace previously led to database import errors.
• Sharing an import will now also share any associated syncs which are active.

Week ending in 7/31/2020

Updates

• All user data files across Platform will expire 30 days after they are created.
• Clicking an export link generated by selecting “export results” on the query page will now open in a new platform tab.
• Schemas without tables or views will now appear in search results within the data pane.
• Documentation on how to test a database connection in Platform is now available here.

Bug Fixes

• Resolved a bug with malformed parameters used when selecting the “Create Report” button. 

Week ending in 7/24/2020

Bug Fixes

• Removed a 404 error message that populated when attempting to populate schema or table drop downs for non-platform databases. Moving forward, users can enter free text to populate the schema and table fields for non-platform databases.
• Updated import and export logging to check whether child job(s) run(s) have failed prior to returning a “Successful Import” message.
• Included regex matching support for grants when users and/or groups are quoted.

Updates

• Users should now be able to open workflow execution links in new tabs either through right-click or cmd+click.

Week ending in 7/17/2020

Bug Fixes

• Users can now add import and export jobs (i.e. database export, CSV import, Salesforce import & export) to workflows via the UI graph view.
• When switching to a different workflow execution, parameters should now update and reflect the parameters that were actually used in the execution.

Updates

• The sample records view in Identity Resolution should now work when multiple phone number columns are mapped for a single source.
• Civis Data Match now avoids creating an empty output table if the job fails due to invalid configuration (e.g. if the input table doesn’t exist).
• Security and infrastructure updates

Week ending in 7/10/2020

Bug Fixes

• We resolved a bug that prevented users from being able to shut down a credentialed notebook successfully.
   

Week ending in 7/3/2020

Updates

• We added logging to SQL script and CSV export runs to increase visibility into use of cached exports and empower users to identify when there is an issue with their results. Now, users should be able to refer to the run logs and understand if a given run used a cached export as well as the date that the cached export was originally created.
• The single table scanner option has been removed from the admin dropdown. This functionality is now available in the Data Pane.

Week ending in 6/26/2020

Updates

• When a SQL Script or CSV Export runs and returns a cached run output, a log line is added to the run log with the date and time the cached output was created.
• We updated workflows so that the execute button becomes disabled when required parameter fields are either missing or invalid.
• Security Updates

Week ending in 6/19/2020

Bugfixes

• We fixed a bug in Google Sheet Imports and Exports that sometimes caused a new Worksheet selection to not save.
• We fixed a bug in Google Sheet Exports that sometimes caused the Worksheet to not clear before the data was written to the Sheet.
• We fixed a bug that sometimes caused existing Google Sheet Exports or Imports to not load. 

Week ending in 6/12/2020

Updates

• Version v1.14.2 of the Python API Client has been released. It includes bug fixes and documentation improvements as well as support for the latest version of Python (3.8). More details can be found here.
• Projects can now store more than 20 child projects.

Bugfixes

• We fixed a bug that caused extra white space padding on Tableau reports.

Week ending in 6/5/2020

Updates

• We updated our Google Sheet Import and Export jobs to reference the Worksheet and Spreadsheet by ID rather than name on the backend. Now, Import and Export jobs won’t break if a Worksheet or Spreadsheet name is changed and users can select Sheets with duplicated names. No action required by users to use this update.
• Security Updates

Week ending in 5/29/2020

Updates

• We deprecated support for specifying partitions in MySQL database syncs. 

Bugfixes

• We fixed a bug in SQL scripts and CSV exports that caused the delimiter to always be a comma, regardless of user specifications.

Week ending in 5/22/2020

Updates

• We updated our Google Sheet Import and Export jobs to support multiple Google credentials per user. Users can now select the credential they want to use from the credential dropdown rather than re-authenticate each time they want to use a different credential.
• We updated our Google Sheet Export to not create an additional blank Sheet when exporting data.
• We updated our export functionality to ensure that files exported using the export/files/csv endpoint have a file extension consistent with the specified compression and column_delimiter. This does not apply when force_multifile is true.

Bugfixes

• We fixed a bug that caused some Workflows to fail when running concurrently.

Week ending in 5/15/2020

Announcements

• Action required ONLY if you reference your Google credential ID in your code. For the vast majority of users who use Google Sheet Imports/Exports via the UI, no action is needed:
  • As part of the improvements we are making to our integration with Google Sheets, users who reference their Google credential ID in their code will need to make a one-time change. After an upcoming update on May 14, 2020, when a user re-authenticates their Google credential, the credential ID will change. Any references to the ID in code will break and need to be updated with the new ID.
• We have deprecated the /tables/{id}/enhancements/{enhancement_type} endpoints. We intend to remove them in 2021. Users should now use the /enhancements/{enhancement_type} API endpoints, which have the same functionality. For example, users should change their code to use the /enhancements/cass-ncoa endpoint instead of /tables/{id}/enhancements/cass-ncoa.

Updates

• We updated the Google authentication process for Google Imports and Exports so that the user only needs to validate their credentials once. Previously, the user had to validate the credentials twice.
• To clarify requirements for upserts in the CSV Imports job, we updated the job to fail earlier if the user did not provide both the primary and last modified keys when creating a new table.
• We added in-line validation for missing but required fields on Database sync jobs.
• We added support for S3 bucket names that include periods when importing or exporting data using the export/files/csv and import/files csv endpoints. 

Bugfixes

• We fixed a bug that caused database imports to fail when importing smalldatetime data types into Postgres databases.
• We fixed a bug that caused margin padding on Tableau public links.

Week ending in 5/8/2020

What’s New

• We added the data_updated_at date to the Data Pane and Table Details Page in Platform. This date is the last time that Civis Platform captured a change in this table. This date is only applicable for Redshift tables. Read more here. 

Updates

• We moved the table description on the Table Details page to be more visible - right below the table name. Users can add descriptions to their tables and use Platform search to find a table based on the description.
• We added a column delimiter (columnDelimiter) parameter to the imports/files/csv and exports/files/csv endpoints.
• When a user selects “truncate long text” for a MYSQL database export, the export now truncates to the max SQLSERVER limit rather than the max VARCHAR redshift limit to ensure the job runs successfully.
• We added a schema_updated_at field to the tables/id API endpoint. This date is the last time that Platform captured a change to the table attributes/structure. It is only applicable for Redshift tables.

Bugfixes

• We fixed a bug that prevented users from committing to github from Platform.

Week ending in 5/1/2020

Updates

• Workflows
  • Users now have the ability to only run one Workflow execution at a time. Users may want to choose this setting to ensure their more recent executions don't catch up to a previously-running execution and fail. Learn more here.
  • To prevent more manual work, canceling a Workflow now cancels all the associated jobs and action executions.
• Matching and Identity Resolution
  • We now support multiple phone number columns (e.g., home phone, mobile phone).
  • We made some changes to the matching algorithm to better handle first or last name values with spaces and/or hyphens.
• We updated our Civis Python API client to v1.14.0. This update removes support for Python 2.7 and 3.4 and adds an .outputs method to retrieve outputs from CivisFuture Objects. More information here.
• Security Updates

Bugfixes

• We fixed a bug that only allowed Workflow owners to update Workflow notifications. Now, anyone with manage or edit permission on the Workflow can update the Workflow.

Week ending in 4/24/2020

Updates

• We released our new files export API endpoint! This endpoint allows users to directly export data to their own S3 bucket and is more performant because it does not run a SQL script to export the data. Find out more here

Bugfixes

• We fixed a bug that sometimes caused the Enhancements index page not to load.

Week ending in 4/17/2020

Updates

• We updated the JDBC driver version for our MySQL database sync job in order to support MySQL 8 database versions. For users who use SSL parameters in their database URL, we automatically updated the settings to the supported parameters. More information here.

Bugfixes

• We fixed a bug that sometimes caused the run history pane and query results not to load. Now, users are able to open the run history pane, view expired files (but not download them), and click “view results” for queries greater than six months (although the actual results are deleted). Users can rerun the query to see the results.
• We fixed a bug that sometimes caused the Enhancements index page not to load.

Week ending in 4/10/2020

Updates

• We completed a rewrite of our state management system that enables Platform to handle more load and reduce the occurrence of stalled jobs, notebooks and service, and missed state notifications e.g. I cancelled my job but it still hasn't been cancelled. This is part of the ongoing stability project prioritized for 2020.
• We now allow users to update the file name and expiration date of S3 files at the Civis files endpoint. Details here.
• We updated our database sync so that only MySQL database syncs can run without a specified schema.
   

Week ending in 4/3/2020

Updates

• We updated our Google Sheet export to only export columns that include data when exporting to a new worksheet. This will help users not run into the Google Sheet cell limit because we are not exporting empty cells.

Bugfixes

• We fixed a bug that caused the table details links on the query tool to break if the table name was mixed case.
• We fixed a bug that prevented table statistics from being updated if the table had a column with an oid data type.

Week ending in 3/27/2020

Bugfixes

• We fixed a bug that allowed users to select "(Display limited to 50 search results)" in the table drop down for import and export jobs.

Week ending in 3/20/2020

Updates

• When adding a new database to Platform via the database sync, users can test the connection before running the actual sync job. We improved this testing experience by adding detailed error messages so users can understand what is incorrect (the username/password, JDBC URL format, etc.)

Bugfixes

• We fixed a bug in our Google Sheet Export that created a new spreadsheet when multiple spreadsheets of the same name were found. This now throws an error instead of creating a new spreadsheet.
• We fixed a bug that caused the update table details button to spin infinitely if the table was altered while the table details refresh was running.

Week Ending 2/24/2020

Updates:

• We updated our Facebook Import to output an aggregated page insights table, which is more streamlined and easy to use.
• We released a new version of the Import from S/FTP script with the ability to add a private key credential.
• We added the ability to filter for archived Workflows on the Workflow index page.
• Users can now add a name to their database when adding it to Platform via the Database Sync job.

Bugfixes:

• We fixed a bug that caused the table details page not to load because of a non-ASCII byte in the column name.
• We fixed an error in dbsyncs where BIT columns for Postgres databases would sometimes fail the sync.

Week Ending 2/18/2020

What's New:

• We now support Database Imports into non-Platform Redshift clusters.
• We now support the use of SSL key pairs for Database Imports with Postgres databases.

Updates:

• We’ve released a new version of civis-python, 1.12.1. This release includes a change to civis.io functions to allow users to pass in multiple files, along with some minor bugfixes For details, see the 1.12.0 and 1.12.1 release notes here: https://github.com/civisanalytics/civis-python/releases
• We have also updated the Python package versions for notebooks and scripts. This includes updates to data science packages such as scikit-learn, pandas, and tensorflow. For details, see the full release notes here: https://github.com/civisanalytics/datascience-python/releases
• If you use Python scripts or notebooks, you will get access to these changes by default.

Bugfixes:

• We fixed a bug that caused some Database Imports with BIT column types to incorrectly cast to boolean when their length was greater than 1.
• We fixed a bug in our Google Sheet Export to more consistently support more than 100k rows of data.
• We fixed a bug that prevented users from adding a Google Sheet Import to a Workflow in the UI.
• We fixed a bug in the Compute Usage Report that caused some jobs to appear to be running even after they had completed.

Week Ending 2/11/2020

Updates:

• We updated the CSV export created from the query tool to default to gzip compression, and we no longer surface the corresponding sql script in the UI.
• We added support for Postgres dates > 9999. The future is now!
• Users can now access the standard action menu for the last object in the projects detail page, which was previously blocked by the help widget.

Bugfixes:

• We fixed a bug in our CSV Import that mistakenly prevented mixed-case imports to Redshift.
• We fixed a bug that caused SQL scripts created via the API with the column delimiter set to a non-comma delimiter option to output files as comma delimited.
• We fixed a bug that caused multiple error messages to appear (rather than a single message) that prevented users from accessing certain areas of Scripts and Imports.

Week Ending 2/4/2020

Updates:

• We updated our email notifications to differentiate between failed and canceled jobs.
• In the Python API client, we updated dataframe_to_civis, csv_to_civis, and civis_file_to_table to support existing_table_rows=upsert and added the following parameters: primary_keys, last_modified_keys, escaped, and execution, which are all useful for upsert mode.
• Also in the Python API client, we updated civis_to_multifile_csv to include the max_file_size parameter, which allows users to control the size of their output files.
• We added a convenience method get_storage_host_id to retrieve the id of a storage host based on its name. This is to be used with client owned s3 buckets.
• We’ve updated the list of supported Notebook Docker images. We support v1.9 and higher for the civis-jupyter-python3 image and v1.8 and v1.10 for the civis-jupyter-r image. More information on supported Docker images can be found here.

Week Ending 1/28/2020

What's New:

• We've updated our Google Sheets integration to support Team Drives. This update includes a new interface to select the worksheet you want to import. No action is required to run existing jobs.

Updates:

• We removed the ability to create Python 2 Notebooks from the top navigation menu. We recommend users create Python 3 Notebooks moving forward.

Bugfixes:

• We fixed a bug that caused the running_as user to update on a script when a user clicked Run Now without intentionally switching the script to run with their credentials.
• We fixed a bug that sometimes caused a Workflow to fail if one of the tasks kicked off an additional Workflow.
• We fixed a bug that caused report viewers to be redirected to the home page if they navigated to a report URL in a new tab or window.

Week Ending 1/21/2020

Updates:

• We added scheduled and run state filters to the workflows index in the Civis API, enabling users to list and filter workflows by these attributes based on the most recent workflow execution.
• We made table scanner (an automatic job that populates the metadata on the table details page) more performant by removing unnecessary data calculations
• We updated SQL Scripts running on a Postgres database to not return an empty file if the result set only contains headers.
• We added support for the Oracle NUMBER datatype in the Database Import job.
• Security Updates.

Bugfixes:

• We fixed a bug in our Oracle Database Import that prevented connections for particular Oracle database versions due to Oracle issue ORA-01882.
• We fixed a bug that caused Google credentials to update from the job's running_as user when a person was assuming the role of another user.
• We fixed a bug that caused DATES to be mapped as VARCHARs when importing data to a Redshift database.
• We fixed a bug that caused auto-complete in the query tool to not display any tables if the schema name contained letters after numbers.

Week Ending 1/14/2020

Bugfixes:

• We fixed a bug that caused archived objects to show up in a Project when not filtering for archived objects.
• We fixed a bug that caused table scanner jobs to fail on Postgres clusters if the table included a column that only had NaN values.
• We fixed a bug so that our query autocomplete works with table names that include numbers.

Week Ending 1/7/2020

Updates:

• It's a new year, and we've updated our R images! We've updated to R 3.6.1 which contains the most recent tidyverse updates. Across notebooks, services, and scripts we've also added geospatial system dependencies and 30+ of the most commonly installed R packages for visualization, geospatial analysis, and modeling. The default shiny image now also includes 15 additional packages useful for shiny development and interactive visualization.
• We've added the following updates to the projects detail page:
  • Display last run information for enhancements (e.g. Civis Data Match)
  • Ability to download files through the standard action menu
• We added a max_file_size parameter to SQL scripts and CSV imports so that users control the max file size on parallel unloads from redshifts
• When creating or editing Certificate credentials there is now a multi-line entry form for the password
• We added include_header and column_delimiter parameters for preprocessing csv files so that users have more control over how the data types in their file are detected
• Security Updates

Bugfixes:

• We fixed a bug that caused sql scripts to fail when a single quote was escaped by a backslash or when a backslash was being used to escape another backslash and not a single quote
• We fixed a bug that caused a Platform redirect error once an object was removed from a project
• We fixed a bug where a valid JDBC url for an Oracle database could not be used for creating a remote host
• We fixed a bug with creating a table when performing a database sync to an Oracle database
• We fixed a bug that caused sql scripts executed with an unload that contained sql with a semi-colon as a string literal to fail
• We fixed a bug that caused sql scripts executed with an unload to return no results on Redshift clusters with concurrency scaling enable
• We fixed a bug where a table scan was using two connections to the Platform data warehouse instead of one