Articles in this section

See more

Release Notes

Avatar
Alex Marino
Updated
Follow

Week Ending 09/21/2024

Updates

  • Made improvements to the Query tool to improve the experience when pasting in large queries.
  • Improved clarity of the manage versions experience when setting up a new Github connection.
  • Added additional info to the GET/database endpoint, including host, port, and database name.

Bug Fixes

  • Fixed broken search links for IDR pipelines.
  • Fixed an issue preventing IDR pipelines from being scheduled via the UI.
  • Fixed an issue preventing container scripts from running when the command was pasted from a Windows machine.
  • Fix for “unrecognized node type: 407” errors occasionally occurring in queries.

Help Articles Updated

Week Ending 09/14/2024

Updates

  • Salesforce Import: The internal logic for handling Salesforce data has been updated, to be more robust for potential issues related to CSV handling (e.g., the comma character being part of the actual data).
  • New container scripts will now use the latest version of datascience-python, currently v8.0, by default.
  • New Python Jupyter notebooks will now use the latest version of the civis-jupyter-python3 Docker image, currently v4.0, by default.
  • The Query tool now includes the option to open your query as a SQL Report in a new “Open As” menu.
  • dbt scripts now have an option to skip dbt docs generation.
  • dbt run outputs will now have job and run IDs appended to their file names.
  • Disk space usage chart in the Platform Usage Overview will now display usage in TB instead of percents.

Help Articles Updated

Week Ending 09/07/2024

Updates

  • The Remote Host Detail and Edit pages have been given a new and updated appearance.

Week Ending 08/31/2024

Bug Fixes

  • Clarified SQL Script errors when the specified git ref doesn’t exist, e.g. a branch was deleted.
  • Fixed Data Crosswalk output for non-existent mappings - they will now be mapped to NULL instead of the string “null”.

Help Articles Updated

Week Ending 08/24/2024

Updates

  • The Table Details page now includes an “Add to Project” button in the top right-hand corner.
  • Added support for dbt retry command.
  • dbt jobs can now be run using pre-release versions of dbt core.

Bug Fixes

  • Fixed a bug preventing some objects from being added to projects.
  • Worflows will now return a more descriptive error message if an execution is started with improper input parameters.
  • Fixed remote host links in Transfer Dependencies side pane.

Week Ending 08/17/2024

Updates

  • Added real-time requirements feedback to the database password reset process.
  • Updated the R docker images to provide new supported versions, see our documentations for details.

Help Articles Updated

Week Ending 08/10/2024

Updates

  • Platform now offers a dbt job type. You can find it under the Code menu in the top nav. To learn more, see our documentation.
  • The default number of query preview rows has been increased from 10 rows to 50 rows.

Bug Fixes

  • Fixed issue preventing language scripts from showing error messages when saving.
  • The “Add to Project” button in the projects side bar has been fixed for Table objects. See our documentation to learn more.

Help Articles Updated

Week Ending 08/03/2024

Updates

  • Google Analytics Import: Updated the maximum allowed number of records returned from a Google Analytics API call from 50,000 to 100,000.
  • The data pane now includes a button to copy the qualified table name to your clipboard.
  • Fixed parameter values are now shown on Template backing Scripts.
  • Added a description attribute to the Files API.
  • Geocoder has been updated to include the latest Canadian addresses.  

Bug Fixes

  • Salesforce Export: Fixed the row count for failed records in the logs.

Week Ending 07/20/2024

Updates

  • New Python Jupyter notebooks will now use the latest version of the civis-jupyter-python3 Docker image, currently v3.1, by default. This latest version is on Python 3.12.
  • Added Tableau Single Sign-On (SSO) which simplifies access to Tableau Server and Desktop with centralized authentication using platform credentials, enhancing security and streamlining user experience. Get started with setup instructions here.

Week Ending 07/13/2024

Bug Fixes

  • Fixed a bug causing first row to be duplicated in BigQuery Exports to Google Sheets.

Week Ending 06/29/2024

Updates

  • The Database Usage page now includes a Query Duration by Phase graph indicating the average amount of time which queries spend in each phase at a given time. Go to Admin -> Platform Usage Overview -> Database -> Query Performance to view the new graph.
  • Added the ability for users to order their favorites on the home page by preference, instead of by when the favorite was added. Using the icons on the right of the table, users can move items up or down or send them to the top or bottom of the list.

Week Ending 06/22/2024

Updates

  • New container scripts will now use the latest version of the Civis datascience-python docker image, currently v7.3.0, by default.

Week Ending 06/08/2024

Updates

  • New quick-start Service Templates are available to help you launch your own dashboards, APIs, and other applications in Civis. We offer example apps for a few different frameworks: Streamlit, Shiny, and Flask under the “Publish” menu. For more information, see our Services Documentation.

Bug Fixes

  • Fixed “Add to Project” button in projects left side pane for Template Scripts.
  • Fixed issue where long Script Parameter names could overflow their column in the Parameters table.

Week Ending 06/01/2024

Updates

  • Updated CASS/NCOA to use a new version of the vendor API for improved stability and support.
  • IDR Pipelines will now error before running if an enforced link column no longer exists.

Bug Fixes

  • Fixed a bug preventing IDR errors from displaying properly.

Week Ending 05/25/2024

Updates

  • JavaScript Scripts have been updated to use V8 engine version 10.2.154.26, allowing access to new JavaScript language features, including but not limited to nicer syntax for classes and improved support for string interpolation and multi-line strings via back-tick strings. See here for a list of popular V8 syntax features: https://developers.google.com/apps-script/guides/v8-runtime#v8-syntax 

Week Ending 05/18/2024

Updates

  • Civis API Python client v2.0.0 has been released. For details, please see the release notes. The documentation has also been updated. If you’re using Python 3.9 or later, it is available through “pip install civis==2.0.0” or equivalent. Highlights for this release:
    • To facilitate development work, the details of the API endpoints and methods are now available to a “civis.APIClient” instance in your local IDE (e.g., VS Code, PyCharm) for auto code suggestions, type checking, etc.
    • In the documentation, the API resources each have their own page for easier navigation.
    • For Civis Platform scripts, a new function “civis.civis_logger()” is available as a drop-in replacement for Python’s built-in “logging.getLogger(__name__)”. This logger has been configured to send INFO-level (or below) logging to stdout so that the log messages show up in black on Civis Platform, and WARNING-level (or above) logging to stderr in red.
    • A “CivisFuture” instance now has a new property “job_url”.

Bug Fixes

  • Fixed issue causing the workflows detail page to crash when loading a custom task with a parameterized credential.
  • Fixed support for database syncs with a source redshift cluster outside of us-east-1.

Week Ending 05/11/2024

Updates

  • No Updates

Bug Fixes

  • Fixed bug where database syncs from Postgres to BigQuery would convert NULLs to FALSE.
  • Fixed bug causing Query tool to display BigQuery boolean values incorrectly.
  • IDR Pipelines can now be added to Projects directly from the Projects side pane.

Week Ending 05/04/2024

Updates

  • Projects detail page now shows breadcrumb navigation for parent projects.

Bug Fixes

  • Snowflake NUMBER columns with decimal digits precision of 0 will now be mapped to the INTEGER data type in db syncs to BigQuery databases. NUMBER columns with decimal digits precision > 0 will continue to be mapped to the FLOAT64 data type.
  • Fixed display of BigQuery boolean values in Query tool results.
  • Fixed duplicate error banners in data side pane.
  • Fixed success banners for the “Add to Workflow” standard action.

Week Ending 04/27/2024

Updates

  • Added ‘Description’ field to Reports.
  • Extended support for DB Syncs to Redshift clusters in all regions
  • Added SQL Reports: a new reporting capability that allows you to quickly create visualizations using SQL statements and then customize your visualizations directly in Platform. For more information, see our documentation or click here to try it out!

Week Ending 04/13/2024

Updates

  • Improved performance of the Remote Hosts API and index page.
    Top nav bar searches may now be sorted by Updated At date.

Bug Fixes

  • Fixed bug preventing projects bar search results from showing in the correct order.

Week Ending 03/16/2024

Updates

  • IDR pipelines can now be cancelled from the UI.

Week Ending 03/09/2024

Updates

  • MySQL Database Syncs no longer require local file loads and are now compatible with MySQL 8 databases using default settings. local_infile may be set to either true or false.
  • IDR will now give you a warning if you are about to navigate away with unsaved changes.

Week Ending 02/11/2024

Updates

  • Added Data Crosswalk, which is a new feature that will allow users to streamline the process of categorizing data values. For more information about this feature attend the webinar next week or visit this blog post. 

Bug Fixes

  • Fixed issue preventing the “Verify Table Row Count” feature from working with BigQuery database syncs.

Week Ending 01/20/2024

Bug Fixes

  • When navigating to the Scripts, Imports, and Exports index pages, users will no longer receive the “Failed to load Scripts” error message. The pages will load correctly.

Week Ending 01/13/2024

Updates

  • New Import from Phone2Action - Advocates has been released. Please reference this documentation for more information.

Help Article Updates

Week Ending 12/16/2023

Updates

  • New Import from DonorDrive Query API has been release. Please reference Import from DonorDrive Query API for more information.
  • New Import from DonorDrive Public REST API. Please reference Import from DonorDrive Public REST API for more information.
  • A breakdown of historical compute usage by instance type can now be viewed from the Platform Usage Overview Monthly Compute Hours. This page can be found by navigating to Admin -> Platform Usage Overview -> Hours Used.
  • The description field on the Table Details page will now show the full Table description text by default.

Bug Fixes

  • In Project view, reports generated by a service now get the "report" icon instead of the "service" icon.

Help Article Updates

Week Ending in 12/09/2023

Updates

  • Container scripts and services can now pull Docker images from ECR and GitHub, in addition to DockerHub. 
  • Civis VPN users should use the new AWS VPN, released during the maintenance window on December 13th. For instructions on how to set up the new VPN, see the Help article, 'New Civis VPN (December 2023) Set-Up Instructions’. The existing VPN will be deprecated on December 13th.

Help Article Updates

Week Ending in 11/18/2023

Updates

  • Added upsert functionality for BigQuery database syncs

Bug Fixes

  • Fixed downloads of files with special characters in file name from the Project detail page

Week Ending in 10/28/2023

Updates

  • Introduced a new threshold on Project size when enabling Auto-Share: Auto-Share may now only be enabled for Projects with 100 or fewer objects in them.

Week Ending in 10/21/2023

Updates

  • A new Export to S3 Template has been published. Exporting Civis Files, Platform tables, and  custom SQL are supported. Reference the Export to S3 documentation for more information.

Week Ending in 10/7/2023

Updates

  • A new Export to S3 Template has been published. Exporting Civis Files, Platform tables, and  custom SQL are supported. Reference the Export to S3 documentation for more information.

Help Article Updates

Week Ending in 10/7/2023

Updates

  • The civis-jupyter-python3 docker image has a new release: <a href=”https://github.com/civisanalytics/civis-jupyter-python3/releases/tag/v3.0.0”>v3.0.0</a>. This image is based off of datascience-python v7.0 and runs Python 3.11.4. It will be used automatically by all new Platform Python Notebooks. Existing Python Notebooks will continue to use the image and Python version which they were created with.
    • If you wish to update an existing Python Notebook to use the new image, you can update its Tag to "3.0".

Help Article Updates

Week Ending in 09/23/2023

Bug Fixes

  • Resolved a bug responsible for page crashes when users attempted to add a Workflow to projects using the 'Add to Project' button.

Week Ending in 09/16/2023

Bug Fixes

  • Fixed a bug with MySQL to BigQuery syncs where timestamps ending in :00 would cause the sync to fail. 
  • Increased the BigQuery maximum String size to the <a href=”https://cloud.google.com/bigquery/quotas#max_row_size”>maximum row size</a> of 100MB. This will allow BigQuery DB Syncs with large string columns to succeed.

Help Article Updates

Week Ending in 09/09/2023

Updates

  • A new feedback form has been added to Platform and can be found via the feedback icon in the top navigation bar. Let us know how we are doing any time by filling out the short poll and/or providing any thoughts and ideas you have. Your feedback helps us to improve!

Bug Fixes

  • Fixed bug which prevented some users from viewing PUO when navigating directly from homepage.

Help Article Updates

 

Week Ending in 09/02/2023

Updates

  • Services have a new feature: Public Link Generation. Public URLs can now be generated from the “Security and Access” settings area for running Services. These URLs will allow anyone to view your Service without logging into Platform.
    • Note: Please make sure that there is no sensitive data in your Service before creating a public link.

 

Week Ending in 08/26/2023

Updates

  • Improved the user experience for importing CSV files, now allowing users to easily delete and replace CSV files.

Help Article Updates

Week Ending in 08/11/2023

Updates

  • The Import from URL template has three new optional parameters: database credential, distribution style, and primary keys.
  • Platform Usage Overview now allows you to view your organization’s historical compute usage for the last 12 months by clicking on “Hours Used” from the Compute landing page.

Bug Fixes

  • No updates

Week Ending in 08/05/2023

Updates

  • No updates

Bug Fixes

  • Fixed dbsync error when syncing Postgres tables with JSON columns to BigQuery.
  • Fixed a dbsync issue which prevented new tables with primary keys from being imported to BigQuery: A NOT ENFORCED clause will now be added to primary key columns when BigQuery tables are created by dbsync jobs. To learn more about the introduction of primary keys in BigQuery, please see <a href=”https://cloud.google.com/blog/products/data-analytics/join-optimizations-with-bigquery-primary-and-foreign-keys”>here</a>.
  • Fixed bug which prevented users from overriding their default notification settings in API POST calls.

Help Article Updates

Week Ending in 07/29/2023

Updates

    • Person Data Standardization Tool can now be created by going to Tools -> Enhancements -> Person Data Standardization in the top nav.  
    • Improved the Script Parameters pane experience by moving the “Add” button below the new parameter form.
    • The datascience-python docker image has a new release: <a href=”https://github.com/civisanalytics/datascience-python/releases/tag/v7.0.0”>v7.0.0</a>. This image runs Python 3.11.4 and will be used automatically by all new Platform Python scripts going forward. Existing Python Scripts will continue to use the image and Python version which they were created with.

Bug Fixes

  • Fixed bug which caused incorrect execution times in history pane of Workflows & Scripts.
  • Fixed issue in Workflows share pane which prevented distribution of email notifications.
  • Fixed a formatting issue with BigQuery Table Create Statements.

Help Article Updates

Week Ending in 07/22/2023

Updates

  • No updates

Bug Fixes

  • No updates

Help Article Updates

  • No updates

Week Ending in 07/15/2023

Updates

  • Enhanced data hygiene experience via UI improvements and expanded documentation for the Person Data Standardization tool.

Bug Fixes

  • Fixed a visibility issue in Query results which blocked the user from reading the last row of results.

Help Article Updates

  • None

 

Week Ending in 07/08/2023

Updates

  • Platform Usage Overview now has a new Database tab that allows users to easily view accurate and actionable metrics on their database’s health. For more information see <a href=”https://civis.zendesk.com/hc/en-us/articles/17484340645901-Database-Status-Overview”> Database Status Overview</a>.

Bug Fixes

  • None

Help Article Updates

  • None

 

 

Updates

  • Enhanced data hygiene experience via UI improvements and expanded documentation for the Person Data Standardization tool.

Bugs

  • Fixed a visibility issue in Query results which blocked the user from reading the last row of results.

Week Ending in 07/01/2023

Updates

  • None

Bug Fixes

  • Fixed bug which caused buttons to be obstructed when sharing reports with lots of users and groups. 
  • Fixed browser tab title mismatch for Table Details page.

Help Article Updates

Week Ending in 06/24/2023

Updates

  • Introducing Flexible Cloud Storage now with support for BigQuery. For more information click here and ask your Account Manager.  
  • Added API endpoint descriptions to our API documentation for select endpoints.
  • Users can now set a credential as their default for a given remote host via the Credentials UI.

Bug Fixes

  • None

Help Article Updates

Week Ending in 06/17/2023

Updates

  • Moved the Platform Usage Overview from “#/admin/usage” to “#/usage”. Existing bookmarks will be re-routed to the new url.

Bug Fixes

  • Fixed a bug which prevented sharing notifications from being sent for Scripts and Credentials. 

Help Article Updates

Week Ending in 06/10/2023

Updates

  • The “Quick Start” Zendesk document has been updated to reflect recent changes to Platform.  For more information see Quick Start Guide
  • Platform now sends SMS 2FA codes for UK phone numbers via alphanumeric sender IDs, to comply with new UK regulations.

Bug Fixes

  • Fixed a bug with the “Verify Table Row Count” feature, which would sometimes cause DB sync jobs to fail.
  • Fixed issue which would hide results text in Query history pane if “0 Rows” are returned.

Help Article Updates

Week Ending in 06/03/2023

Updates

  • Admins can now send a ‘Reset Password’ or ‘Welcome to Platform’ email to users by selecting the “Send Access Email” option via the User Management dropdown.
  • Report-only users from organizations with extra US Federal Government compliance features enabled will now be required to authorize via 2FA each time they log into Platform

Bug Fixes

  • No bug fixes

Help Article Updates

 

Week Ending in 05/27/2023

Updates

  • No updates  

Bug Fixes

  • No bug fixes

Help Article Updates

Week Ending in 05/20/2023

Updates

  • Your user ID number can now be accessed from the My Profile page.  
  • Improved helper text language on Project share pane.
  • Removed ‘nominatim' provider support from the Geocode Addresses tool.

Bug Fixes

  • Fixed the issue of missing data for the “page insights” table from the Facebook Page Insights Import template.
  • Fixed a bug in the Salesforce Export template, where it previously did not use the user-supplied database credential and now it does.
    • If a Salesforce Export job previously ran successfully but now starts to fail due to database credential errors, try supplying your default database credential to the script.
  • Fixed issue which caused Platform to crash if Announcements pane was opened from Workflows page.
  • Fixed issue where Query Results tables automatically replaced empty or ‘0’ value strings with "NULL".
  • Fixed stacking error banner issue when switching between Table Details pages.

Help Article Updates

Week Ending in 05/13/2023

Updates

  • Report viewers and standard users can now favorite Reports by clicking the star icon on the Reports Detail page. 
  • Added the ability to cancel a running query from the main query page.

Bug Fixes

  • Fixed a bug in Salesforce Imports where the import would sometimes use the user’s default database credential instead of the user-supplied database credential argument.
    • If you notice a Salesforce Import start failing with credential errors, try supplying your default database credential to the script.

Help Article Updates

Week Ending in 05/06/2023

Updates

  • The Salesforce Import template now has additional parameters to configure the imported data: diststyle, distkey, sortkey1, and sortkey2.
  • 'View Details' link in Project pane now can be opened in a new tab when right-clicked.
  • Improved Services page UI to clearly denote buttons are unclickable for users with read-only access.

Bug Fixes

  • Fixed “Last Seen At” column on User Index page to show correct user value.

Help Article Updates

Week Ending in 04/29/2023

Updates

  • Improved clarity of top navigation menu:
    • Clarified Query page navigation button by including text next to the database icon. 
    • Add a red dot badge to the notification icon (in top right corner) to highlight when you have an unseen announcement.
    • Object index page links have been moved from headers to dedicated “View All” links in the top navigation menu.
  • New features for the Google Analytics 4 import template. For both features highlighted below, the details are in the migration documentation.   
    • To facilitate the migration of existing GA import custom scripts from UA to GA4, we have developed a tool that translates the script parameters “dimensions”, “metrics”, and “sort” from the UA-specific dimensions/metrics to their GA4 equivalent.
      • Please refer to the section “Automatically Converting Parameter Values from UA to GA4” in the migration documentation. 
    • The Google Analytics 4 import template now supports more complex filtering logic (e.g., “city is any of {London, Paris} AND dayOfWeek is 5 or 6 AND month is not 12”).
    • We highly recommend you clone your existing custom script jobs to test the GA4 property to prevent issues when making changes to your production code.
  • The Favorites widget now displays the user’s current count of favorites for easier tracking and management. 

Bug Fixes

  • Fixed ability to view active users after deselecting the "show deactivated" box on the Table Privileges tab.

Help Article Updates

 

Week Ending in 04/22/2023

Updates

    • We highly recommend you clone your existing custom script jobs to test the GA4 property to prevent issues when making changes to your production code. 

Bug Fixes

  • Fixed custom script links in navigation bar search.

Help Article Updates

Week Ending in 04/15/2023

Updates

  • The new Dashboard is live:
      • Activity Metrics, for a better overview of current Platform activity
      • Favorites, which now includes the ability to favorite Scripts, Workflows, Imports, and Exports in addition to Projects
      • Organization Details, to give a better understanding of your org structure and support options
      • Quick Links, to provide easier access to important locations across Platform
      • Feedback is welcome!

Bug Fixes

  • Fixed a bug that was causing Platform pages to error out arbitrarily. 

Help Article Updates

Week Ending in 04/08/2023

Updates

  • To maintain the relevance of favorites, when a Project is archived it will automatically be unfavorited.

Bug Fixes

  • No Updates

Help Article Updates

 

Week Ending in 04/01/2023

Updates

    • The new Activity View is live as we set the stage for a more productive homepage.
      • The upgraded Activity View loads faster so you can get results sooner
      • Filter options have been clarified so that you can narrow down results efficiently
      • The Author filter has been changed to Owner for more relevant results

Bug Fixes

  • A fix has been made to correct some workflow page crashes that occurred when an incorrect job number was used. 

Help Article Updates

 

Week Ending in 03/25/2023

Updates

    • None

Bug Fixes

  • None

Help Article Updates

 

Week Ending in 03/18/2023

Updates

  • The following updates have been made to the Salesforce integration:
    • For incremental sync on import, we fixed a bug that prevented the import job from importing newly created data on the same day as the import job runs.  
    • When importing, the created Civis Platform table now has diststyle set to “key” and distkey set to “id”.
    • When a custom SQL query is provided to export data to Salesforce, the required “id” column can now be specified more flexibly as “SELECT foo.id, …” or “SELECT foo AS id, …“, in addition to “SELECT id, …“.
  • The Database Privileges tab on the Table Details page has been updated:
    • Users can now view members of Database groups for a clearer understanding of how user access is granted via the Privilege Source column. 
    • Filtering by user account status is now supported for auditing and security purposes. 

Bug Fixes

  • None

Help Article Updates

Week Ending in 03/11/2023

Updates

  • The Project creation process has been improved. A new “Create Project” template encourages unique Project Names and provides the opportunity to set Description, Notes, and Auto-Share before creation. A cancel button enables you to stop the creation process at any point.

Bug Fixes

  • None

Help Article Updates

Week Ending in 03/04/2023

Updates

  • Org Admins will now receive a security notification email if a user with access to one or more non-Civis managed databases is deactivated. The email will list databases which the user may retain access to, and provide instructions on how to expire or drop their database user account, if desired.

Bug Fixes

  • None

Help Article Updates

Week Ending in 02/25/2023

Updates

  • Workflows used as tasks within another workflow are now viewable using the ‘eye’ icon on the Info tab

Bug Fixes

  • None

Help Article Updates

Week Ending in 02/18/2023

Updates

  • The “Resume Workflows” functionality has been restored to the UI in the new Workflows page.  Paused workflows will automatically display the Resume Workflows button in place of the Retry Failed Tasks button. 
  • Reminder: A reminder that as per Civis' Password Expiration Policy implemented in 2022, Civis requires Platform passwords to be changed once a year. Users will receive notification emails everyday for 10 days prior to their Platform password expiring. For more information, please see documentation: https://civis.zendesk.com/hc/en-us/articles/360059204612-Reset-Password

Bug Fixes

  • None

Help Article Updates

  • None

 

Week Ending in 02/11/2023

Updates

  • No Updates
  • Reminder: A reminder that as per Civis' Password Expiration Policy implemented in 2022, Civis requires Platform passwords to be changed once a year. Users will receive notification emails everyday for 10 days prior to their Platform password expiring. For more information, please see documentation: https://civis.zendesk.com/hc/en-us/articles/360059204612-Reset-Password

Bug Fixes

  • None

Help Article Updates

  • None

 

Week Ending in 02/04/2023

Updates

  • A user's Civis-managed database password will be automatically expired when a user’s Platform account is deactivated. Reactivating the Platform user's account will un-expire the associated database account. For more information see User Deactivation Document.
  • Various small fixes have been made to the new Workflows page based on user feedback.

Bug Fixes

  • None

Help Article Updates

Week Ending in 01/28/2023

Updates

Bug Fixes

  • Fixed bug which caused an error when attempting to preview query results older than 30 days in the Query pane.

Help Article Updates

Week ending in 01/21/2023

Updates

  • Partial Workflow functionality has been added to the Workflows Beta. 

Help Article Updates

  • none

Week ending in 01/14/2023

Updates

  • None

Help Article Updates

Week ending in 01/07/2023

Updates

  • None

Help Article Updates

Week ending in 12/31/2022

Updates

  • None

Help Article Updates

Week ending in 12/24/2022

Updates

  • None

Help Article Updates

Week ending in 12/10/2022

Updates

  • You can now choose the default notification setting for all NEW jobs, workflows, and services via the Preferences pane under the initials menu.  For more information see Notifications

Help Article Updates

Week ending in 12/03/2022

Updates

  • The Workflows 2.0 Beta is now live!  You can switch between the Beta and Current versions from the Workflows page.
  • The Salesforce Export template has two new parameters (MAXIMUM ROW ERRORS and EXIT JOB ON FAILURE) for controlling the export run’s behavior when there are records that have failed to export to Salesforce. Please see the Salesforce Export documentation for details.

Week ending in 11/26/2022

Updates

  • There are no updates this week. 

Week ending in 11/19/2022

Updates

  • New feature allows users to clone Projects and associated objects from the standard menu options.  For more information see Introduction to Projects.

Help Article Updates

Week ending in 11/12/2022

Updates

  • Improved save confirmation for Notify Pane.

Week ending in 11/05/2022

Bug Fixes

  • Fixed bug which would sometimes cause capacity metrics to be double counted on Platform Usage Overview graphs.

Week ending in 10/29/2022

Updates

  • Improved experience of project archiving by adding success notification pop-ups.

Bug Fixes

  • Fixed bug in Admin Center modals which caused some dropdowns to display behind checkboxes.

Help Article Updates

Week ending in 10/22/2022

  • Fixed bug in the data side pane which did not allow users to copy values from the first column in the column definition table. 
  • Fixed a bug which prevented Table Scanner from running for Postgres databases.
  • Improved discoverability of recent Projects by updating the Project side pane to show “Recently Updated” projects.

Help Article Updates

Week ending in 10/15/2022

  • Salesforce Migration notes and deprecation plans are forthcoming. The tentative date for deprecation of the old integration is 1/31/22 so we recommend you start moving to the new one as soon as possible.
  • Enhanced data side pane UI with new icons, sort functionality for column definitions list, and improved visibility of distkey & sortkey.
  • Fixed bug in data pane search which could cause excessive “Failed to load tables” errors.
  • Improved UI by adding explanatory tooltips for Archive/Unarchive menu options.

Help Article Updates

Week ending in 10/08/2022

  • Improved error messaging and clearly delineated archived objects in the UI to reduce confusion about permissible actions. 

Week ending in 10/01/2022

Updates

  • All Org Admin users will now have the Table Tag Admin role (some Org Admins may have already had these abilities) and be able to grant that role to additional groups, should they so desire. The Table Tag Admin role allows users to create and manage database table tags for their organization. Please see Introduction to Table Tags for more information on using Table Tags.

Week ending in 09/10/2022

Updates

  • Notebooks and Reports can now be transferred to teammates in the UI via the Transfer Ownership side pane.

Bug Fixes

  • None

Help Article Updates

 

Week ending in 09/03/2022

Updates

  • To improve the credential process, users are now able to utilize Google OAuth while assuming the role of a robot. For information see the Robot Users section in Zendesk. 
  • Projects and contents can now be transferred to teammates in the UI via the Transfer Ownership side pane.

Bug Fixes

  • None

Help Article Updates

Updated- Robot Users

 

Week ending in 08/20/2022

Updates

  • Civis Platform Key Features documentation has been broken out into individual articles to improve keyword searches. For information on Platform key features see Civis Platform Key Features section in Zendesk.

Bug Fixes

  • None

Help Article Updates

Updated -Export Civis Data to NGPVAN
Updated - Redshift to Bigquery
New - Troubleshooting Version Control
Updated - Version Control

 

Week ending in 08/06/2022

Updates

  • Users can run a partial query by highlighting only the desired SQL prior to running
  • Text wrapping within Query has been implemented
  • Clarified role by renaming the Admin role "Group Admin" to "Team Admin". For information on roles capabilities see Platform Roles in Zendesk.
  • Increased flexibility of email address management by enabling Organization and Team Admins to update user email addresses.

Bug Fixes

  • The ability to cancel a running query has been restored

Help Article Updates

Week ending in 07/30/2022

Updates

  • Improved user experience of Transfer Ownership process by adding banners to show success and error feedback messages, and by making error messages provide more detailed descriptions of what went wrong.

Bug Fixes

  • The “Reset 2FA” button is now disabled when you don’t have permission to reset 2FA for a given user.

Help Article Updates

None 

Week ending in 07/23/2022

Updates

  • New Auto-Share feature allows automatic sharing of all current and future objects with users and groups shared on a Project, at their respective access levels. For more information see Auto-Share Projects in Zendesk.
  • Improved user experience of Credentials Details page by adding feedback banners for share errors.
  • New Zendesk article Updating Groups document on how to update groups in Platform.
  • Improved Projects API usability with more detailed error messages when adding an object to a project.
  • Expanded sharing and collaboration capabilities by allowing objects with validation errors to be shared.

Bug Fixes

  • None

Help Article Updates

Week ending in 07/16/2022

Updates

  • New admin control for user management of 2FA. For more information see Two Factor Authentication (2FA) in Zendesk.
  • Added feedback banners for credential sharing actions.
  • Enable ability to view user detail page for all users you share a group with.

Bug Fixes

  • None

Help Article Updates

 

Week ending in 07/16/2022

Updates

  • New admin control for user management of 2FA. For more information see Two Factor Authentication (2FA) in Zendesk.
  • Added feedback banners for credential sharing actions.
  • Enable ability to view user detail page for all users you share a group with.

Bug Fixes

  • None

Help Article Updates

Week ending in 07/09/2022

Updates

  • Increased security for users by sending a notification email to the users old address when an email address change has been requested. 

Bug Fixes

  • None

Week ending in 07/02/2022

Updates

  • “Publish to Report” button has been updated to “Publish” on the Services Details Page.

Bug Fixes

  • None

Week ending in 06/25/2022

Updates

  • Added the ability to preview up to 1000 results within Query 2.0
  • Query History pane will now stay open unless closed by the user
  • New Zendesk article Creating Groups details the process and requirements for new group creation in Platform.
  • Added feedback banners for “Add to Project” and “Share” Pane actions for Scripts and Workflows.

Bug Fixes

  • None

Week ending in 06/18/2022

Updates

  • None

Bug Fixes

  • None

Help Article Updates

Week ending in 06/11/2022

Updates

  • Links to previously run queries, such as those on the landing page, will now correctly load the linked query. 
  • Improved messages for error handling for “Add to Projects” actions that do not follow Platform’s sharing criteria.

Bug Fixes

  • None

Help Article Updates

Week ending in 06/04/2022

Updates

  • Added a quick query button to the data pane to make it easier to query a table being viewed
  • Added ID and duration to the script history pane
  • Updated and clarified error messaging for the scripts manage versions pane
  • Added the ability to resize the Query writing/results pane

Bug Fixes

  • None

Help Article Updates

Week ending in 05/28/2022

Updates

Bug Fixes

  • Fixed an issue where the Edit User modal would not open for some users.
  • Improved troubleshooting in Projects by providing a clearer error message when updating a project fails or when an object fails to add to a project.
  • Fixed UI inconsistency between user index and user detail page. Admins are now able to edit a user's active/inactive status, SSO requirement, and groups directly from the user index page.

Help Article Updates

Week ending in 05/21/2022

Updates

  • The new Query tool now allows queries to be written while a previous query is still running
  • The Container Script page now has validation applied to prevent overly long scripts

Bug Fixes

  • The query history scroll should no longer ‘jump’ or get stuck, but should allow scrolling forwards and backwards as intended. 
  • The query history should now display short queries properly
  • The copy and create script buttons should now always be visible in Query, even if disabled.

Help Article Updates

  • None

Week ending in 05/14/2022

Updates

  • Added Control+Enter as a hotkey shortcut to run queries in the new Query tool.
  • Added the “All Time” option to search beyond 90 days in the Query History

Bug Fixes

  • Corrected an issue in Query that caused some leading 0s to be deleted
  • Corrected an issue in Query History that prevented deletion in some cases

Help Article Updates

  • None

Week ending in 05/07/2022

Updates

No Updates

Bug Fixes

Fixed a bug that prevented Projects with archived objects from being transferred. Please note that archived objects will be skipped when transferring project dependencies.

Help Article Updates

Week ending in 04/29/2022

Updates

No Updates

Bug Fixes

No Bug Fixes

Help Article Updates

Week ending in 04/22/2022

Updates

No Updates

Bug Fixes

  • Corrected a bug in the new Query tool which showed duplicate lines returned that occurred when an ‘ID’ column returned non-unique results

Week ending in 04/15/2022

No Updates

Week ending in 04/08/2022

Updates

  • The enhanced Query tool is available!  For more details, go here.

Bug Fixes

  • Fixed a bug that caused script templates to create duplicate Scripts.
  • Fixed a bug that in rare occasions caused a hanging state if users created multiple Scripts via the UI consecutively.

 

Week ending in 04/01/2022

Updates

  • The password expiration window for all users (excluding non-FedRamp and SSO w/o VPN) has been extended to allow for issue resolution and security policy improvement. Civis will continue to require authentication either via a secure Identity Provider or 2FA (password + an expiring one-time code).
  •  Suspended users receiving an email notification of their account suspension will be requested to reach out to their organizational Admins to unsuspend their account. Organizational Admins can reference the following: https://civis.zendesk.com/hc/en-us/articles/4426900580493-How-to-Unsuspend-Users-in-Civis-Platform for instructions on how to unsuspend users.

Bug Fixes

  • Fixed a bug that prevented capacity data from showing in Platform Usage Overview historical graphs.

Week ending in 3/25/2022

No Updates

Week ending in 3/18/2022

Updates

  • We’ve released Token Authentication for Services. Service Token Authentication allows users and their web applications to utilize tokens to access Services such as APIs or deployed models from outside of Platform. These tokens may be created/removed via the Service UI or API. For more information about Service Token Authentication, please read here.

Bug Fixes

  • Fixed an error which prevented Tables from being added to Projects with the “Share All Project Objects” feature enabled.

Week ending in 3/4/2022

Updates

  • The Automate pane within the Scripts page has been updated to be easier to use and requires fewer clicks to set a schedule.  You can also schedule scripts on a monthly cadence. 
  • Project links on the homepage have been updated to be more descriptive.

Bug Fixes

  • Custom scripts’ Last Run timestamps are now populated

Week ending in 2/25/2022

Updates

  • Providing additional information on a previous update, the “Runs per Hour” field for scheduled jobs has been deprecated in the API for all jobs but not in the UI. 

Bug Fixes

  • We fixed the bug regarding false error messages when unsuspending users.

Week ending in 2/18/2022

Updates

  • The “Runs per Hour” field for scheduled jobs has been deprecated.  All existing scheduled jobs have been converted to the equivalent value in minutes.

Bug Fixes

  • We fixed the bug around not being able to change users’ primary groups and secondary groups.

Week ending in 2/11/2022

Updates

  • When user accounts are changed to a robot account, that account will be emailed informing the user that their account has been changed to a robot account.

Week ending in 2/04/2022

Updates

  • There has been an enhancement to the Superadmin feature. Users who enter Superadmin mode now have ‘Manage’ permissions on reports and projects within their organization.

Bug Fixes

  • The pagination buttons have been fixed for the Script Template page for when there are more than 20 custom scripts in a template.

Week ending in 1/28/2022

Updates

  • The “Help” option in the initials dropdown was renamed to “Documentation” to clarify that this link will take you to our Zendesk documentation.
  • We updated our messaging around email changes to clarify that confirmation of the new email address is necessary in order for the change to take effect.

Week ending in 1/21/2022

Updates

  • After creating a new user, you can click on their username in the notification popup that will open up the new user’s detail page.

Week ending in 1/14/2022

Updates

No Updates

Bug Fixes

  • We fixed a bug in Admin Center so that users can now search groups using partial strings.  For example, if you type “reporting”, you will get groups whose name contains “reporting.” For more information about Admin Center, please read here.  
  • We fixed a bug in Projects where NCOA jobs have correct links in the project folders.

Week Ending 1/7/2021

No Updates

Week ending in 12/31/2021

Updates

  • There has been an enhancement to the Superadmin feature. Users who enter this Superadmin mode can now view reports and projects within their organization.  For details about the Superadmin mode, read here.
  • Updated Tableau server from version 2020.4.4 to 2021.3.6. You can find more info. here.

Week ending in 12/24/2021

Updates

  • We updated our Civis Python API client to v1.16.0.  You can pip-install it (e.g., pip install civis==1.16.0), and it’s now available by default in new Python scripts, container scripts, and Python notebooks on Platform.  This update includes bug fixes and documentation improvements.  More information here.

Week ending in 12/17/2021

No Updates

Week ending in 12/10/2021

Updates

  • We’ve released Permission Sets for Services. Permission Sets allow you to store and manage each user’s permissions in Platform in regards to Services. Whenever a user views your app, it can retrieve their permissions via the Platform API and act accordingly. For more information about Permission Sets, please read here.

Week ending in 12/03/2021

No Updates

Week ending in 11/26/2021

No Updates

Week ending in 11/19/2021

Updates

  • We released a Civis Platform connector for Google Data Studio; you can find the connector in Google Data Studio’s Community Connectors.  With the Civis Platform connector, you can now connect to any Platform tables and data when building Data Studio reports.  For more details, please read here.

Week ending in 11/12/2021

No Updates

Week ending in 11/05/2021

Updates

  • The legacy Users page has been sunsetted. Please use Admin Center to view, edit, and manage your users.  For more information about Admin Center, read here. 

Bug Fixes

  • We fixed a bug so that you can view the settings of a service-backed Report while the service is deploying. 

Week ending in 10/29/2021

Bug Fixes

  • We addressed a performance issue in the Admin Center; now you can input user information without any time delay. 

Week ending in 10/22/2021

No Updates

Week ending in 10/15/2021

No Updates

Week ending in 10/8/2021

Updates

  • We released an updated Admin Center experience.  We added additional search filters for you to look up users and groups; we also improved the load and search performance so that you can see the content quicker.  For more details, read here.

Week ending in 9/17/2021

Bug Fixes

  • We fixed a bug that prevented you from removing a file from a project.  

Week ending in 9/3/2021

Bug Fixes

  • We fixed a bug in the Admin Center so that you can add users to the newly created groups in the Edit User modal without refreshing the page.

Week ending in 8/27/2021

Updates

  • We have made Matching Algorithm modifications to address the NULL name conflicts. To avoid potential false positives and clustering issues the model now will assign scores of 0 when the first or last name is not available.

Week ending in 8/20/2021

Bug Fixes

  • We fixed a bug that prevented report viewers from creating new projects. 

Week ending in 8/6/2021

Updates

  • To give admins more flexibility and control over their compute, we’ve enabled admins to view and set default compute partitions for a particular group.  To do so, admins can go to Admin Center, locate the group, and click to view the group’s details.  Learn more about default partitions here.

Week ending in 7/30/2021

Updates

  • For security, we introduced a new suspension status that will suspend, or block Platform logins, for users who have not logged into Platform for over 90 days.  To improve user experience, Admins can now quickly unsuspend users in the Admin Center UI.

Week ending in 7/23/2021

Updates

  • We’ve updated the Group Detail page in the Admin Center such that when you go to a specific group’s Detail page, you can easily navigate to see a group member’s details by clicking on the user in the Members table.

Week ending in 7/16/2021

No Updates

Week ending in 7/9/2021

Updates

  • We’ve updated our running as logic for Platform Scripts. PATCH requests via the API will no longer set the running as user to the requester automatically. Requester must explicitly set the running as user to themself before making changes to relevant fields.

Week ending in 7/2/2021

Updates

  • We’ve updated the UI to better ensure role assumption rules; if you are assuming a role, all Assume Role options are disabled.  This is because you cannot assume another role while assuming a role.

Bug Fixes

  • We’ve fixed bugs related to the sharing experience so that you can auto-share projects that have tables or Archived objects; the tables and Archived objects are skipped. 

Week ending in 6/25/2021

Updates

  • We updated the Notebooks and Services experiences where users can create new credentials in Notebooks and Services.

Week ending in 6/18/2021

Updates

  1. We improved the Admin Center experience so that feedback banners will no longer stack on top of each other and will clear when you leave the page.

Week ending in 6/11/2021

Updates

  1. We added additional metadata to the user detail page in the Admin Center, including the date the user account is created, updated, and last seen at.  
  2. We improved the scripts experience to enable users to create new credentials in scripts. 

Week ending in 6/4/2021

Updates

  1. We added additional metadata to the group detail page in the Admin Center, including the organization the selected group belongs to.  
  2. We’ve improved the script parameters experience to enable clients to edit existing parameters and add allowed values from the UI. Learn more.

 

Bug Fixes

Week ending in 5/21/2021

Updates

  1. We released a Query Search API so that users can look up past queries with the endpoint /search/queries. For details, take a look at the API docs.  
  2. We’ve added helpful metadata on the group detail page in Admin Center including created at and updated at timestamps.

Bug Fixes

  1. We fixed a bug in Query so that users can now see the output from non-SELECT statements such as EXPLAIN or SHOW.

Week ending in 5/14/2021

Updates

  • We updated the Credentials page to show which remote host the Database type Credentials are linked to.  
  • We added a new suspension status that will suspend, or block Platform logins, for users who have not logged into Platform for over 90 days. This is a new security feature.
  • We updated our logic on data update detection so that the Data Updated timestamp is also updated on CREATE TABLE statements that insert data.  For more details on the Data Updated timestamp, check out our documentation here. 
  • We released new Transfer Ownership endpoints that enable users to assign a new owner to any Platform object, streamlining team collaboration and handoffs. Learn more. 

Bug Fixes

  • We fixed a bug that was causing the Identity Resolution interface to incorrectly show some types of failed runs as still in progress.

Week ending in 5/7/2021

Bug Fixes

  • We fixed a Query bug so that when you write and execute multiple SQL statements, we will now execute all of these statements and return the results from the last SQL statement.

Week ending in 4/30/2021

Updates

  • On 4/29/2021, Civis will be removing support for R client versions older than 1.6.1. Please see our documentation for more details on how to update affected.
  • We released a new version of the Table Details page; we updated its look and feel to improve readability.  For more details on Table Details functionality, please read documentation.  

Bug Fixes

  • We continued to make improvements on the “running as” behavior for Database Imports, including displaying the source and destination dropdown with the right credentials in the UI.  

Week ending in 4/23/2021

Updates

  • We updated the R environment to R4.0.4.   
  • We made a few improvements to the Admin Center page, include moving this page to the top of the Admin menu dropdown for quick access and loading the Users by default to streamline admin workflow.   
  • We implemented stronger Platform password requirements. 

Bug Fixes

  • We fixed the “running as” behavior for database imports such that whenever the current  user is using their default credentials, it should substitute with the runner’s default credential.  The custom credential should stay the same if both users have access. 

Week ending in 4/16/2021

Bug Fixes

  • We fixed a validation issue with Database Imports so that you will no longer get error messages until you finish selecting the destination and source schema and table fields. 

Week ending in 4/9/2021

Bug Fixes

  • Further improvements and bug fixes to the Credentials page, including adding a browser page title

Week ending in 4/2/2021

Updates

  • On Container Scripts, we’ve added an auto-complete feature when searching and selecting  Git Repository. 

Week ending in 3/26/2021

Updates

  • We added a group member count on the Script’s share pane to help users see who is shared on a script.
  • We released the new credentials page, which includes a redesign and feature improvements making it easier for users to access and manage (i.e. create, edit, share, delete) their credentials. In addition, users will now receive an email notification if any credentials they own have been modified. To access the new credentials page, go to Admin > Credentials. For more details, check out our credentials documentation here.

Bug Fixes

  • We fixed some validation errors related to the creation of Table Tags. Table Tags documentation is found here. 
  • We fixed a bug which prevented feedback banners from showing up on the user profile page.

Week ending in 3/19/2021

Updates

  • We added the ability to edit a template note from the template detail page, previously only accessible via the API.
  • We strengthened the Civis Platform password criteria. Moving forward, any newly created Civis Platform passwords must have a minimum length of 18 characters. To update your Civis Platform password, navigate to the Initials Menu > My Profile. 
  • Platform services which fail to start up within one hour will now be automatically terminated. 

Bug Fixes

  • We’ve fixed bugs related to the new Scripts experience including: read-only code formatting, adding the template note on custom scripts, removing the close button, and preventing edits to the script name field for those without permission.

Week ending in 3/12/2021

Bug Fixes

  • We resolved a Query parsing issue where spacing and new lines are correctly handled in “SELECT INTO” statements.

Week ending in 3/5/2021

Updates

  • We updated error messages within the job logs to indicate when a job cannot run within the memory or CPU resources available in your organization. To monitor your organization’s compute resources, go to Admin > Platform Usage Overview.

Bug Fixes

  • We’ve fixed bugs related to the new Scripts experience including: loading issues when navigating between scripts and searching within the Script code box.

Week ending in 2/26/2021

Updates

  • We updated the /table_tags/ endpoint so that you can easily look up tag IDs given a tag name.  For details on Table Tag endpoints, please go here.  Table Tags documentation is found here. 
  • In Files Import, we exposed the Auto-Detect option for the “First row is a header” setting to improve transparency. Civis will automatically detect if the first row is a header, but in case we get it wrong we give you the ability to manually set it.  For more details, please read our documentation. 
  • In Database Imports, we improved the Verify Table Row Count logic so that the check does not fail when new rows are added to the source table after the sync began.  For more details, please read our documentation. 
  • In scripts, we added a “Publish as Template” side pane to make a script’s publicized status more clear and accessible.
  • Under your initials > “My Profile”, we improved the experience of adding a new Git repository including adding tooltip messages and disabling the “Add Repository” button if there isn’t a valid repo selected. 

Bug Fixes

  • We resolved a Search bug so that Google Sheet Exports will appear in the search results.  
  • We resolved a Database Import bug to correctly validate table names and raise an error when the table names contain invalid characters.  For details on what characters Database Imports support, please read our documentation. 
  • We resolved a Query bug such that when you click on the “Open as Script” action of the executed query, you will be directed to the SQL scripts page.  

Week ending in 2/19/2021

Updates

  • New & Improved Scripts: we released our Scripts redesign which enhances a core Platform feature with streamlined design, superior performance, feature improvements. For more information, read our help documentation. 

Bug Fixes

  • We resolved a File Import issue such that when the flat file only contains a header row or a single data row without the line break character, the import will not fail. 

Week ending in 2/12/2021

Updates

  • We launched a new feature called Table Tags. Table Tags enable you and your team to organize and label data across schemas for efficient collaboration. For example, you can tag your production or golden tables to guide analysts to important datasets.  For more details, check out our feature documentation.
  • We improved our error messages in Database Imports so that when a database sync fails, you can see which tables failed in the error log.
  • We launched the active workloads table in Platform Usage Overview, which enables you to monitor and cancel workloads running on your organization’s resources. You can find the active workloads table by navigating to Admin > Platform Usage Overview and selecting an instance type. For more information, check out our feature documentation.

Week ending in 2/5/2021

Updates

  • On 02-01-2021, we have stopped supporting Python API Client versions older than 1.12.1, as well as some older versions of docker images.  For more details, please go here. 
  • For all top-right navigation menus (i.e. Admin, Initials, and Database menu), you can now right-click and open menu items in a new tab.
  • All credentials API endpoints now include a “user” response field.

Bug Fixes

  • We fixed a bug in the Query tool to preserve consecutive spaces.  Specifically, if a string has consecutive spaces, the results will be rendered with the same number of consecutive spaces.  
  • We fixed a bug in the Database Import where the list of tables in the destination table dropdown will appear based on user input.   

Week ending in 1/29/2021

Updates

  • We added error messages in Google Sheet Export to inform users that they can’t enter multiple SQL statements in the export. 
  • Platform Usage Overview now includes a message to contact support@civisanalytics.com in order to learn more about increasing your organization's Platform compute resources. 

Bug Fixes

  • We fixed a bug in Database Import and Export that when using a non-default Platform database credential, the schema and table dropdowns reflect the list of schemas and tables that credential has access to on the Platform database. 
  • We fixed a bug on the Facebook Import that returned only a subset of the expected data.

Week ending in 1/22/2021

Updates

  • For the /tables/{id}/columns endpoint, we updated the API parameter description for distinct_count to clarify that it accounts for NULL values.  
  • Similar to notebooks and services, run history logs for container/r/python scripts will now indicate when the workload cannot run with the configured memory/CPU parameters because the organization is nearing its maximum compute resource usage. To monitor your organization’s compute resources, check out Platform Usage Overview.

Bug Fixes

  • For database imports, we fixed a couple case sensitivity issues for tables in case-insensitive databases.  If the source database is case insensitive, we downcase the source table's column names to preserve consistency across case insensitive databases.   For more details on case sensitivity support, please refer to this documentation.

Week ending in 1/15/2021

Updates

  • For database imports, we added case sensitivity support for column names for Postgres, MSSQL, and MySQL.  For more detail, see documentation. 
  • Within a user’s profile page in the admin center, we added a true/false robot user field to indicate whether the account is a robot user. To edit the robot user field, navigate to the edit user modal. 
  • Security updates

Week ending in 12/25/2020

Updates

  1. We removed the column tooltips from the Table Details pane to improve readability.  
  2. We added additional warnings in database sync jobs when the primary key used in the sync is not of a sequential data type.   For reference, documentation on primary key setting is available here. 
  3. Shortcut Assume Role: Under the profile menu in Platform, we’ve added shortcut options to your most recently assumed roles from Admin Center in Platform. This significantly reduces the number of clicks needed to switch roles in Platform and retains the user's page while switching roles.
  4. Expanded Privileges when Assuming Role: Now when assuming a robot’s role you can create and edit credentials and generate API keys. Previously, this required users to login as the robot account rather than assuming its role, which can be painful because of shared passwords. Note that this requires the user to be marked as a robot in Admin Center.
  5. We improved the retry behavior for CASS/NCOA jobs to improve stability

Bug Fixes

  1. We fixed a database sync bug so that users with access to multiple Snowflake databases do not have issues running database syncs.  
  2. We fixed links to CASS/NCOA jobs to point to the Enhancements page
  3. We fixed a CASS/NCOA permissions issue that caused some jobs with different users for author and “running as” to fail

Week ending in 12/18/2020

Updates

  • We updated various dependencies to improve the long-term reliability of Identity Resolution and Civis Data Match, and we improved the matching algorithm’s use of frequency information related to first and last names.
  • Documentation on Platform roles now available here.

Bug Fixes

  • We fixed an export to SQL Server bug so that integer data types are properly mapped to SQL server. 
  • We fixed a couple Google Sheet export bugs such that: 
    • When using the “Create Document” export option and the destination spreadsheet name is left empty, exports will not fail and still be able to save results.
    • When the destination spreadsheet is updated, export will save results to the updated spreadsheet.

Week ending in 12/11/2020

Updates

  • IDR usage limit enforcement has been initialized.  Clients will receive email notifications as they approach and exceed usage milestones. For more information read IDR Usage Policy.
  • The “Services” dropdown (located underneath the top-right initials menu) has been renamed to “Remote Hosts”.

Bug Fixes

  • We fixed a SQL script but so that a runner’s access to input and output files are validated appropriately and said runner can access these files with proper access.
  • We resolved an issue that caused duplicate “Failed to get outputs. Request failed with status code 503” error messages on Platform workloads. 

Week ending in 12/4/2020

No updates

Week ending in 11/27/2020

Bug Fixes

  • We fixed Google Sheet Exports so that users can export SQL queries into google sheets up to 5 million cells

Week ending in 11/20/2020

Bug Fixes

  • We fixed a CASS/NCOA bug so that users can access the output files quickly. 
  • We fixed a Google Sheet Export bug so that when users create or update the sync job of a Google Sheet Export, the proper credential is used.

Week ending in 11/13/2020

No updates due to Platform code freeze.

Week ending in 11/6/2020

Updates

  • Documentation on how to interpret maximum memory & cpu usage now available here.
  • Service reports can now be archived in Platform and via the API. To archive a service report in Platform, navigate to the report’s standard action menu and select “Archive”.

Week ending in 10/31/2020

Bug Fixes

  • We fixed the “Add Repository” button in the add git repo side pane.

Week ending in 10/24/2020

Updates

  • Updated the “Monthly Compute Hours” field on Platform Usage Overview to account for hours used on multiple instance types.

Bug Fixes

  • We fixed a Redshift import bug when a system table returned 0 results.  
  • We resolved a Salesforce objects bulk API import bug which was creating columns that were too wide for Redshift's varchar column width.

Week ending in 10/17/2020

Updates

  • Logging for maximum CPU used and maximum memory used is now available for python, R, and container scripts as well as notebooks and services.

Bug Fixes

  • We fixed a Google Sheets Export bug so that the export will not fail when there are extra whitespaces after a semicolon.
  • We fixed an expiration setting bug so that users can patch an unexpired file and set a NULL expires_at date.

Week ending in 10/9/2020

Updates

  • We simplified the database import experience by removing deprecated setting Chunking Method from Database Import
  • We updated our Civis Python API client to v1.15.0.  This update improves retry behavior that should help when network connections are unreliable and removes Python 3.5 support.  More information here. 
  • We updated the Civis Python Notebook, Python Script, and Container Script Docker Images
    • base datascience-python version → v6.3.0
    • civis-jupyter-python3 →  v2.3.0

Bug Fixes

  • Resolved a bug causing the Import Multi from Civis script to fail because of invalid UTF-8 characters

Week ending in 10/2/2020

Updates

  • Users can now view enhanced error messages for failed pipelines in the Identity Resolution tool. New troubleshooting documentation on Common IDR Errors can be found in Zendesk.

Bug Fixes

  • Resolved a bug showing incorrect load errors on CSV file imports. 
  • Resolved a bug on database syncs so that users can now make constant updates to their source tables while incrementally syncing to Platform. 
  • Resolved a bug on CSV File file imports so that users will get correct error messages when a Redshift copy fails.
  • Resolved a bug where a workflow execution task failed though the failed icon did not appear.

Week ending in 9/25/2020

Updates

  • Users can now view enhanced error messages for failed pipelines in the Identity Resolution tool. New troubleshooting documentation on Common IDR Errors can be found in Zendesk.

Bug Fixes

  • Resolved a bug showing incorrect load errors on CSV file imports. 
  • Resolved a bug causing the Import Multi from Civis script to fail because of invalid UTF-8 characters.

Week ending in 9/18/2020

Bug Fixes

  • Resolved a bug so that users can view the full table name in Table Details, even if the table name is very long 
  • Resolved a bug so that database sync will not fail if the update_column contains null values

Week ending in 9/11/2020

Updates

  • To improve client support and issue resolution on database syncs, we standardized our implementation logic on how we import and export data across databases.  

Bug Fixes

  • Resolved a bug that automatically redirected users to a workflow execution page after navigating to a new pane in Platform.

Week ending in 9/4/2020

Updates

  • Identity Resolution customers can now view their monthly pipeline usage via our new IDR Usage Dashboard found under Account in the upper right hand corner of the IDR homepage. 
  • New Script Template Detail Page allows users to manage template sharing and monitor Custom Scripts created from a template all in one place.
  • Custom Scripts created from a template now have a clean new look.
  • Custom Scripts in runner context now have resource fields under Settings (CPU, Memory and Disk Space). Previously, Custom Scripts inherited the resource settings from the backing script, but now have added flexibility to allow for resource optimization.
  • We updated our workflows documentation to provide examples of passing parameters into scripts within workflows. More details available here.

Bug Fixes

  • We fixed a bug preventing users from deploying services with max resources.

Week ending in 8/28/2020

Updates

  • Users can hide executed queries in the query pane so that they can quickly sift through query history to locate relevant executed queries 
  • Security updates

Bug Fixes

  • A Google Sheets Export job is not intended for executing an arbitrary number of SQL statements to prepare data for export.  Users now receive validation errors when their Google Sheet Export jobs contain more than one SQL statement.

Week ending in 8/21/2020

Updates

  • We updated the show workflows pane so that users can now open any listed workflows on a new tab.
  • Users can now sync tables whose names are reserved words onto their database.
  • Database syncs now support SLL encrypted connections for Oracle. More details available here.

Week ending in 8/14/2020

Updates

  • The R environments on Civis Platform will be upgraded to R 4.0.0 on 2020-08-10. It contains a breaking change to a default option when reading data into memory: following the upgrade, the stringsAsFactors option will be set to FALSE instead of TRUE in data.frame, read.table, and read.csv/read_civis. For more info, see this page.
  • Workflows now support the action std.fail, which can be used to manually fail a task in a workflow. More details available here.
  • Platform Usage Overview is now available to view and monitor your organization’s compute resources. More details available here.

Bug Fixes

  • Fixed a bug that caused Workflow’s UI to be blank when using a conditional fail.
  • Resolved a bug being raised in the POST /workflows endpoint that led to a “(500) An unexpected condition was encountered” error message.
  • Removed double ‘add’ button on sharing pane.

Week ending in 8/7/2020

Updates

  • Documentation on Encrypting Database Connections with SSL in Database Syncs now available here.

Bug Fixes

  • Resolved a data type mapping error for INT and DATETIME data types when importing from Oracle databases.
  • JDBC remote host urls can no longer start with whitespace. The extra whitespace previously led to database import errors. 
  • Sharing an import will now also share any associated syncs which are active.

Week ending in 7/31/2020

Updates

  • All user data files across Platform will expire 30 days after they are created. 
  • Clicking an export link generated by selecting “export results” on the query page will now open in a new platform tab.
  • Schemas without tables or views will now appear in search results within the data pane. 
  • Documentation on how to test a database connection in Platform is now available here.

Bug Fixes

  • Resolved a bug with malformed parameters used when selecting the “Create Report” button. 

Week ending in 7/24/2020

Bug Fixes

  • Removed a 404 error message that populated when attempting to populate schema or table drop downs for non-platform databases. Moving forward, users can enter free text to populate the schema and table fields for non-platform databases. 
  • Updated import and export logging to check whether child job(s) run(s) have failed prior to returning a “Successful Import” message.
  • Included regex matching support for grants when users and/or groups are quoted.

Updates

  • Users should now be able to open workflow execution links in new tabs either through right-click or cmd+click.

Week ending in 7/17/2020

Bug Fixes

  • Users can now add import and export jobs (i.e. database export, CSV import, Salesforce import & export) to workflows via the UI graph view.
  • When switching to a different workflow execution, parameters should now update and reflect the parameters that were actually used in the execution.

Updates

  • The sample records view in Identity Resolution should now work when multiple phone number columns are mapped for a single source.
  • Civis Data Match now avoids creating an empty output table if the job fails due to invalid configuration (e.g. if the input table doesn’t exist).
  • Security and infrastructure updates

Week ending in 7/10/2020

Bug Fixes

  • We resolved a bug that prevented users from being able to shut down a credentialed notebook successfully.
     

Week ending in 7/3/2020

Updates

  • We added logging to SQL script and CSV export runs to increase visibility into use of cached exports and empower users to identify when there is an issue with their results. Now, users should be able to refer to the run logs and understand if a given run used a cached export as well as the date that the cached export was originally created.
  • The single table scanner option has been removed from the admin dropdown. This functionality is now available in the Data Pane.

 

Week ending in 6/26/2020

Updates

  • When a SQL Script or CSV Export runs and returns a cached run output, a log line is added to the run log with the date and time the cached output was created.
  • We updated workflows so that the execute button becomes disabled when required parameter fields are either missing or invalid.
  • Security Updates

 

Week ending in 6/19/2020

Bugfixes

  • We fixed a bug in Google Sheet Imports and Exports that sometimes caused a new Worksheet selection to not save.
  • We fixed a bug in Google Sheet Exports that sometimes caused the Worksheet to not clear before the data was written to the Sheet.
  • We fixed a bug that sometimes caused existing Google Sheet Exports or Imports to not load. 

 

Week ending in 6/12/2020

Updates

  • Version v1.14.2 of the Python API Client has been released. It includes bug fixes and documentation improvements as well as support for the latest version of Python (3.8). More details can be found here.
  • Projects can now store more than 20 child projects.

Bugfixes

  • We fixed a bug that caused extra white space padding on Tableau reports.

Week ending in 6/5/2020

Updates

  • We updated our Google Sheet Import and Export jobs to reference the Worksheet and Spreadsheet by ID rather than name on the backend. Now, Import and Export jobs won’t break if a Worksheet or Spreadsheet name is changed and users can select Sheets with duplicated names. No action required by users to use this update.
  • Security Updates

 

Week ending in 5/29/2020

Updates

  • We deprecated support for specifying partitions in MySQL database syncs. 

Bugfixes

  • We fixed a bug in SQL scripts and CSV exports that caused the delimiter to always be a comma, regardless of user specifications.

 

Week ending in 5/22/2020

Updates

  • We updated our Google Sheet Import and Export jobs to support multiple Google credentials per user. Users can now select the credential they want to use from the credential dropdown rather than re-authenticate each time they want to use a different credential.
  • We updated our Google Sheet Export to not create an additional blank Sheet when exporting data.
  • We updated our export functionality to ensure that files exported using the export/files/csv endpoint have a file extension consistent with the specified compression and column_delimiter. This does not apply when force_multifile is true.

Bugfixes

  • We fixed a bug that caused some Workflows to fail when running concurrently.

 

Week ending in 5/15/2020

Announcements

  • Action required ONLY if you reference your Google credential ID in your code. For the vast majority of users who use Google Sheet Imports/Exports via the UI, no action is needed:
    • As part of the improvements we are making to our integration with Google Sheets, users who reference their Google credential ID in their code will need to make a one-time change. After an upcoming update on May 14, 2020, when a user re-authenticates their Google credential, the credential ID will change. Any references to the ID in code will break and need to be updated with the new ID. 
  • We have deprecated the /tables/{id}/enhancements/{enhancement_type} endpoints. We intend to remove them in 2021. Users should now use the /enhancements/{enhancement_type} API endpoints, which have the same functionality. For example, users should change their code to use the /enhancements/cass-ncoa endpoint instead of /tables/{id}/enhancements/cass-ncoa.

Updates

  • We updated the Google authentication process for Google Imports and Exports so that the user only needs to validate their credentials once. Previously, the user had to validate the credentials twice. 
  • To clarify requirements for upserts in the CSV Imports job, we updated the job to fail earlier if the user did not provide both the primary and last modified keys when creating a new table.
  • We added in-line validation for missing but required fields on Database sync jobs. 
  • We added support for S3 bucket names that include periods when importing or exporting data using the export/files/csv and import/files csv endpoints. 

Bugfixes

  • We fixed a bug that caused database imports to fail when importing smalldatetime data types into Postgres databases. 
  • We fixed a bug that caused margin padding on Tableau public links.

 

Week ending in 5/8/2020

What’s New

  • We added the data_updated_at date to the Data Pane and Table Details Page in Platform. This date is the last time that Civis Platform captured a change in this table. This date is only applicable for Redshift tables. Read more here. 

Updates

  • We moved the table description on the Table Details page to be more visible - right below the table name. Users can add descriptions to their tables and use Platform search to find a table based on the description. 
  • We added a column delimiter (columnDelimiter) parameter to the imports/files/csv and exports/files/csv endpoints. 
  • When a user selects “truncate long text” for a MYSQL database export, the export now truncates to the max SQLSERVER limit rather than the max VARCHAR redshift limit to ensure the job runs successfully.
  • We added a schema_updated_at field to the tables/id API endpoint. This date is the last time that Platform captured a change to the table attributes/structure. It is only applicable for Redshift tables.

Bugfixes

  • We fixed a bug that prevented users from committing to github from Platform.

Week ending in 5/1/2020

Updates

  • Workflows
    • Users now have the ability to only run one Workflow execution at a time. Users may want to choose this setting to ensure their more recent executions don't catch up to a previously-running execution and fail. Learn more here.
    • To prevent more manual work, canceling a Workflow now cancels all the associated jobs and action executions.
  • Matching and Identity Resolution
    • We now support multiple phone number columns (e.g., home phone, mobile phone).
    • We made some changes to the matching algorithm to better handle first or last name values with spaces and/or hyphens.
  • We updated our Civis Python API client to v1.14.0. This update removes support for Python 2.7 and 3.4 and adds an .outputs method to retrieve outputs from CivisFuture Objects. More information here.
  • Security Updates

Bugfixes

  • We fixed a bug that only allowed Workflow owners to update Workflow notifications. Now, anyone with manage or edit permission on the Workflow can update the Workflow.

Week ending in 4/24/2020

Updates

  • We released our new files export API endpoint! This endpoint allows users to directly export data to their own S3 bucket and is more performant because it does not run a SQL script to export the data. Find out more here

Bugfixes

  • We fixed a bug that sometimes caused the Enhancements index page not to load.

Week ending in 4/17/2020

Updates

  • We updated the JDBC driver version for our MySQL database sync job in order to support MySQL 8 database versions. For users who use SSL parameters in their database URL, we automatically updated the settings to the supported parameters. More information here.

Bugfixes

  • We fixed a bug that sometimes caused the run history pane and query results not to load. Now, users are able to open the run history pane, view expired files (but not download them), and click “view results” for queries greater than six months (although the actual results are deleted). Users can rerun the query to see the results. 
  • We fixed a bug that sometimes caused the Enhancements index page not to load.

Week ending in 4/10/2020

Updates

  • We completed a rewrite of our state management system that enables Platform to handle more load and reduce the occurrence of stalled jobs, notebooks and service, and missed state notifications e.g. I cancelled my job but it still hasn't been cancelled. This is part of the ongoing stability project prioritized for 2020. 
  • We now allow users to update the file name and expiration date of S3 files at the Civis files endpoint. Details here. 
  • We updated our database sync so that only MySQL database syncs can run without a specified schema.

Week ending in 4/3/2020

Updates

  • We updated our Google Sheet export to only export columns that include data when exporting to a new worksheet. This will help users not run into the Google Sheet cell limit because we are not exporting empty cells.

Bugfixes

  • We fixed a bug that caused the table details links on the query tool to break if the table name was mixed case. 
  • We fixed a bug that prevented table statistics from being updated if the table had a column with an oid data type.

Week ending in 3/27/2020

Bugfixes

  • We fixed a bug that allowed users to select "(Display limited to 50 search results)" in the table drop down for import and export jobs.

Week ending in 3/20/2020

Updates

  • When adding a new database to Platform via the database sync, users can test the connection before running the actual sync job. We improved this testing experience by adding detailed error messages so users can understand what is incorrect (the username/password, JDBC URL format, etc.)

Bugfixes

  • We fixed a bug in our Google Sheet Export that created a new spreadsheet when multiple spreadsheets of the same name were found. This now throws an error instead of creating a new spreadsheet.
  • We fixed a bug that caused the update table details button to spin infinitely if the table was altered while the table details refresh was running.

 

Week Ending 2/24/2020

Updates:

  • We updated our Facebook Import to output an aggregated page insights table, which is more streamlined and easy to use. 
  • We released a new version of the Import from S/FTP script with the ability to add a private key credential.
  • We added the ability to filter for archived Workflows on the Workflow index page.
  • Users can now add a name to their database when adding it to Platform via the Database Sync job.

Bugfixes:

  • We fixed a bug that caused the table details page not to load because of a non-ASCII byte in the column name.
  • We fixed an error in dbsyncs where BIT columns for Postgres databases would sometimes fail the sync.



Week Ending 2/18/2020

What's New:

  • We now support Database Imports into non-Platform Redshift clusters.
  • We now support the use of SSL key pairs for Database Imports with Postgres databases.

Updates:

  • We’ve released a new version of civis-python, 1.12.1. This release includes a change to civis.io functions to allow users to pass in multiple files, along with some minor bugfixes For details, see the 1.12.0 and 1.12.1 release notes here: https://github.com/civisanalytics/civis-python/releases
  • We have also updated the Python package versions for notebooks and scripts. This includes updates to data science packages such as scikit-learn, pandas, and tensorflow. For details, see the full release notes here: https://github.com/civisanalytics/datascience-python/releases
  • If you use Python scripts or notebooks, you will get access to these changes by default.

Bugfixes:

  • We fixed a bug that caused some Database Imports with BIT column types to incorrectly cast to boolean when their length was greater than 1.
  • We fixed a bug in our Google Sheet Export to more consistently support more than 100k rows of data.
  • We fixed a bug that prevented users from adding a Google Sheet Import to a Workflow in the UI. 
  • We fixed a bug in the Compute Usage Report that caused some jobs to appear to be running even after they had completed.

 

Week Ending 2/11/2020

Updates:

  • We updated the CSV export created from the query tool to default to gzip compression, and we no longer surface the corresponding sql script in the UI.
  • We added support for Postgres dates > 9999. The future is now!
  • Users can now access the standard action menu for the last object in the projects detail page, which was previously blocked by the help widget.

Bugfixes:

  • We fixed a bug in our CSV Import that mistakenly prevented mixed-case imports to Redshift.
  • We fixed a bug that caused SQL scripts created via the API with the column delimiter set to a non-comma delimiter option to output files as comma delimited.
  • We fixed a bug that caused multiple error messages to appear (rather than a single message) that prevented users from accessing certain areas of Scripts and Imports.

Week Ending 2/4/2020

Updates:

  • We updated our email notifications to differentiate between failed and canceled jobs. 
  • In the Python API client, we updated dataframe_to_civis, csv_to_civis, and civis_file_to_table to support existing_table_rows=upsert and added the following parameters: primary_keys, last_modified_keys, escaped, and execution, which are all useful for upsert mode.
  • Also in the Python API client, we updated civis_to_multifile_csv to include the max_file_size parameter, which allows users to control the size of their output files.
  • We added a convenience method get_storage_host_id to retrieve the id of a storage host based on its name. This is to be used with client owned s3 buckets.
  • We’ve updated the list of supported Notebook Docker images. We support v1.9 and higher for the civis-jupyter-python3 image and v1.8 and v1.10 for the civis-jupyter-r image. More information on supported Docker images can be found here.

Week Ending 1/28/2020

What's New:

  • We've updated our Google Sheets integration to support Team Drives. This update includes a new interface to select the worksheet you want to import. No action is required to run existing jobs.

Updates:

  • We removed the ability to create Python 2 Notebooks from the top navigation menu. We recommend users create Python 3 Notebooks moving forward.

Bugfixes:

  • We fixed a bug that caused the running_as user to update on a script when a user clicked Run Now without intentionally switching the script to run with their credentials. 
  • We fixed a bug that sometimes caused a Workflow to fail if one of the tasks kicked off an additional Workflow.
  • We fixed a bug that caused report viewers to be redirected to the home page if they navigated to a report URL in a new tab or window.

Week Ending 1/21/2020

Updates:

  • We added scheduled and run state filters to the workflows index in the Civis API, enabling users to list and filter workflows by these attributes based on the most recent workflow execution.
  • We made table scanner (an automatic job that populates the metadata on the table details page) more performant by removing unnecessary data calculations
  • We updated SQL Scripts running on a Postgres database to not return an empty file if the result set only contains headers.
  • We added support for the Oracle NUMBER datatype in the Database Import job.
  • Security Updates.

Bugfixes:

  • We fixed a bug in our Oracle Database Import that prevented connections for particular Oracle database versions due to Oracle issue ORA-01882.
  • We fixed a bug that caused Google credentials to update from the job's running_as user when a person was assuming the role of another user.
  • We fixed a bug that caused DATES to be mapped as VARCHARs when importing data to a Redshift database.
  • We fixed a bug that caused auto-complete in the query tool to not display any tables if the schema name contained letters after numbers.

Week Ending 1/14/2020

Bugfixes:

  • We fixed a bug that caused archived objects to show up in a Project when not filtering for archived objects.
  • We fixed a bug that caused table scanner jobs to fail on Postgres clusters if the table included a column that only had NaN values.
  • We fixed a bug so that our query autocomplete works with table names that include numbers.

Week Ending 1/7/2020

Updates:

  • It's a new year, and we've updated our R images! We've updated to R 3.6.1 which contains the most recent tidyverse updates. Across notebooks, services, and scripts we've also added geospatial system dependencies and 30+ of the most commonly installed R packages for visualization, geospatial analysis, and modeling. The default shiny image now also includes 15 additional packages useful for shiny development and interactive visualization.
  • We've added the following updates to the projects detail page:
    • Display last run information for enhancements (e.g. Civis Data Match)
    • Ability to download files through the standard action menu
  • We added a max_file_size parameter to SQL scripts and CSV imports so that users control the max file size on parallel unloads from redshifts
  • When creating or editing Certificate credentials there is now a multi-line entry form for the password
  • We added include_header and column_delimiter parameters for preprocessing csv files so that users have more control over how the data types in their file are detected
  • Security Updates

Bugfixes:

  • We fixed a bug that caused sql scripts to fail when a single quote was escaped by a backslash or when a backslash was being used to escape another backslash and not a single quote
  • We fixed a bug that caused a Platform redirect error once an object was removed from a project
  • We fixed a bug where a valid JDBC url for an Oracle database could not be used for creating a remote host
  • We fixed a bug with creating a table when performing a database sync to an Oracle database
  • We fixed a bug that caused sql scripts executed with an unload that contained sql with a semi-colon as a string literal to fail
  • We fixed a bug that caused sql scripts executed with an unload to return no results on Redshift clusters with concurrency scaling enable
  • We fixed a bug where a table scan was using two connections to the Platform data warehouse instead of one

Week Ending 12/17/2019

Updates:

  • Notice: Civis R Client will be disabled for v1.5.1 and prior
    • We are making changes to our infrastructure to make it more stable and secure. As a reminder, we will disable the Civis R Client v1.5.1 and prior on December 18, 2019. Please upgrade to v1.6.0 or higher to successfully use the client. If you use any of the following Docker images in any R Scripts, Notebooks, or Shiny applications, they will also need to be upgraded to at least the following versions:
      • datascience-r 2.8.0
      • civis-jupyter-r 1.6.0
      • civis-services-shiny 1.2.0
    • Docker image versions used in R Notebooks and Shiny Applications can be updated via the UI. Docker image versions used in Scripts can be updated via the API. Please email support@civisanalytics.com with any questions.
  • We've added the following updates to the Projects detail page:
    • Filters for active, archived, and hidden objects in the Project detail page
    • Disabling the 'me' checkbox if current user is not an author on any of the objects
    • File ID visibility next to file name [Ex: My_Platform_File (123456)]

Bugfixes:

  • We fixed a bug that caused SQL Scripts to fail if the query included a semicolon in the SELECT statement.
  • We updated our Workflow logic so that if a Workflow execution fails to be added to Mistral (our workflow service), we now retry adding it to Mistral rather than failing the execution in Platform.

 

 

Week Ending 12/10/2019

What's New:

  • We have released the following updates for Platform Projects:
    • Favoriting: Users can favorite projects and easily locate them on the left-hand projects side pane.
    • Nested Projects: Projects can now be nested within other projects.
    • Sharing: When sharing a project, users now have the ability to share all objects within the project (note: excludes tables, objects in nested projects).
    • Notes Side Pane: New side pane with markdown notes section
    • Standardized Platform Design and Behavior: Standard Action menu (e.g. share, archive) found at the project and object level
    • And More! Project descriptions and visibility into object meta-data 
    • To learn more, visit our Introduction to Projects Help Documentation.

Updates:

Bugfixes:

  • We fixed a bug that allowed users to create an infinite loop in a Workflow by having a task kick off itself. 

Week Ending 12/3/2019

Updates:

  • Civis Matching: We improved the use of latitude and longitude fields for finding potential matches and for computing match scores.
  • We added support for Redshift data type GEOMETRY.

Bugfixes:

  • When a user unloads data from Platform via civis_to_multifile_csv the manifest entry's filename extension will now include both its delimiter (csv) and its compression (.gz) where relevant.
  • We fixed a bug that caused the table scanner job (an automatic job that populates the metadata on the data pane and table details page) to fail if a table had a column that was dropped and the remaining columns had comments.
  • We fixed a bug that prevented users from importing multiple CSV files to Postgres within the same Import.

Week Ending 11/26/2019

What's New:

  • We’ve added the capability to embed values and tables defined in Python, R and container scripts in job notification emails. To leverage this feature, your script should upload any values or tables you want to include in your email as a run output JSONValue named "email_outputs". More information here.

Week Ending 11/19/2019

What's New:

  • We’ve added support for stall warnings on Workflows. Similar to the stall warning feature for jobs, this feature will alert you if a Workflow execution is taking longer than expected, enabling you to monitor and address any issues with key Workflows for your organization. More information can be found here.

Updates:

  • We added type loosening capabilities to the /imports/files/csv endpoint so that column types can be updated to fit the data being imported rather than causing the import to fail. For example, varchar (42) might be created as varchar (1024) to accommodate data larger than 42 characters.

Bugfixes:

  • We fixed a bug that caused the file contained within an exported zip archive to also have a .zip extension.
  • We fixed a bug in our Google Sheet export job that overwrote data in a column when the column name was duplicative.
  • We fixed a bug that broke the zip compression of exported files. 
  • We removed table level comments from column comments on the table details page. Now table level comments will only appear on the data pane.

Week Ending 11/12/2019

Updates:

  • We updated Tableau public links to no longer redirect logged in users to the full Civis Platform Reports page. This will prevent users from sharing the wrong report URL (the URL asks users to sign in) to non-Platform users. 
  • We updated our Notebook health checks to wait longer before throwing a “failed health check” warning to allow for more computationally heavy commands to finish running.

Bugfixes:

  • We updated our database imports to retry rather than fail if the job throws a reconnection error. 
  • We fixed an issue that caused multi-file export manifest entries to be improperly decompressed on download.

Week Ending 11/5/2019

Updates:

  • We updated our Database import so that users can specify the modified_at column that will be used for incremental syncs.

Bugfixes:

  • We fixed a bug that prevented users from clicking “run with my credentials” on a Google Sheet Export. Users are now able to switch the job to run with their credentials, but will need to manually refresh the page after clicking “run with my credentials” in order for the changes to take effect. 
  • We fixed a bug that caused errors when a user used the new CSV import endpoints for an upsert with delayed execution and changed the destination table column names.

 

Week Ending 10/29/2019

Updates:

  • On Friday 11/1/2019, we are deprecating support for the Export People to VAN, Export a list to VAN, and Export a score to VAN Labs jobs. Additionally, we are removing support for the Export Queue Table to VAN template script. You will no longer be able to create new jobs of these types.
    • We recommend leveraging our new script templates instead: Export Civis Data to NGPVAN (template id: 19204) and Export Saved List to VAN (template id: 30573). Please reach out to your Client Success Analyst for access to these scripts.
  • We added a limit of 100 to the number of action executions that can be kicked off concurrently for a workflow task.

Bugfixes:

  • We fixed a bug that caused MySQL Database Imports to error when updating data on a table with a primary key when there was no schema.

Week Ending 10/22/2019

Bugfixes:

  • We fixed a bug that caused imports to intermittently fail with "Couldn't find Database::Table" errors.
  • We fixed a bug that caused all Boolean values to convert to 0 when imported into a MySQL database.
  • We fixed a bug that prevented the sync of Boolean data types from a Redshift or Postgres databases to all Database types when creating a new table.

Week Ending 10/15/2019

What's New:

  • A New and Improved Notebooks Page! Now, when you access the Notebooks pages in Platform (Python3, Python2, or R), you will see new and updated capabilities!
    • Standardized Platform Design and Behavior: Action buttons (share, clone, schedule, etc.) are now found under a standard action menu at the top of the page. We also updated the side pane design to be more visually and functionally consistent based on user feedback.
    • A new full screen experience that gives you additional room to work in your Notebook.
    • Visibility and Status: Accurate warnings and error messages so you can understand when something went wrong and how to correct it. 
    • Stability and More Stability: Backend updates to ensure Notebooks are reliable and always ready to use.

Updates:

  • Security and stability updates 

Week Ending 10/8/2019

What's New:

  • We added a new file cleaning endpoint that gives users the flexibility to select how they want to preprocess their file before uploading it into Platform. This includes header row, delimiter, compression and column type detection as well as addressing file properties (i.e., conversion to UTF-8, removal of byte order markers, etc.) that may prevent importing the file into a database. This does not include preprocessing the data in the file. This endpoint is to be used with our streamlined CSV Import API endpoint so that users can balance their import performance versus preprocessing needs.

Updates:

  • We are making changes to our infrastructure to make it more stable and secure. As part of this change, we are deprecating the Civis R Client v1.5.1 and prior on December 9, 2019. Please upgrade to v1.6.0 or higher to successfully use the client. If you use any of the following Docker images in any R Scripts, Notebooks, or Shiny applications, they will also need to be upgraded to at least the following versions:
    • datascience-r 2.8.0
    • civis-jupyter-r 1.6.0
    • civis-services-shiny 1.2.0
  • Matching
    • We added clearer, earlier error messages in Civis Data Match if the input table has null primary keys for some records.
    • We made minor improvements and bug fixes to the use of location information in Civis Data Match and Identity Resolution.
    • We made a minor fix to how name fields are preprocessed for use in Civis Data Match and Identity Resolution.
  • We optimized Platform's tracking of Redshift schema and table grants to reduce the workload on Redshift.

Week Ending 10/1/2019

Updates:

  • We deprecated Vertica Database syncs. 
  • We added name (name of the import) and redshiftDestinationOptions.diststyle (the diststyle to add to a table) fields to the imports/files/csv endpoint. More information about this endpoint can be found here. 

Bugfixes:

  • We fixed a bug that broke Google Sheet Imports if there was a unicode character (such as an emoji or &) in the sheet title. 
  • We fixed a bug that caused VAN List imports to fail if the list name contained a slash. 
  • We fixed a bug that caused reports to display only partially when viewed on a Chromebook or in Safari.

Week Ending 9/24/2019

Updates:

  • We made improvements to our infrastructure to better handle periods of increased load.

Bugfixes:

  • We fixed a bug in the nightly table scanner job (an automatic job that populates the metadata on the data pane and table details page) to ensure it runs successfully when syncing information about schemas. 
  • We fixed an issue that caused imports to fail due to the case of column names not matching the case of columns for already existing tables. 

Week Ending 9/17/2019

Updates:

  • We made an update so that changing your database password will take effect more quickly. 
  • We updated our imports to fold distkeys to lowercase when importing data into a Redshift database. This had previously prevented imports from running successfully.

Bugfixes:

  • We fixed a workflow bug that caused updates to the YAML file to not always save when the page was refreshed.  
  • We fixed a bug that prevented a Google Sheet import from running successfully if the sheet name contained a period.
  • We fixed a bug in our geocoding job that prevented the job from running successfully. 
  • We fixed a bug that showed the old Platform UI when a user updated their two factor preferences.

Week Ending 9/10/2019

Updates:

  • We updated the Civis R client to v2.1.0
    • This update diminishes some of the pain of rate limits. We now automatically retry all endpoints if a user has been rate limited (respecting the ‘Retry-After’ header). Additionally, the user can specify a global polling interval to help prevent rate-limiting issues when running many long-running processes in parallel. You can see detailed changes in the CHANGELOG and browse the new documentation here.
  • Updated datascience-r docker image to v3.1.0
    • rocker/verse -> 3.6.0
    • civis-r -> 2.1.0
    • civis-python -> 1.11.0
  • Updated civis-jupyter-r to v1.8.0
    • rocker/verse -> 3.6.0
    • R -> 3.6.0
    • civis-r -> 2.1.0
  • Updated civis-services-shiny to v1.4.0
    • Upgraded base image datascience-r from 3.0.0 -> 3.1.0
  • We made reliability updates that keep Platform up with minimal downgraded functionality if Elasticsearch (part of our backend infrastructure) goes down.
  • We deprecated Silverpop and Catalist jobs since they are no longer supported. Civis provides other tools that can be used to move this data into Platform. 

Bugfixes:

  • We fixed a bug that prevented users from testing their RemoteHost connection on the “Add Source” side pane.
  • We fixed a bug that prevented Geocoding jobs from running successfully.

Week Ending 9/3/2019

Updates:

  • We reduced the number of connections opened when running a query or SQL script.
  • We added an error message to the Scripts page that warns a user if they are using a Docker image that they do not have permission to use.
  • We updated our two factor login options so that an organization's administrator can decide which type of two factor authentication (SMS vs. app) users should use.

Bugfixes:

  • We fixed a bug that sent some users multiple SMS codes when they tried to log in to Platform with SMS two factor authentication.
  • We fixed a bug to prevent inactive users from receiving job email notifications.
  • We fixed a bug that caused data from CSV imports to sometimes be encoded incorrectly.
  • We fixed a bug that caused SQL scripts to fail on Postgres clusters if there was a semicolon at the end of the script.
  • We fixed a bug that caused table scanner (an automatic job that populates metadata on the table details page and data pane) to not pick up statements that included "TRUNCATE TABLE..."
  • We fixed a Google Sheet export bug that created a blank Google Sheet even if the export failed because the worksheet name was blank.

Week Ending 8/27/2019

Updates:

  • We made performance improvements to the Table Scanner and Database Status jobs. These jobs are automatically run by Platform and are used to populate the data pane. 
  • Truncate, revoke, and grant statements (when performed outside SQL Scripts and the Civis Query tool) are now picked up by the Database Status job, and the appropriate updates will now show up in the data pane.
  • We removed the “Visualize the Query” button from the query tool due to downgraded functionality. 
  • We removed the “Import from Box” and “Export from Box” options from the Import and Export menus due to downgraded functionality. 
  • We updated a misleading Notebook error messages to no longer print “User successfully terminated the service” when the Notebook shutdown by itself.

Bugfixes:

  • We fixed a typo on the preference menu.

Week Ending 8/20/2019

Updates:

  • To keep users informed about the activity on their account, we added an email notification (sent to the email address associated with the user’s account) when a user creates an API key.

Bugfixes:

  • We fixed a bug so that you can now view a custom script even if you are not shared on the template. 
  • We fixed a bug on the workflow execution page that was causing some parts of the page to not always load.

Week Ending 8/13/2019

Updates:

  • We removed the underutilized priority Redshift query queue in order to reallocate additional memory to the general Redshift query queue. This should result in improved query performance for users.
  • We optimized the performance of SQL queries used by a Platform backend job (“table scanner”) that is used to populate the data pane and the metadata in the table details page.
  • We removed query caching for late binding views to ensure query results are always up-to-date.

Bugfixes:

  • There was a bug that prevented column metadata from being populated in the table details page for columns that are oid data types, and thus caused the nightly table scanner job to fail. This is now resolved.

Week Ending 8/6/2019 

Updates:

  • We no longer support query caching for Postgres databases. 
  • After reviewing the new Redshift default behavior of column compression and conferring with the AWS Redshift team directly, we decided to switch back to the new default behavior. We did this because we believe that this setting will provide the best user experience across all possible data sets and data load scenarios. Specifically, we expect to see a decrease in CPU usage and execution time for Import jobs. We also experienced some intermittent bugs in our database sync tool that were associated with forcing column compression. These should now be resolved.
  • Geocoding Enhancement:
    • We improved the interface between backend systems and client clusters to fix an issue with long-running geocoding jobs.
  • Matching:
    • We improved sensitivity to the frequency of names. Matching on rare names (e.g., “Keanu”) should lead to larger relative increases in scores than matching on common names (e.g., “John”).
    • We added support for partial matching on ZIP codes by prefix, since people often move to nearby ZIP codes (e.g., 60614, 60657).
    • We improved heuristics and weighting around date of birth features.
    • We added a fix to ensure duplicate primary keys that don’t match any Civis Data records are removed from the output.

Bugfixes

  • We fixed a bug that prevented some accented characters from being imported correctly into Redshift.
  • We fixed a bug so that views will have their metadata refreshed overnight if their dependent table’s metadata is also refreshed.
  • We fixed a bug so that SQL scripts will output the refreshed data from a view if the underlying table is updated. 
  • We fixed a bug where notebooks would not load if the notebook became too big.
  • We fixed a bug on the report share pane so that it displays all the groups a user has permission to grant on. 
  • We fixed a bug so that tables starting with “pg” will appear in the data pane. 

Week Ending 7/30/2019

Bugfixes:

  • We fixed a bug so that a query cache for a view is invalidated if the query cache for the associated table is invalidated. Before the fix, views sometimes were not updated even when their associated table was updated. 
  • We fixed a bug so that users can now see column details in the data pane for views with no schema binding.
  • We fixed a bug that caused SFTP exports to fail when exporting large amounts of data. 
  • We fixed a bug so that when a user alters the schema name, the tables within that schema won’t disappear from the data pane.
  • We fixed a bug so that columns that contain the characters ‘zip’ don’t automatically map to the varchar datatype. 
  • Security Updates

Week Ending 7/23/2019

Updates:

  • We added the template script ID to the More Script Templates side pane. Now users can easily find the template script ID in the UI, rather than using the API. 
  • We added database and port to the list of available environment variables when using the database parameter in scripts. This will allow users to connect easily to clusters directly in their scripts. 
  • We updated our email notification so that they contain the entire job log. Previously the email was sent before all the logs were printed. 
  • We made a frontend update to improve the load time of all import, script, workflow, report, and service detail pages.
  • We moved the job status (success or failure) to the beginning of the subject line of the workflow notification emails. 
  • We updated our Google Sheet export so that the errors are more understandable. Now the errors give information about rerunning the job or checking job settings when a job fails. 
  • We added last_successful_run parameter to the scripts/custom API endpoint so that users can debug their script and find the last time it ran successfully.

Bugfixes:

  • We experienced latency issues with some Platform features last week due to issues with our underlying job queueing system. We took actions to fix the issues and all jobs should be running as expected. 
  • We fixed a bug that made tables disappear in the data pane if a user updated the schema name. Now the tables will remain visible even if the schema name is updated.
  • When importing data in Platform, if a diststyle is not set, it will now default to EVEN, rather than AUTO. This was the default behavior of redshift before Amazon made some changes last week. We are bringing this default back manually because the AWS changes introduced bugs.
  • We fixed a bug that prevented users from importing data to Postgres databases.

Week Ending 7/16/2019

What's New:

  • We added an import script that enables users to move tables into platform from Google BigQuery. Navigate to Code -> More Script Templates -> Import Big Query Table.

Updates:

  • We turned on column compression for all Imports. This is because Amazon released a change to Redshift during the week of June 30th that changed the default behavior of how column compression (COMPUPDATE parameter) is handled in the Redshift COPY command. Turning on column compression should make Import jobs behave as closely as possible to how they did before the change to Redshift, however we are working closely with the AWS Redshift team to better understand their change and the impact it has on Civis Platform. For more information, see Redshift cluster version history.
  • We now convert a file to utf-8 encoding if we are unable to detect the character set encoding of the file. This will prevent import jobs from failing since the data must be encoded as utf-8 to load onto the database. This update will resolve the error “undefined method `shellescape' for nil:NilClass” users saw in the past. 
  • We updated our /scripts endpoint so that the Scripts page loads more quickly.

Bugfixes:

  • We fixed a bug that caused Projects to not always load in the project side pane.
  • We fixed a bug to hide the help widget in report full screen mode.
  • We fixed a bug in our Database import, most commonly seen in MySQL database imports, that will prevent the error “ERROR: gzip: unexpected end of stream”.

Week Ending 7/9/2019

Upgrades and Updates:

  • We released a new fullscreen experience for reports. Now, when a user clicks on the fullscreen icon, the data pane and report title is hidden. Users can “un-fullscreen” their report by clicking on the exit button located at the top of the report.
  • We added the ability to sort by relevance on the script template side pane, so users can find the script templates they want more easily.
  • In Platform search results, we hid the count of “all” objects when filtering only for “mine” for a more clear user experience.

Bugfixes:

  • We fixed a bug so that users can now update the title of a service report (a shiny report, for example).
  • We fixed a bug with our query editor to resolve the “wrong argument type nil” error message.
  • We fixed a bug so that the list of available Redshift clusters is in alphabetical order.
  • We fixed a bug with our Google Sheet Import so that “first row is header = auto” now runs successfully.
  • We fixed a bug so that table scanner (an automatic job that populates the metadata on the table details page) works on tables without owners.

Week Ending 7/2/2019

Bugfixes:

  • We updated our Google Doc export to support worksheet titles of any length with accented characters.

Week Ending 6/25/2019

Upgrades and Updates:

  • We updated our Rails version to v5.2.3 to continue security best practices.
  • We modified our Mobile Commons import integration to be backwards compatible with old field names, while still supporting the latest API return values for new users of the integration or anyone who has updated their tables with the new DDL
  • We added additional fields to the jobs/[id] endpoint:
    • successEmailSubject (the subject line of a success notification email)
    • successEmailBody (the contents of a success notification email)
    • runningAsUser (the user account that the job runs as)
    • runByUser (the user that kicked off the job)
  • We deprecated the ability to email reports due to unreliable functionality.
  • We updated civis-r docker image to v2.0.0. You can see detailed changes in the CHANGELOG and browse the new documentation here.
    • Breaking changes:
      • write_civis_file now uploads data frames as CSVs to S3 by default. So simply writing:  id <- write_civis_file(df)
        will upload the data frame as a CSV.
      • Previously this function would upload a data frame as an RDS object instead of a CSV. This should be more in line with what users expect when uploading data frames, and helps increase portability between Python and R. To minimize the extent of breaking changes in code, read_civis now reads file IDs as CSVs by default as well.
    • New Features:
      • We have some new functions that simplify working with Civis Scripts. It’s now much easier to read and write script outputs and to run templates and arbitrary expressions in containers. These functions should help make your productionizing efforts a little easier!
      • civis_script annotates an integer as a valid custom, container, or other platform script.
      • A new method read_civis.civis_script is added to easily read script outputs into memory.
      • fetch_output_file_ids returns file ids of run outputs for a civis_script.
      • fetch_output returns the output of scripts_list_*_runs_outputs for a civis_script.
      • write_job_output adds File run outputs to scripts.
      • run_civis can be used to run an arbitrary R expression in a Civis Platform container.
      • run_template can be used to run a template job, and return file IDs of results.
      • To see some examples about how these new functions work, take a look at our new vignette, Productionizing with Civis Scripts, which explains how to automate tasks related to scripts using civis-r and illustrates these new helper functions.
  • We updated datascience-r docker image to v3.0.0
    • rocker/verse -> 3.5.3
    • civis-r -> 2.0.0
    • civis-python -> 1.10.0
    • added requirements.txt
    • added package zoo
  • Updated civis-services-shiny docker image to v1.3.0
  • Upgraded base image datascience-r from 2.8.0 -> 3.0.0

Bugfixes:

  • We fixed a bug in our support for views without schema binding so that the column information now appears in the data pane when a user updates the underlying table.
  • We fixed a bug so that the result of a TRUNCATE query updates the stats on the table details page and outputs the correct result of the query after the TRUNCATE statement. Previously we cached the old query results, so the output didn’t reflect updates made by the TRUNCATE statement.

Week Ending 6/18/2019

Upgrades and Updates:

  • We added the ability to use the new, more streamlined [CSV Import API endpoint](https://platform.civisanalytics.com/spa/#/api#post_imports_files_csv) to import files directly from your S3 bucket. As a reminder, this API endpoint has a number of benefits, including:
    • Faster importing
    • A new “upsert” mode for more efficient data refreshes
    • The ability to import a file containing a subset of the destination table’s columns
    • The ability to import multiple files to the same table in a single run
    • For more information about this endpoint see the help documentation.
  • We adjusted our column name cleaning logic used in Imports from Facebook, Files, Google Analytics, Google Sheet, SFTP, and URL to preserve all leading numbers in a column name, not just the first one.
  • We added a check for the number of input records to the Address Correction job. A run will fail if the input is empty.
  • We added a new endpoint GET /jobs/{id}/runs/{run_id}/outputs for getting a list of run outputs for any job run.
  • The “Help Docs” link was renamed “Help” for simplicity.
  • We added a detectedInfo object to the response for GET /files/{id} to house more information about your file, including whether it has headers, what the delimiter is, and what the column names and data types are. We will be adding a new job soon that you can run to populate that information.

Bug Fixes:

  • We fixed some issues with Reports created from Services:
    • We fixed an issue preventing these reports from being renamed.
    • We fixed the fullscreen report view so there is only one scrollbar in long reports.
    • We fixed an issue preventing form submission in these reports.
  • We fixed an issue so that multiple Platform organizations can use the same Single Sign-On (SSO) configuration settings.

Week Ending 6/11/2019

Bug Fixes:

  • We fixed a bug where tables wouldn't properly appear in the data pane when being granted via a statement that included a comma-separate list of tables. Previously only the first table is the list would appear in the data pane.
  • We fixed a bug on the Database Status page that prevented some users from loading the page.
  • We fixed a bug so that users can now import a large number of files concurrently from an SFTP into Platform.

Week Ending 6/4/2019

Upgrades and Updates:

  • We added support for timestamp data without timezones in Postgres Databases on Platform, so users can import their data without altering it before importing into Platform.

Bug Fixes:

  • We fixed an intermittent bug that some users were experiencing, where jobs either failed with 503 errors or failed to complete at all. Jobs are now completing successfully.
  • We fixed a bug in SQL Scripts so that they no longer return an output file from the previous run if the most recent SQL query has no results.
  • We fixed a bug in our Export Queue Table to NGPVAN so that null values for the contactTypeID field can be exported from Platform into VAN.

Week Ending 5/28/2019

Upgrades and Updates:

  • For Civis Data Match, Table Deduplication, and Data Unification jobs, we updated the way we handle large data to increase efficiency and cut down on runtime.
  • We updated CivisML to v2.2.4 which includes a bugfix for report rendering issues.
  • We updated our Salesforce Export error logging to display more helpful error messages to the user.

Bug Fixes:

  • We fixed a bug that prevented some queries and imports from running successfully.

Week Ending 5/21/2019

Upgrades and Updates:

  • We updated our Database Import job to allow users to leave the destination table name blank when using a wildcard for the source table name. If the destination table name is left blank, the tables will inherit the names from the source.
  • We made minor improvements to the Civis Data Match model for the Civis Data Match job and Identity Resolution Pipelines.
  • We updated all of the Civis Notebook Docker Images
    • civis-jupyter-python2 to 1.6.1
    • civis-jupyter-python3 to 1.12.0
    • civis-jupyter-r to 1.6.1

Bug Fixes:

  • We fixed a Workflows bug so that the visual representation of a Workflow appears in the UI even if you set advanced features - specifically conditional task dependencies (on success, on error, etc.).
  • We fixed a data type issue that caused Geocoding jobs to fail when low-accuracy responses were recorded.

Week Ending 5/14/2019

Upgrades and Updates:

  • We updated how we create match targets to use the latest information on duplicate records and records of the deceased.

Bug Fixes:

  • We fixed a bug so the new CSV import API endpoint ( /imports/files/csv) respects the max error parameter. Users can use this parameter to specify the maximum number of rows to exclude from their import if those rows contain data that cannot be imported into Redshift.
  • We fixed two bugs on our MobileCommons import so that users can successfully import Members and Broadcast data into Platform.

Week Ending 5/7/2019

Upgrades and Updates:

  • We updated Jupyter Python 3 Notebooks to use version 1.11 of the civis-jupyter-python3 image. There is one small update that enables an upcoming feature.

Bug Fixes:

  • We fixed a handful of issues with the new Reports page, including:
    • An issue where some reports weren’t loading
    • We made the page title match the report’s title
    • An issue with searching for users and groups in the Sharing pane
    • An issue where the Help widget wasn’t loading
  • We removed broken Help links in File Imports.

Week Ending 4/30/2019

Upgrades and Updates:

  • We added support for wildcard characters in our Database Imports. Now, you can use the “%” symbol in your source tablename to sync all tables in a schema or a subset of tables in a schema with a specific prefix or suffix.
  • We updated the Container Script datascience-python default Docker image tag to v5.0.0 and Notebook civis-jupyter-notebook version to v0.4.5. Existing Container Scripts and Notebook Docker images will not be updated, but new scripts will have the updated Docker image tag so that users have the most current packages.

Bug Fixes:

  • We fixed a bug so that users can now create Remote Hosts from the Remote Host index page in Platform. Users can create Remote Hosts when they want to connect to external databases, for example, VAN, Blue State Digital, or a MySQL Database.


Week Ending 4/23/2019

What's New:

  • We added a new, more streamlined CSV import: /imports/files/csv. This API endpoint has a number of benefits, including:
    • Faster importing
    • A new “upsert” mode for more efficient data refreshes
    • The ability to import a file containing a subset of the destination table’s columns
    • The ability to import multiple files to the same table in a single run
    • For more information about this endpoint see the help documentation.
  • We released a new reports page (Tableau, HTML, and Services) with new and updated capabilities. Changes include:
    • A standard action menu for all global action buttons so that you have a more clean and streamlined UI experience
    • Save buttons and warnings on all side panes so that you understand what changes you are making to a job

Week Ending 4/16/2019

Upgrades and Updates:

  • We added deployment logs to Notebooks (under the run history button) so you can understand what is happening under the hood while your notebook is starting up.
  • We renamed the "type" field in the "column" object to "sqlType" and added a new "civisDataType" field that assigns a more generic data type to each column for the following endpoints:
    • GET /tables/:id
    • GET /tables/:id/columns
  • We added the ability to update the docker image tag via the API so that you can use the most up to date docker image on R and Python scripts.
  • Added
    • CivisFuture has the job_id and run_id property attributes
  • Fixed
    • Polling will treat None responses generated by spotty internet connections like responses with a non-DONE state.
  • We updated civis-python client to v1.10.0. There are two small updates in this release- one update to make it easier for you to access basic information about your jobs and one bug fix.

Bug Fixes 

  • We fixed a bug so that you will only receive an email notification when your phone number is updated on the profile page. Previously, emails were sent even if no changes were made and you pressed the save button.
  • A bug in the modeling wizard caused it to break for some users. Now everyone can use the modeling wizard to build models without writing code.
  • We fixed a typo on the URL import description on the top navigation menu.

Week Ending 4/9/2019

Upgrades and Updates:

  • You can now hover over the failed status icon in a workflow execution log to quickly and easily see any failure messages.
  • We added security recommendations on the SQL script parameter side pane to explain how you can use variables in your code to prevent sql-injection attacks.
  • We added the ability to import the `money` data type into a Postgres database so that you don’t need to transform your data before importing it into Civis.

Bugfixes:

  • We fixed a bug so that you can toggle between Tableau reports loaded in multiple tabs.
  • We fixed a bug to populate the `type` parameter description for the `/remote_hosts` API endpoint so that you know which remote host types Platform currently supports.

Week Ending 4/2/2019

 Upgrades and Updates:

  • We deprecated the `enhancements/person_matching` endpoint. This was the final step in the rollout of our new matching endpoints, which were released in 2018. These new endpoints accomplish a few things: They automatically implement any updates made to the Civis matching algorithm; and, they allow you to add matching jobs to workflows and schedule and share match jobs. The new endpoints are:
  • We made API searches more granular: Now, you can filter API search results by `type = jobs` so that the returned results include any data jobs (scripts, imports and exports), but not results from static workflows and notebooks.
  • Where do we send you when an error occurs? Previously, you’d be routed either to a working page - or a blank one. We’ve updated this so that, whenever an error occurs, you’ll always be taken back to a working page
  • Security Update to our API

Week Ending 3/26/2019

Upgrades and Updates:

  • We added an error notification to notify you when resource constraints prevent you from starting a Notebook.

Bugfixes:

  • We added the ability to import nvarchar data types into a Postgres database. Now, you don’t need to transform your data before it is imported into Platform, regardless of whether you’re using Redshift or Postgres as your database of choice.
  • We updated job logging to ensure that jobs won’t fail when the logs are really long.

Week Ending 3/19/2019

Upgrades and Updates:

  • We updated our Google Sheet import to allow apostrophes and backslashes in the Sheet title so that you don’t need to alter your title before importing it into Platform.
  • We updated the “alter group” command in the query tool to include schema grants so that new users granted to the group can immediately start working with their data.
  • Security Updates
  • We’ve updated CivisML to v2.3.3. Along with several other improvements, this version also fixes a bug about resource allocation for job parallelization so that jobs can run more quickly.
  • We’ve updated datascience-python docker image to v5.0.0. As part of this upgrade, a wide set of services also received an upgrade that won’t change your experience working in Platform.
    • Ubuntu 14.04 -> 18.04
    • python 3.6.4 -> 3.7.1
    • conda 4.3.30 -> 4.6.8
    • explicitly installs click=6.7
    • awscli 1.15.4 -> 1.16.121
    • beautifulsoup4 4.5.3 -> 4.7.1
    • botocore 1.10.4 -> 1.12.111
    • boto 2.46.1 -> 2.49.0
    • boto3 1.7.4 -> 1.9.111
    • bqplot 0.10.2 -> 0.11.5
    • cloudpickle 0.5.2 -> 0.8.0
    • cython 0.27.3 -> 0.29.6
    • dask 0.17.2 -> 1.1.4
    • ipython 6.1.0 -> 7.3.0
    • ipywidgets 7.1.0 -> 7.4.2
    • jinja2 2.9.6 -> 2.10
    • jsonschema 2.5.1 -> 3.0.1
    • libtiff 4.0.6 -> 4.0.10
    • libxml2 2.9.2 -> 2.9.8
    • matplotlib 2.2.2 -> 3.0.3
    • notebook 5.4.1 -> 5.7.5
    • numexpr 2.6.2 -> 2.6.9
    • numpy 1.13.3 -> 1.16.2
    • openblas 0.2.20 -> 0.3.5
    • pandas 0.22.0 -> 0.24.1
    • patsy 0.4.1 -> 0.5.1
    • psycopg2 2.6.2 -> 2.7.7
    • pyarrow 0.8.0 -> 0.12.1
    • pytest 3.1.3 -> 4.3.0
    • pyyaml 3.12 -> 3.13
    • requests 2.18.4 -> 2.21.0
    • s3fs 0.1.2 -> 0.2.0
    • seaborn 0.8 -> 0.9.0
    • scipy 1.0.1 -> 1.2.0
    • scikit-learn 0.19.1 -> 0.19.2
    • statsmodels 0.8.0 -> 0.9.0
    • urllib3 1.22 -> 1.24.1
    • xgboost 0.6a2 -> 0.81
    • civis 1.9.0 -> 1.9.4
    • civisml-extensions 0.1.8 -> 0.1.10
    • dropbox 7.1.1 -> 9.3.0
    • glmnet 2.0.0 -> 2.1.1
    • muffnn 2.1.0 -> 2.2.0
    • pubnub 4.0.13 -> 4.1.2
    • requests-toolbelt 0.8.0 -> 0.9.1
    • tensorflow 1.7.0 -> 1.13.
    • Changed
    • New Packages
    • Package Updates
  • Finally, we updated civis-jupyter-python2 docker image in notebooks to v1.6.0. There are several services updates included in this upgrade that do not affect user experience or functionality:
    • civis 1.9.0 -> 1.9.4
    • civisml-extensions 0.1.8 -> 0.1.10
    • numpy 1.13.3 -> 1.14.3 (fixes an error with the tensorflow binary)
    • Package Updates

Bugfixes:

  • We fixed a bug so that all users with permissions on a table can now click the Update Table Details button on the Table details page to refresh the table metadata on that page and in the Data Pane.

Week Ending 3/12/2019

Upgrades and Updates:

  • We’ve changed the way Platform handles imports from Salesforce: Now, Platform will retry importing partial batches of data from Salesforce rather than automatically failing after an error and requiring you to restart the entire import.
  • You can now set up Tableau user filtering without the need for our Client Success team to update each user’s Platform profile.
  • We’ve also made it easier for you to filter for the set of credentials you need for a specific database by adding a parameter (remoteHostId) to the GET /credential endpoint.

Bugfixes:

  • We fixed a bug that prevented you from updating the settings of a Google Sheet Export if you are not the owner of the export.
  • We fixed a bug that was causing sporadic 404 errors when loading Service reports.

Week Ending 3/5/2019

What’s New:

  • User level filtering to Tableau reports, which was in Beta for a few customers, is now Generally Available. You can now honor user-level filters set up on your Tableau reports using a new toggle in the sharing pane.
  • We’ve also added an API endpoint to create report templates to save you some time when creating reports multiple times.
  • For geocoding, you’re now able to parse an address within a geocoding job for use in other analyses, and we’ve added support for a few additional geocoding providers that integrate seamlessly with Platform (Nominatim and geocoder_ca [for Canadian geographies]).
  • Finally, we’ve added support for IDENTITY column types for Postgres-to-Redshift Database imports, making it easier for you to move data from one to the other.

Upgrades and Updates:

  • We’ve updated the sharing email notification to the correct URL link for Reports created from Services.
  • We’ve added some updates to help Services and Notebooks start faster.
  • Finally, we’ve updated the Civis Python client to v1.9.3. The key highlights are:
    • The SQL command get_table_id can now handle quoted `schema.tablename`.
    • Run cancellation endpoints now return correct empty responses rather than an error.

Bugfixes:

  • We’ve fixed a permissioning issue where a user needed to be the manager of a person in their entire set of groups in order to change their permissions in a single group. Now, permissions can be managed on an individual group level.
  • Fixed-height reports didn't automatically load with a scroll bar. That has been fixed.

Week Ending 2/26/2019

  • Added the ability to run particular segments of a Workflow, instead of the entire thing.
  • Added the ability for scripts to output any JSON value (e.g string, integer, array, dictionary) as an Output in the Run History pane. Tasks in a workflow can pass this value to the next task.
  • Added the ability to send a notification email to users or groups you are sharing an object with via the sharing side pane.
  • Added custom filtering to the logs in the Deployment History pane to filter by time increments.
  • Added a Bitbucket integration to Notebooks and SQL, R, Python, and JavaScript Scripts.
  • Added a new POST /tables/scan endpoint for updating table info and metadata in the data pane.
  • Added more specific error messages when available in Google Sheet Imports and Exports.
  • Added a warning if the table created by a Salesforce Import includes truncated values or records.
  • Added scriptType parameter to GET /templates/scripts/{id}.
  • Added support for session reuse for FTP_TLS servers to the Import from S/FTP job.
  • Fixed an issue causing the job that refreshes the data pane and table metadata to fail when trying to refresh data on a large number of tables.
  • Fixed an issue with the /match_targets/ endpoints.

Week Ending 2/19/2019

  • Released new Script parameter type, Database, which displays dropdowns for both selecting a Platform database and database credential to be referenced in the Script.
  • Added the option to set a fixed value for a Script parameter, using the optional “value” field in the “params” object in /scripts/. Learn more here.
  • Added the option to restrict a Script parameter to certain allowed values, using the “allowedValues” field in the “params” object in /scripts/. This displays as a dropdown in the UI for the Script. Learn more here.
  • Updated Civis R client to v1.6.1
    • Write_civis.numeric now correctly syncs with headers
    • Write_civis gains header, credential_id and import_args arguments to more flexibly import data
    • Added coef.civis_ml function which returns civis_ml model coefficients in the format of stats::coef
    • Added get_feature_importance to civis_ml_utils
  • Updated datascience-r docker image to v2.8.0
    • rocker/verse -> 3.5.2
    • civis-r -> 1.6.1
    • civis-python -> 1.9.3
  • Updated civis-jupyter-r to v1.6.0
    • rocker/verse -> 3.5.2
    • R -> 3.5.2
    • civis-r -> 1.6.1
  • Updated civis-services-shiny to v1.2.0
    • Upgraded base image datascience-r from 2.7.0 -> 2.8.0
  • Fixed a 500 error upon login
  • Fixed an issue where Notebooks would fail to start
  • Fixed an issue where users were seeing a blank page upon opening a new tab
  • Fixed a credential issue in Shapefile Imports
  • Fixed an issue in Database Exports when quotes inside a field were followed by a line feed character
  • Deprecated Person Matching job in Labs
  • Began deprecation of the Box Import and Export jobs
  • Hid the “New Report” button on the Reports page for Report-Viewer users
  • Minor updates to API endpoints to enable future features
  • Deprecate the CONSOLE_API_KEY environment variable inside of Container, Python, and R scripts in Platform.

Week Ending 2/8/2019

  • Updated Civis python client to v1.9.3
    • Fixed a table formatting issue in the scoring step of CivisML when no primary key column is present
    • Added explicit support for Python 3.7
    • Note: Civis-supported docker images for notebooks and container scripts continue to use v1.9.2 by default. They will be updated soon.
  • Updated Civis Data Match, Table Deduplication, and Data Unification jobs to use person-matching v3.2.1.
    • Improved matching on last name, address unit, zip code, and incomplete birthdates (only month or birthday)
    • Improved handling of ties between match pairs if only best match is desired
    • Improved matching on email only or on phone number only, when name fields are not provided
  • Better handle user-defined data types in the job that populates table metadata
  • Fixed 500 errors in CivisML job
  • Fixed an issue with NULL fields in Database Export jobs when the destination is a database that isn’t Redshift, MySQL, Vertica, or Postgres.
  • Fixed an issue in Database Export jobs when exporting numeric fields with high precision when the destination is a database that isn’t Redshift, MySQL, Vertica, or Postgres
  • Fixed an issue in Database Export jobs when exporting string fields that contain backslashes when the destination is a database that isn’t Redshift, Vertica, or Postgres
  • Made some changes to prepare for the upcoming deprecation of the Person Matching job in Labs for most users
  • Made some updates to API endpoints to enable upcoming features
  • Security updates

Week Ending 2/1/2019

  • Updated CivisML to v2.2.2
    • Catch feature selection errors in child jobs and replace with a clearer error message
    • Round predictions to the nearest 1e-20 for Redshift compatibility
    • Enforce float32 precision for predictions to prevent Redshift overflow errors
    • Scoring tables no longer contain a "None" column when no primary key is specified
    • Raise a RuntimeError if there are more than 5000 levels in a column
    • Emit a warning if the column levels during transform don't overlap at all with levels from fitting
  • Added a new “Person Data Standardization” tool under Tools > Enhancements. This job takes a table with personally identifiable information (PII; e.g., names, addresses, dates of birth), and standardizes the PII. See this article for more information.
  • Added new Import and Export jobs for working with shapefiles and geodatabases (For use with Postgres only)
  • Added new enhancements/geocode endpoints that include the full set of features standard for Platform jobs -- adding to a Project, sharing, etc
  • Added an error message when there is an issue loading a page that tells the user to refresh their browser
  • Minor styling changes to the left Projects pane
  • Fixed a bug for exporting to MS Sql Server for NULL numeric columns and numeric columns with a precision over 10

Week Ending 1/25/2019

  • The File Import job in the UI now supports importing files up to 5 TB in size.
  • The primaryKeys parameter at the /tables/{id} endpoint automatically updates to match the database after a table scan.
  • Minor updates to API documentation

Week Ending 1/18/2019

  • Fixed a bug where logging in always redirected to the homepage, even if the user originally entered a more specific URL within Platform.
  • Added a new “Welcome to Civis” walkthrough for new users.
  • Made improvements to performance during periods of heavy usage.
  • Security updates.

Week Ending 1/11/2019

  • Fixed an issue with Container Scripts that reference large GitHub repositories.
  • Fixed an issue where workflow executions would fail to start.

Week Ending 12/28/2018

  • Added new parameters for primaryKeys and lastModifiedKeys to the /tables/{id} endpoints.

Week Ending 12/21/2018

  • Improved error messages on imports when multiple line endings are detected in the file.
  • The LIST /imports API endpoint now allows you to filter database imports by the source.
  • Created the GET /jobs/:id/runs/:run_id/logs API endpoint to see logs for any job type.
  • Fixed a bug in Address Correction where a temporary table was not always dropped.
  • Security updates.

Week Ending 12/14/2018

  • Fixed an issue where Imports from File, Google Sheet, S/FTP, or URL would fail if the column names specified as the dist key or sort key were cleaned by the job.
  • Fixed an issue in the Address Correction job causing an “undefined method `each' for true:TrueClass” error.
  • Clarified in the API docs the expected value for the limitingSQL parameter in /enhancements/cass-ncoa endpoints.
  • Removed the POST /tables endpoint.
  • Security updates.

Week Ending 12/7/2018

  • Updated the Civis Python API client to v1.9.2.
    • Improved handling of S3 connection errors by civis.io.civis_to_file.
  • Prevented Address Correction jobs from failing if the input table doesn’t have metadata.
  • Fixed an error in Address Correction where temp tables weren’t being dropped sometimes.
  • Removed outdated IP addresses from the UI for setting up a New Database and replaced with a link to a help center article with the current addresses.
  • Fixed an issue where the column names and data types weren’t editable in the File Import job UI if the file has quoted headers.
  • Fixed a bug with setting session authorization in Query.
  • Fixed an issue with fixed credentials on script templates.
  • Prevented unnecessary job failures in Imports from Facebook, File, Google Analytics, SFTP, and URL.
  • Fixed an issue where Google Sheet imports were failing when some headers were blank.
  • Fixed an issue with the table preview in File Imports when the table had CR line endings.
  • User information is now passed to services via service tokens.
  • Allow authors of SQL script templates to set fixed credentials on the template in runner context and non-fixed credentials in author context.
  • Security updates.

Week Ending 11/16/2018

  • Reduced page load times.
  • Upgraded Tableau Server from 10.5 to 2018.2.
  • Upgraded the Python API Client to v1.9.1.
  • Fixed a bug in civis.io.civis_to_multifile_csv.
  • Fixed a bug in CivisML due to a recent pandas update.
  • Added more documentation for Windows users.
  • Other documentation updates.
  • Added file IDs on the project’s page for files in the project.
  • Fixed an issue where tables created from Database Imports weren’t showing up right away in the left pane.
  • Improved error messages in File Imports.
  • Improved efficiency of scaling down instances after using Notebooks and Services.

Week Ending 11/9/2018

  • Italicized the names of table views in the left pane.
  • The width of the left pane is now adjustable by clicking on its right edge and dragging left or right.
  • Fixed a typo in “More Enhancements” menu.
  • Security updates

Week Ending 11/2/2018

  • Added the memory used by a notebook deployment to the deployment logs, available at GET /notebooks/{id}/deployments/{deployment_id}/logs.
  • Fixed an issue where committing a Notebook was sometimes not responsive.
  • Fixed an issue causing SQL scripts to fail with “Run Error: Connection timed out” error.
  • Fixed an issue where cancelled SQL scripts were stuck in “running” state and were not sending failure notification emails.
  • Fixed an issue where clicking on a model on the Models page would fail to load the page for that Model, redirecting the user back to the Models page.
  • Added “adapter” field to the response from the GET /databases/ endpoint to indicate the type of database.
  • Added a new GET /databases/{id} endpoint which responds with the database id, name, and adapter.
  • Changed "Change Redshift Password" on My Profile page to "Change Database Password".
  • Fixed an issue where dropping a table did not also drop its ontology, which prevented a new table with the same name from being created in the future.
  • Added GET /clusters/kubernetes/{id}/deployment_stats endpoint which returns active and queued jobs.
  • Security updates.

Week Ending 10/26/2018

  • New Platform navigation!
  • Updated login page to reflect new Civis logo and color scheme.
  • Updated favicon to be new Civis logo.
  • Added CPU, memory, and disk space parameters to Imports from S/FTP.
  • Fixed an issue where the API keys page would automatically refresh when a new key was created.
  • Improved the reliability of updates to the information in the data pane and table details.
  • Fixed an issue with Salesforce imports encountering end of file errors
  • Added instanceMaxDisk to the /clusters/kubernetes endpoint.
  • Upgraded Civis Data Match, Table Deduplication, and Data Unification jobs to use person-matching v3.1.2.
    • Increased the speed of the Civis Data Match job.
    • Minor improvements to the handling of phone numbers, latitude, and longitude.
      • E.g. the algorithm now ignores common fake phone numbers like 1234567890.
    • Output table name is now always lowercase.

Week Ending 10/19/2018

  • Added support for curly brackets and other symbols in remote host JDBC URLs, enabling connections to databases with spaces in their names.
  • Added support for published SQL scripts to have either fixed or user-supplied credentials regardless of author context.
  • Improved performance of Imports from Facebook, File, Google Analytics, SFTP, and URL when “if table exists” is set to append or wipe.
  • Security updates

Week Ending 10/12/2018

  • Fixed an issue where some Imports and Exports were not showing up in the job search in Workflows.
  • Fixed an issue where Notebooks were not taking up the whole height of the screen.
  • Added export templates to the GET /search/ API endpoint.
  • Changed the email footer on success notifications so it shows the email address of the user who runs the job instead of the owner of the job.
  • Fixed an issue with our maximum memory calculations on Notebooks and Services. The maximum memory users can allocate was reduced slightly.
  • Security updates.

Week Ending 10/5/2018

  • Upgraded Civis Data Match, Table Deduplication, and Data Unification jobs to use person-matching v3.0.1
    • Correction: Previous version used was v2.1.1. The job was not updated to 3.0.0 during week of 2018.09.14-2018.09.20
    • Updated model used for scoring
    • Refactored the Data Unification job to cut runtime in about half and reduce memory usage by about 20%
    • Fixed a bug with matching on nicknames
    • Fixed a bug in preparing data for matching
    • Improved handling of special characters
    • Fixed an issue with parsing data in the jobs
    • Jobs now fail when there are parsing issues instead of attempting to match records
  • Fixed broken links to help documentation in File Imports jobs
  • Searching on the Users page now supports search terms with more than 10 characters
  • Improved the reliability of updates to the information in the data pane and table details, and reduced the cluster resources consumed by update jobs.
  • Fixed an issue that prevented grant queries from succeeding
  • Security updates

Week Ending 9/28/2018

  • Improved detection and handling of delimiters and line endings in Imports from File, Box, Facebook, Google Analytics, S/FTP, and URL.
  • Implemented logic that prevents jobs and deployments from failing when there is an issue with logs.
  • Made service deployments with a custom application port more reliable.
  • Fixed a bug where a table wouldn’t show up in the data pane after its name was changed.
  • Resolved error messages showing up in Import from Facebook jobs.
  • Updated our Import from Facebook job to use v3.0 of the facebook-sdk.

Week Ending 9/21/2018

  • Upgraded Civis Data Match, Table Deduplication, and Data Unification jobs to use person-matching v3.0.0.
    • Changed model used for scoring.
    • Improved handling of special characters.
    • Fixed a bug with matching on nicknames.
    • Fixed a bug in preparing data for matching.
  • Added a preview of the Service below the Settings. Removed the option to deploy a preview.
  • Improved the handling of “execution expired” errors.
  • Security updates.

Week Ending 9/14/2018

  • Upgraded Civis Data Match, Table Deduplication, and Data Unification jobs to use person-matching v2.1.1
    • Security update
  • Removed the thumbnail view from the Reports page
  • Improved performance of SQL scripts
  • Fixed an issue where dropping a table did not also drop its ontology, which prevented a new table with the same name from being created in the future
  • Fixed an issue with services that use the viewer's API key
  • Fixed an issue causing rate card imports to fail
  • Performance improvements
  • Security updates

Week Ending 9/7/2018

  • Fixed an issue where an already running job that a workflow tries to kick off shows as idle instead of failed.
  • Added the option for new users to use an authenticator app for two-factor authentication when they set up their account.
  • Renamed the “Redshift credential” parameter type in the Scripts to “Database credential”
  • Security update

Week Ending 8/31/2018

  • Increased the security of the Civis VPN login by implementing a temporary ban after failed login attempts.
  • Added additional vulnerability detection to Tableau and Platform servers.
  • Fixed an issue where Import from S/FTP jobs sometimes timed out.
  • Updated Services to improve hardware utilization .
  • Fixed an issue that prevented the export icon displaying in project details
  • Fixed an error causing script jobs to fail intermittently.

Week Ending 8/24/2018

  • Changed the Project description default to be blank instead of “default project description”.
  • Added the ability for organizations to have some of their users use Single Sign On to log into Platform and others log in using passwords. Contact Client Success if you’re interested in implementing this for your organization.
  • Rounded the upper memory limit for Notebooks and Services down to nearest integer.
  • Fixed an issue where Services that are set to be always on were still going to sleep.

Week Ending 8/17/2018

  • Fixed an issue where R scripts using the ‘civis’ package were failing
  • Fixed an issue where some jobs in a workflow were marked as “failed” even though they never ran
  • Made some changes to Salesforce imports to fix the “End of File Reached” error
  • Fixed an issue causing Google Sheet imports to fail
  • Improved reliability of logs for Services
  • Added support for importing and exporting larger files via S/FTP
  • Fixed an issue where the provideAPIKey attribute was being cleared when making any PATCH /reports/services request
  • Fixed a misspelling in the API documentation
  • A new version of the R client enables hyperband for stacking estimators in CivisML. If an API key is present but the auto-generation fails, we now revert to the default API client.

Week Ending 8/10/2018

  • Fixed an issue that was causing notebooks to drop their connection
  • Fixed an issue where the workflow graph was not rendering if there were symbols in job names
  • Made reports default to full-screen when they’re opened from a project
  • Fixed an issue where the data pane was not updating right away to reflect new or dropped schemas when the create/drop statement used “create/drop schema if not exists”
  • Improved the reliability of service logs
  • Made the error message more helpful when a service cannot be deployed because there are no available instances
  • Added support for importing and exporting larger files via S/FTP
  • Added better error messaging to S/FTP imports and export

Week Ending 8/3/2018

  • Only the first 10 unique table names listed in a query get linked to their Table Details page. This is to improve performance on the Query page.
  • Fixed an issue where renamed schemas weren’t getting updated in the data pane right away if there was a line break in the ALTER SCHEMA query.
  • Added an error message to the Disk Space field on Container Scripts if an invalid value is entered.
  • Added CIVIS_SERVICE_ID and CIVIS_DEPLOYMENT_ID environment variables into the container run in a service.
  • Security updates

Week Ending 7/27/2018

  • Added a button on the My Profile page that can trigger a password-reset email.
  • Fixed an issue where Tableau reports would shift slightly when a user hovered over a tooltip within them.
  • Fixed an issue with updating the details of tables and schemas with long names.

Week Ending 7/20/2018

  • Limited all schema dropdowns in Imports to the first 50 schemas by default to avoid page freezes.
  • Fixed an issue where some logs in the Run History weren’t showing up.
  • Added information about the last run of a job to the GET /search/ response.
  • Security updates.

Week Ending 7/13/2018

  • Environment variables in scripts are no longer case sensitive.
  • Reduced the likelihood of imports failing due to connection issues.
  • Fixed an issue which was preventing some Container Scripts from finishing.
  • Fixed an issue where schemas granted or revoked with lowercase “grant” or “revoke” statements weren’t being updated right away in the data pane.

Week Ending 7/6/2018

  • Added a fix for the “Failed to send a recent heartbeat” error on some jobs.
  • Added validation to prevent a script created from a script template from being published as a new template.
  • Fixed alphabetical sorting in search.
  • Fixed a bug where exports to a Google Sheet with A1 notation in the name didn’t work correctly.
  • Added the archived status of app instances and releases to the GET API endpoints.
  • Security updates

Week Ending 6/29/2018

  • Added new API endpoints under /enhancements for the Civis Data Match, Data Unification, and Table Deduplication jobs.
  • Fixed a bug that caused imports from Postgres databases to fail.
  • Reverted last week's fix for GRANT statements on schemas after we found it caused jobs with GRANT statements to take too long and are working on a more permanent solution.

Week Ending 6/22/2018

  • Changed how we handle system failures so jobs are considered failed instead of canceled
  • Fixed an issue where grant statements on schemas were not working if the schema was lowercase
  • Added an error to the GET /scripts endpoint if the supplied type isn’t one of the supported ones
  • Enabled services to use a custom endpoint for health checks
  • Security updates

Week Ending 6/15/2018

  • In order to improve performance, limited the number of run outputs that can be referenced by a workflow to the first 100 that each job produces.
  • Added a “state” parameter to the credentials endpoint, which can be used for VAN credentials.

Week Ending 6/8/2018

  • Made some wording changes to API documentation to generalize terminology in places where it was too specific.
  • Fixed an issue that was causing some scripts to fail.
  • Fixed an issue where editing two-factor authentication settings was disabled for some users.
  • Fixed an issue where some credentials weren’t showing up in the credential dropdown in Query.
  • Fixed an issue where Services could not be deployed without a git repository.

Week Ending 6/1/2018

  • Nothing new this week, but exciting updates are in the works!

Week Ending 5/25/2018

  • Added Workflow ID and Execution ID as environment variables to Workflows available in the Configuration pane. They are accessible via `<% $.get('__env').workflow_id %>` and `<% $.get('__env').execution_id %>` in the YAML.

Week Ending 5/18/2018

  • Added more helpful details to the error messages for SQL scripts.
  • Cleaned up error messages in Query.
  • Fixed an issue with sending success notifications causing SQL script jobs to fail.
  • Fixed an issue with the syntax for BIGINT columns in the “Create Statement” tab in table details.
  • Fixed an issue with setting up two-factor authentication for some phone numbers.

Week Ending 5/11/2018

  • Fixed an issue where File Imports were ignoring the first row of data if it was not a header row.
  • Fixed an issue where Services could not be deployed without a Git repository.
  • Fixed an issue where users with their Redshift cluster on certain servers could not create a batch file import.
  • Fixed an issue in Media Optimizer when inexpensive spots were used.

Week Ending 5/4/2018

  • Upgraded the Civis API Python Client to v1.9.0
    • Fixed an issue which could prevent `civis.io.civis_to_file` from returning the entire Civis File object when writing to a user-supplied buffer.
    • Redshift schema / tablename splitting now correctly handles quoted identifiers.
    • Documentation fixes and improvements
    • Added `civis.io.export_to_civis_file` function.
    • Default to creating new APIClient objects with `resources="all"` parameter setting, instead of the previous `resources="base"`.
    • New version of CivisML (v2.2.0)
  • Upgraded CivisML to v2.2.0
    • When making predictions for multi-output models, users may request to only score a subset of outputs. Use the "dvs_to_predict" parameter in the API clients' "predict" functions.
    • Users may now train (scikit-learn-compatible) model objects in their local environments, then upload them to Civis Platform for CivisML to score. Use via the "civis.ml.ModelPipeline.register_pretrained_model" method in the Python API client v1.9.
    • Added an `if_exists="truncate"` option for writing output score tables.
    • Include model metrics with the training metadata (the "metadata.json" output file) instead of only in the "metrics.json" output file.
    • Now truncate column names in predictions if they would be over Redshift's column name length limit
    • Fixed a bug where we could make the wrong decision about how to parallelize a prediction job if the Civis Platform's table scanner hadn't run yet
    • Fixed a bug where random forest and extra trees models would train trees much deeper than necessary, resulting in excessively long training times and large memory usage. (These tree models now default to a max depth of 7, instead of unlimited, and hyperband no longer explores unlimited depth trees.)
    • When making predictions, write Redshift tables with 4-byte floats instead of 8-byte floats to reduce disk usage
    • In pre-defined stacking models, change the step name "sparse_logistic" to "logistic_with_ridge" and change "sparse_linear_regressor" to "ridge", to match the algorithms we're actually using.
  • Upgraded the Civis search functionality and improved reliability.
  • Upgraded Ruby to 2.3.7.
  • Fixed an issue that caused Report Snapshots to stop working.

Week Ending 4/27/2018

  • Updated the Address Correction job so it automatically removes double quotes and commas from data, preventing job failures.
  • Fixed an issue where someone running a notebook created by someone else couldn’t shut it down. All notebooks now run as the user who starts it.
  • Fixed a bug where some logs weren’t showing up in the Run History.

Week Ending 4/20/2018

  • Added links to the R Client documentation in the API docs
  • Added an API endpoint for retrieving logs for Notebook deployments.
  • Added an API endpoint to /jobs to cancel a run.
  • Updated the Workflow configuration code box so it grows when the code exceeds the default height.
  • Added error messages to the Address Correction job for when the input file is too small (less than 100 records) or if a specified column does not exist.
  • Updated the name of the browser tab to match the name of the current page when using Civis apps.
  • Added a clearer error message in Media Optimizer for when the selected date range and the date range on the selected rate card do not overlap.
  • Fixed a Media Optimizer bug where optimizations were not using the full budget when certain constraints were set.

Week Ending 4/13/2018

  • Updated the user drop-down menu to include direct links to the Help Center and the newly-optimized API documentation.
  • Fixed a bug that was preventing schedule parameters from being set through the API.
  • Fixed a bug that was preventing git credentials from being copied while cloning a Notebook.

Week Ending 4/6/2018

  • Fixed a bug that was causing unexpected behavior in advanced options toggling for imports with multiple syncs.

Week Ending 3/30/2018

  • Updated search to include published scripts.
  • Fixed a bug that was causing Workflows icons to fail to render.

Week Ending 3/23/2018

  • Added an API endpoint that gives users the ability to post log messages to running Container, Custom, Python, and R scripts.
  • Fixed a bug in Salesforce imports that was causing silent failures when records were missing.

Week Ending 3/16/2018

  • Labs-created person matches will now only include relevant match targets.
  • Updated the icon in Query that copies your past code to the Query editor.
  • Fixed a bug that allowed for the logging of credential parameter details.

Week Ending 3/9/2018

  • Released Workflow Parameterization.
  • Fixed a bug in version control that prevented users from configuring and committing to GitHub repositories.

Week Ending 3/2/2018

  • Added job schedule information to /jobs API endpoint.
  • Patched a SAML vulnerability that could allow for authentication bypass via incorrect XML. You can read more about the vulnerability here.
  • Fixed a bug that was causing errors when linking to custom scripts in the Triggers section of the Scheduling/Automation pane.
  • Fixed a bug that was preventing scheduled Workflows from running.
  • Fixed a bug in Platform Search that was preventing search results from properly populating.

Week Ending 2/23/2018

  • Updated Scripts to disable the “run now” button on a job if it is already queued or running.
  • Updated compute scaling so that Script workers are provisioned instantly in response to the number of queued jobs.
  • Fixed a bug in the Help Center that was causing document search to be hidden by chat.
  • Fixed a bug that was preventing newly granted tables and schemas from showing up in the data pane.
  • Fixed a bug in scheduled jobs that was causing some jobs to run twice.

Week Ending 2/16/2018

  • Upgraded CivisML to v2.1.2
    • Fixed an issue where not all exceptions were being caught while writing to the out-of-sample table.
    • Changed to install custom dependencies before running CivisML training jobs, so that user-specified package versions override default CivisML versions.
    • Added a fix to prevent stacking models from failing.
    • Added a more useful error message for nulls in multi-target models.
    • Fixed a typo in FIT_PARAMS argument of deploy templates.
    • Updated the Civis Python API client from v1.7.2 to v1.8.1 to include a patch for logging errors which was released with v1.8.1.
    • Worked around a bug in scikit-learn's code which prevented
      us from running multioutput-multiclass models.
    • Changed N_JOBS to allow floats in deploy template.
    • Created new test protocols and updated acceptance tests.
    • Restricted prediction jobs so that they don't launch child jobs from the child jobs.
    • Added guardrails to check that both schema and table names are present.
    • Added instructions for changing a Docker image tag in case of a broken release.
    • Increased maximum cancel timeout on CivisML jobs from 60 to 600 seconds to allow extra time for child jobs to be shut down.
  • Fixed an issue where workflows were not running on schedule.
  • Improved performance when there is a large demand of resources on Platform.
  • Changed the error shown when attempting to use an API key that is inactive because it belongs to an inactive user.

Week Ending 2/9/2018

  • Added a credential parameter to Notebooks. You can now include a credential when launching your Notebooks environment so you can use the credential while working in your Notebook without exposing the credential details in plain text.
  • Added civis.workflows.execute action to Workflows -- now Workflows can include other Workflows by using this action type and specifying a workflow_id parameter as a task input.
  • Changed Docker tag pinning for Notebooks to pin to minor versions instead of patch versions. The default tag will now be of the form "1.6" instead of "1.6.2" to enable bug fixes to be deployed to pinned images.
  • Increased the maximum cancel timeout for a job from 60 to 600 seconds to allow for enough time to cleanly cancel complex jobs.
  • Decreased the max CPU value in Notebooks to accommodate overhead CPU required by the notebooks server. Updated all existing notebooks' CPU usage to values that do not exceed the new maximum.

Week Ending 2/2/2018

  • Fixed a bug where GRANT queries took a long time to run.
  • Fixed a bug that was preventing job logs from being included in notification email.

Week Ending 1/26/2018

  • Added the ability to search on job ID in the Civis Platform search bar.
  • Updated Python and R scripts to automatically use the latest Docker image on script creation.
  • Removed the Notes field from script templates that include a custom UI.
  • Improved Civis performance during the execution of scheduled jobs.
  • Removed the ability to include Unicode characters in a workflow to prevent the workflow from being saved in an unreadable state.

Week Ending 1/19/2018

  • Added more explicit error messages when trying to execute an invalid Workflow via the API.
  • When viewing the execution for a Workflow, the YAML code box is grayed out to indicate the code is not editable.
  • Changed the highlighting of the Workflows graph to indicate all immediate upstream and downstream tasks when hovering over a task.
  • Fixed a bug in Workflows where tasks with extremely long error messages weren't properly updating their task state.

Week Ending 1/12/2018

  • Added a "Description" field to Workflows so users can include helpful descriptive text for other members of their organization.
  • Changed the Workflows execution side panel to indicate which execution is currently being viewed.
  • Fixed a bug that was preventing Notebooks with unicode characters from starting.
  • Fixed a bug where Civis was not properly recognizing the width of columns with a CHAR data type.
  • Fixed a bug in Workflows so that longer error messages are handled gracefully.
  • Fixed a bug in the Workflows graph that was causing incorrect job status icon to be displayed.
  • Fixed a bug that was causing the run history pane to show a job as "Queued".
  • Fixed a bug in the User Details page so that user’s primary group is displayed properly.

Week Ending 1/5/2018

  • In Workflows, YAML configuration error messages will now specifically mention missing or invalid actions.

Week Ending 12/22/2017

  • Added highlighting to the Workflows execution history panel to indicate which execution is currently being viewed.
  • When creating a new task from the graph view in Workflows, the task editing panel will automatically open.
  • Fixed a bug that was preventing published custom Scripts from being unpublishable.
  • Fixed a bug in scheduled Workflows that was causing errors with overlapping execution schedules.

Week Ending 12/15/2017

  • Past Query results are now collapsed by default to improve page performance.
  • Changed how API requests are handled so that background requests will be dropped when the system is overloaded to accelerate recovery.
  • Updated the Docker image for R scripts to datascience-r 2.3.0. New R scripts created in Platform now have the Civis R API Client installed automatically.
  • Fixed a bug in Query autocomplete that was causing text to be inserted incorrectly into the query.

Week Ending 12/8/2017

  • Updated the look of the user details page to reduce white space.
  • Fixed a bug in Python script cloning so that the script’s required resources are also cloned.
  • Fixed a bug in File imports that was causing errors when overriding column names that were initially duplicates.
  • Fixed a security vulnerability in Notebooks.

Week Ending 12/1/2017

  • Upgraded datascience-python Docker image to 3.3.0. New Python Scripts and Python Notebooks will use this image by default.
    • New packages:
      • civisml-extensions 0.1.5
      • dask 0.15.4
      • s3fs 0.1.2
    • Package Updates
      • civis 1.6.2 -> 1.7.1
      • boto3 1.4.4 -> 1.4.5
      • matplotlib 2.0.2 -> 2.1.0
      • numpy 1.13.1 -> 1.13.3
      • pandas 0.20.3 -> 0.21.0
      • pyarrow 0.5.0 -> 0.7.1
      • scikit-learn 0.19.0 -> 0.19.1
      • cloudpickle 0.3.1 -> 0.5.1
      • muffnn 1.1.2 -> 1.2.0
      • pubnub 4.0.12 -> 4.0.13
      • tensorflow 1.2.1 -> 1.4.0
  • Moved conda to version 4.3.30.
  • Improved the performance of job indices by loading user lists when opening the sharing pane rather than when the index page loaded.
  • Added ability to set Memory and CPU allocations for Python and R Scripts.
  • Fixed a bug that was allowing Queries to be executed in browser after a session timeout.
  • Fixed a bug in Query autocomplete to allow autocompletion on table names that include numbers.
  • Fixed a bug when validating schema and table names in Imports.
  • Fixed a bug in Imports so that a new table is not created if there is an error in the Import.

Week Ending 11/17/2017

  • Improved the performance of queries executed in the UI by moving API-initiated queries to separate resources.
  • Fixed a bug in Workflows where an execution would fail if a job's error message container non-ASCII characters.

Week Ending 11/10/2017

  • Python scripts now use v3-series datascience-python images. v3 includes the Civis Python API client v1.5.2 which includes CivisML capabilities.
  • Added the ability for users to change when their Notebook will time out.
  • Added the ability to set CPU, memory, and disk space on Python and R scripts.
  • Changed the vertical height of Tableau reports accessed using public links so it matches how the report looks in Platform.
  • Changed export encoding behavior so it doesn’t cause Postgres export jobs to fail.
  • Increased the memory available for Database Export jobs.
  • Improved the performance on all index pages.
  • Fixed a bug where a workflow wasn’t marking tasks as complete after they finished.
  • Fixed an error where queries were sometimes exposing the ruby error instead of the SQL error.
  • Fixed issues with exporting data to Vertica.
  • Fixed a bug where user admins could not see inactive users.
  • Fixed incorrect warnings that a user’s platform session had timed out.
  • Fixed a bug where the timezone wasn’t showing up in the automation pane.
  • Fixed a bug where fullscreen reports loaded with blank white space on the right side of the screen.

Week Ending 11/3/2017

  • Changed Python script editor to convert all tabs to spaces so as to better align with recommended style conventions.
  • Performance improvements for index page filtering.
  • Fixed bugs causing errors when cloning certain types of scripts.

Week Ending 10/27/2017

  • Added support for using Redshift's TIMESTAMPTZ (time stamp with time zone) data type.
  • Improved the formatting of EXPLAIN plans in Query to make it more readable.
  • Changed the error message for Container Scripts to indicate when a run failed due to memory constraints.
  • Changed the icon on Workflow tasks with errors to be an exclamation point instead of an "X" to be consistent with other Civis icons.
  • Fixed a bug where SQL Scripts that resulted in a null result set were linking files from the last run that contained results.
  • Fixed a bug where a user running a job owned by a different user could cause a job queue.
  • Fixed a bug in Workflows that was causing a locked Workflow to appear editable after an execution.

Week Ending 10/20/2017

  • Added the ability to use numbers in Civis usernames.
  • Added the ability for run outputs to be set during script shutdown so the script writer can store debug information about the job.
  • Added helpful descriptive text to the advanced options section of File Imports.
  • Changed the width of tabbing in the Python editor from 2 to 4 characters to align with recommended style conventions.
  • Added an error message to Container Scripts to indicate when the command input exceeds the character limit.
  • Added a default value of 0 to the max errors field in File Imports.
  • Fixed a bug in Database Imports that was preventing connections to external clusters.

Week Ending 10/13/2017

  • New “Import from S/FTP” and “Import from URL” jobs can now be created from the “New Import” menu and previously created jobs can be found in Imports.
  • New “Export to S/FTP” jobs can now be created from the “New Export” menu and previously created jobs can be found in Exports.
  • SQL Scripts are no longer listed as Exports. They only appear on the Scripts index now.
  • Fixed a bug in two-factor authentication that allowed 2FA code reuse under certain conditions.
  • Fixed bugs in Database Exports to Vertica where incorrect syntax was being used to create the destination table.
  • Fixed a bug in Notebooks where custom packages were not preserved when a notebook was cloned.

Week Ending 10/6/2017

  • Added a new script parameter type of Multi line String to make it easier to format long blocks of text.
  • Improved the performance of an import's page when listing source and destination databases.
  • Fixed a bug in Salesforce and Database Imports where errors were not being properly surfaced.
  • Fixed a bug in File Imports where column name overrides weren't working on files without header rows.
  • Fixed a bug where Report Only users were able to see the Enhancements index.
  • Fixed a bug in Database Imports where the source or destination doesn't have a schema.

Week Ending 9/29/2017

  • Upgraded Container Scripts to use the datascience-python Docker image version 3.2.0.
    • scikit-learn 0.18.2 -> 0.19.0
    • civis 1.6.0 -> 1.6.2
    • requests 2.14.2 -> 2.18.4
    • urllib3 1.19 -> 1.22
  • Improved the performance of filtering on the homepage activity filter.
  • Improved the performance of the Models index.
  • The data pane will quickly reflect changes made by additional ALTER SCHEMA commands.
  • Added detailed logging to Imports and Export to better communicate the progress of the job.
  • Added a clearer indication to File Imports that column names are editable.
  • Fixed a bug in Address Correction where the output table name didn't appear if the output schema had no tables in it.
  • Fixed a bug when searching for tables in Database Imports.
  • Fixed a bug that was preventing Exports from being removed from a Project.
  • Fixed a bug where Container Scripts built from a template were inaccurately routing the URL after closing the script.
  • Fixed a bug that allowed non-administrators the ability to view the details of inactive users in their organization.
  • Fixed a bug in Database Imports to enable the import of ENUM data types and columns with the UNSIGNED designation.

Week Ending 9/22/2017

  • Schemas without any tables in them now show up in the data pane, are re-scanned when changed using queries, and can be selected in schema dropdowns.
  • The “Export Results” link in Query now links to scripts page instead of jobs page to make it easier to download query results.
  • Improved the performance of run history pane by eliminating unnecessary API calls.
  • Code windows automatically expand as text lines increase to avoid having a scroll bar within the code window.
  • Fixed a bug in scripts that allowed users to accidentally and irreparably delete text from the code window.
  • Fixed a bug that was creating an unnecessary error message in sync jobs.
  • Fixed a bug in scheduling custom scripts.

Week Ending 9/15/2017

  • Upgraded Civis Python API Client to v1.6.2
    • Added ResourceWarning for Python 2.7.
    • Added TypeError for multi-indexed dataframes when used as input to CivisML.
    • ModelPipeline.from_existing will warn if users attempt to recreate a model trained with a newer version of CivisML, and fall back on the most recent prediction template it knows of.
    • Make the PaginatedResponse returned by LIST endpoints a full iterator, which also makes the iterator=True parameter work in Python 2.
    • When using civis.io.civis_to_csv, emit a warning on SQL queries which return no results instead of allowing a cryptic IndexError to surface.
    • Fixed the example code snippet for civis.io.civis_to_multifile_csv and added more details on its return dict in the docstring.
    • Pinned down sphinx_rtd_theme and numpydoc in dev-requirements.txt for building the documentation.
  • Updated error message when creating batch imports to be more informative.
  • Improved the performance of the CASS autosave feature.
  • Change group selection in user pages to typeahead search.
  • Removed the “active” checkbox from user creation page.
  • Disabled the editability of email field in user detail page to match API behavior.
  • Fixed a bug in CASS/NCOA jobs that was overwriting existing triggers.

Week Ending 9/8/2017

  • Replaced the error displayed with a warning when a download callback query made via the Civis API client returns an empty result.
  • When users call a list endpoint with the iterator argument from the Civis API client the response is now an iterable object.

Week Ending 9/1/2017

  • Added an expires_at parameter to GET /files/:id to enable generated file links to last up to 36 hours.
  • Changed the error message when importing ZIP files to be more informative.
  • Fixed a bug in imports that was causing duplicate column errors.

Week Ending 8/25/2017

  • Upgraded CivisML to v1.1.1
    • Fixed a bug which prevented users from grid searching over a calibrated classifier model.
    • Improve handling of tables containing a column with "date" type by now correctly reading SQL "date" type as a string.
    • Added the ability for tables to have integer column names.
    • Improved the handling of errors raised in prediction child jobs.
    • Enabled retries for failed prediction child jobs.
  • Improved the performance of the homepage by restricting it to only show recent activity (from the last 30 days).
  • When creating a new table via imports, the primary key column is now wrapped in double quotes to support column names with spaces.
  • Fixed a bug where scripts that contained a GRANT...ON ALL TABLES IN SCHEMA... SQL statement were failing.
  • Fixed a bug that prevented users from being able to clone imports.
  • Fixed a visual bug in the primary key field in the Score Model job where the "X" icon overlapped with the key name.

Week Ending 8/18/2017

  • Updating Google authentication on an export will now redirect back to the export rather than an import.
  • When importing a file with over 1,600 columns (Redshift's max column count), the error will now be properly displayed.
  • Added a “CreateStatement” tab to the Table Details page that shows the SQL table definition (DDL) making it easier to create duplicate or modified versions of the table.
  • Fixed a typo in the New Script pane.

Week Ending 8/11/2017

  • Added tooltips to the left side navigation icons when the navigation pane is collapsed.
  • Improved job failure messages by replacing System Errors with the full error.
  • Implemented minor UI changes to the New Import and New Script side panels.
  • Fixed a bug in the modeling wizard when selecting the sparse continuous option.

Week Ending 8/4/2017

  • Upgraded CivisML to v1.1
    • Added the ability to install custom dependencies when using CivisML via the API client.
    • Added null indicator column to categorical expansion to more correctly handle missing values in features.
    • Improved cross-validation to perform better with ordered datasets.
    • Improved the reliability of model training when encountering AWS errors.
    • Improved the reliability of model scoring when fetching results from successful runs.
    • Fixed a bug where modeling would fail if column names were reserved Redshift words.
  • Links from the Reports index page will now open the report in full-screen mode
  • Updated Imports text to reflect available data sources and direct users to help documentation

Week Ending 7/28/2017

  • Added a "note" parameter to the /scripts/templates API endpoint.  Users can now use POST or PATCH requests to add custom descriptive text to the top of a script template.
  • Increased the maximum number of displayed run outputs per job run to 10.  This will accommodate additional outputs of CivisML jobs.
  • Closing a script built from a template will now properly take you to the Scripts index.
  • Fixed a rare bug in Export to Google Sheet where the entire result set wasn't getting exported.

Week Ending 7/21/2017

  • Improved the front-end performance of job status updates.
  • Improved the message for job credential permission errors to be more descriptive.
  • Improved the handling of canceled jobs in the Python API Client.
  • Added support for tables and schemas showing up quickly in the data pane after adding a Redshift user to a Redshift group or running a GRANT ... ON ALL TABLES IN SCHEMA ... command.
  • Fixed a bug in how credentials are used when running a script built from a template.
  • Fixed a bug in the retry logic of file uploads.
  • Fixed a bug in saving user preferences on index pages.

Week Ending 7/14/2017

  • Improved the person matching algorithm when including email as a match field.
  • Changed the default If Table Exists setting to "Fail" on Google Sheet imports to maintain consistency with other imports and to default to the most conservative option.
  • Reworded job status notification emails to lead with the status to better communicate the results of jobs with long names.
  • Fixed a bug where CivisML models were linking to the incorrect script page.

Week Ending 7/7/2017

  • Files added to a Project will now appear in the UI. Files can be downloaded or removed from the project directly from the project page.
  • Improved the performance of table caching in CivisML when starting multiple model scoring jobs from the same table.
  • Fixed a bug in API documentation generation.

Week Ending 6/30/2017

  • Updated GET /scripts documentation to include "containers" as a type parameter value.
  • Fixed a bug in imports to remove stray period in the destination schema.table field.

Week Ending 6/23/2017

  • Improved the performance of the Models index.
  • Added clarification on the File Import page about which file types are supported.
  • Fixed a bug where the first run of a Database Import could fail.

Week Ending 6/16/2017

  • Improved the error message on File Imports when not specifying a destination schema.
  • Fixed a bug in Export to Google Sheet jobs where the SQL code could prevent users from editing the job.

Week Ending 6/9/2017

  • Added a helpful error message when a user attempts to add an invalid phone number to a Civis account.
  • Improved the Civis - Tableau integration for more reliable automatic importing of Tableau reports to Civis.
  • Fixed a bug in email triggers.
  • Fixed a bug in displaying SQL views with TEXT type columns.
  • Fixed a bug in the Google Sheets integration to reduce error messages when attempting to access a Google Sheet owned by someone else.

Week Ending 6/2/2017

  • Fixed a bug when applying secondary groups in user creation.
  • Fixed a bug to ensure Redshift credentials are added when adding a new user.
  • Fixed a bug in credential switching when cloning an Export to Google Sheet.
  • Fixed a bug in job cancelation to speed up cancel response time.
  • Fixed a bug in canceling Import syncs.

Week Ending 5/26/2017

  • Updated matching algorithm to improve table-to-table match rate involving zip codes.
  • Improved the performance of the data pane.
  • Added additional logging to Report Snapshot jobs to improve troubleshooting.
  • Performance improvements on the Query page.
  • Fixed a bug to allow viewing of run logs.
  • Fixed a bug so only SQL scripts can be linked to reports.
  • Fixed a bug in delimiter toggle for file imports.
  • Fixed a bug in table scanner column count.

Week Ending 5/19/2017

  • Enabled users to add a credential as a run output for a job via the API.
  • Migrated all Google Sheet Import syncs into the updated Imports UI.
  • Added a cancel_timeout parameter to Python, R, and Container scripts to allow users to insert buffer time for a container’s jobs to be canceled before the container is stopped
  • Updated API documentation for Export to Google Sheets and Run Outputs to add clarity.
  • Updated run log transfer to replace all characters outside of the Latin-1 character set with question marks to avoid an error while sending the logs via API.
  • Enabled the data pane to be widened the full width of the window to accommodate long schema and table names.
  • Performance improvements when listing custom scripts derived from a specific templateId.
  • Performance improvements on the Exports and Reports index pages.
  • Performance improvements on the home page.
  • Fixed a bug in Google API rate limits.
  • Fixed a bug in table details Javascript.
  • Fixed a bug that was incorrectly labeling run output files as expired.

Week Ending 5/12/2017

  • Released CivisML replacing our old modeling architecture and released support for Gradient Boosting Trees, Random Forests, and Extra Trees algorithms.  Read about using CivisML in a script template here.
  • Added clarity to acceptable file types for file imports.
  • Added better visual confirmation during table scan after initial file import.
  • Added pagination to the /jobs endpoint to prevent a request from overloading web servers.
  • Added expanded error logs to notification emails.
  • Fixed a bug in the Assume Role feature for admin accounts.
  • Fixed a bug in table UNLOAD SQL statement.

Week Ending 5/5/2017

  • Redesign of homepage to give more flexibility and relevance to your landing page view.
  • Added autocomplete to Query to help streamline your data exploration.
  • Added data side panel to Imports, Scripts, Models, and Exports to make it easier to find information about your data while you’re working.
  • Added additional logging to Imports jobs to increase understanding of job performance.
  • Added database names to scripts when user has access to more than one database to provide environment clarity when executing jobs.
  • Added clarifying text to Python scripts parameter panel.
  • Added trailing whitespace validation to server names.
  • Reordered search filters to match the job types in Civis' left sidebar.
  • Removed ability to create deprecated types of jobs from the Labs page.
  • Fixed a bug in links from success & failure emails.
  • Fixed a bug in container script credentials.
  • Fixed a bug in tables linked from query.

Week Ending 4/28/2017

  • Changed run history labels in scripts to reflect timestamp of each run to add more clarity to the job’s history.
  • Released Client User Management.
  • 3X improvement to response time for GET API actions.
  • Fixed a bug in the Modeling workflow predictor column selection.
  • Fixed a bug in Report Snapshot performance.
  • Fixed a bug where tables with comments weren't showing up in Query.
  • Fixed a bug in two-factor authentication.

Week Ending 4/21/2017

  • Improved integration with Google Sheets to streamline imports and exports.
  • Added links to run outputs to the Run History pane.
  • Changed descriptive language from "Google Docs" to "Google Sheets".
  • Upgraded Ruby to 2.3.4.
  • Upgraded Bundler 1.14.6.
  • Fixed a bug in SQL script notification emails.
  • Fixed a bug in script runner credentials.
  • Fixed a bug in Tableau where users were periodically being prompted to log in to Tableau Server.
  • Fixed a bug in CASS/NCOA job connections.
  • Fixed a security vulnerability to prevent a SQL injection attack.

Week Ending 4/14/2017

  • Removed the ability to DELETE objects from Civis in favor of /archive endpoints.
  • Added clarity to run names in the Run History pane.
  • The homepage activity feed will now display one entry per import instead of multiple entries.
  • Upgraded Devise gem to 4.2 and Devise-Two-Factor to 3.0. Devise is used in authentication.
  • Upgraded rubyzip gem to 1.2.1.
  • Fixed a bug with Tableau reports prompting users to log in to Tableau Server.

Week Ending 4/7/2017

  • Improved Civis scalability by changing how Query results are stored.
  • Added phone number validation to user profiles to improve 2FA SMS reliability.
  • Fixed a bug with full screen report toggling.
  • Fixed bugs with Google Doc integrations.

Week Ending 3/31/2017

  • Switched the default Python Docker image from datascience-base to datascience-python.
  • Upgraded nokogiri to v1.7.1.  Nokogiri is an open-source parser used with various imports and exports.
  • Fixed a bug in uploading basefiles for person matching.
  • Fixed visible jobs on the /users/<username> page.

Week Ending 3/24/2017

  • 6X improvement to response time on the Scripts index and associated API endpoints.
  • Updated the icon for idle jobs.
  • Added support for much larger job chains in a workflow.
  • Improved the performance of the Imports index when filtering by Destination.
  • Upgraded the Civis Python API Client to v1.4.0.
  • Fixed a bug in model scoring.
  • Fixed a bug in imports when dropping an empty table.

Week Ending 3/17/2017

  • Tableau report snapshots now use server-side rendering to better handle more complex visualizations.
  • The Last Run sorting option has been restored to all indices.
  • Fixed a bug where canceled Person Match jobs were blocking subsequent match jobs.
  • Fixed various bug with Google Sheet imports.

Week Ending 3/10/2017

  • Updated the left side menu. Civis jobs are now grouped by functionality, which will make it easier to navigate through your work.
  • In Query, the Copy Output to Clipboard function has been refactored to not use Flash.
  • Background jobs created by modeling are now hidden to avoid confusion and clutter on the homepage and associated indices.
  • 4X improvement to response time on the Imports, Models, and Projects indices and associated API endpoints.
  • Fixed a bug in public script templates to allow templated script creation by any user.

Week Ending 3/3/2017

  • Upgraded Rails to 4.2.8.
  • Fixed a bug with Salesforce Import where user and client credentials weren't loading properly.
  • Fixed a bug with Imports where the "If Table Exists" option of "Drop" couldn't drop an empty table.

Week Ending 2/24/2017

  • Upgraded Tableau Server to 10.1
  • Added more detailed error messages to Database Imports.
  • Added support for non-numeric ID columns in Database Imports.
  • Reduced Modeling memory consumption defaults.
  • Added a helpful error message when you try to import an empty sheet in a Google Doc.
  • Fixed a bug in Database Imports where users couldn't select a source database/credential they were permissioned on.

Week Ending 2/17/2017

  • The API now allows you to add an object as an output of a run and then access that output via the runId.
  • Fixed a bug in exports where a cloned export was redirecting to the /#/imports page.
  • Fixed a bug in generating script output links.

Week Ending 2/10/2017

  • Exposed the bocce_clusters API endpoint to improve visibility into Bocce jobs (such as Container Scripts).
  • Disabled index sorting by last run to improve performance.
  • Fixed a bug with cloning Imports and Exports.
  • Fixed a bug in scoring models.
  • Deprecated an unused API endpoint.
  • Fixed a bug with Google Doc Imports to allow for null values in the rightmost column.

Week Ending 2/3/2017

  • Tableau reports are archivable.
  • Fixed a JavaScript bug in the modeling workflow.

Week Ending 1/27/2017

  • Users with "view" permissions on a script can now access the run history pane.
  • Added the cancel button to Imports.
  • New Civis account creation links now expire after 7 days instead of 3 days to accommodate accounts created just before the weekend.
  • Fixed a visual bug in Imports filters that caused the list to spill off the screen.
  • Upgraded Elasticsearch to 2.0.
  • Rearranged past dates on the Query page to add clarity when scrolling through the query history.
  • Updated text in the Sharing pane to add clarity when a job is shared to a person hidden from the user.

Week Ending 1/20/2017

  • Users have more flexible CSV settings when using the POST /scripts/sql API endpoint.
  • Added the runner_user_id environment variable to the SQL Scripts API to add clarity between the user IDs of a script's author and a script's runner. 

Week Ending 1/13/2017

  • Allow access to credentials that have been shared with a user's non-primary groups.
  • Upgrade Bundler to 1.13.6.
  • Upgrade Ruby to 2.3.3.
  • Additional sorting and filtering options on the /scripts/custom API endpoint.
  • Users can now archive scripts with archived templates.
  • Updated scripts documentation to clarify author parameter.
  • Users can match an S3 file prefix with a file's owner for jobs run by the non-author.
  • Restore "New Database" button to Database Import.
  • Fixed a bug where users had difficulty editing notifications after a job has run.
  • Fixed a bug where the notifications pane in Imports wouldn't open. 

Week Ending 12/16/2016

  • Improved organization and usability in Search.
  • Allow “run script to update report” action when report is created in query to mimic script-backed report behavior.
  • Allow resources allocated to a script template to be set at runtime using parameters to give template users more control over scripts.
  • More intuitive handling of headers in CSV and Google Doc imports to reduce import errors.
  • Bug fixes in report modification.

Week Ending 12/2/2016

  • Search request optimization.
  • Bug fix in CMO channel mapping storage.
  • Bug fix to restore permissions in Tableau workbooks.
  • Modeling wizard - Instead of creating a model object at the start of the wizard, collect all settings and then create and build the model when the wizard is completed.
  • Remove unnecessary Redshift usage.
  • Sync notification bug fixes.
  • File read and validation for spot order model.

Related articles

Comments

0 comments

Please sign in to leave a comment.