Articles in this section

See more

Civis Platform Release Notes 2019

Avatar
Ryan Jewell
Updated
Follow

Week Ending 12/17/2019

Updates:

  • Notice: Civis R Client will be disabled for v1.5.1 and prior
    • We are making changes to our infrastructure to make it more stable and secure. As a reminder, we will disable the Civis R Client v1.5.1 and prior on December 18, 2019. Please upgrade to v1.6.0 or higher to successfully use the client. If you use any of the following Docker images in any R Scripts, Notebooks, or Shiny applications, they will also need to be upgraded to at least the following versions:
      • datascience-r 2.8.0
      • civis-jupyter-r 1.6.0
      • civis-services-shiny 1.2.0
    • Docker image versions used in R Notebooks and Shiny Applications can be updated via the UI. Docker image versions used in Scripts can be updated via the API. Please email support@civisanalytics.com with any questions.
  • We've added the following updates to the Projects detail page:
    • Filters for active, archived, and hidden objects in the Project detail page
    • Disabling the 'me' checkbox if current user is not an author on any of the objects
    • File ID visibility next to file name [Ex: My_Platform_File (123456)]

Bugfixes:

  • We fixed a bug that caused SQL Scripts to fail if the query included a semicolon in the SELECT statement.
  • We updated our Workflow logic so that if a Workflow execution fails to be added to Mistral (our workflow service), we now retry adding it to Mistral rather than failing the execution in Platform.

 

 

Week Ending 12/10/2019

What's New:

  • We have released the following updates for Platform Projects:
    • Favoriting: Users can favorite projects and easily locate them on the left-hand projects side pane.
    • Nested Projects: Projects can now be nested within other projects.
    • Sharing: When sharing a project, users now have the ability to share all objects within the project (note: excludes tables, objects in nested projects).
    • Notes Side Pane: New side pane with markdown notes section
    • Standardized Platform Design and Behavior: Standard Action menu (e.g. share, archive) found at the project and object level
    • And More! Project descriptions and visibility into object meta-data
    • To learn more, visit our Introduction to Projects Help Documentation.

Updates:

Bugfixes:

  • We fixed a bug that allowed users to create an infinite loop in a Workflow by having a task kick off itself. 

Week Ending 12/3/2019

Updates:

  • Civis Matching: We improved the use of latitude and longitude fields for finding potential matches and for computing match scores.
  • We added support for Redshift data type GEOMETRY.

Bugfixes:

  • When a user unloads data from Platform via civis_to_multifile_csv the manifest entry's filename extension will now include both its delimiter (csv) and its compression (.gz) where relevant.
  • We fixed a bug that caused the table scanner job (an automatic job that populates the metadata on the data pane and table details page) to fail if a table had a column that was dropped and the remaining columns had comments.
  • We fixed a bug that prevented users from importing multiple CSV files to Postgres within the same Import.

Week Ending 11/26/2019

What's New:

  • We’ve added the capability to embed values and tables defined in Python, R and container scripts in job notification emails. To leverage this feature, your script should upload any values or tables you want to include in your email as a run output JSONValue named "email_outputs". More information here.

Week Ending 11/19/2019

What's New:

  • We’ve added support for stall warnings on Workflows. Similar to the stall warning feature for jobs, this feature will alert you if a Workflow execution is taking longer than expected, enabling you to monitor and address any issues with key Workflows for your organization. More information can be found here.

Updates:

  • We added type loosening capabilities to the /imports/files/csv endpoint so that column types can be updated to fit the data being imported rather than causing the import to fail. For example, varchar (42) might be created as varchar (1024) to accommodate data larger than 42 characters.

Bugfixes:

  • We fixed a bug that caused the file contained within an exported zip archive to also have a .zip extension.
  • We fixed a bug in our Google Sheet export job that overwrote data in a column when the column name was duplicative.
  • We fixed a bug that broke the zip compression of exported files.
  • We removed table level comments from column comments on the table details page. Now table level comments will only appear on the data pane.

Week Ending 11/12/2019

Updates:

  • We updated Tableau public links to no longer redirect logged in users to the full Civis Platform Reports page. This will prevent users from sharing the wrong report URL (the URL asks users to sign in) to non-Platform users.
  • We updated our Notebook health checks to wait longer before throwing a “failed health check” warning to allow for more computationally heavy commands to finish running.

Bugfixes:

  • We updated our database imports to retry rather than fail if the job throws a reconnection error.
  • We fixed an issue that caused multi-file export manifest entries to be improperly decompressed on download.

Week Ending 11/5/2019

Updates:

  • We updated our Database import so that users can specify the modified_at column that will be used for incremental syncs.

Bugfixes:

  • We fixed a bug that prevented users from clicking “run with my credentials” on a Google Sheet Export. Users are now able to switch the job to run with their credentials, but will need to manually refresh the page after clicking “run with my credentials” in order for the changes to take effect.
  • We fixed a bug that caused errors when a user used the new CSV import endpoints for an upsert with delayed execution and changed the destination table column names.

 

Week Ending 10/29/2019

Updates:

  • On Friday 11/1/2019, we are deprecating support for the Export People to VAN, Export a list to VAN, and Export a score to VAN Labs jobs. Additionally, we are removing support for the Export Queue Table to VAN template script. You will no longer be able to create new jobs of these types.
    • We recommend leveraging our new script templates instead: Export Civis Data to NGPVAN (template id: 19204) and Export Saved List to VAN (template id: 30573). Please reach out to your Client Success Analyst for access to these scripts.
  • We added a limit of 100 to the number of action executions that can be kicked off concurrently for a workflow task.

Bugfixes:

  • We fixed a bug that caused MySQL Database Imports to error when updating data on a table with a primary key when there was no schema.

Week Ending 10/22/2019

Bugfixes:

  • We fixed a bug that caused imports to intermittently fail with "Couldn't find Database::Table" errors.
  • We fixed a bug that caused all Boolean values to convert to 0 when imported into a MySQL database.
  • We fixed a bug that prevented the sync of Boolean data types from a Redshift or Postgres databases to all Database types when creating a new table.

Week Ending 10/15/2019

What's New:

  • A New and Improved Notebooks Page! Now, when you access the Notebooks pages in Platform (Python3, Python2, or R), you will see new and updated capabilities!
    • Standardized Platform Design and Behavior: Action buttons (share, clone, schedule, etc.) are now found under a standard action menu at the top of the page. We also updated the side pane design to be more visually and functionally consistent based on user feedback.
    • A new full screen experience that gives you additional room to work in your Notebook.
    • Visibility and Status: Accurate warnings and error messages so you can understand when something went wrong and how to correct it.
    • Stability and More Stability: Backend updates to ensure Notebooks are reliable and always ready to use.

Updates:

  • Security and stability updates 

Week Ending 10/8/2019

What's New:

  • We added a new file cleaning endpoint that gives users the flexibility to select how they want to preprocess their file before uploading it into Platform. This includes header row, delimiter, compression and column type detection as well as addressing file properties (i.e., conversion to UTF-8, removal of byte order markers, etc.) that may prevent importing the file into a database. This does not include preprocessing the data in the file. This endpoint is to be used with our streamlined CSV Import API endpoint so that users can balance their import performance versus preprocessing needs.

Updates:

  • We are making changes to our infrastructure to make it more stable and secure. As part of this change, we are deprecating the Civis R Client v1.5.1 and prior on December 9, 2019. Please upgrade to v1.6.0 or higher to successfully use the client. If you use any of the following Docker images in any R Scripts, Notebooks, or Shiny applications, they will also need to be upgraded to at least the following versions:
    • datascience-r 2.8.0
    • civis-jupyter-r 1.6.0
    • civis-services-shiny 1.2.0
  • Matching
    • We added clearer, earlier error messages in Civis Data Match if the input table has null primary keys for some records.
    • We made minor improvements and bug fixes to the use of location information in Civis Data Match and Identity Resolution.
    • We made a minor fix to how name fields are preprocessed for use in Civis Data Match and Identity Resolution.
  • We optimized Platform's tracking of Redshift schema and table grants to reduce the workload on Redshift.

Week Ending 10/1/2019

Updates:

  • We deprecated Vertica Database syncs.
  • We added name (name of the import) and redshiftDestinationOptions.diststyle (the diststyle to add to a table) fields to the imports/files/csv endpoint. More information about this endpoint can be found here. 

Bugfixes:

  • We fixed a bug that broke Google Sheet Imports if there was a unicode character (such as an emoji or &) in the sheet title.
  • We fixed a bug that caused VAN List imports to fail if the list name contained a slash.
  • We fixed a bug that caused reports to display only partially when viewed on a Chromebook or in Safari.

Week Ending 9/24/2019

Updates:

  • We made improvements to our infrastructure to better handle periods of increased load.

Bugfixes:

  • We fixed a bug in the nightly table scanner job (an automatic job that populates the metadata on the data pane and table details page) to ensure it runs successfully when syncing information about schemas.
  • We fixed an issue that caused imports to fail due to the case of column names not matching the case of columns for already existing tables. 

Week Ending 9/17/2019

Updates:

  • We made an update so that changing your database password will take effect more quickly.
  • We updated our imports to fold distkeys to lowercase when importing data into a Redshift database. This had previously prevented imports from running successfully.

Bugfixes:

  • We fixed a workflow bug that caused updates to the YAML file to not always save when the page was refreshed.
  • We fixed a bug that prevented a Google Sheet import from running successfully if the sheet name contained a period.
  • We fixed a bug in our geocoding job that prevented the job from running successfully.
  • We fixed a bug that showed the old Platform UI when a user updated their two factor preferences.

Week Ending 9/10/2019

Updates:

  • We updated the Civis R client to v2.1.0
    • This update diminishes some of the pain of rate limits. We now automatically retry all endpoints if a user has been rate limited (respecting the ‘Retry-After’ header). Additionally, the user can specify a global polling interval to help prevent rate-limiting issues when running many long-running processes in parallel. You can see detailed changes in the CHANGELOG and browse the new documentation here.
  • Updated datascience-r docker image to v3.1.0
    • rocker/verse -> 3.6.0
    • civis-r -> 2.1.0
    • civis-python -> 1.11.0
  • Updated civis-jupyter-r to v1.8.0
    • rocker/verse -> 3.6.0
    • R -> 3.6.0
    • civis-r -> 2.1.0
  • Updated civis-services-shiny to v1.4.0
    • Upgraded base image datascience-r from 3.0.0 -> 3.1.0
  • We made reliability updates that keep Platform up with minimal downgraded functionality if Elasticsearch (part of our backend infrastructure) goes down.
  • We deprecated Silverpop and Catalist jobs since they are no longer supported. Civis provides other tools that can be used to move this data into Platform. 

Bugfixes:

  • We fixed a bug that prevented users from testing their RemoteHost connection on the “Add Source” side pane.
  • We fixed a bug that prevented Geocoding jobs from running successfully.

Week Ending 9/3/2019

Updates:

  • We reduced the number of connections opened when running a query or SQL script.
  • We added an error message to the Scripts page that warns a user if they are using a Docker image that they do not have permission to use.
  • We updated our two factor login options so that an organization's administrator can decide which type of two factor authentication (SMS vs. app) users should use.

Bugfixes:

  • We fixed a bug that sent some users multiple SMS codes when they tried to log in to Platform with SMS two factor authentication.
  • We fixed a bug to prevent inactive users from receiving job email notifications.
  • We fixed a bug that caused data from CSV imports to sometimes be encoded incorrectly.
  • We fixed a bug that caused SQL scripts to fail on Postgres clusters if there was a semicolon at the end of the script.
  • We fixed a bug that caused table scanner (an automatic job that populates metadata on the table details page and data pane) to not pick up statements that included "TRUNCATE TABLE..."
  • We fixed a Google Sheet export bug that created a blank Google Sheet even if the export failed because the worksheet name was blank.

Week Ending 8/27/2019

Updates:

  • We made performance improvements to the Table Scanner and Database Status jobs. These jobs are automatically run by Platform and are used to populate the data pane.
  • Truncate, revoke, and grant statements (when performed outside SQL Scripts and the Civis Query tool) are now picked up by the Database Status job, and the appropriate updates will now show up in the data pane.
  • We removed the “Visualize the Query” button from the query tool due to downgraded functionality.
  • We removed the “Import from Box” and “Export from Box” options from the Import and Export menus due to downgraded functionality.
  • We updated a misleading Notebook error messages to no longer print “User successfully terminated the service” when the Notebook shutdown by itself.

Bugfixes:

  • We fixed a typo on the preference menu.

Week Ending 8/20/2019

Updates:

  • To keep users informed about the activity on their account, we added an email notification (sent to the email address associated with the user’s account) when a user creates an API key.

Bugfixes:

  • We fixed a bug so that you can now view a custom script even if you are not shared on the template.
  • We fixed a bug on the workflow execution page that was causing some parts of the page to not always load.

Week Ending 8/13/2019

Updates:

  • We removed the underutilized priority Redshift query queue in order to reallocate additional memory to the general Redshift query queue. This should result in improved query performance for users.
  • We optimized the performance of SQL queries used by a Platform backend job (“table scanner”) that is used to populate the data pane and the metadata in the table details page.
  • We removed query caching for late binding views to ensure query results are always up-to-date.

Bugfixes:

  • There was a bug that prevented column metadata from being populated in the table details page for columns that are oid data types, and thus caused the nightly table scanner job to fail. This is now resolved.

Week Ending 8/6/2019 

Updates:

  • We no longer support query caching for Postgres databases.
  • After reviewing the new Redshift default behavior of column compression and conferring with the AWS Redshift team directly, we decided to switch back to the new default behavior. We did this because we believe that this setting will provide the best user experience across all possible data sets and data load scenarios. Specifically, we expect to see a decrease in CPU usage and execution time for Import jobs. We also experienced some intermittent bugs in our database sync tool that were associated with forcing column compression. These should now be resolved.
  • Geocoding Enhancement:
    • We improved the interface between backend systems and client clusters to fix an issue with long-running geocoding jobs.
  • Matching:
    • We improved sensitivity to the frequency of names. Matching on rare names (e.g., “Keanu”) should lead to larger relative increases in scores than matching on common names (e.g., “John”).
    • We added support for partial matching on ZIP codes by prefix, since people often move to nearby ZIP codes (e.g., 60614, 60657).
    • We improved heuristics and weighting around date of birth features.
    • We added a fix to ensure duplicate primary keys that don’t match any Civis Data records are removed from the output.

Bugfixes

  • We fixed a bug that prevented some accented characters from being imported correctly into Redshift.
  • We fixed a bug so that views will have their metadata refreshed overnight if their dependent table’s metadata is also refreshed.
  • We fixed a bug so that SQL scripts will output the refreshed data from a view if the underlying table is updated.
  • We fixed a bug where notebooks would not load if the notebook became too big.
  • We fixed a bug on the report share pane so that it displays all the groups a user has permission to grant on.
  • We fixed a bug so that tables starting with “pg” will appear in the data pane. 

Week Ending 7/30/2019

Bugfixes:

  • We fixed a bug so that a query cache for a view is invalidated if the query cache for the associated table is invalidated. Before the fix, views sometimes were not updated even when their associated table was updated.
  • We fixed a bug so that users can now see column details in the data pane for views with no schema binding.
  • We fixed a bug that caused SFTP exports to fail when exporting large amounts of data.
  • We fixed a bug so that when a user alters the schema name, the tables within that schema won’t disappear from the data pane.
  • We fixed a bug so that columns that contain the characters ‘zip’ don’t automatically map to the varchar datatype.
  • Security Updates

Week Ending 7/23/2019

Updates:

  • We added the template script ID to the More Script Templates side pane. Now users can easily find the template script ID in the UI, rather than using the API.
  • We added database and port to the list of available environment variables when using the database parameter in scripts. This will allow users to connect easily to clusters directly in their scripts.
  • We updated our email notification so that they contain the entire job log. Previously the email was sent before all the logs were printed.
  • We made a frontend update to improve the load time of all import, script, workflow, report, and service detail pages.
  • We moved the job status (success or failure) to the beginning of the subject line of the workflow notification emails.
  • We updated our Google Sheet export so that the errors are more understandable. Now the errors give information about rerunning the job or checking job settings when a job fails.
  • We added last_successful_run parameter to the scripts/custom API endpoint so that users can debug their script and find the last time it ran successfully.

Bugfixes:

  • We experienced latency issues with some Platform features last week due to issues with our underlying job queueing system. We took actions to fix the issues and all jobs should be running as expected.
  • We fixed a bug that made tables disappear in the data pane if a user updated the schema name. Now the tables will remain visible even if the schema name is updated.
  • When importing data in Platform, if a diststyle is not set, it will now default to EVEN, rather than AUTO. This was the default behavior of redshift before Amazon made some changes last week. We are bringing this default back manually because the AWS changes introduced bugs.
  • We fixed a bug that prevented users from importing data to Postgres databases.

Week Ending 7/16/2019

What's New:

  • We added an import script that enables users to move tables into platform from Google BigQuery. Navigate to Code -> More Script Templates -> Import Big Query Table.

Updates:

  • We turned on column compression for all Imports. This is because Amazon released a change to Redshift during the week of June 30th that changed the default behavior of how column compression (COMPUPDATE parameter) is handled in the Redshift COPY command. Turning on column compression should make Import jobs behave as closely as possible to how they did before the change to Redshift, however we are working closely with the AWS Redshift team to better understand their change and the impact it has on Civis Platform. For more information, see Redshift cluster version history.
  • We now convert a file to utf-8 encoding if we are unable to detect the character set encoding of the file. This will prevent import jobs from failing since the data must be encoded as utf-8 to load onto the database. This update will resolve the error “undefined method `shellescape' for nil:NilClass” users saw in the past.
  • We updated our /scripts endpoint so that the Scripts page loads more quickly.

Bugfixes:

  • We fixed a bug that caused Projects to not always load in the project side pane.
  • We fixed a bug to hide the help widget in report full screen mode.
  • We fixed a bug in our Database import, most commonly seen in MySQL database imports, that will prevent the error “ERROR: gzip: unexpected end of stream”.

Week Ending 7/9/2019

Upgrades and Updates:

  • We released a new fullscreen experience for reports. Now, when a user clicks on the fullscreen icon, the data pane and report title is hidden. Users can “un-fullscreen” their report by clicking on the exit button located at the top of the report.
  • We added the ability to sort by relevance on the script template side pane, so users can find the script templates they want more easily.
  • In Platform search results, we hid the count of “all” objects when filtering only for “mine” for a more clear user experience.

Bugfixes:

  • We fixed a bug so that users can now update the title of a service report (a shiny report, for example).
  • We fixed a bug with our query editor to resolve the “wrong argument type nil” error message.
  • We fixed a bug so that the list of available Redshift clusters is in alphabetical order.
  • We fixed a bug with our Google Sheet Import so that “first row is header = auto” now runs successfully.
  • We fixed a bug so that table scanner (an automatic job that populates the metadata on the table details page) works on tables without owners.

Week Ending 7/2/2019

Bugfixes:

  • We updated our Google Doc export to support worksheet titles of any length with accented characters.

Week Ending 6/25/2019

Upgrades and Updates:

  • We updated our Rails version to v5.2.3 to continue security best practices.
  • We modified our Mobile Commons import integration to be backwards compatible with old field names, while still supporting the latest API return values for new users of the integration or anyone who has updated their tables with the new DDL
  • We added additional fields to the jobs/[id] endpoint:
    • successEmailSubject (the subject line of a success notification email)
    • successEmailBody (the contents of a success notification email)
    • runningAsUser (the user account that the job runs as)
    • runByUser (the user that kicked off the job)
  • We deprecated the ability to email reports due to unreliable functionality.
  • We updated civis-r docker image to v2.0.0. You can see detailed changes in the CHANGELOG and browse the new documentation here.
    • Breaking changes:
      • write_civis_file now uploads data frames as CSVs to S3 by default. So simply writing:  id <- write_civis_file(df)
        will upload the data frame as a CSV.
      • Previously this function would upload a data frame as an RDS object instead of a CSV. This should be more in line with what users expect when uploading data frames, and helps increase portability between Python and R. To minimize the extent of breaking changes in code, read_civis now reads file IDs as CSVs by default as well.
    • New Features:
      • We have some new functions that simplify working with Civis Scripts. It’s now much easier to read and write script outputs and to run templates and arbitrary expressions in containers. These functions should help make your productionizing efforts a little easier!
      • civis_script annotates an integer as a valid custom, container, or other platform script.
      • A new method read_civis.civis_script is added to easily read script outputs into memory.
      • fetch_output_file_ids returns file ids of run outputs for a civis_script.
      • fetch_output returns the output of scripts_list_*_runs_outputs for a civis_script.
      • write_job_output adds File run outputs to scripts.
      • run_civis can be used to run an arbitrary R expression in a Civis Platform container.
      • run_template can be used to run a template job, and return file IDs of results.
      • To see some examples about how these new functions work, take a look at our new vignette, Productionizing with Civis Scripts, which explains how to automate tasks related to scripts using civis-r and illustrates these new helper functions.
  • We updated datascience-r docker image to v3.0.0
    • rocker/verse -> 3.5.3
    • civis-r -> 2.0.0
    • civis-python -> 1.10.0
    • added requirements.txt
    • added package zoo
  • Updated civis-services-shiny docker image to v1.3.0
  • Upgraded base image datascience-r from 2.8.0 -> 3.0.0

Bugfixes:

  • We fixed a bug in our support for views without schema binding so that the column information now appears in the data pane when a user updates the underlying table.
  • We fixed a bug so that the result of a TRUNCATE query updates the stats on the table details page and outputs the correct result of the query after the TRUNCATE statement. Previously we cached the old query results, so the output didn’t reflect updates made by the TRUNCATE statement.

Week Ending 6/18/2019

Upgrades and Updates:

  • We added the ability to use the new, more streamlined [CSV Import API endpoint](https://platform.civisanalytics.com/spa/#/api#post_imports_files_csv) to import files directly from your S3 bucket. As a reminder, this API endpoint has a number of benefits, including:
    • Faster importing
    • A new “upsert” mode for more efficient data refreshes
    • The ability to import a file containing a subset of the destination table’s columns
    • The ability to import multiple files to the same table in a single run
    • For more information about this endpoint see the help documentation.
  • We adjusted our column name cleaning logic used in Imports from Facebook, Files, Google Analytics, Google Sheet, SFTP, and URL to preserve all leading numbers in a column name, not just the first one.
  • We added a check for the number of input records to the Address Correction job. A run will fail if the input is empty.
  • We added a new endpoint GET /jobs/{id}/runs/{run_id}/outputs for getting a list of run outputs for any job run.
  • The “Help Docs” link was renamed “Help” for simplicity.
  • We added a detectedInfo object to the response for GET /files/{id} to house more information about your file, including whether it has headers, what the delimiter is, and what the column names and data types are. We will be adding a new job soon that you can run to populate that information.
     

Bug Fixes:

  • We fixed some issues with Reports created from Services:
    • We fixed an issue preventing these reports from being renamed.
    • We fixed the fullscreen report view so there is only one scrollbar in long reports.
    • We fixed an issue preventing form submission in these reports.
  • We fixed an issue so that multiple Platform organizations can use the same Single Sign-On (SSO) configuration settings.

Week Ending 6/11/2019

Bug Fixes:

  • We fixed a bug where tables wouldn't properly appear in the data pane when being granted via a statement that included a comma-separate list of tables. Previously only the first table is the list would appear in the data pane.
  • We fixed a bug on the Database Status page that prevented some users from loading the page.
  • We fixed a bug so that users can now import a large number of files concurrently from an SFTP into Platform.

Week Ending 6/4/2019

Upgrades and Updates:

  • We added support for timestamp data without timezones in Postgres Databases on Platform, so users can import their data without altering it before importing into Platform.

Bug Fixes:

  • We fixed an intermittent bug that some users were experiencing, where jobs either failed with 503 errors or failed to complete at all. Jobs are now completing successfully.
  • We fixed a bug in SQL Scripts so that they no longer return an output file from the previous run if the most recent SQL query has no results.
  • We fixed a bug in our Export Queue Table to NGPVAN so that null values for the contactTypeID field can be exported from Platform into VAN.

Week Ending 5/28/2019

Upgrades and Updates:

  • For Civis Data Match, Table Deduplication, and Data Unification jobs, we updated the way we handle large data to increase efficiency and cut down on runtime.
  • We updated CivisML to v2.2.4 which includes a bugfix for report rendering issues.
  • We updated our Salesforce Export error logging to display more helpful error messages to the user.

Bug Fixes:

  • We fixed a bug that prevented some queries and imports from running successfully.

Week Ending 5/21/2019

Upgrades and Updates:

  • We updated our Database Import job to allow users to leave the destination table name blank when using a wildcard for the source table name. If the destination table name is left blank, the tables will inherit the names from the source.
  • We made minor improvements to the Civis Data Match model for the Civis Data Match job and Identity Resolution Pipelines.
  • We updated all of the Civis Notebook Docker Images
    • civis-jupyter-python2 to 1.6.1
    • civis-jupyter-python3 to 1.12.0
    • civis-jupyter-r to 1.6.1

Bug Fixes:

  • We fixed a Workflows bug so that the visual representation of a Workflow appears in the UI even if you set advanced features - specifically conditional task dependencies (on success, on error, etc.).
  • We fixed a data type issue that caused Geocoding jobs to fail when low-accuracy responses were recorded.

Week Ending 5/14/2019

Upgrades and Updates:

  • We updated how we create match targets to use the latest information on duplicate records and records of the deceased.

Bug Fixes:

  • We fixed a bug so the new CSV import API endpoint ( /imports/files/csv) respects the max error parameter. Users can use this parameter to specify the maximum number of rows to exclude from their import if those rows contain data that cannot be imported into Redshift.
  • We fixed two bugs on our MobileCommons import so that users can successfully import Members and Broadcast data into Platform.

Week Ending 5/7/2019

Upgrades and Updates:

  • We updated Jupyter Python 3 Notebooks to use version 1.11 of the civis-jupyter-python3 image. There is one small update that enables an upcoming feature.

Bug Fixes:

  • We fixed a handful of issues with the new Reports page, including:
    • An issue where some reports weren’t loading
    • We made the page title match the report’s title
    • An issue with searching for users and groups in the Sharing pane
    • An issue where the Help widget wasn’t loading
  • We removed broken Help links in File Imports.

Week Ending 4/30/2019

Upgrades and Updates:

  • We added support for wildcard characters in our Database Imports. Now, you can use the “%” symbol in your source tablename to sync all tables in a schema or a subset of tables in a schema with a specific prefix or suffix.
  • We updated the Container Script datascience-python default Docker image tag to v5.0.0 and Notebook civis-jupyter-notebook version to v0.4.5. Existing Container Scripts and Notebook Docker images will not be updated, but new scripts will have the updated Docker image tag so that users have the most current packages.

Bug Fixes:

  • We fixed a bug so that users can now create Remote Hosts from the Remote Host index page in Platform. Users can create Remote Hosts when they want to connect to external databases, for example, VAN, Blue State Digital, or a MySQL Database.

Week Ending 4/23/2019

What's New:

  • We added a new, more streamlined CSV import: /imports/files/csv. This API endpoint has a number of benefits, including:
    • Faster importing
    • A new “upsert” mode for more efficient data refreshes
    • The ability to import a file containing a subset of the destination table’s columns
    • The ability to import multiple files to the same table in a single run
    • For more information about this endpoint see the help documentation.
  • We released a new reports page (Tableau, HTML, and Services) with new and updated capabilities. Changes include:
    • A standard action menu for all global action buttons so that you have a more clean and streamlined UI experience
    • Save buttons and warnings on all side panes so that you understand what changes you are making to a job

Week Ending 4/16/2019

Upgrades and Updates:

  • We added deployment logs to Notebooks (under the run history button) so you can understand what is happening under the hood while your notebook is starting up.
  • We renamed the "type" field in the "column" object to "sqlType" and added a new "civisDataType" field that assigns a more generic data type to each column for the following endpoints:
    • GET /tables/:id
    • GET /tables/:id/columns
  • We added the ability to update the docker image tag via the API so that you can use the most up to date docker image on R and Python scripts.
  • Added
    • CivisFuture has the job_id and run_id property attributes
  • Fixed
    • Polling will treat None responses generated by spotty internet connections like responses with a non-DONE state.
  • We updated civis-python client to v1.10.0. There are two small updates in this release- one update to make it easier for you to access basic information about your jobs and one bug fix.

Bug Fixes 

  • We fixed a bug so that you will only receive an email notification when your phone number is updated on the profile page. Previously, emails were sent even if no changes were made and you pressed the save button.
  • A bug in the modeling wizard caused it to break for some users. Now everyone can use the modeling wizard to build models without writing code.
  • We fixed a typo on the URL import description on the top navigation menu.

Week Ending 4/9/2019

Upgrades and Updates:

  • You can now hover over the failed status icon in a workflow execution log to quickly and easily see any failure messages.
  • We added security recommendations on the SQL script parameter side pane to explain how you can use variables in your code to prevent sql-injection attacks.
  • We added the ability to import the `money` data type into a Postgres database so that you don’t need to transform your data before importing it into Civis.

Bugfixes:

  • We fixed a bug so that you can toggle between Tableau reports loaded in multiple tabs.
  • We fixed a bug to populate the `type` parameter description for the `/remote_hosts` API endpoint so that you know which remote host types Platform currently supports.

Week Ending 4/2/2019

 Upgrades and Updates:

  • We deprecated the `enhancements/person_matching` endpoint. This was the final step in the rollout of our new matching endpoints, which were released in 2018. These new endpoints accomplish a few things: They automatically implement any updates made to the Civis matching algorithm; and, they allow you to add matching jobs to workflows and schedule and share match jobs. The new endpoints are:
  • We made API searches more granular: Now, you can filter API search results by `type = jobs` so that the returned results include any data jobs (scripts, imports and exports), but not results from static workflows and notebooks.
  • Where do we send you when an error occurs? Previously, you’d be routed either to a working page - or a blank one. We’ve updated this so that, whenever an error occurs, you’ll always be taken back to a working page
  • Security Update to our API

Week Ending 3/26/2019

Upgrades and Updates:

  • We added an error notification to notify you when resource constraints prevent you from starting a Notebook.

Bugfixes:

  • We added the ability to import nvarchar data types into a Postgres database. Now, you don’t need to transform your data before it is imported into Platform, regardless of whether you’re using Redshift or Postgres as your database of choice.
  • We updated job logging to ensure that jobs won’t fail when the logs are really long.

Week Ending 3/19/2019

Upgrades and Updates:

  • We updated our Google Sheet import to allow apostrophes and backslashes in the Sheet title so that you don’t need to alter your title before importing it into Platform.
  • We updated the “alter group” command in the query tool to include schema grants so that new users granted to the group can immediately start working with their data.
  • Security Updates
  • We’ve updated CivisML to v2.3.3. Along with several other improvements, this version also fixes a bug about resource allocation for job parallelization so that jobs can run more quickly.
  • We’ve updated datascience-python docker image to v5.0.0. As part of this upgrade, a wide set of services also received an upgrade that won’t change your experience working in Platform.
    • Ubuntu 14.04 -> 18.04
    • python 3.6.4 -> 3.7.1
    • conda 4.3.30 -> 4.6.8
  • explicitly installs click=6.7
  • awscli 1.15.4 -> 1.16.121
  • beautifulsoup4 4.5.3 -> 4.7.1
  • botocore 1.10.4 -> 1.12.111
  • boto 2.46.1 -> 2.49.0
  • boto3 1.7.4 -> 1.9.111
  • bqplot 0.10.2 -> 0.11.5
  • cloudpickle 0.5.2 -> 0.8.0
  • cython 0.27.3 -> 0.29.6
  • dask 0.17.2 -> 1.1.4
  • ipython 6.1.0 -> 7.3.0
  • ipywidgets 7.1.0 -> 7.4.2
  • jinja2 2.9.6 -> 2.10
  • jsonschema 2.5.1 -> 3.0.1
  • libtiff 4.0.6 -> 4.0.10
  • libxml2 2.9.2 -> 2.9.8
  • matplotlib 2.2.2 -> 3.0.3
  • notebook 5.4.1 -> 5.7.5
  • numexpr 2.6.2 -> 2.6.9
  • numpy 1.13.3 -> 1.16.2
  • openblas 0.2.20 -> 0.3.5
  • pandas 0.22.0 -> 0.24.1
  • patsy 0.4.1 -> 0.5.1
  • psycopg2 2.6.2 -> 2.7.7
  • pyarrow 0.8.0 -> 0.12.1
  • pytest 3.1.3 -> 4.3.0
  • pyyaml 3.12 -> 3.13
  • requests 2.18.4 -> 2.21.0
  • s3fs 0.1.2 -> 0.2.0
  • seaborn 0.8 -> 0.9.0
  • scipy 1.0.1 -> 1.2.0
  • scikit-learn 0.19.1 -> 0.19.2
  • statsmodels 0.8.0 -> 0.9.0
  • urllib3 1.22 -> 1.24.1
  • xgboost 0.6a2 -> 0.81
  • civis 1.9.0 -> 1.9.4
  • civisml-extensions 0.1.8 -> 0.1.10
  • dropbox 7.1.1 -> 9.3.0
  • glmnet 2.0.0 -> 2.1.1
  • muffnn 2.1.0 -> 2.2.0
  • pubnub 4.0.13 -> 4.1.2
  • requests-toolbelt 0.8.0 -> 0.9.1
  • tensorflow 1.7.0 -> 1.13.
  • Changed
  • New Packages
  • Package Updates
  • Finally, we updated civis-jupyter-python2 docker image in notebooks to v1.6.0. There are several services updates included in this upgrade that do not affect user experience or functionality:
    • civis 1.9.0 -> 1.9.4
    • civisml-extensions 0.1.8 -> 0.1.10
    • numpy 1.13.3 -> 1.14.3 (fixes an error with the tensorflow binary)
  • Package Updates

Bugfixes:

  • We fixed a bug so that all users with permissions on a table can now click the Update Table Details button on the Table details page to refresh the table metadata on that page and in the Data Pane.

Week Ending 3/12/2019

Upgrades and Updates:

  • We’ve changed the way Platform handles imports from Salesforce: Now, Platform will retry importing partial batches of data from Salesforce rather than automatically failing after an error and requiring you to restart the entire import.
  • You can now set up Tableau user filtering without the need for our Client Success team to update each user’s Platform profile.
  • We’ve also made it easier for you to filter for the set of credentials you need for a specific database by adding a parameter (remoteHostId) to the GET /credential endpoint.

Bugfixes:

  • We fixed a bug that prevented you from updating the settings of a Google Sheet Export if you are not the owner of the export.
  • We fixed a bug that was causing sporadic 404 errors when loading Service reports.

Week Ending 3/5/2019

What’s New:

  • User level filtering to Tableau reports, which was in Beta for a few customers, is now Generally Available. You can now honor user-level filters set up on your Tableau reports using a new toggle in the sharing pane.
  • We’ve also added an API endpoint to create report templates to save you some time when creating reports multiple times.
  • For geocoding, you’re now able to parse an address within a geocoding job for use in other analyses, and we’ve added support for a few additional geocoding providers that integrate seamlessly with Platform (Nominatim and geocoder_ca [for Canadian geographies]).
  • Finally, we’ve added support for IDENTITY column types for Postgres-to-Redshift Database imports, making it easier for you to move data from one to the other.

Upgrades and Updates:

  • We’ve updated the sharing email notification to the correct URL link for Reports created from Services.
  • We’ve added some updates to help Services and Notebooks start faster.
  • Finally, we’ve updated the Civis Python client to v1.9.3. The key highlights are:
  •  
    • The SQL command get_table_id can now handle quoted `schema.tablename`.
    • Run cancellation endpoints now return correct empty responses rather than an error.

Bugfixes:

  • We’ve fixed a permissioning issue where a user needed to be the manager of a person in their entire set of groups in order to change their permissions in a single group. Now, permissions can be managed on an individual group level.
  • Fixed-height reports didn't automatically load with a scroll bar. That has been fixed.

Week Ending 2/26/2019

  • Added the ability to run particular segments of a Workflow, instead of the entire thing.
  • Added the ability for scripts to output any JSON value (e.g string, integer, array, dictionary) as an Output in the Run History pane. Tasks in a workflow can pass this value to the next task.
  • Added the ability to send a notification email to users or groups you are sharing an object with via the sharing side pane.
  • Added custom filtering to the logs in the Deployment History pane to filter by time increments.
  • Added a Bitbucket integration to Notebooks and SQL, R, Python, and JavaScript Scripts.
  • Added a new POST /tables/scan endpoint for updating table info and metadata in the data pane.
  • Added more specific error messages when available in Google Sheet Imports and Exports.
  • Added a warning if the table created by a Salesforce Import includes truncated values or records.
  • Added scriptType parameter to GET /templates/scripts/{id}.
  • Added support for session reuse for FTP_TLS servers to the Import from S/FTP job.
  • Fixed an issue causing the job that refreshes the data pane and table metadata to fail when trying to refresh data on a large number of tables.
  • Fixed an issue with the /match_targets/ endpoints.

Week Ending 2/19/2019

  • Released new Script parameter type, Database, which displays dropdowns for both selecting a Platform database and database credential to be referenced in the Script.
  • Added the option to set a fixed value for a Script parameter, using the optional “value” field in the “params” object in /scripts/. Learn more here.
  • Added the option to restrict a Script parameter to certain allowed values, using the “allowedValues” field in the “params” object in /scripts/. This displays as a dropdown in the UI for the Script. Learn more here.
  • Updated Civis R client to v1.6.1
    • Write_civis.numeric now correctly syncs with headers
    • Write_civis gains header, credential_id and import_args arguments to more flexibly import data
    • Added coef.civis_ml function which returns civis_ml model coefficients in the format of stats::coef
    • Added get_feature_importance to civis_ml_utils
  • Updated datascience-r docker image to v2.8.0
    • rocker/verse -> 3.5.2
    • civis-r -> 1.6.1
    • civis-python -> 1.9.3
  • Updated civis-jupyter-r to v1.6.0
    • rocker/verse -> 3.5.2
    • R -> 3.5.2
    • civis-r -> 1.6.1
  • Updated civis-services-shiny to v1.2.0
    • Upgraded base image datascience-r from 2.7.0 -> 2.8.0
  • Fixed a 500 error upon login
  • Fixed an issue where Notebooks would fail to start
  • Fixed an issue where users were seeing a blank page upon opening a new tab
  • Fixed a credential issue in Shapefile Imports
  • Fixed an issue in Database Exports when quotes inside a field were followed by a line feed character
  • Deprecated Person Matching job in Labs
  • Began deprecation of the Box Import and Export jobs
  • Hid the “New Report” button on the Reports page for Report-Viewer users
  • Minor updates to API endpoints to enable future features
  • Deprecate the CONSOLE_API_KEY environment variable inside of Container, Python, and R scripts in Platform.

Week Ending 2/8/2019

  • Updated Civis python client to v1.9.3
    • Fixed a table formatting issue in the scoring step of CivisML when no primary key column is present
    • Added explicit support for Python 3.7
    • Note: Civis-supported docker images for notebooks and container scripts continue to use v1.9.2 by default. They will be updated soon.
  • Updated Civis Data Match, Table Deduplication, and Data Unification jobs to use person-matching v3.2.1.
    • Improved matching on last name, address unit, zip code, and incomplete birthdates (only month or birthday)
    • Improved handling of ties between match pairs if only best match is desired
    • Improved matching on email only or on phone number only, when name fields are not provided
  • Better handle user-defined data types in the job that populates table metadata
  • Fixed 500 errors in CivisML job
  • Fixed an issue with NULL fields in Database Export jobs when the destination is a database that isn’t Redshift, MySQL, Vertica, or Postgres.
  • Fixed an issue in Database Export jobs when exporting numeric fields with high precision when the destination is a database that isn’t Redshift, MySQL, Vertica, or Postgres
  • Fixed an issue in Database Export jobs when exporting string fields that contain backslashes when the destination is a database that isn’t Redshift, Vertica, or Postgres
  • Made some changes to prepare for the upcoming deprecation of the Person Matching job in Labs for most users
  • Made some updates to API endpoints to enable upcoming features
  • Security updates

Week Ending 2/1/2019

  • Updated CivisML to v2.2.2
    • Catch feature selection errors in child jobs and replace with a clearer error message
    • Round predictions to the nearest 1e-20 for Redshift compatibility
    • Enforce float32 precision for predictions to prevent Redshift overflow errors
    • Scoring tables no longer contain a "None" column when no primary key is specified
    • Raise a RuntimeError if there are more than 5000 levels in a column
    • Emit a warning if the column levels during transform don't overlap at all with levels from fitting
  • Added a new “Person Data Standardization” tool under Tools > Enhancements. This job takes a table with personally identifiable information (PII; e.g., names, addresses, dates of birth), and standardizes the PII. See this article for more information.
  • Added new Import and Export jobs for working with shapefiles and geodatabases (For use with Postgres only)
  • Added new enhancements/geocode endpoints that include the full set of features standard for Platform jobs -- adding to a Project, sharing, etc
  • Added an error message when there is an issue loading a page that tells the user to refresh their browser
  • Minor styling changes to the left Projects pane
  • Fixed a bug for exporting to MS Sql Server for NULL numeric columns and numeric columns with a precision over 10

Week Ending 1/25/2019

  • The File Import job in the UI now supports importing files up to 5 TB in size.
  • The primaryKeys parameter at the /tables/{id} endpoint automatically updates to match the database after a table scan.
  • Minor updates to API documentation

Week Ending 1/18/2019

  • Fixed a bug where logging in always redirected to the homepage, even if the user originally entered a more specific URL within Platform.
  • Added a new “Welcome to Civis” walkthrough for new users.
  • Made improvements to performance during periods of heavy usage.
  • Security updates.

Week Ending 1/11/2019

  • Fixed an issue with Container Scripts that reference large GitHub repositories.
  • Fixed an issue where workflow executions would fail to start.

Related articles

Comments

0 comments

Please sign in to leave a comment.