Civis Platform Release Notes 2016-2017

Week Ending 12/22/2017

• Added highlighting to the Workflows execution history panel to indicate which execution is currently being viewed.
• When creating a new task from the graph view in Workflows, the task editing panel will automatically open.
• Fixed a bug that was preventing published custom Scripts from being unpublishable.
• Fixed a bug in scheduled Workflows that was causing errors with overlapping execution schedules.

Week Ending 12/15/2017

• Past Query results are now collapsed by default to improve page performance.
• Changed how API requests are handled so that background requests will be dropped when the system is overloaded to accelerate recovery.
• Updated the Docker image for R scripts to datascience-r 2.3.0. New R scripts created in Platform now have the Civis R API Client installed automatically.
• Fixed a bug in Query autocomplete that was causing text to be inserted incorrectly into the query.

Week Ending 12/8/2017

• Updated the look of the user details page to reduce white space.
• Fixed a bug in Python script cloning so that the script’s required resources are also cloned.
• Fixed a bug in File imports that was causing errors when overriding column names that were initially duplicates.
• Fixed a security vulnerability in Notebooks.
   

Week Ending 12/1/2017

• Upgraded datascience-python Docker image to 3.3.0. New Python Scripts and Python Notebooks will use this image by default.
  • New packages:
    • civisml-extensions 0.1.5
    • dask 0.15.4
    • s3fs 0.1.2
  • Package Updates
    • civis 1.6.2 -> 1.7.1
    • boto3 1.4.4 -> 1.4.5
    • matplotlib 2.0.2 -> 2.1.0
    • numpy 1.13.1 -> 1.13.3
    • pandas 0.20.3 -> 0.21.0
    • pyarrow 0.5.0 -> 0.7.1
    • scikit-learn 0.19.0 -> 0.19.1
    • cloudpickle 0.3.1 -> 0.5.1
    • muffnn 1.1.2 -> 1.2.0
    • pubnub 4.0.12 -> 4.0.13
    • tensorflow 1.2.1 -> 1.4.0
• Moved conda to version 4.3.30.
• Improved the performance of job indices by loading user lists when opening the sharing pane rather than when the index page loaded.
• Added ability to set Memory and CPU allocations for Python and R Scripts.
• Fixed a bug that was allowing Queries to be executed in browser after a session timeout.
• Fixed a bug in Query autocomplete to allow autocompletion on table names that include numbers.
• Fixed a bug when validating schema and table names in Imports.
• Fixed a bug in Imports so that a new table is not created if there is an error in the Import.

Week Ending 11/17/2017

• Improved the performance of queries executed in the UI by moving API-initiated queries to separate resources.
• Fixed a bug in Workflows where an execution would fail if a job's error message container non-ASCII characters.

Week Ending 11/10/2017

• Python scripts now use v3-series datascience-python images. v3 includes the Civis Python API client v1.5.2 which includes CivisML capabilities.
• Added the ability for users to change when their Notebook will time out.
• Added the ability to set CPU, memory, and disk space on Python and R scripts.
• Changed the vertical height of Tableau reports accessed using public links so it matches how the report looks in Platform.
• Changed export encoding behavior so it doesn’t cause Postgres export jobs to fail.
• Increased the memory available for Database Export jobs.
• Improved the performance on all index pages.
• Fixed a bug where a workflow wasn’t marking tasks as complete after they finished.
• Fixed an error where queries were sometimes exposing the ruby error instead of the SQL error.
• Fixed issues with exporting data to Vertica.
• Fixed a bug where user admins could not see inactive users.
• Fixed incorrect warnings that a user’s platform session had timed out.
• Fixed a bug where the timezone wasn’t showing up in the automation pane.
• Fixed a bug where fullscreen reports loaded with blank white space on the right side of the screen.

Week Ending 11/3/2017

• Changed Python script editor to convert all tabs to spaces so as to better align with recommended style conventions.
• Performance improvements for index page filtering.
• Fixed bugs causing errors when cloning certain types of scripts.

Week Ending 10/27/2017

• Added support for using Redshift's TIMESTAMPTZ (time stamp with time zone) data type.
• Improved the formatting of EXPLAIN plans in Query to make it more readable.
• Changed the error message for Container Scripts to indicate when a run failed due to memory constraints.
• Changed the icon on Workflow tasks with errors to be an exclamation point instead of an "X" to be consistent with other Civis icons.
• Fixed a bug where SQL Scripts that resulted in a null result set were linking files from the last run that contained results.
• Fixed a bug where a user running a job owned by a different user could cause a job queue.
• Fixed a bug in Workflows that was causing a locked Workflow to appear editable after an execution.

Week Ending 10/20/2017

• Added the ability to use numbers in Civis usernames.
• Added the ability for run outputs to be set during script shutdown so the script writer can store debug information about the job.
• Added helpful descriptive text to the advanced options section of File Imports.
• Changed the width of tabbing in the Python editor from 2 to 4 characters to align with recommended style conventions.
• Added an error message to Container Scripts to indicate when the command input exceeds the character limit.
• Added a default value of 0 to the max errors field in File Imports.
• Fixed a bug in Database Imports that was preventing connections to external clusters.

Week Ending 10/13/2017

• New “Import from S/FTP” and “Import from URL” jobs can now be created from the “New Import” menu and previously created jobs can be found in Imports.
• New “Export to S/FTP” jobs can now be created from the “New Export” menu and previously created jobs can be found in Exports.
• SQL Scripts are no longer listed as Exports. They only appear on the Scripts index now.
• Fixed a bug in two-factor authentication that allowed 2FA code reuse under certain conditions.
• Fixed bugs in Database Exports to Vertica where incorrect syntax was being used to create the destination table.
• Fixed a bug in Notebooks where custom packages were not preserved when a notebook was cloned.
   

Week Ending 10/6/2017

• Added a new script parameter type of Multi line String to make it easier to format long blocks of text.
• Improved the performance of an import's page when listing source and destination databases.
• Fixed a bug in Salesforce and Database Imports where errors were not being properly surfaced.
• Fixed a bug in File Imports where column name overrides weren't working on files without header rows.
• Fixed a bug where Report Only users were able to see the Enhancements index.
• Fixed a bug in Database Imports where the source or destination doesn't have a schema.

Week Ending 9/29/2017

• Upgraded Container Scripts to use the datascience-python Docker image version 3.2.0.
  • scikit-learn 0.18.2 -> 0.19.0
  • civis 1.6.0 -> 1.6.2
  • requests 2.14.2 -> 2.18.4
  • urllib3 1.19 -> 1.22
• Improved the performance of filtering on the homepage activity filter.
• Improved the performance of the Models index.
• The data pane will quickly reflect changes made by additional ALTER SCHEMA commands.
• Added detailed logging to Imports and Export to better communicate the progress of the job.
• Added a clearer indication to File Imports that column names are editable.
• Fixed a bug in Address Correction where the output table name didn't appear if the output schema had no tables in it.
• Fixed a bug when searching for tables in Database Imports.
• Fixed a bug that was preventing Exports from being removed from a Project.
• Fixed a bug where Container Scripts built from a template were inaccurately routing the URL after closing the script.
• Fixed a bug that allowed non-administrators the ability to view the details of inactive users in their organization.
• Fixed a bug in Database Imports to enable the import of ENUM data types and columns with the UNSIGNED designation.

Week Ending 9/22/2017

• Schemas without any tables in them now show up in the data pane, are re-scanned when changed using queries, and can be selected in schema dropdowns.
• The “Export Results” link in Query now links to scripts page instead of jobs page to make it easier to download query results.
• Improved the performance of run history pane by eliminating unnecessary API calls.
• Code windows automatically expand as text lines increase to avoid having a scroll bar within the code window.
• Fixed a bug in scripts that allowed users to accidentally and irreparably delete text from the code window.
• Fixed a bug that was creating an unnecessary error message in sync jobs.
• Fixed a bug in scheduling custom scripts.

Week Ending 9/15/2017

• Upgraded Civis Python API Client to v1.6.2
  • Added ResourceWarning for Python 2.7.
  • Added TypeError for multi-indexed dataframes when used as input to CivisML.
  • ModelPipeline.from_existing will warn if users attempt to recreate a model trained with a newer version of CivisML, and fall back on the most recent prediction template it knows of.
  • Make the PaginatedResponse returned by LIST endpoints a full iterator, which also makes the iterator=True parameter work in Python 2.
  • When using civis.io.civis_to_csv, emit a warning on SQL queries which return no results instead of allowing a cryptic IndexError to surface.
  • Fixed the example code snippet for civis.io.civis_to_multifile_csv and added more details on its return dict in the docstring.
  • Pinned down sphinx_rtd_theme and numpydoc in dev-requirements.txt for building the documentation.
• Updated error message when creating batch imports to be more informative.
• Improved the performance of the CASS autosave feature.
• Change group selection in user pages to typeahead search.
• Removed the “active” checkbox from user creation page.
• Disabled the editability of email field in user detail page to match API behavior.
• Fixed a bug in CASS/NCOA jobs that was overwriting existing triggers.

Week Ending 9/8/2017

• Replaced the error displayed with a warning when a download callback query made via the Civis API client returns an empty result.
• When users call a list endpoint with the iterator argument from the Civis API client the response is now an iterable object.

Week Ending 9/1/2017

• Added an expires_at parameter to GET /files/:id to enable generated file links to last up to 36 hours.
• Changed the error message when importing ZIP files to be more informative.
• Fixed a bug in imports that was causing duplicate column errors.

Week Ending 8/25/2017

• Upgraded CivisML to v1.1.1
  • Fixed a bug which prevented users from grid searching over a calibrated classifier model.
  • Improve handling of tables containing a column with "date" type by now correctly reading SQL "date" type as a string.
  • Added the ability for tables to have integer column names.
  • Improved the handling of errors raised in prediction child jobs.
  • Enabled retries for failed prediction child jobs.
• Improved the performance of the homepage by restricting it to only show recent activity (from the last 30 days).
• When creating a new table via imports, the primary key column is now wrapped in double quotes to support column names with spaces.
• Fixed a bug where scripts that contained a GRANT...ON ALL TABLES IN SCHEMA... SQL statement were failing.
• Fixed a bug that prevented users from being able to clone imports.
• Fixed a visual bug in the primary key field in the Score Model job where the "X" icon overlapped with the key name.

Week Ending 8/18/2017

• Updating Google authentication on an export will now redirect back to the export rather than an import.
• When importing a file with over 1,600 columns (Redshift's max column count), the error will now be properly displayed.
• Added a “CreateStatement” tab to the Table Details page that shows the SQL table definition (DDL) making it easier to create duplicate or modified versions of the table.
• Fixed a typo in the New Script pane.

Week Ending 8/11/2017

• Added tooltips to the left side navigation icons when the navigation pane is collapsed.
• Improved job failure messages by replacing System Errors with the full error.
• Implemented minor UI changes to the New Import and New Script side panels.
• Fixed a bug in the modeling wizard when selecting the sparse continuous option.

Week Ending 8/4/2017

• Upgraded CivisML to v1.1
  • Added the ability to install custom dependencies when using CivisML via the API client.
  • Added null indicator column to categorical expansion to more correctly handle missing values in features.
  • Improved cross-validation to perform better with ordered datasets.
  • Improved the reliability of model training when encountering AWS errors.
  • Improved the reliability of model scoring when fetching results from successful runs.
  • Fixed a bug where modeling would fail if column names were reserved Redshift words.
• Links from the Reports index page will now open the report in full-screen mode
• Updated Imports text to reflect available data sources and direct users to help documentation

Week Ending 7/28/2017

• Added a "note" parameter to the /scripts/templates API endpoint.  Users can now use POST or PATCH requests to add custom descriptive text to the top of a script template.
• Increased the maximum number of displayed run outputs per job run to 10.  This will accommodate additional outputs of CivisML jobs.
• Closing a script built from a template will now properly take you to the Scripts index.
• Fixed a rare bug in Export to Google Sheet where the entire result set wasn't getting exported.

Week Ending 7/21/2017

• Improved the front-end performance of job status updates.
• Improved the message for job credential permission errors to be more descriptive.
• Improved the handling of canceled jobs in the Python API Client.
• Added support for tables and schemas showing up quickly in the data pane after adding a Redshift user to a Redshift group or running a GRANT ... ON ALL TABLES IN SCHEMA ... command.
• Fixed a bug in how credentials are used when running a script built from a template.
• Fixed a bug in the retry logic of file uploads.
• Fixed a bug in saving user preferences on index pages.

Week Ending 7/14/2017

• Improved the person matching algorithm when including email as a match field.
• Changed the default If Table Exists setting to "Fail" on Google Sheet imports to maintain consistency with other imports and to default to the most conservative option.
• Reworded job status notification emails to lead with the status to better communicate the results of jobs with long names.
• Fixed a bug where CivisML models were linking to the incorrect script page.

Week Ending 7/7/2017

• Files added to a Project will now appear in the UI. Files can be downloaded or removed from the project directly from the project page.
• Improved the performance of table caching in CivisML when starting multiple model scoring jobs from the same table.
• Fixed a bug in API documentation generation.

Week Ending 6/30/2017

• Updated GET /scripts documentation to include "containers" as a type parameter value.
• Fixed a bug in imports to remove stray period in the destination schema.table field.

Week Ending 6/23/2017

• Improved the performance of the Models index.
• Added clarification on the File Import page about which file types are supported.
• Fixed a bug where the first run of a Database Import could fail.

Week Ending 6/16/2017

• Improved the error message on File Imports when not specifying a destination schema.
• Fixed a bug in Export to Google Sheet jobs where the SQL code could prevent users from editing the job.

Week Ending 6/9/2017

• Added a helpful error message when a user attempts to add an invalid phone number to a Civis account.
• Improved the Civis - Tableau integration for more reliable automatic importing of Tableau reports to Civis.
• Fixed a bug in email triggers.
• Fixed a bug in displaying SQL views with TEXT type columns.
• Fixed a bug in the Google Sheets integration to reduce error messages when attempting to access a Google Sheet owned by someone else.

Week Ending 6/2/2017

• Fixed a bug when applying secondary groups in user creation.
• Fixed a bug to ensure Redshift credentials are added when adding a new user.
• Fixed a bug in credential switching when cloning an Export to Google Sheet.
• Fixed a bug in job cancelation to speed up cancel response time.
• Fixed a bug in canceling Import syncs.

Week Ending 5/26/2017

• Updated matching algorithm to improve table-to-table match rate involving zip codes.
• Improved the performance of the data pane.
• Added additional logging to Report Snapshot jobs to improve troubleshooting.
• Performance improvements on the Query page.
• Fixed a bug to allow viewing of run logs.
• Fixed a bug so only SQL scripts can be linked to reports.
• Fixed a bug in delimiter toggle for file imports.
• Fixed a bug in table scanner column count.

Week Ending 5/19/2017

• Enabled users to add a credential as a run output for a job via the API.
• Migrated all Google Sheet Import syncs into the updated Imports UI.
• Added a cancel_timeout parameter to Python, R, and Container scripts to allow users to insert buffer time for a container’s jobs to be canceled before the container is stopped
• Updated API documentation for Export to Google Sheets and Run Outputs to add clarity.
• Updated run log transfer to replace all characters outside of the Latin-1 character set with question marks to avoid an error while sending the logs via API.
• Enabled the data pane to be widened the full width of the window to accommodate long schema and table names.
• Performance improvements when listing custom scripts derived from a specific templateId.
• Performance improvements on the Exports and Reports index pages.
• Performance improvements on the home page.
• Fixed a bug in Google API rate limits.
• Fixed a bug in table details Javascript.
• Fixed a bug that was incorrectly labeling run output files as expired.

Week Ending 5/12/2017

• Released CivisML replacing our old modeling architecture and released support for Gradient Boosting Trees, Random Forests, and Extra Trees algorithms.  Read about using CivisML in a script template here.
• Added clarity to acceptable file types for file imports.
• Added better visual confirmation during table scan after initial file import.
• Added pagination to the /jobs endpoint to prevent a request from overloading web servers.
• Added expanded error logs to notification emails.
• Fixed a bug in the Assume Role feature for admin accounts.
• Fixed a bug in table UNLOAD SQL statement.

Week Ending 5/5/2017

• Redesign of homepage to give more flexibility and relevance to your landing page view.
• Added autocomplete to Query to help streamline your data exploration.
• Added data side panel to Imports, Scripts, Models, and Exports to make it easier to find information about your data while you’re working.
• Added additional logging to Imports jobs to increase understanding of job performance.
• Added database names to scripts when user has access to more than one database to provide environment clarity when executing jobs.
• Added clarifying text to Python scripts parameter panel.
• Added trailing whitespace validation to server names.
• Reordered search filters to match the job types in Civis' left sidebar.
• Removed ability to create deprecated types of jobs from the Labs page.
• Fixed a bug in links from success & failure emails.
• Fixed a bug in container script credentials.
• Fixed a bug in tables linked from query.

Week Ending 4/28/2017

• Changed run history labels in scripts to reflect timestamp of each run to add more clarity to the job’s history.
• Released Client User Management.
• 3X improvement to response time for GET API actions.
• Fixed a bug in the Modeling workflow predictor column selection.
• Fixed a bug in Report Snapshot performance.
• Fixed a bug where tables with comments weren't showing up in Query.
• Fixed a bug in two-factor authentication.

Week Ending 4/21/2017

• Improved integration with Google Sheets to streamline imports and exports.
• Added links to run outputs to the Run History pane.
• Changed descriptive language from "Google Docs" to "Google Sheets".
• Upgraded Ruby to 2.3.4.
• Upgraded Bundler 1.14.6.
• Fixed a bug in SQL script notification emails.
• Fixed a bug in script runner credentials.
• Fixed a bug in Tableau where users were periodically being prompted to log in to Tableau Server.
• Fixed a bug in CASS/NCOA job connections.
• Fixed a security vulnerability to prevent a SQL injection attack.

Week Ending 4/14/2017

• Removed the ability to DELETE objects from Civis in favor of /archive endpoints.
• Added clarity to run names in the Run History pane.
• The homepage activity feed will now display one entry per import instead of multiple entries.
• Upgraded Devise gem to 4.2 and Devise-Two-Factor to 3.0. Devise is used in authentication.
• Upgraded rubyzip gem to 1.2.1.
• Fixed a bug with Tableau reports prompting users to log in to Tableau Server.

Week Ending 4/7/2017

• Improved Civis scalability by changing how Query results are stored.
• Added phone number validation to user profiles to improve 2FA SMS reliability.
• Fixed a bug with full screen report toggling.
• Fixed bugs with Google Doc integrations.

Week Ending 3/31/2017

• Switched the default Python Docker image from datascience-base to datascience-python.
• Upgraded nokogiri to v1.7.1.  Nokogiri is an open-source parser used with various imports and exports.
• Fixed a bug in uploading basefiles for person matching.
• Fixed visible jobs on the /users/<username> page.

Week Ending 3/24/2017

• 6X improvement to response time on the Scripts index and associated API endpoints.
• Updated the icon for idle jobs.
• Added support for much larger job chains in a workflow.
• Improved the performance of the Imports index when filtering by Destination.
• Upgraded the Civis Python API Client to v1.4.0.
• Fixed a bug in model scoring.
• Fixed a bug in imports when dropping an empty table.

Week Ending 3/17/2017

• Tableau report snapshots now use server-side rendering to better handle more complex visualizations.
• The Last Run sorting option has been restored to all indices.
• Fixed a bug where canceled Person Match jobs were blocking subsequent match jobs.
• Fixed various bug with Google Sheet imports.

Week Ending 3/10/2017

• Updated the left side menu. Civis jobs are now grouped by functionality, which will make it easier to navigate through your work.
• In Query, the Copy Output to Clipboard function has been refactored to not use Flash.
• Background jobs created by modeling are now hidden to avoid confusion and clutter on the homepage and associated indices.
• 4X improvement to response time on the Imports, Models, and Projects indices and associated API endpoints.
• Fixed a bug in public script templates to allow templated script creation by any user.

Week Ending 3/3/2017

• Upgraded Rails to 4.2.8.
• Fixed a bug with Salesforce Import where user and client credentials weren't loading properly.
• Fixed a bug with Imports where the "If Table Exists" option of "Drop" couldn't drop an empty table.

Week Ending 2/24/2017

• Upgraded Tableau Server to 10.1
• Added more detailed error messages to Database Imports.
• Added support for non-numeric ID columns in Database Imports.
• Reduced Modeling memory consumption defaults.
• Added a helpful error message when you try to import an empty sheet in a Google Doc.
• Fixed a bug in Database Imports where users couldn't select a source database/credential they were permissioned on.

Week Ending 2/17/2017

• The API now allows you to add an object as an output of a run and then access that output via the runId.
• Fixed a bug in exports where a cloned export was redirecting to the /#/imports page.
• Fixed a bug in generating script output links.

Week Ending 2/10/2017

• Exposed the bocce_clusters API endpoint to improve visibility into Bocce jobs (such as Container Scripts).
• Disabled index sorting by last run to improve performance.
• Fixed a bug with cloning Imports and Exports.
• Fixed a bug in scoring models.
• Deprecated an unused API endpoint.
• Fixed a bug with Google Doc Imports to allow for null values in the rightmost column.

Week Ending 2/3/2017

• Tableau reports are archivable.
• Fixed a JavaScript bug in the modeling workflow.

Week Ending 1/27/2017

• Users with "view" permissions on a script can now access the run history pane.
• Added the cancel button to Imports.
• New Civis account creation links now expire after 7 days instead of 3 days to accommodate accounts created just before the weekend.
• Fixed a visual bug in Imports filters that caused the list to spill off the screen.
• Upgraded Elasticsearch to 2.0.
• Rearranged past dates on the Query page to add clarity when scrolling through the query history.
• Updated text in the Sharing pane to add clarity when a job is shared to a person hidden from the user.

Week Ending 1/20/2017

• Users have more flexible CSV settings when using the POST /scripts/sql API endpoint.
• Added the runner_user_id environment variable to the SQL Scripts API to add clarity between the user IDs of a script's author and a script's runner. 

Week Ending 1/13/2017

• Allow access to credentials that have been shared with a user's non-primary groups.
• Upgrade Bundler to 1.13.6.
• Upgrade Ruby to 2.3.3.
• Additional sorting and filtering options on the /scripts/custom API endpoint.
• Users can now archive scripts with archived templates.
• Updated scripts documentation to clarify author parameter.
• Users can match an S3 file prefix with a file's owner for jobs run by the non-author.
• Restore "New Database" button to Database Import.
• Fixed a bug where users had difficulty editing notifications after a job has run.
• Fixed a bug where the notifications pane in Imports wouldn't open. 

Week Ending 12/16/2016

• Improved organization and usability in Search.
• Allow “run script to update report” action when report is created in query to mimic script-backed report behavior.
• Allow resources allocated to a script template to be set at runtime using parameters to give template users more control over scripts.
• More intuitive handling of headers in CSV and Google Doc imports to reduce import errors.
• Bug fixes in report modification.

Week Ending 12/2/2016

• Search request optimization.
• Bug fix in CMO channel mapping storage.
• Bug fix to restore permissions in Tableau workbooks.
• Modeling wizard - Instead of creating a model object at the start of the wizard, collect all settings and then create and build the model when the wizard is completed.
• Remove unnecessary Redshift usage.
• Sync notification bug fixes.
• File read and validation for spot order model.