Civis Platform Release Notes 2018

Week Ending 12/28/2018

• Added new parameters for primaryKeys and lastModifiedKeys to the /tables/{id} endpoints.

Week Ending 12/21/2018

• Improved error messages on imports when multiple line endings are detected in the file.
• The LIST /imports API endpoint now allows you to filter database imports by the source.
• Created the GET /jobs/:id/runs/:run_id/logs API endpoint to see logs for any job type.
• Fixed a bug in Address Correction where a temporary table was not always dropped.
• Security updates.

Week Ending 12/14/2018

• Fixed an issue where Imports from File, Google Sheet, S/FTP, or URL would fail if the column names specified as the dist key or sort key were cleaned by the job.
• Fixed an issue in the Address Correction job causing an “undefined method `each' for true:TrueClass” error.
• Clarified in the API docs the expected value for the limitingSQL parameter in /enhancements/cass-ncoa endpoints.
• Removed the POST /tables endpoint.
• Security updates.

Week Ending 12/7/2018

• Updated the Civis Python API client to v1.9.2.
  • Improved handling of S3 connection errors by civis.io.civis_to_file.
• Prevented Address Correction jobs from failing if the input table doesn’t have metadata.
• Fixed an error in Address Correction where temp tables weren’t being dropped sometimes.
• Removed outdated IP addresses from the UI for setting up a New Database and replaced with a link to a help center article with the current addresses.
• Fixed an issue where the column names and data types weren’t editable in the File Import job UI if the file has quoted headers.
• Fixed a bug with setting session authorization in Query.
• Fixed an issue with fixed credentials on script templates.
• Prevented unnecessary job failures in Imports from Facebook, File, Google Analytics, SFTP, and URL.
• Fixed an issue where Google Sheet imports were failing when some headers were blank.
• Fixed an issue with the table preview in File Imports when the table had CR line endings.
• User information is now passed to services via service tokens.
• Allow authors of SQL script templates to set fixed credentials on the template in runner context and non-fixed credentials in author context.
• Security updates.

Week Ending 11/16/2018

• Reduced page load times.
• Upgraded Tableau Server from 10.5 to 2018.2.
• Upgraded the Python API Client to v1.9.1.
• Fixed a bug in civis.io.civis_to_multifile_csv.
• Fixed a bug in CivisML due to a recent pandas update.
• Added more documentation for Windows users.
• Other documentation updates.
• Added file IDs on the project’s page for files in the project.
• Fixed an issue where tables created from Database Imports weren’t showing up right away in the left pane.
• Improved error messages in File Imports.
• Improved efficiency of scaling down instances after using Notebooks and Services.

Week Ending 11/9/2018

• Italicized the names of table views in the left pane.
• The width of the left pane is now adjustable by clicking on its right edge and dragging left or right.
• Fixed a typo in “More Enhancements” menu.
• Security updates

Week Ending 11/2/2018

• Added the memory used by a notebook deployment to the deployment logs, available at GET /notebooks/{id}/deployments/{deployment_id}/logs.
• Fixed an issue where committing a Notebook was sometimes not responsive.
• Fixed an issue causing SQL scripts to fail with “Run Error: Connection timed out” error.
• Fixed an issue where cancelled SQL scripts were stuck in “running” state and were not sending failure notification emails.
• Fixed an issue where clicking on a model on the Models page would fail to load the page for that Model, redirecting the user back to the Models page.
• Added “adapter” field to the response from the GET /databases/ endpoint to indicate the type of database.
• Added a new GET /databases/{id} endpoint which responds with the database id, name, and adapter.
• Changed "Change Redshift Password" on My Profile page to "Change Database Password".
• Fixed an issue where dropping a table did not also drop its ontology, which prevented a new table with the same name from being created in the future.
• Added GET /clusters/kubernetes/{id}/deployment_stats endpoint which returns active and queued jobs.
• Security updates.

Week Ending 10/26/2018

• New Platform navigation!
• Updated login page to reflect new Civis logo and color scheme.
• Updated favicon to be new Civis logo.
• Added CPU, memory, and disk space parameters to Imports from S/FTP.
• Fixed an issue where the API keys page would automatically refresh when a new key was created.
• Improved the reliability of updates to the information in the data pane and table details.
• Fixed an issue with Salesforce imports encountering end of file errors
• Added instanceMaxDisk to the /clusters/kubernetes endpoint.
• Upgraded Civis Data Match, Table Deduplication, and Data Unification jobs to use person-matching v3.1.2.
  • Increased the speed of the Civis Data Match job.
  • Minor improvements to the handling of phone numbers, latitude, and longitude.
    • E.g. the algorithm now ignores common fake phone numbers like 1234567890.
  • Output table name is now always lowercase.

Week Ending 10/19/2018

• Added support for curly brackets and other symbols in remote host JDBC URLs, enabling connections to databases with spaces in their names.
• Added support for published SQL scripts to have either fixed or user-supplied credentials regardless of author context.
• Improved performance of Imports from Facebook, File, Google Analytics, SFTP, and URL when “if table exists” is set to append or wipe.
• Security updates

Week Ending 10/12/2018

• Fixed an issue where some Imports and Exports were not showing up in the job search in Workflows.
• Fixed an issue where Notebooks were not taking up the whole height of the screen.
• Added export templates to the GET /search/ API endpoint.
• Changed the email footer on success notifications so it shows the email address of the user who runs the job instead of the owner of the job.
• Fixed an issue with our maximum memory calculations on Notebooks and Services. The maximum memory users can allocate was reduced slightly.
• Security updates.

Week Ending 10/5/2018

• Upgraded Civis Data Match, Table Deduplication, and Data Unification jobs to use person-matching v3.0.1
  • Correction: Previous version used was v2.1.1. The job was not updated to 3.0.0 during week of 2018.09.14-2018.09.20
  • Updated model used for scoring
  • Refactored the Data Unification job to cut runtime in about half and reduce memory usage by about 20%
  • Fixed a bug with matching on nicknames
  • Fixed a bug in preparing data for matching
  • Improved handling of special characters
  • Fixed an issue with parsing data in the jobs
  • Jobs now fail when there are parsing issues instead of attempting to match records
• Fixed broken links to help documentation in File Imports jobs
• Searching on the Users page now supports search terms with more than 10 characters
• Improved the reliability of updates to the information in the data pane and table details, and reduced the cluster resources consumed by update jobs.
• Fixed an issue that prevented grant queries from succeeding
• Security updates

Week Ending 9/28/2018

• Improved detection and handling of delimiters and line endings in Imports from File, Box, Facebook, Google Analytics, S/FTP, and URL.
• Implemented logic that prevents jobs and deployments from failing when there is an issue with logs.
• Made service deployments with a custom application port more reliable.
• Fixed a bug where a table wouldn’t show up in the data pane after its name was changed.
• Resolved error messages showing up in Import from Facebook jobs.
• Updated our Import from Facebook job to use v3.0 of the facebook-sdk.

Week Ending 9/21/2018

• Upgraded Civis Data Match, Table Deduplication, and Data Unification jobs to use person-matching v3.0.0.
  • Changed model used for scoring.
  • Improved handling of special characters.
  • Fixed a bug with matching on nicknames.
  • Fixed a bug in preparing data for matching.
• Added a preview of the Service below the Settings. Removed the option to deploy a preview.
• Improved the handling of “execution expired” errors.
• Security updates.

Week Ending 9/14/2018

• Upgraded Civis Data Match, Table Deduplication, and Data Unification jobs to use person-matching v2.1.1
  • Security update
• Removed the thumbnail view from the Reports page
• Improved performance of SQL scripts
• Fixed an issue where dropping a table did not also drop its ontology, which prevented a new table with the same name from being created in the future
• Fixed an issue with services that use the viewer's API key
• Fixed an issue causing rate card imports to fail
• Performance improvements
• Security updates

Week Ending 9/7/2018

• Fixed an issue where an already running job that a workflow tries to kick off shows as idle instead of failed.
• Added the option for new users to use an authenticator app for two-factor authentication when they set up their account.
• Renamed the “Redshift credential” parameter type in the Scripts to “Database credential”
• Security update

Week Ending 8/31/2018

• Increased the security of the Civis VPN login by implementing a temporary ban after failed login attempts.
• Added additional vulnerability detection to Tableau and Platform servers.
• Fixed an issue where Import from S/FTP jobs sometimes timed out.
• Updated Services to improve hardware utilization .
• Fixed an issue that prevented the export icon displaying in project details
• Fixed an error causing script jobs to fail intermittently.

Week Ending 8/24/2018

• Changed the Project description default to be blank instead of “default project description”.
• Added the ability for organizations to have some of their users use Single Sign On to log into Platform and others log in using passwords. Contact Client Success if you’re interested in implementing this for your organization.
• Rounded the upper memory limit for Notebooks and Services down to nearest integer.
• Fixed an issue where Services that are set to be always on were still going to sleep.

Week Ending 8/17/2018

• Fixed an issue where R scripts using the ‘civis’ package were failing
• Fixed an issue where some jobs in a workflow were marked as “failed” even though they never ran
• Made some changes to Salesforce imports to fix the “End of File Reached” error
• Fixed an issue causing Google Sheet imports to fail
• Improved reliability of logs for Services
• Added support for importing and exporting larger files via S/FTP
• Fixed an issue where the provideAPIKey attribute was being cleared when making any PATCH /reports/services request
• Fixed a misspelling in the API documentation
• A new version of the R client enables hyperband for stacking estimators in CivisML. If an API key is present but the auto-generation fails, we now revert to the default API client.

Week Ending 8/10/2018

• Fixed an issue that was causing notebooks to drop their connection
• Fixed an issue where the workflow graph was not rendering if there were symbols in job names
• Made reports default to full-screen when they’re opened from a project
• Fixed an issue where the data pane was not updating right away to reflect new or dropped schemas when the create/drop statement used “create/drop schema if not exists”
• Improved the reliability of service logs
• Made the error message more helpful when a service cannot be deployed because there are no available instances
• Added support for importing and exporting larger files via S/FTP
• Added better error messaging to S/FTP imports and export

Week Ending 8/3/2018

• Only the first 10 unique table names listed in a query get linked to their Table Details page. This is to improve performance on the Query page.
• Fixed an issue where renamed schemas weren’t getting updated in the data pane right away if there was a line break in the ALTER SCHEMA query.
• Added an error message to the Disk Space field on Container Scripts if an invalid value is entered.
• Added CIVIS_SERVICE_ID and CIVIS_DEPLOYMENT_ID environment variables into the container run in a service.
• Security updates

Week Ending 7/27/2018

• Added a button on the My Profile page that can trigger a password-reset email.
• Fixed an issue where Tableau reports would shift slightly when a user hovered over a tooltip within them.
• Fixed an issue with updating the details of tables and schemas with long names.

Week Ending 7/20/2018

• Limited all schema dropdowns in Imports to the first 50 schemas by default to avoid page freezes.
• Fixed an issue where some logs in the Run History weren’t showing up.
• Added information about the last run of a job to the GET /search/ response.
• Security updates.

Week Ending 7/13/2018

• Environment variables in scripts are no longer case sensitive.
• Reduced the likelihood of imports failing due to connection issues.
• Fixed an issue which was preventing some Container Scripts from finishing.
• Fixed an issue where schemas granted or revoked with lowercase “grant” or “revoke” statements weren’t being updated right away in the data pane.

Week Ending 7/6/2018

• Added a fix for the “Failed to send a recent heartbeat” error on some jobs.
• Added validation to prevent a script created from a script template from being published as a new template.
• Fixed alphabetical sorting in search.
• Fixed a bug where exports to a Google Sheet with A1 notation in the name didn’t work correctly.
• Added the archived status of app instances and releases to the GET API endpoints.
• Security updates

Week Ending 6/29/2018

• Added new API endpoints under /enhancements for the Civis Data Match, Data Unification, and Table Deduplication jobs.
• Fixed a bug that caused imports from Postgres databases to fail.
• Reverted last week's fix for GRANT statements on schemas after we found it caused jobs with GRANT statements to take too long and are working on a more permanent solution.

Week Ending 6/22/2018

• Changed how we handle system failures so jobs are considered failed instead of canceled
• Fixed an issue where grant statements on schemas were not working if the schema was lowercase
• Added an error to the GET /scripts endpoint if the supplied type isn’t one of the supported ones
• Enabled services to use a custom endpoint for health checks
• Security updates

Week Ending 6/15/2018

• In order to improve performance, limited the number of run outputs that can be referenced by a workflow to the first 100 that each job produces.
• Added a “state” parameter to the credentials endpoint, which can be used for VAN credentials.

Week Ending 6/8/2018

• Made some wording changes to API documentation to generalize terminology in places where it was too specific.
• Fixed an issue that was causing some scripts to fail.
• Fixed an issue where editing two-factor authentication settings was disabled for some users.
• Fixed an issue where some credentials weren’t showing up in the credential dropdown in Query.
• Fixed an issue where Services could not be deployed without a git repository.

Week Ending 6/1/2018

• Nothing new this week, but exciting updates are in the works!

Week Ending 5/25/2018

• Added Workflow ID and Execution ID as environment variables to Workflows available in the Configuration pane. They are accessible via `<% $.get('__env').workflow_id %>` and `<% $.get('__env').execution_id %>` in the YAML.

Week Ending 5/18/2018

• Added more helpful details to the error messages for SQL scripts.
• Cleaned up error messages in Query.
• Fixed an issue with sending success notifications causing SQL script jobs to fail.
• Fixed an issue with the syntax for BIGINT columns in the “Create Statement” tab in table details.
• Fixed an issue with setting up two-factor authentication for some phone numbers.

Week Ending 5/11/2018

• Fixed an issue where File Imports were ignoring the first row of data if it was not a header row.
• Fixed an issue where Services could not be deployed without a Git repository.
• Fixed an issue where users with their Redshift cluster on certain servers could not create a batch file import.
• Fixed an issue in Media Optimizer when inexpensive spots were used.

Week Ending 5/4/2018

• Upgraded the Civis API Python Client to v1.9.0
  • Fixed an issue which could prevent `civis.io.civis_to_file` from returning the entire Civis File object when writing to a user-supplied buffer.
  • Redshift schema / tablename splitting now correctly handles quoted identifiers.
  • Documentation fixes and improvements
  • Added `civis.io.export_to_civis_file` function.
  • Default to creating new APIClient objects with `resources="all"` parameter setting, instead of the previous `resources="base"`.
  • New version of CivisML (v2.2.0)
• Upgraded CivisML to v2.2.0
  • When making predictions for multi-output models, users may request to only score a subset of outputs. Use the "dvs_to_predict" parameter in the API clients' "predict" functions.
  • Users may now train (scikit-learn-compatible) model objects in their local environments, then upload them to Civis Platform for CivisML to score. Use via the "civis.ml.ModelPipeline.register_pretrained_model" method in the Python API client v1.9.
  • Added an `if_exists="truncate"` option for writing output score tables.
  • Include model metrics with the training metadata (the "metadata.json" output file) instead of only in the "metrics.json" output file.
  • Now truncate column names in predictions if they would be over Redshift's column name length limit
  • Fixed a bug where we could make the wrong decision about how to parallelize a prediction job if the Civis Platform's table scanner hadn't run yet
  • Fixed a bug where random forest and extra trees models would train trees much deeper than necessary, resulting in excessively long training times and large memory usage. (These tree models now default to a max depth of 7, instead of unlimited, and hyperband no longer explores unlimited depth trees.)
  • When making predictions, write Redshift tables with 4-byte floats instead of 8-byte floats to reduce disk usage
  • In pre-defined stacking models, change the step name "sparse_logistic" to "logistic_with_ridge" and change "sparse_linear_regressor" to "ridge", to match the algorithms we're actually using.
• Upgraded the Civis search functionality and improved reliability.
• Upgraded Ruby to 2.3.7.
• Fixed an issue that caused Report Snapshots to stop working.

Week Ending 4/27/2018

• Updated the Address Correction job so it automatically removes double quotes and commas from data, preventing job failures.
• Fixed an issue where someone running a notebook created by someone else couldn’t shut it down. All notebooks now run as the user who starts it.
• Fixed a bug where some logs weren’t showing up in the Run History.

Week Ending 4/20/2018

• Added links to the R Client documentation in the API docs
• Added an API endpoint for retrieving logs for Notebook deployments.
• Added an API endpoint to /jobs to cancel a run.
• Updated the Workflow configuration code box so it grows when the code exceeds the default height.
• Added error messages to the Address Correction job for when the input file is too small (less than 100 records) or if a specified column does not exist.
• Updated the name of the browser tab to match the name of the current page when using Civis apps.
• Added a clearer error message in Media Optimizer for when the selected date range and the date range on the selected rate card do not overlap.
• Fixed a Media Optimizer bug where optimizations were not using the full budget when certain constraints were set.

Week Ending 4/13/2018

• Updated the user drop-down menu to include direct links to the Help Center and the newly-optimized API documentation.
• Fixed a bug that was preventing schedule parameters from being set through the API.
• Fixed a bug that was preventing git credentials from being copied while cloning a Notebook.

Week Ending 4/6/2018

• Fixed a bug that was causing unexpected behavior in advanced options toggling for imports with multiple syncs.

Week Ending 3/30/2018

• Updated search to include published scripts.
• Fixed a bug that was causing Workflows icons to fail to render.

Week Ending 3/23/2018

• Added an API endpoint that gives users the ability to post log messages to running Container, Custom, Python, and R scripts.
• Fixed a bug in Salesforce imports that was causing silent failures when records were missing.

Week Ending 3/16/2018

• Labs-created person matches will now only include relevant match targets.
• Updated the icon in Query that copies your past code to the Query editor.
• Fixed a bug that allowed for the logging of credential parameter details.

Week Ending 3/9/2018

• Released Workflow Parameterization.
• Fixed a bug in version control that prevented users from configuring and committing to GitHub repositories.

Week Ending 3/2/2018

• Added job schedule information to /jobs API endpoint.
• Patched a SAML vulnerability that could allow for authentication bypass via incorrect XML. You can read more about the vulnerability here.
• Fixed a bug that was causing errors when linking to custom scripts in the Triggers section of the Scheduling/Automation pane.
• Fixed a bug that was preventing scheduled Workflows from running.
• Fixed a bug in Platform Search that was preventing search results from properly populating.

Week Ending 2/23/2018

• Updated Scripts to disable the “run now” button on a job if it is already queued or running.
• Updated compute scaling so that Script workers are provisioned instantly in response to the number of queued jobs.
• Fixed a bug in the Help Center that was causing document search to be hidden by chat.
• Fixed a bug that was preventing newly granted tables and schemas from showing up in the data pane.
• Fixed a bug in scheduled jobs that was causing some jobs to run twice.

Week Ending 2/16/2018

• Upgraded CivisML to v2.1.2
  • Fixed an issue where not all exceptions were being caught while writing to the out-of-sample table.
  • Changed to install custom dependencies before running CivisML training jobs, so that user-specified package versions override default CivisML versions.
  • Added a fix to prevent stacking models from failing.
  • Added a more useful error message for nulls in multi-target models.
  • Fixed a typo in FIT_PARAMS argument of deploy templates.
  • Updated the Civis Python API client from v1.7.2 to v1.8.1 to include a patch for logging errors which was released with v1.8.1.
  • Worked around a bug in scikit-learn's code which prevented
    us from running multioutput-multiclass models.
  • Changed N_JOBS to allow floats in deploy template.
  • Created new test protocols and updated acceptance tests.
  • Restricted prediction jobs so that they don't launch child jobs from the child jobs.
  • Added guardrails to check that both schema and table names are present.
  • Added instructions for changing a Docker image tag in case of a broken release.
  • Increased maximum cancel timeout on CivisML jobs from 60 to 600 seconds to allow extra time for child jobs to be shut down.
• Fixed an issue where workflows were not running on schedule.
• Improved performance when there is a large demand of resources on Platform.
• Changed the error shown when attempting to use an API key that is inactive because it belongs to an inactive user.

Week Ending 2/9/2018

• Added a credential parameter to Notebooks. You can now include a credential when launching your Notebooks environment so you can use the credential while working in your Notebook without exposing the credential details in plain text.
• Added civis.workflows.execute action to Workflows -- now Workflows can include other Workflows by using this action type and specifying a workflow_id parameter as a task input.
• Changed Docker tag pinning for Notebooks to pin to minor versions instead of patch versions. The default tag will now be of the form "1.6" instead of "1.6.2" to enable bug fixes to be deployed to pinned images.
• Increased the maximum cancel timeout for a job from 60 to 600 seconds to allow for enough time to cleanly cancel complex jobs.
• Decreased the max CPU value in Notebooks to accommodate overhead CPU required by the notebooks server. Updated all existing notebooks' CPU usage to values that do not exceed the new maximum.

Week Ending 2/2/2018

• Fixed a bug where GRANT queries took a long time to run.
• Fixed a bug that was preventing job logs from being included in notification email.

Week Ending 1/26/2018

• Added the ability to search on job ID in the Civis Platform search bar.
• Updated Python and R scripts to automatically use the latest Docker image on script creation.
• Removed the Notes field from script templates that include a custom UI.
• Improved Civis performance during the execution of scheduled jobs.
• Removed the ability to include Unicode characters in a workflow to prevent the workflow from being saved in an unreadable state.

Week Ending 1/19/2018

• Added more explicit error messages when trying to execute an invalid Workflow via the API.
• When viewing the execution for a Workflow, the YAML code box is grayed out to indicate the code is not editable.
• Changed the highlighting of the Workflows graph to indicate all immediate upstream and downstream tasks when hovering over a task.
• Fixed a bug in Workflows where tasks with extremely long error messages weren't properly updating their task state.

Week Ending 1/12/2018

• Added a "Description" field to Workflows so users can include helpful descriptive text for other members of their organization.
• Changed the Workflows execution side panel to indicate which execution is currently being viewed.
• Fixed a bug that was preventing Notebooks with unicode characters from starting.
• Fixed a bug where Civis was not properly recognizing the width of columns with a CHAR data type.
• Fixed a bug in Workflows so that longer error messages are handled gracefully.
• Fixed a bug in the Workflows graph that was causing incorrect job status icon to be displayed.
• Fixed a bug that was causing the run history pane to show a job as "Queued".
• Fixed a bug in the User Details page so that user’s primary group is displayed properly.

Week Ending 1/5/2018

• In Workflows, YAML configuration error messages will now specifically mention missing or invalid actions.