Dataflow and Ibexa Dataflow 7: UTC dates, richer job tracking, stronger security

Since our last article on Dataflow (externalised logs and automated history cleanup), Ibexa Dataflow has gone from version 6.1.2 to 7.0.0. Richer job tracking, product attribute comparison for Ibexa Commerce, security fixes and, with 7.0, dates stored in UTC: here are the details.

Richer job tracking

  • Delete a pending job straight from the interface (6.2).
  • Download a job's log file (6.4), and better exception handling in the streamed log output (6.3).
  • Type filter, a counter on the dashboard, average run time on the scheduled dataflows tab and improved options display (6.5).
  • Job window with "Information" and "Log" tabs (6.6). The ?job=<id> parameter opens a job's details directly, for example from an Ibexa notification.
  • Nicer form windows: focus on the first field, fields and errors reset on close, submit button disabled while processing (6.6).

Ibexa Commerce: product attribute comparison

The NotModifiedProductFilter filter now compares product attributes, type by type (6.7). It no longer skips the update of a product whose attribute has changed or been cleared, for example a checkbox set to null .

  • Two interfaces let you customise the comparison: ProductAttributesComparatorInterface and ProductAttributeValueComparatorInterface .
  • A comparator for the ibexa_seo field (6.5) and a fix for the BillingAddressComparator (6.1.2).

Security: upgrade to at least 6.5.1

Version 6.5.1 fixes four issues:

  • a missing permission check when editing a scheduled dataflow (broken access control);
  • an XSS flaw in the streamed log output;
  • enabling and disabling a scheduled dataflow now requires a POST request;
  • a 404 error, rather than a 500, when a job or scheduled dataflow ID does not exist.

Version 7: dates move to UTC

This is a backward-incompatible change. Dates of scheduled dataflows and jobs are now read and stored in UTC, whatever PHP's timezone, and displayed in the user's timezone. The forms that depended on both PHP's timezone and the user's are fixed.

  • Ibexa Dataflow 7 requires Dataflow 7 and PHP 8.4.
  • The UserTimezoneAwareDateTimeType and UserTimezoneAwareDateTimeTransformer form classes are removed: forms now use Ibexa's date and time picker.
  • A new command converts the dates entered in forms to UTC.

Updating your existing dates

Back up your tables before you start: the conversion commands are not idempotent, run each of them only once. Run them first with --dry-run , then without that option when the result looks right, and deploy the new version.

Dates written by PHP (start_time and end_time ) were stored in PHP's timezone. Dates entered in forms (next and requested_date ) depend on the version of the bundle that created the row.

1. Dates written by PHP. They are always converted by the Dataflow bundle's command:

   

php bin/console code-rhapsodie:dataflow:convert-dates-to-utc [--timezone=Europe/Paris] --dry-run 
   

2. Dates entered in forms. What to do depends on where the rows came from:

  • Rows created with Ibexa 3 (ezdataflow-bundle 3.x): the value stored in PHP's timezone is correct. Simply add the --next and --requested-date options to the previous command.
  • Rows created with Ibexa 4 or later: the old forms stored the UTC time shifted by the difference between PHP's timezone and the user's. The original date can only be recovered if you tell which timezone your users had (the same for all of them) and which one PHP had. Do not use the --next and --requested-date options on these rows.
   

php bin/console code-rhapsodie:ibexa-dataflow:convert-dates --user-timezone=Europe/Paris [--php-timezone=UTC] --dry-run 
   

The --php-timezone option defaults to the configured PHP timezone.

Beware of tables that mix both origins, for example rows migrated from Ibexa 3 and others created later with Ibexa 4 or later. Rows are not tagged by origin and the commands convert every row: some of them would end up shifted. Check the result with --dry-run and fix the rows of the other origin separately, for example with an SQL query on a range of IDs. The details are in the repository's UPGRADE.md file.

And Dataflow?

You have to upgrade from version 5 to version 6, Dataflow adds columns to the jobs table (the exceptions and their count). Update your database schema:

   

php bin/console code-rhapsodie:dataflow:database-schema --update 
   

Want to go further and try Dataflow: https://github.com/code-rhapsodie/dataflow-bundle/#installation

Try Ibexa Dataflow