View Source Changelog

All notable changes to this project will be documented in this file.

  • Added for new features.
  • Changed for changes in existing functionality.
  • Deprecated for soon-to-be removed features.
  • Removed for now removed features.
  • Fixed for any bug fixes.
  • Security in case of vulnerabilities.

The format is based on Keep a Changelog, and this project adheres to Semantic Versioning.

[Unreleased]

Added

Changed

  • Added Claude integration in job chat #2403
  • OPENAI_API_KEY renamed to AI_ASSISTANT_API_KEY #2403

Fixed

[v2.10.8] - 2024-12-18

Added

  • Add ability to retry or cancel AI Assistant error responses for user messages #2704

Changed

  • Adjusted AI Assistant documentation to note that messages may be monitored

[v2.10.7] - 2024-12-13 🎂 ❤️ Happy Birthday, Mom!

Added

  • Clear AI assistant's chat input after a message is sent #2781
  • Allow different rules and action for delete user. #2500
  • Handle errors from the AI Assistant more gracefully #2741

Changed

  • Make the AdaptorRegistry cache path configurable #2780

Fixed

  • Delete user modal no longer uses the same id as the underlying user record. #2751
  • Use workflow activation limiter on index toggle. #2777

[v2.10.6] - 2024-12-10

Added

  • Handle errors from the AI Assistant more gracefully #2741

Changed

  • Updated the About the AI Assistant help text
  • Make user email verification optional. Defaults to false #2755

    ⚠️ The default was behaviour was to always require email verification. Set REQUIRE_EMAIL_VERIFICATION to true to revert to the old behaviour.

  • Enhance AI assistant panel UI #2497
  • Allow superusers to be created via the user UI. #2719

Fixed

  • Fix Priority and Scope Issues in Inspector Key Bindings #2770
  • Fixed an issue where sometimes adaptor docs won't load in the Inspector #2749
  • Return a 422 when a duplicate key is sent to the collections post/put_all API #2752
  • Do not require the user's password when a superuser updates a user. #2757

[v2.10.5] - 2024-12-04

Added

  • Enable Tab Key for Indenting Text in AI Assistant Input Box #2407
  • Ctrl/Cmd + Enter to Send a Message to the AI Assistant #2406
  • Add styles to AI chat messages #2484
  • Auditing when enabling/disabling a workflow #2697
  • Ability to enable/disable a workflow from the workflow editor #2698

Changed

  • Insert all on a collection with the same timestamp #2711
  • AI Assistant: Show disclaimer once every day per user #2481
  • AI Assistant: Scroll to new message when it arrives #2409
  • AI Assistant: Set vertical scrollbar below the session title #2477
  • AI Assistant: Increase size of input box for easier handling of large inputs #2408
  • Bumped dependencies
  • Extend display of audit events to cater for deletions. #2701
  • Kafka documentation housekeeping. #2414

Fixed

  • Collections controller sending an invalid response body when a item doesn't exist #2733
  • AI Assistant: Text in the form gets cleared when you change the editor content #2739

[v2.10.4] - 2024-11-22

Added

  • Support dynamic json schema email format validation. #2664
  • Audit snapshot creation #2601
  • Allow filtering collection items by updated_before and updated_after. #2693
  • Add support for SMTP email configuration #2699 ⚠️️ Please note that EMAIL_ADMIN defaults to lightning@example.com in production environments

Fixed

  • Fix cursor for small limit on collections request #2683
  • Disable save and run actions on deleted workflows #2170
  • Distinguish active and inactive sort arrows in projects overview table #2423
  • Fix show password toggle icon gets flipped after changing the password value #2611

[v2.10.3] - 2024-11-13

Added

  • Disable monaco command palette in Input and Log viewers #2643
  • Make provision for non-User actors when creating Audit entries. #2601

Fixed

  • Superusers can't update users passwords #2621
  • Attempt to reduce memory consumption when generating UsageTracking reports. #2636

[v2.10.2] - 2024-11-14

Added

  • Audit history exports events #2637

Changed

  • Ignore Plug.Conn.InvalidQueryError in Sentry #2672
  • Add Index to dataclip_id on runs and work_orders tables to speed up deletion PR#2677

Fixed

  • Error when the logger receives a boolean #2666

[v2.10.1] - 2024-11-13

Fixed

  • Fix metadata loading as code-assist in the editor #2669
  • Fix Broken Input Dataclip UI #2670

[v2.10.0] - 2024-11-13

Changed

  • Increase collection items value limit to 1M characters #2661

Fixed

  • Fix issues loading suggestions for code-assist #2662

[v2.10.0-rc.2] - 2024-11-12

Added

  • Bootstrap script to help install and configure the Lightning app for development #2654

Changed

  • Upgrade dependencies #2624
  • Hide the collections and fhir-jembi adaptors from the available adaptors list #2648
  • Change column name for "Last Activity" to "Last Modified" on Projects list #2593

Fixed

  • Fix LiveView crash when pressing "esc" on inspector #2622
  • Delete project data in batches to avoid timeouts in the db connection #2632
  • Fix MetadataService crashing when errors are encountered #2659

[v2.10.0-rc.1] - 2024-11-08

Changed

  • Reduce transaction time when fetching collection items by fetching upfront #2645

[v2.10.0-rc.0] - 2024-11-07

Added

  • Adds a UI for managing collections #2567
  • Introduces collections, a programatic workflow data sharing resource. #2551

[v2.9.15] - 2024-11-06

Added

  • Added some basic editor usage tips to the docs panel #2629
  • Create audit events when the retention periods for a project's dataclips and history are modified. #2589

Changed

  • The Docs panel in the inspector will now be closed by default #2629
  • JSDoc annotations are removed from code assist descriptions #2629
  • Show project name during delete confirmation #2634

Fixed

  • Fix misaligned margins on collapsed panels in the inspector #2571
  • Fix sorting directions icons in projects table in the project dashboard page #2631
  • Fixed an issue where code-completion prompts don't load properly in the inspector #2629
  • Fixed an issue where namespaces (like http.) don't appear in code assist prompts #2629

[v2.9.14] - 2024-10-31

Added

  • Additional documentation and notification text relating to the importance of alternate storage for Kafka triggers. #2614
  • Add support for run memory limit option #2623

Changed

  • Enforcing MFA for a project can be enforced by the usage limiter #2607
  • Add extensions for limiting retention period #2618

[v2.9.13] - 2024-10-28

Changed

  • Add responsible ai disclaimer to arcade video #2610

[v2.9.12] - 2024-10-25

Fixed

  • Fix editor panel buttons gets out of shape on smaller screens #2278
  • Do not send empty strings in credential body to the worker #2585
  • Refactor projects dashboard page and fix bug on last activity column #2593

[v2.9.11] - 2024-10-23

Added

  • Optionally write Kafka messages that can not be persisted to the file system. #2386
  • Add MessageRecovery utility code to restore Kafka messages that were pesisted to the file system. #2386
  • Projects page welcome section: allow users to learn how to use the app thru Arcade videos #2563
  • Store user preferences in database #2564

Changed

  • Allow users to to preview password fields in credential forms #2584
  • Remove superuser flag for oauth clients creation #2417
  • Make URL validator more flexible to support URLs with dashes and other cases #2417

Fixed

  • Fix retry many workorders when built for job #2597
  • Do not count deleted workflows in the projects table #2540

[v2.9.10] - 2024-10-16

Added

  • Notify users when a Kafka trigger can not persist a message to the database. #2386
  • Support kafka trigger type in the provisioner #2506

Fixed

  • Fix work order retry sorting and avoids loading dataclips #2581
  • Fix editor panel overlays output panel when scrolled #2291

[v2.9.9] - 2024-10-09

Changed

  • Make project description multiline in project.yaml #2534
  • Do not track partition timestamps when ingesting Kafka messages. #2531
  • Always use the initial_offset_reset_policy when enabling a Kafka pipeline. #2531
  • Add plumbing to simulate a persistence failure in a Kafka trigger pipeline. #2386

Fixed

  • Fix Oban errors not getting logged in Sentry #2542
  • Perform data retention purging in batches to avoid timeouts #2528
  • Fix editor panel title gets pushed away when collapsed #2545
  • Mark unfinished steps having finished runs as lost #2416

[v2.9.8] - 2024-10-03

Added

  • Ability for users to to retry Runs and create manual Work Orders from the job inspector #2496 #2496

Fixed

  • Fix panel icons overlays on top title when collapsed #2537

[v2.9.7] - 2024-10-02

Added

  • Enqueues many work orders retries in the same transaction per Oban job. #2363
  • Added the ability to retry rejected work orders. #2391

Changed

  • Notify other present users when the promoted user saves the workflow #2282
  • User email change: Add debounce on blur to input forms to avoid validation after every keystroke #2365

Fixed

  • Use timestamps sent from worker when starting and completing runs #2434
  • User email change: Add debounce on blur to input forms to avoid validation after every keystroke #2365

Fixed

  • User email change: Send notification of change to the old email address and confirmation to the new email address #2365
  • Fixes filters to properly handle the "rejected" status for work orders. #2391
  • Fix item selection (project / billing account) in the context switcher #2518
  • Export edge condition expressions as multiline in project spec #2521
  • Fix line spacing on AI Assistant #2498

[v2.9.6] - 2024-09-23

Added

Changed

  • Increase minimum password length to 12 in accordance with ASVS 4.0.3 recommendation V2.1.2 #2507
  • Changed the public sandbox (https://demo.openfn.org) setup script to use welcome12345 passwords to comply with a 12-character minimum

Fixed

  • Dataclip selector always shows that the dataclip is wiped even when the job wasn't run #2303
  • Send run channel errors to sentry #2515

[v2.9.5] - 2024-09-18

Changed

  • Hide export history button when no workorder is rendered in the table #2440
  • Improve docs for running lightning locally #2499 #2499

Fixed

  • Fix empty webhook URL when switching workflow trigger type #2050
  • Add quotes when special YAML characters are present in the exported project #2446
  • In the AI Assistant, don't open the help page when clicking the Responsible AI Link #2511

[v2.9.4] - 2024-09-16

Changed

  • Responsible AI review of AI Assistant #2478
  • Improve history export page UI #2442
  • When selecting a node in the workflow diagram, connected edges will also be highlighted #2396

Fixed

  • Fix AI Assitant crashes on a job that is not saved yet #2479
  • Fix jumpy combobox for scope switcher #2469
  • Fix console errors when rending edge labels in the workflow diagram
  • Fix tooltip on export workorder button #2430

[v2.9.3] - 2024-09-11

Added

  • Add utility module to seed a DB to support query performance analysis. #2441

Changed

  • Enhance user profile page to add a section for updating basic information #2470
  • Upgraded Heroicons to v2.1.5, from v2.0.18 #2483
  • Standardize link-uuid style for uuid chips
  • Updated PromEx configuration to align with custom Oban naming. #2488

[v2.9.2] - 2024-09-09

Changed

  • Temporarily limit AI to @openfn emails while testing #2482

[v2.9.1] - 2024-09-09

Fixed

  • Provisioner creates invalid snapshots when doing CLI deploy #2461 #2460

    This is a fix for future Workflow updates that are deployed by the CLI and Github integrations. Unfortunately, there is a high likelihood that your existing snapshots could be incorrect (e.g. missing steps, missing edges). In order to fix this, you will need to manually create new snapshots for each of your workflows. This can be done either by modifying the workflow in the UI and saving it. Or running a command on the running instance:

    alias Lightning.Repo
alias Lightning.Workflows.{Workflow, Snapshot}

Repo.transaction(fn ->
  snapshots =
    Repo.all(Workflow)
    |> Enum.map(&Workflow.touch/1)
    |> Enum.map(&Repo.update!/1)
    |> Enum.map(fn workflow ->
      {:ok, snapshot} = Snapshot.create(workflow)
      snapshot
    end)

 {:ok, snapshots}
end)

[v2.9.0] - 2024-09-06

Added

  • Limit AI queries and hook the increment of AI queries to allow usage limiting. #2438
  • Persist AI Assistant conversations and enable it for all users #2296

Changed

  • Rename new_table component to table. #2448

Fixed

  • Fix workflow_id presence in state.json during Github sync #2445

[v2.8.2] - 2024-09-04

Added

  • Change navbar colors depending on scope. #2449
  • Add support for configurable idle connection timeouts via the IDLE_TIMEOUT environment variable. #2443

Changed

  • Allow setup_user command to be execute from outside the container with /app/bin/lightning eval Lightning.Setup.setup_user/3
  • Implement a combo-box to make navigating between projects easier #241
  • Updated vulnerable version of micromatch. #2454

[v2.8.1] - 2024-08-28

Changed

  • Improve run claim query by removing extraneous sorts #2431

[v2.8.0] - 2024-08-27

Added

  • Users are now able to export work orders, runs, steps, logs, and dataclips from the History page. #1698

Changed

  • Add index over run_id and step_id in run_steps to improve worker claim speed. #2428
  • Show Github Error messages as they are to help troubleshooting #2156
  • Allow Setup_utils.setup_user to be used for the initial superuser creation.
  • Update to code assist in the Job Editor to import namespaces from adaptors. #2432

Fixed

  • Unable to remove/reconnect github app in lightning after uninstalling directly from Github #2168
  • Github sync buttons available even when usage limiter returns error PR#2390
  • Fix issue with the persisting of a Kafka message with headers. #2402
  • Protect against race conditions when updating partition timestamps for a Kafka trigger. #2378

[v2.7.19] - 2024-08-19

Added

  • Pass the user_id param on check usage limits. #2387

[v2.7.18] - 2024-08-17

Added

  • Ensure that all users in an instance have a confirmed email address within 48 hours #2389

Changed

  • Ensure that all the demo accounts are confirmed by default #2395

Fixed

  • Removed all Kafka trigger code that ensured that message sequence is honoured for messages with keys. Functionality to ensure that message sequence is honoured will be added in the future, but in an abstraction that is a better fit for the current Lightning design. #2362
  • Dropped the trigger_kafka_messages table that formed part of the Kafka trigger implementation, but which is now obsolete given the removal of the code related to message sequence preservation. #2362

[v2.7.17] - 2024-08-14

Added

  • Added an iex command to setup a user, an apiToken, and credentials so that it's possible to get a fully running lightning instance via external shell script. (This is a tricky requirement for a distributed set of local deployments) #2369 and #2373
  • Added support for very basic project-credential management (add, associate with job) via provisioning API. #2367

Changed

  • Enforced uniqueness on credential names by user. #2371
  • Use Swoosh to format User models into recipients #2374
  • Bump default CLI to @openfn/cli@1.8.1

Fixed

  • When a Workflow is deleted, any associated Kafka trigger pipelines will be stopped and deleted. #2379

[v2.7.16] - 2024-08-07

Fixed

  • @ibrahimwickama fixed issue that prevented users from creating new workflows if they are running in an http environment (rather than localhost or https). #2365

[v2.7.15] - 2024-08-07

Changed

  • Kafka messages without keys are synchronously converted into a Workorder, Dataclip and Run. Messages with keys are stored as TriggerKafkaMessage records, however the code needed to process them has been disabled, pending removal. #2351

[v2.7.14] - 2024-08-05

Changed

  • Use standard styles for link, fix home button in breadcrumbs #2354

[v2.7.13] - 2024-08-05

Changed

  • Don't log 406 Not Acceptable errors to Sentry #2350

Fixed

  • Correctly handle floats in LogMessage #2348

[v2.7.12] - 2024-07-31

Changed

  • Make root layout configurable #2310
  • Use snapshots when initiating Github Sync #1827
  • Move runtime logic into module #2338
  • Use AccountHook Extension to register new users invited in a project #2341
  • Standardized top bars across the UI with a navigable breadcrumbs interface #2299

Fixed

  • Limit frame size of worker socket connections #2339
  • Limit number of days to 31 in cron trigger dropdown #2331

[v2.7.11] - 2024-07-26

Added

  • Expose more Kafka configuration at instance-level. #2329

Fixed

  • Table action css tweaks #2333

[v2.7.10]

Added

  • A rudimentary optimisation for Kafka messages that do not have a key as the sequence of these messages can not be guaranteed. #2323

Fixed

  • Fix an intermittent bug when trying to intern Kafka offset reset policy. #2327

[v2.7.9] - 2024-07-24

Changed

  • CSS - standardized some more tailwind components PR#2324

[v2.7.8] - 2024-07-24

Changed

  • Enable End to End Integration tests #2187
  • Make selected Kafka trigger parameters configurable via ENV vars. #2315
  • Use the Oauth2 revocation_endpoint to revoke token access (1) before attempting to reauthorize and (2) when users schedule a credential for deletion #2314
  • Standardized tailwind alerts #2314
  • Standardized link tailwind style (and provided link-plain, link-info, link-error, and link-warning) #2314

Fixed

  • Fix work order URL in failure alerts #2305
  • Fix error when handling existing encrypted credentials #2316
  • Fix job editor switches to the snapshot version when body is changed #2306
  • Fix misaligned "Retry from here" button on inspector page #2308

[v2.7.7] - 2024-07-18

Added

  • Add experimental support for triggers that consume message from a Kafka cluster #1801
  • Workflows can now specify concurrency, allowing runs to be executed syncronously or to a maximum concurrency level. Note that this applies to the default FifoRunQueue only. #2022
  • Invite Non-Registered Users to a Project #2288

Changed

  • Make modal close events configurable #2298

Fixed

  • Prevent Oauth credentials from being created if they don't have a refresh_token #2289 and send more helpful error data back to the worker during token refresh failure #2135
  • Fix CLI deploy not creating snapshots for workflows #2271

[v2.7.6] - 2024-07-11

Fixed

  • UsageTracking crons are enabled again (if config is enabled) #2276
  • UsageTracking metrics absorb the fact that a step's job_id may not currently exist when counting unique jobs #2279
  • Adjusted layout and text displayed when preventing simultaneous edits to accommodate more screen sizes #2277

[v2.7.5] - 2024-07-10

Changed

  • Prevent two editors from making changes to the same workflow at the same time #1949
  • Moved the Edge Condition Label field to the top of the form, so it's always visible #2236
  • Update edge condition labels in the Workflow Diagram to always show the condition type icon and the label #2236

Fixed

  • Do Not Require Lock Version In URL Parameters #2267
  • Trim erroneous spaces on user first and last names #2269

[v2.7.4] - 2024-07-06

Changed

  • When the entire log string is a valid JSON object, pretty print it with a standard JSON.stringify(str, null, 2) but if it's something else then let the user do whatever they want (e.g., if you write console.log('some', 'cool', state.data) we won't mess with it.) #2260

Fixed

  • Fixed sticky toggle button for switching between latest version and a snapshot of a workflow #2264

[v2.7.3] - 2024-07-05

Changed

  • Bumped the ws-worker to v1.3

Fixed

  • Fix issue when selecting different steps in RunViewer and the parent liveview not being informed #2253
  • Stopped inspector from crashing when looking for a step by a run/job combination #2201
  • Workflow activation only considers new and changed workflows #2237

[v2.7.2] - 2024-07-03

Changed

  • Allow endpoint plugs to be injected at compile time. #2248
  • All models to use the public schema. #2249
  • In the workflow diagram, smartly update the view when adding new nodes #2174
  • In the workflow diagram, remove the "autofit" toggle in the control bar

Fixed

  • Remove prompt parameter from the authorization URL parameters for the Generic Oauth Clients #2250
  • Fixed react key error #2233
  • Show common functions in the Docs panel #1733

[v2.7.1] - 2024-07-01

Changed

  • Update email copies #2213

Fixed

  • Fix jumpy cursor in the Job editor. #2229
  • Rework syncing behaviour to prevent changes getting thrown out on a socket reconnect. #2007

[v2.7.0] - 2024-06-26

Added

  • Use of snapshots for displaying runs and their associated steps in the History page. #1825
  • Added view-only mode for rendering workflows and runs in the Workflow Canvas and the Inspector page using snapshots, with the option to switch between a specific snapshot version and the latest version. Edit mode is available when displaying the latest version. #1843
  • Allow users to delete steps sssociated with runs in the Workflow Canvas #2027
  • Link to adaptor /src from inspector.
  • Prototype AI Assistant for working with job code. #2193

Changed

  • Reverted behaviour on "Rerun from here" to select the Log tab. #2202
  • Don't allow connections between an orphaned node and a Trigger#2188
  • Reduce the minimum zoom in the workflow diagram #2214

Fixed

[v2.6.3] - 2024-06-19

Changed

  • Added a notice on application start about anonymous public impact reporting and its importance for the sustainability of Digital Public Goods and Digital Public Infrastructure.
  • Increase default WORKER_MAX_RUN_DURATION_SECONDS to 300 to match the ws-worker default so if people don't set their timeout via ENV, at least the two match up.

[v2.6.2] - 2024-06-13

Fixed

  • Fix vanishing Docs panel when Editor panel is collapsed and opened again #2195
  • Maintain tab when RunViewer remounts/push state drops tab hash #2199

[v2.6.1] - 2024-06-12

Changed

  • Erlang to 26.2.5
  • Update debian bookworm from 20240130 to 20240513.
  • Return 403s when Provisioning API fails because of usage limits #2182
  • Update email notification for changing retention period #2066
  • Return 415s when Webhooks are sent Content-Types what are not supported. #2180
  • Updated the default step text

Fixed

  • Rewrite TabSelector (now Tabbed) components fixing a number of navigation issues #2051

[v2.6.0] - 2024-06-05

Added

  • Support multiple edges leading to the same step (a.k.a., "drag & drop") #2008

Changed

Fixed

[v2.5.5] - 2024-06-05

Added

  • Replace LiveView Log Viewer component with React Monaco #1863

Changed

  • Bump default CLI to @openfn/cli@1.3.2
  • Don't show deprecated adaptor versions in the adaptor version picklist (to be followed by some graceful deprecation handling/warning in later work) #2169
  • Refactor count workorders to reuse search code #2121
  • Updated provisioning error message to include workflow and job names #2140

Fixed

  • Don't let two deploy workflows run at the same time to prevent git collisions #2044
  • Stopped sending emails when creating a starter project #2161

[v2.5.4] - 2024-05-31

Added

  • CORS support #2157
  • Track users emails preferences #2163

Changed

  • Change Default Text For New Job Nodes #2014
  • Persisted run options when runs are created, not when they are claimed. This has the benefit of "locking in" the behavior desired by the user at the time they demand a run, not whenever the worker picks it up. #2085
  • Made RUN_GRACE_PERIOD_SECONDS a configurable ENV instead of 20% of the WORKER_MAX_RUN_DURATION #2085

Fixed

  • Stopped Janitor from calling runs lost if they have special runtime options #2079
  • Dataclip Viewer now responds to page resize and internal page layout #2120

[v2.5.3] - 2024-05-27

Changed

  • Stop users from creating deprecated Salesforce and GoogleSheets credentials. #2142
  • Delegate menu customization and create menu components for reuse. #1988

Fixed

  • Disable Credential Save Button Until All Form Fields Are Validated #2099
  • Fix Credential Modal Closure Error When Workflow Is Unsaved #2101
  • Fix error when socket reconnects and user is viewing a run via the inspector #2148

[v2.5.2] - 2024-05-23

Fixed

  • Preserve custom values (like apiVersion) during token refresh for OAuth2 credentials #2131

[v2.5.1] - 2024-05-21

Fixed

  • Don't compile Phoenix Storybook in production and test environments #2119
  • Improve performance and memory consumption on queries and logic for digest mailer #2121

[v2.5.0] - 2024-05-20

Fixed

  • When a refresh token is updated, save it! #2124

[v2.5.0-pre4] - 2024-05-20

Fixed

  • Fix duplicate credential type bug #2100
  • Ensure Global OAuth Clients Accessibility for All Users #2114

[v2.5.0-pre3] - 2024-05-20

Fixed

  • Fix credential not added automatically after being created from the canvas. #2105
  • Replace the "not working?" prompt by "All good, but if your credential stops working, you may need to re-authorize here.". #2102
  • Fix Generic Oauth credentials don't get included in the refresh flow #2106

[v2.5.0-pre2] - 2024-05-17

Changed

  • Replace LiveView Dataclip component with React Monaco bringing large performance improvements when viewing large dataclips. #1872

[v2.5.0-pre] - 2024-05-17

Added

  • Allow users to build Oauth clients and associated credentials via the user interface. #1919

[v2.4.14] - 2024-05-16

Changed

  • Refactored image and version info #2097

Fixed

  • Fixed issue where updating adaptor name and version of job node in the workflow canvas crashes the app when no credential is selected #99
  • Removes stacked viewer after switching tabs and steps. #2064

[v2.4.13] - 2024-05-16

Fixed

  • Fixed issue where updating an existing Salesforce credential to use a sandbox endpoint would not properly re-authenticate. #1842
  • Navigate directly to settings from url hash and renders default panel when there is no hash. #1971

[v2.4.12] - 2024-05-15

Fixed

  • Fix render settings default panel on first load #1971

[v2.4.11] - 2024-05-15

Changed

  • Upgraded Sentry to v10 for better error reporting.

[v2.4.10] - 2024-05-14

Fixed

  • Fix the "reset demo" script by disabling the emailing that was introduced to the create_project function. #2063

[v2.4.9] - 2024-05-14

Changed

  • Bumped @openfn/ws-worker to 1.1.8

Fixed

  • Correctly pass max allowed run time into the Run token, ensuring it's valid for the entirety of the Runs execution time #2072

[v2.4.8] - 2024-05-13

Added

  • Add Github sync to usage limiter #2031

Changed

  • Remove illogical cancel buttons on user/pass change screen #2067

Fixed

  • Stop users from configuring failure alerts when the limiter returns error #2076

[v2.4.7] - 2024-05-11

Fixed

  • Fixed early worker token expiry bug #2070

[v2.4.6] - 2024-05-08

Added

  • Allow for automatic resubmission of failed usage tracking report submissions. 1789
  • Make signup feature configurable #2049
  • Apply runtime limits to worker execution #2015
  • Limit usage for failure alerts #2011

[v2.4.5] - 2024-05-07

Fixed

  • Fix provioning API calls workflow limiter without the project ID #2057

[v2.4.4] - 2024-05-03

Added

  • Benchmarking script that simulates data from a cold chain. #1993

Changed

  • Changed Snapshot get_or_create_latest_for to accept multis allow controlling of which repo it uses.
  • Require exactly one owner for each project #1991

Fixed

  • Fixed issue preventing credential updates #1861

[v2.4.3] - 2024-05-01

Added

  • Allow menu items customization #1988
  • Workflow Snapshot support #1822
  • Fix sample workflow from init_project_for_new_user #2016

Changed

  • Bumped @openfn/ws-worker to 1.1.6

Fixed

  • Assure workflow is always passed to Run.enqueue #2032
  • Fix regression on History page where snapshots were not preloaded correctly #2026

[v2.4.2] - 2024-04-24

Fixed

  • Fix missing credential types when running Lightning using Docker #2010
  • Fix provisioning API includes deleted workflows in project state #2001

[v2.4.1] - 2024-04-19

Fixed

  • Fix github cli deploy action failing to auto-commit #1995

[v2.4.1-pre] - 2024-04-18

Added

  • Add custom metric to track the number of finalised runs. #1790

Changed

  • Set better defaults for the GitHub connection creation screen #1994
  • Update submission_status for any Usagetracking.Report that does not have it set. #1789

[v2.4.0] - 2024-04-12

Added

  • Allow description below the page title #1975
  • Enable users to connect projects to their Github repos and branches that they have access to #1895
  • Enable users to connect multiple projects to a single Github repo #1811

Changed

  • Change all System.get_env calls in runtime.exs to use dotenvy #1968
  • Track usage tracking submission status in new field #1789
  • Send richer version info as part of usage tracking submission. #1819

Fixed

  • Fix sync to branch only targetting main branch #1892
  • Fix enqueue run without the workflow info #1981

[v2.3.1] - 2024-04-03

Changed

  • Run the usage tracking submission job more frequently to reduce the risk of Oban unavailability at a particular time. #1778
  • Remove code supporting V1 usage tracking submissions. #1853

Fixed

  • Fix scrolling behaviour on inspector for small screens #1962
  • Fix project picker for users with many projects #1952

[v2.3.0] - 2024-04-02

Added

  • Support for additional paths on a webhook URL such as /i/<uuid>/Patient #1954
  • Support for a GET endpoint to "check" webhook URL availability #1063
  • Allow external apps to control the run enqueue db transaction #1958

[v2.2.2] - 2024-04-01

Changed

  • Changed dataclip search from string LIKE to tsvector on keys and values. While this will limit partial string matching to the beginning of words (not the middle or end) it will make searching way more performant #1939
  • Translate job error messages using errors.po file #1935
  • Improve the UI/UX of the run panel on the inspector for small screens #1909

Fixed

  • Regular database timeouts when searching across dataclip bodies #1794

[v2.2.1] - 2024-03-27

Added

  • Enable users to connect to their Github accounts in preparation for streamlined GitHub project sync setup #1894

Fixed

  • Apply usage limit to bulk-reruns #1931
  • Fix edge case that could result in duplicate usage tracking submissions. #1853
  • Fix query timeout issue on history retention deletion #1937

[v2.2.0] - 2024-03-21

Added

  • Allow admins to set project retention periods #1760
  • Automatically wipe input/output data after their retention period #1762
  • Automatically delete work order history after their retention period #1761

Changed

  • When automatically creating a project for a newly registered user (via the INIT_PROJECT_FOR_NEW_USER=true environment variable) that user should be the owner of the project. #1927
  • Give priority to manual runs (over webhook requests and cron) so that active users on the inspector don't have to wait ages for thier work during high load periods #1918

[v2.1.0] - 2024-03-20

Added

  • TSVector index to log_lines, and gin index to dataclips #1898
  • Add API Version field to Salesforce OAuth credentials #1838

Changed

  • Replace v1 usage tracking with v2 usage tracking. #1853

[v2.0.10]

Changed

  • Updated anonymous usage tracker submissions #1853

[v2.0.9] - 2024-03-19

Added

  • Support for smaller screens on history and inspector. #1908
  • Polling metric to track number of available runs. #1790
  • Allows limiting creation of new runs and retries. #1754
  • Add specific messages for log, input, and output tabs when a run is lost #1757
  • Soft and hard limits for runs created by webhook trigger. #1859
  • Publish an event when a new user is registered #1873
  • Adds ability to add project collaborators from existing users #1836
  • Added ability to remove project collaborators #1837
  • Added new usage tracking submission code. #1853

Changed

  • Upgrade Elixir to 1.16.2
  • Remove all values from .env.example. #1904

Fixed

  • Verify only stale project credentials #1861

[v2.0.8] - 2024-02-29

Fixed

  • Show flash error when editing stale project credentials #1795
  • Fixed bug with Github sync installation on docker-based deployments #1845

[v2.0.6] - 2024-02-29

Added

  • Automatically create Github workflows in a target repository/branch when users set up a Github repo::OpenFn project sync #1046
  • Allows limiting creation of new runs and retries. #1754

Changed

  • Change bucket size used by the run queue delay custom metric. #1790
  • Require setting IS_RESETTABLE_DEMO to "yes" via ENV before allowing the destructive Demo.reset_demo/0 function from being called. #1720
  • Remove version display condition that was redundant due to shadowing #1819

Fixed

  • Fix series of sentry issues related to OAuth credentials #1799

[v2.0.5] - 2024-02-25

Fixed

  • Fixed error in Credentials without sanbox field set; only display sandbox field for Salesforce oauth credentials. #1798

[v2.0.4] - 2024-02-24

Added

  • Display and edit OAuth credentials scopes#1706

Changed

  • Stop sending operating_system_detail to the usage tracker #1785

Fixed

  • Make handling of usage tracking errors more robust. #1787
  • Fix inspector shows selected dataclip as wiped after retying workorder from a non-first step #1780

[v2.0.3] - 2024-02-21

Added

  • Actual metrics will now be submitted by Lightning to the Usage Tracker. #1742
  • Added a support link to the menu that goes to the instance admin contact #1783

Changed

  • Usage Tracking submissions are now opt-out, rather than opt-in. Hashed UUIDs to ensure anonymity are default. #1742

  • Usage Tracking submissions will now run daily rather than hourly. #1742

  • Bumped @openfn/ws-worker to v1.0 (this is used in dev mode when starting the worker from your mix app: RTM=true iex -S mix phx.server)

  • Bumped @openfn/cli to v1.0 (this is used for adaptor docs and magic)

Fixed

  • Non-responsive workflow canvas after web socket disconnection #1750

[v2.0.2] - 2024-02-14

Fixed

  • Fixed a bug with the OAuth2 credential refresh flow that prevented GoogleSheets jobs from running after token expiration #1735

[v2.0.1] - 2024-02-13

Changed

  • Renamed ImpactTracking to UsageTracking #1729
  • Block github installation if there's a pending installation in another project #1731

Fixed

  • Expand work order button balloons randomly #1737
  • Editing credentials doesn't work from project scope #1743

[v2.0.0] - 2024-02-10

At the time of writing there are no more big changes planned and testing has gone well. Thanks to everyone who's helped to kick the tyres during the "rc" phase. There are still a lot of new features coming, so please:

Head to docs.openfn.org for product documentation and help with v1 to v2 migration.

Changed

  • Bump @openfn/worker to v0.8.1
  • Only show GoogleSheets and Salesforce credential options if Oauth clients are registered with the instance via ENV #1734

Fixed

  • Use standard table type for webhook auth methods #1514
  • Make disabled button for "Connect to GitHub" clear, add tooltip #1732

[v2.0.0-rc12] - 2024-02-09

Added

  • Add RunQueue extension to allow claim customization. #1715
  • Add support for Salesforce OAuth2 credentials #1633

Changed

  • Use PAYLOAD_SIZE_KB in k6 load testing script, set thresholds on wait time, set default payload size to 2kb

Fixed

  • Adds more detail to work order states on dashboard #1677
  • Fix Output & Logs in inspector fails to show sometimes #1702

[v2.0.0-rc11] - 2024-02-08

Fixed

  • Bumped Phoenix LiveView from 0.20.4 to 0.20.5 to fix canvas selection issue #1724

[v2.0.0-rc10] - 2024-02-08

Changed

  • Implemented safeguards to prevent deletion of jobs with associated run history #1570

Fixed

  • Fixed inspector dataclip body not getting updated after dataclip is wiped #1718
  • Fixed work orders getting retried despite having wiped dataclips #1721

[v2.0.0-rc9] 2024-02-05

Added

  • Persist impact tracking configuration and reports #1684
  • Add zero-persistence project setting #1209
  • Wipe dataclip after use when zero-persistence is enabled #1212
  • Show appropriate message when a wiped dataclip is viewed #1211
  • Disable selecting work orders having wiped dataclips in the history page #1210
  • Hide rerun button in inspector when the selected step has a wiped dataclip #1639
  • Add rate limiter to webhook endpoints and runtime limiter for runs. #639

Fixed

  • Prevented secret scrubber from over-eagerly adding *** between all characters if an empty string secret was provided as a credential field value (e.g., {"username": "come-on-in", "password": ""}) #1585
  • Fixed permissions issue that allowed viewer/editor to modify webhook auth methods. These permissions only belong to project owners and admins #1692
  • Fixed bug that was duplicating inbound http_requests, resulting in unnecessary data storage #1695
  • Fixed permissions issue that allowed editors to set up new Github connections #1703
  • Fixed permissions issue that allowed viewers to initiate syncs to github #1704
  • Fixed inspector view stuck at processing when following a crashed run #1711
  • Fixed inspector dataclip selector not getting updated after running manual run #1714

[v2.0.0-rc8] - 2024-01-30

Added

  • Shim code to interact with the Impact Tracking service #1671

Changed

  • Standardized naming of "attempts" to "runs". This had already been done in the front-end, but this change cleans up the backend, the database, and the interface with the worker. Make sure to run migrations and update your ENV/secrets to use WORKER_RUNS_PRIVATE_KEY rather than WORKER_ATTEMPTS_PRIVATE_KEY #1657
  • Required @openfn/ws-worker@0.8.0 or above.

[v2.0.0-rc7] - 2024-01-26

Added

  • Store webhook request headers in Dataclips for use in jobs. #1638

Changed

  • Display http_request dataclips to the user as they will be provided to the worker as "input" state to avoid confusion while writing jobs. 1664
  • Named-spaced all worker environment variables with WORKER_ and added documentation for how to configure them. #1672
  • Bumped to @openfn/ws-worker@0.6.0
  • Bumped to @openfn/cli@0.4.15

Fixed

  • Fix Run via Docker #1653
  • Fix remaining warnings, enable "warnings as errors" #1642
  • Fix workflow dashboard bug when viewed for newly created workflows with only unfinished run steps. #1674

[v2.0.0-rc5] - 2024-01-22

Changed

  • Made two significant backend changes that don't impact UI/UX but require migrations and should make Lightning developer lives easier by updating parts of the backend to match terms now used in the frontend:
    • Renamed the Runs model and table to Steps #1571
    • Renamed the AttemptRuns model and table to AttemptSteps #1571

[v2.0.0-rc4] - 2024-01-19

Added

  • Scrub output dataclips in the UI to avoid unintentional secret exposure #1606

Changed

  • Bump to @openfn/cli@0.4.14
  • Do not persist the active tab setting on the job editor #1504
  • Make condition label optional #1648

Fixed

  • Fix credential body getting leaked to sentry incase of errors #1600
  • Fixed validation on Javascript edge conditions #1602
  • Removed unused code from run_live directory #1625
  • Edge condition expressions not correctly being handled during provisioning #openfn/kit#560

[v2.0.0-rc3] 2024-01-12

Added

  • Custom metric to track stalled attempts #1559
  • Dashboard with project and workflow stats #755
  • Add search by ID on the history page #1468
  • Custom metric to support autoscaling #1607

Changed

  • Bumped CLI version to 0.4.13
  • Bumped worker version to 0.5.0
  • Give project editors and viewers read only access to project settings instead #1477

Fixed

  • Throw an error when Lightning.MetadataService.get_adaptor_path/1 returns an adaptor path that is nil #1601
  • Fix failure due to creating work order from a newly created job #1572
  • Fixes on the dashboard and links #1610 and #1608

[2.0.0-rc2] - 2024-01-08

Fixed

  • Restored left-alignment for step list items on run detail and inspector a6e4ada
  • Inspector: fixed attempt/run language for "skipped" tooltip fd7dd0c
  • Inspector: fixed failure to save during "save & run" from inspector #1596
  • Inspector: fixed key bindings for save & run (retry vs. new work order) getting overridden when user focuses on the Monaco editor #1596

[2.0.0-rc1] - 2024-01-05

Why does this repo go from v0 to v2.0?

Lightning is the 2nd version of the OpenFn platform. While much of the core technology is the same, there are breaking changes between v1.105 (pre-2024) and v2 ("OpenFn Lightning").

For customers using OpenFn v1, a migration guide will be provided at docs.openfn.org

Added

  • Link to the job inspctor for a selected run from the history interface #1524
  • Reprocess an existing work order from the job inspector by default (instead of always creating a new work order) #1524
  • Bumped worker to support edge conditions between trigger and first job "@openfn/ws-worker": "^0.4.0"

Changed

  • Updated naming to prepare for v2 release #1248; the major change is that each time a work order (the typical unit of business value for an organization, e.g. "execute workflow ABC for patient 123") is executed, it is called a "run". Previously, it was called an "attempt". The hierarchy is now:

    Build-Time: Projects > Workflows > Steps
Run-Time: Work Orders > Runs > Steps

    Note the name changes here are reflected in the UI, but not all tables/models will be changed until 1571 is delivered.

[v0.12.2] - 2023-12-24

Changed

  • Bumped worker to address occasional git install issue "@openfn/ws-worker": "^0.3.2"

Fixed

  • Fix RuntimeError: found duplicate ID "google-sheets-inner-form" for GoogleSheetsComponent #1578
  • Extend export script to include new JS expression edge type #1540
  • Fix regression for attempt viewer log line highlighting #1589

[v0.12.1] - 2023-12-21

Changed

  • Hide project security setting tab from non-authorized users #1477

Fixed

  • History page crashes if job is removed from workflow after it's been run #1568

[v0.12.0] - 2023-12-15

Added

  • Add ellipsis for long job names on the canvas #1217
  • Fix Credential Creation Page UI #1064
  • Custom metric to track Attempt queue delay #1556
  • Expand work order row when a workorder_id is specified in the filter #1515
  • Allow Javascript expressions as conditions for edges #1498

Changed

  • Derive dataclip in inspector from the attempt & step #1551
  • Updated CLI to 0.4.10 (fixes logging)
  • Changed UserBackupToken model to use UTC timestamps (6563cb77)
  • Restore FK relationship between work_orders and attempts pending a decision re: further partitioning. #1254

Fixed

  • New credential doesn't appear in inspector until refresh #1531
  • Metadata not refreshing when credential is updated #791
  • Adjusted z-index for Monaco Editor's sibling element to resolve layout conflict #1329
  • Demo script sets up example Runs with their log lines in a consistant order. #1487
  • Initial credential creation changes show after as null rather a value #1118
  • AttemptViewer flashing/rerendering when Jobs are running #1550
  • Not able to create a new Job when clicking the Check icon on the placeholder #1537
  • Improve selection logic on WorkflowDiagram #1220

[v0.11.0] - 2023-12-06

Added

  • Improved UI when manually creating Attempts via the Job Editor #1474
  • Increased the maximum inbound webhook request size to 10MB and added protection against very large payloads with a 100MB "max_skip_body_length" #1247

Changed

  • Use the internal port of the web container for the worker configuration in docker-compose setup. #1485

[v0.10.6] - 2023-12-05

Changed

  • Limit entries count on term work orders search #1461
  • Scrub log lines using multiple credentials samples #1519
  • Remove custom telemetry plumbing. 1259
  • Enhance UX to prevent modal closure when Monaco/Dataclip editor is focused #1510

Fixed

  • Use checkbox on boolean credential fields rather than a text input field #1430
  • Allow users to retry work orders that failed before their first run was created #1417
  • Fix to ensure webhook auth modal is closed when cancel or close are selected. #1508
  • Enable user to reauthorize and obtain a new refresh token. #1495
  • Save credential body with types declared on schema #1518

[v0.10.5] - 2023-12-03

Changed

  • Only add history page filters when needed for simpler multi-select status interface and shorter page URLs #1331
  • Use dynamic Endpoint config only on prod #1435
  • Validate schema field with any of expected values #1502

Fixed

  • Fix for liveview crash when token expires or gets deleted after mount #1318
  • Remove two obsolete methods related to Run: Lightning.Invocation.delete_run and Lightning.Invocation.Run.new_from. #1254
  • Remove obsolete field previous_id from runs table. #1254
  • Fix for missing data in 'created' audit trail events for webhook auth methods #1500

[v0.10.4] - 2023-11-30

Changed

  • Increased History search timeout to 30s #1461

Fixed

  • Tooltip text clears later than the background #1094
  • Temporary fix to superuser UI for managing project users #1145
  • Fix for adding ellipses on credential info on job editor heading #1428

[v0.10.3] - 2023-11-28

Added

  • Dimmed/greyed out triggers and edges on the canvas when they are disabled #1464
  • Async loading on the history page to improve UX on long DB queries #1279
  • Audit trail events for webhook auth (deletion method) change #1165

Changed

  • Sort project collaborators by first name #1326
  • Work orders will now be set in a "pending" state when retries are enqueued. #1340
  • Avoid printing 2FA codes by default #1322

Fixed

  • Create new workflow button sizing regression #1405
  • Google credential creation and automatic closing of oAuth tab #1109
  • Exporting project breaks the navigation of the page #1440

[v0.10.2] - 2023-11-21

Changed

  • Added max_frame_size to the Cowboy websockets protocol options in an attempt to address #1421

[v0.10.1] - 2023-11-21

Fixed

  • Work Order ID was not displayed properly in history page #1423

[v0.10.0] - 2023-11-21

🚨 Breaking change warning! 🚨

This release will contain breaking changes as we've significantly improved both the workflow building and execution systems.

Nodes and edges

Before, workflows were represented as a list of jobs and triggers. For greater flexibility and control of complex workflows, we've moved towards a more robust "nodes and edges" approach. Where jobs in a workflow (a node) can be connected by edges.

Triggers still exist, but live "outside" the directed acyclic graph (DAG) and are used to automatically create work orders and attempts.

We've provided migrations that bring v0.9.3 workflows in line with the v0.10.0 requirements.

Scalable workers

Before, Lightning spawned child processes to execute attempts in sand-boxed NodeVMs on the same server. This created inefficiencies and security vulnerabilities. Now, the Lightning web server adds attempts to a queue and multiple worker applications can pull from that queue to process work.

In dev mode, this all happens automatically and on one machine, but in most high-availability production environments the workers will be on another server.

Attempts are now handled entirely by the workers, and they report back to Lightning. Exit reasons, final attempt states, error types and error messages are either entirely new or handled differently now, but we have provided migration scripts that will work to bring most v0.9.3 runs, attempts, and work orders up to v0.10.0, though the granularity of v0.9.3 states and exits will be less than v0.10.0 and the final states are not guaranteed to be accurate for workflows with multiple branches and leaf nodes with varying exit reasons.

The migration scripts can be run with a single function call in SetupUtils from a connect iex session:

Lightning.SetupUtils.approximate_state_for_attempts_and_workorders()

Note that (like lots of other functionality in SetupUtils, calling this function is a destructive action and you should only do it if you've backed up your data and you know what you're doing.)

As always, we recommend backing up your data before migrating. (And thanks for bearing with us as we move towards our first stable Lightning release.)

Added

  • Fix flaky job name input behavior on error #1218
  • Added a hover effect on copy and add button for adaptors examples #1297
  • Migration helper code to move from v0.9.3 to v0.10.0 added to SetupUtils #1363
  • Option to start with RTM=false iex -S mix phx.server for opting out of the dev-mode automatic runtime manager.
  • Webhook Authentication Methods database and CRUD operations #1152
  • Creation and Edit of webhook webhook authentication methods UI #1149
  • Add webhook authentication methods overview methods in the canvas #1153
  • Add icon on the canvas for triggers that have authentication enabled #1157
  • Require password/2FA code before showing password and API Key for webhook auth methods #1200
  • Restrict live dashboard access to only superusers, enable DB information and OS information #1170 OS information #1170
  • Expose additional metrics to LiveDashboard #1171
  • Add plumbing to dump Lightning metrics during load testing #1178
  • Allow for heavier payloads during load testing #1179
  • Add dynamic delay to help mitigate flickering test #1195
  • Add a OpenTelemetry trace example #1189
  • Add plumbing to support the use of PromEx #1199
  • Add warning text to PromEx config #1222
  • Track and filter on webhook controller state in :telemetry metrics #1192
  • Secure PromEx metrics endpoint by default #1223
  • Partition log_lines table based on attempt_id #1254
  • Remove foreign key from attempts in preparation for partitioning work_orders #1254
  • Remove Workflows.delete_workflow. It is no longer in use and would require modification to not leave orphaned attempts given the removal of the foreign key from attempts. #1254
  • Show tooltip for cloned runs in history page #1327
  • Have user create workflow name before moving to the canvas #1103
  • Allow PromEx authorization to be disabled #1483

Changed

  • Updated vulnerable JS libraries, postcss and semver #1176
  • Update "Delete" to "Delete Job" on Job panel and include javascript deletion confirmation #1105
  • Move "Enabled" property from "Jobs" to "Edges" #895
  • Incorrect wording on the "Delete" tooltip #1313

Fixed

  • Fixed janitor lost query calculation #1400
  • Adaptor icons load gracefully #1140
  • Selected dataclip gets lost when starting a manual work order from the inspector interface #1283
  • Ensure that the whole edge when selected is highlighted #1160
  • Fix "Reconfigure Github" button in Project Settings #1386
  • Make janitor also clean up runs inside an attempt #1348
  • Modify CompleteRun to return error changeset when run not found #1393
  • Drop invocation reasons from DB #1412
  • Fix inconsistency in ordering of child nodes in the workflow diagram #1406

[v0.9.3] - 2023-09-27

Added

  • Add ellipsis when adaptor name is longer than the container allows #1095
  • Webhook Authentication Methods database and CRUD operations #1152

Changed

  • Prevent deletion of first job of a workflow #1097

Fixed

  • Fix long name on workflow cards #1102
  • Fix highlighted Edge can get out of sync with selected Edge #1099
  • Creating a new user without a password fails and there is no user feedback #731
  • Crash when setting up version control #1112

[v0.9.2] - 2023-09-20

Added

  • Add "esc" key binding to close job inspector modal #1069

Changed

  • Save icons from the adaptors repo locally and load them in the job editor #943

[v0.9.1] - 2023-09-19

Changed

  • Modified audit trail to handle lots of different kind of audit events #271/#44
  • Fix randomly unresponsive job panel after job deletion #1113

[v0.9.0] - 2023-09-15

Added

  • Add favicons #1079
  • Validate job name in placeholder job node #1021
  • Bring credential delete in line with new GDPR interpretation #802
  • Make job names unique per workflow #1053

Changed

  • Enhanced the job editor/inspector interface #1025

Fixed

  • Finished run never appears in inspector when it fails #1084
  • Cannot delete some credentials via web UI #1072
  • Stopped the History table from jumping when re-running a job #1100
  • Fixed the "+" button when adding a job to a workflow #1093

[v0.8.3] - 2023-09-05

Added

  • Render error when workflow diagram node is invalid #956

Changed

  • Restyle history table #1029
  • Moved Filter and Search controls to the top of the history page #1027

Fixed

  • Output incorrectly shows "this run failed" when the run hasn't yet finished #1048
  • Wrong label for workflow card timestamp #1022

[v0.8.2] - 2023-08-31

Fixed

  • Lack of differentiation between top of job editor modal and top menu was disorienting. Added shadow.

[v0.8.1] - 2023-08-31

Changed

  • Moved Save and Run button to bottom of the Job edit modal #1026
  • Allow a manual work order to save the workflow before creating the work order #959

[v0.8.0] - 2023-08-31

Added

  • Introduces Github sync feature, users can now setup our github app on their instance and sync projects using our latest portability spec #970
  • Support Backup Codes for Multi-Factor Authentication 937
  • Log a warning in the console when the Editor/docs component is given latest #958
  • Improve feedback when a Workflow name is invalid #961
  • Show that the jobs' body is invalid #957
  • Reimplement skipped CredentialLive tests #962
  • Reimplement skipped WorkflowLive.IndexTest test #964
  • Show GitHub installation ID and repo link to help setup/debugging for version control 1059

Fixed

  • Fixed issue where job names were being incorrectly hyphenated during project.yaml export #1050
  • Allows the demo script to set a project id during creation to help with cli deploy/pull/Github integration testing.
  • Fixed demo project_repo_connection failing after nightly demo resets 1058
  • Fixed an issue where the monaco suggestion tooltip was offset from the main editor 1030

[v0.7.3] - 2023-08-15

Changed

  • Version control in project settings is now named Export your project #1015

Fixed

  • Tooltip for credential select in Job Edit form is cut off #972
  • Dataclip type and state assembly notice for creating new dataclip dropped during refactor #975

[v0.7.2] - 2023-08-10

Changed

  • NodeJs security patch 1009

Fixed

[v0.7.1] - 2023-08-04

Fixed

  • Fixed flickery icons on new workflow job creation.

[v0.7.0] - 2023-08-04

Added

  • Project owners can require MFA for their users 892

Changed

  • Moved to Elixir 1.15 and Erlang 26.0.2 to sort our an annoying ElixirLS issue that was slowing down our engineers.
  • Update Debian base to use bookworm (Debian 12) for our Docker images
  • Change new credential modal to take up less space on the screen #931
  • Placeholder nodes are now purely handled client-side

Fixed

  • Fix issue creating a new credential from the Job editor where the new credential was not being set on the job. #951
  • Fix issue where checking a credential type radio button shows as unchecked on first click. #976
  • Return the pre-filled workflow names #971
  • Fix version reporting and external reset_demo() call via Application.spec()#1010
  • Fixed issue where entering a placeholder name through the form would result an in unsaveable workflow #1001
  • Ensure the DownloadController checks for authentication and authorisation.

[v0.7.0-pre5] - 2023-07-28

Changed

  • Unless otherwise specified, only show work orders with activity in last 14 days #968

[v0.7.0-pre4] - 2023-07-27

Changed

  • Don't add cast fragments if the search_term is nil #968

[v0.7.0-pre3] - 2023-07-26

Fixed

  • Fixed an issue with newly created edges that prevented downstream jobs 977

[v0.7.0-pre2] - 2023-07-26

Note that this is a pre-release with a couple of known bugs that are tracked in the Nodes and Edges epic.

Added

  • Added ability for a user to enable MFA on their account; using 2FA apps like Authy, Google Authenticator etc #890
  • Write/run sql script to convert triggers #875
  • Export projects as .yaml via UI #249

Changed

  • In v0.7.0 we change the underlying workflow building and execution infrastructure to align with a standard "nodes and edges" design for directed acyclic graphs (DAGs). Make sure to run the migrations! 793

Fixed

  • Propagate url pushState/changes to Workflow Diagram selection #944
  • Fix issue when deleting nodes from the workflow editor #830
  • Fix issue when clicking a trigger on a new/unsaved workflow #954

[0.6.7] - 2023-07-13

Added

  • Add feature to bulk rerun work orders from a specific step in their workflow; e.g., "rerun these 50 work orders, starting each at step 4." #906

Fixed

  • Oban exception: "value too long" when log lines are longer than 255 chars #929

[0.6.6] - 2023-06-30

Added

  • Add public API token to the demo site setup script
  • Check and renew OAuth credentials when running a job #646

Fixed

  • Remove google sheets from adaptors list until supporting oauth flow #792
  • Remove duplicate google sheets adaptor display on credential type picklist #663
  • Fix demo setup script for calling from outside the app on Kubernetes deployments #917

[0.6.5] - 2023-06-22

Added

  • Ability to rerun work orders from start by selecting one of more of them from the History page and clicking the "Rerun" button. #659

Fixed

  • Example runs for demo incorrect #856

[0.6.3] - 2023-06-15

Fixed

  • Prevent saving null log lines to the database, fix issue with run display #866

[0.6.2] - 2023-06-09

Fixed

  • Fixed viewer permissions for delete workflow

  • Fixed bug with workflow cards #859

[0.6.1] - 2023-06-08

Fixed

  • Fixed bug with run logs #864

  • Correctly stagger demo runs to maintain order #856

  • Remove Timex use from SetupUtils in favor of DateTime to fix issue when calling it in escript.

[0.6.0]- 2023-04-12

Added

  • Create sample runs when generating sample workflow #821
  • Added a provisioning api for creating and updating projects and their workflows See: PROVISIONING.md #641
  • Add ability for a superuser to schedule deletion, cancel deletion, and delete projects #757
  • Add ability for a project owner to schedule deletion, cancel deletion, and delete projects #746

Changed

  • Ability to store run log lines as rows in a separate table #514

Fixed

  • Incorrect project digest queries #768]
  • Fix issue when purging deleted users #747
  • Generate a random name for Workflows when creating one via the UI. #828
  • Handle error when deleting a job with runs. #814

[0.5.2]

Added

  • Add workflow_edges table in preparation for new workflow editor implementation #794
  • Stamped credential_id on run directly for easier auditing of the history interface. Admins can now see which credential was used to run a run. #800
  • Better errors when using magic functions: "no magic yet" and "check credential" #812

Changed

  • The delete-project function now delete all associated activities #759

Fixed

[0.5.1] - 2023-04-12

Added

  • Added ability to create and revoke personal API tokens #147
  • Add last-used at to API tokens #722
  • Improved "save" for job builder; users can now press Ctrl + S or ⌘ + S to save new or updated jobs job panel will not close. (Click elsewhere in the canvas or click the "Close" button to close.) #568
  • Add filtered search params to the history page URL #660

Changed

  • The secret scrubber now ignores booleans 690

Fixed

  • The secret scrubber now properly handles integer secrets from credentials 690
  • Updated describe-package dependency, fixing sparkles in adaptor-docs 657
  • Clicks on the workflow canvas were not lining up with the nodes users clicked on; they are now 733
  • Job panel behaves better when collapsed 774

[0.5.0] - 2023-04-03

Added

  • Magic functions that fetch real metadata from connected systems via credentials and suggest completions in the job builder (e.g., pressing control-space when setting the orgUnit attribute for a DHIS2 create operation will pull the actual list of orgUnits with human readable labels and fill in their orgUnit codes upon enter.)670
  • A "metadata explorer" to browse actual system metadata for connected instances. 658
  • Resizable job builder panel for the main canvas/workflow view. 681

Changed

  • Display timezone for cron schedule—it is always UTC. #716
  • Instance administrators can now configure the interval between when a project owner or user requests deletion and when these records are purged from the database. It defaults to 7, but by providing a PURGE_DELETED_AFTER_DAYS environment variable the grace period can be altered. Note that setting this variable to 0 will make automatic purging never occur but will still make "deleted" projects and users unavailable. This has been requested by certain organizations that must retain audit logs in a Lightning instance. 758

Fixed

  • Locked CLI version to @openfn/cli@0.0.35. #761

[0.4.8] - 2023-03-29

Added

  • Added a test harness for monitoring critical parts of the app using Telemetry #654

Changed

  • Set log level to info for runs. Most of the debug logging is useful for the CLI, but not for Lightning. In the future the log level will be configurable at instance > project > job level by the superuser and any project admin.
  • Renamed license file so that automagic github icon is less confusing

Fixed

  • Broken links in failure alert email #732
  • Registration Submission on app.openfn.org shows internal server error in browser #686
  • Run the correct runtime install mix task in Dockerfile-dev #541
  • Users not disabled when scheduled for deletion #719

[0.4.6] - 2023-03-23

Added

  • Implement roles and permissions across entire app #645
  • Fix webhook URL (https://<<HOST_URL>>/i/cae544ab-03dc-4ccc-a09c-fb4edb255d7a) for the OpenHIE demo workflow 448
  • Phoenix Storybook for improved component development
  • Load test for webhook endpoint performance #645
  • Notify user via email when they're added to a project #306
  • Added notify user via email when their account is created #307

Changed

  • Improved errors when decoding encryption keys for use with Cloak. #684
  • Allow users to run ANY job with a custom input. #629

Fixed

  • Ensure JSON schema form inputs are in the same order as they are written in the schema #685

[0.4.4] - 2023-03-10

Added

  • Users can receive a digest email reporting on a specified project. #638 #585

[0.4.3] - 2023-03-06

Added

  • Tooltips on Job Builder panel #650

Changed

  • Upgraded to Phoenix 1.7 (3945856)

Fixed

  • Issue with FailureAlerter configuration missing in prod mode.

[0.4.2] - 2023-02-24

Added

  • A user can change their own email #247
  • Added a SCHEMAS_PATH environment variable to override the default folder location for credential schemas #604
  • Added the ability to configure Google Sheets credentials #536
  • Function to import a project #574

Changed

  • Users cannot register if they have not selected the terms and conditions #531

Fixed

  • Jobs panel slow for first open after restart #567

[0.4.0] - 2023-02-08

Added

  • Added a Delete job button in Inspector
  • Filter workflow runs by text/value in run logs or input body
  • Drop "configuration" key from Run output dataclips after completion
  • Ability to 'rerun' a run from the Run list
  • Attempts and Runs update themselves in the Runs list
  • Configure a project and workflow for a new registering user
  • Run a job with a custom input
  • Added plausible analytics
  • Allow user to click on Webhook Trigger Node to copy webhook URL on workflow diagram
  • Allow any user to delete a credential that they own
  • Create any credential through a form except for OAuth
  • Refit all diagram nodes on browser and container resize
  • Enable distributed Erlang, allowing any number of redundant Lightning nodes to communicate with each other.
  • Users can set up realtime alerts for a project

Changed

  • Better code-assist and intelliense in the Job Editor
  • Updated @openfn/workflow-diagram to 0.4.0
  • Make plus button part of job nodes in Workflow Diagram
  • Updated @openfn/adaptor-docs to 0.0.5
  • Updated @openfn/describe-package to 0.0.10
  • Create an follow a manual Run from the Job Inspector
  • View all workflows in a project on the workflows index page
  • Move @openfn/workflow-diagram into the application, the NPM module is now deprecated.
  • Remove workflow name from first node
  • Move the used parts of @openfn/engine into the application.
  • [BREAKING CHANGE] Ported mix openfn.install.runtime into application, use mix lightning.install_runtime.
  • [BREAKING CHANGE] Introduced @openfn/cli as the new runtime for Jobs
  • Rename a workflow through the page heading
  • Hide the dataclips tab for beta
  • Make adaptor default to common@latest
  • Remove jobs list page
  • Better error handling in the docs panel
  • Disable credential ownership transfer in dev and prod environments
  • Add project settings page
  • Change Work Order filters to apply to the aggregate state of the work order and not the run directly
  • Enable jobs by default
  • Set log level to info
  • Add Beta checkbox to register page
  • User roles and permissions

Fixed

  • Don't consider disabled jobs when calculating subsequent runs
  • Fixed overflow on Job Editor Tooltips
  • Fixed auto-scroll when adding a new snippet in the Job Editor
  • Fixed common operation typings in Job Editor

[0.3.1] - 2022-11-22

Fixed

  • Fixed bug that tried to execute HTML scripts in dataclips
  • Fixed bug that prevented work orders from displaying in the order of their last run, descending.
  • Remove alerts after set timeout or close

[0.3.0] - 2022-11-21

Added

  • Add seed data for demo site
  • Create adaptor credentials through a form
  • Configure cron expressions through a form
  • View runs grouped by work orders and attempts
  • Run an existing Job with any dataclip uuid from the Job form

Changed

  • Redirect users to projects list page when they click on Admin Settings menu
  • Move job, project, input and output Dataclips to Run table
  • Reverse the relationship between Jobs and Triggers. Triggers now can exist on their own; setting the stage for branching and merging workflows
  • Updated Elixir and frontend dependencies
  • [BREAKING CHANGE] Pipeline now uses Work Orders, previous data is not compatible.
  • Runs, Dataclips and Attempts now all correctly use usec resolution timestamps.
  • Upgraded LiveView to 0.18.0
  • Upgraded Elixir to 1.14.1 and OTP 25
  • Workflow Job editor now behaves like a panel
  • Split JobLive.InspectorFormComponent into different plug-able subcomponents
  • Ensure new jobs with cron triggers receive a default frequency
  • Webhooks are now referenced by the trigger id instead of job id.
  • Filter runs by status
  • Filter runs by workflow
  • Filter runs by date
  • View a job run from the runs history
  • View latest matching inputs to run a job with

[0.2.0] - 2022-09-12

Changed

  • [BREAKING CHANGE] Add Workflow model, Jobs now belong to a Workflow This is a breaking change to the schema.
  • Use Node.js 18, soon to be in LTS.
  • Visualize success/fail triggers in workflow diagram.
  • Move WorkflowDiagram related actions from DashboardLive into WorkflowLive
  • Move WorkflowDiagram component into liveview, so that we can subscribe to channels (i.e. updating of the diagram when someone changes something).
  • Integrate @openfn/workflow-diagram@0.0.8 and use the new Store interface for updating it.
  • Remove component_mounted event from WorkflowDiagram hook, using a MutationObserver and a Base64 encoded JSON payload.
  • Fixed an issue where the compiler component would try and load a 'nothing adaptor', added a condition to check an adaptor is actually selected.
  • Removed previous Workflow CTE queries, replaced by the introduction of the Workflow model, see (https://github.com/OpenFn/Lightning/blob/53da6883483e7d8d078783f348da327d1dd72d20/lib/lightning/workflows.ex#L111-L119).

[0.1.13] - 2022-08-29

Added

  • Allow administrators to configure OIDC providers for authentication (note that this is just for authenticating, not yet for creating new accounts via OIDC)
  • Add Monaco editor to the step/job panel
  • Allow users to delete their own accounts. Schedule their user and credentials data for deletion when they do.
  • Allow superusers to delete a user account. Schedule the user's credentials and user data for deletion when they do.
  • If a user is scheduled for deletion, disable their account and prevent them from logging in.
  • The 'User profile' and 'Credentials' page now have a sidebar menu

Changed

  • Project users now have one of the following roles: viewer, editor, admin, owner
  • Users only have the following roles: user, superuser

[0.1.12] - 2022-08-15

Added

  • Transfer credential ownership to another user.
  • Create credentials via a form interface*
  • Show "projects with access" in credentials list view.
  • Show job in runs list and run view.
  • Added roles and permissions to workflows and history page #645

*The form is defined by a JSON schema provided by an adaptor, in most cases: e.g., language-dhis2 provides a single schema which defines the required attributes for state.configuration, while language-common provides multiple credential schemas like "oauth" or "basic auth" which define attributes for state.configuration and which might be used by lots of different jobs.)

Fixed

  • User menu (top right) appears on top of all other components.
  • User profile screen integrated with the rest of the liveview app.

[0.1.11] - 2022-08-05

Fixed

  • Fixed logging in Runner when :debug log level used; note that this caused crashes in Oban

[0.1.10] - 2022-08-05

Added

  • Credential auditing
  • Build/version information display for easier debugging

Fixed

  • Fixed a bug that enqueued cron-triggered jobs even when they were disabled

[0.1.9] - 2022-07-27

Added

  • Navigate to user profile or credentials page and log out through the user icon dropdown
  • Create and edit dataclips
  • Add a production tag to credentials
  • View a dropdown of operations and their description for the language-common v2.0.0-rc2 adaptor (this pattern to be rolled out across adaptors)

Changed

  • Navigate between projects through a project picker on the navbar

Fixed

  • Run Lightning with docker

Security

  • Sensitive credential values are scrubbed from run logs
  • All credentials are encrypted at REST

[0.1.7] - 2022-06-24

Added

  • Run a job with a cron trigger
  • Queue jobs via Oban/Postgres
  • Edit jobs via the workflow canvas

[0.1.6] - 2022-06-07

Added

  • Register, log in and log out of an account

  • Allow superusers and admin users to create projects

  • Allow admin users to create or disable a user’s account

  • Allow superusers for local deployments to create users and give them access to project spaces

  • Create and edit a job with a webhook, flow/fail or cron trigger

  • Create and edit credentials for a job

  • Copy a job's webhook URL

  • View all workflows in a project visually

  • Deploy lightning locally with Docker

  • Enable a job to automatically process incoming requests

  • Run a job with a webhook or flow/fail trigger

  • View job runs along with their logs, exit code, start and end time

  • View data clips that have initiated job runs (http requests for webhooks, run results)