Lightning.Kickstart (Lightning v2.19.0-pre)

View Source

Declarative, idempotent kickstarting of Lightning from a scenario file.

Given a plain map (typically decoded from a YAML/JSON scenario file), this module creates users (with optional API tokens), credentials and projects, and provisions each project's workflows through Lightning.Projects.Provisioner — the same engine that backs the /api/provision HTTP API.

It serves local development (bin/e2e --scenario) and external test harnesses: boot Lightning into a known state, read the manifest, drive the public APIs.

It is not a way to provision a live instance. There is no release entry point, and run/1 refuses to run outside dev and test — see "Safety". Deploying state to a running instance is what /api/provision and the CLI are for.

Idempotency

Re-running the same scenario converges instead of duplicating:

  • Users are matched by email, credentials by {owner, name}, and both are reused when they already exist.
  • API tokens are signed JWTs and cannot be supplied; when api_token: true the user's oldest existing API token is reused, and one is generated only if none exists. Tokens are surfaced through manifest/1.
  • Projects, workflows, triggers, jobs and edges get deterministic ids (UUIDv5-style, derived from a project/workflow's name and from a record's key in the workflow spec) unless an explicit id is given, so the provisioner upserts them on subsequent runs.
  • Project members are added or have their role updated, never removed.

Renaming a record changes its derived id, and what that costs depends on the record:

  • Renaming a job, trigger or edge deletes the old row and creates a new one - Workflow declares has_many :jobs, on_replace: :delete. A renamed webhook trigger answers at a new /i/<id> URL.
  • Renaming a workflow or collection fails, naming the record: the old one is still on the project and the provisioner treats the document as the complete set. Pin an explicit id on a workflow you intend to rename and its jobs, triggers and edges keep their ids too.
  • Renaming a project creates a new one and leaves the old one alone.

The whole run happens in a single transaction — a failing scenario leaves the database untouched.

Safety

Kickstarting creates users (including superusers), and applies a scenario as the desired state: records it declares are overwritten from the file, so a job body edited in the editor is reverted on the next run. That is fine for a database that exists to be thrown away, and wrong for one anybody relies on — so it is confined to dev and test, and run/1 raises anywhere else.

Mix tasks aren't shipped in a release, so mix lightning.kickstart cannot be reached in production at all; the environment check exists to also stop someone calling this module directly from a remote console.

Scenario shape

users:
  - email: amy@openfn.org          # required
    first_name: Amy
    superuser: true
    api_token: true                # generate/reuse a token, see manifest
    # password defaults to "welcome12345"

credentials:
  - name: dhis2-prod               # required, unique per scenario
    owner: amy@openfn.org          # required, a user declared above
    schema: dhis2
    body:
      password: ${env:DHIS2_PASSWORD}  # explicit env interpolation

projects:
  - name: my-project               # required, url-safe
    description: An example project # optional
    members:                       # required, exactly one owner
      - { email: amy@openfn.org, role: owner }
    credentials: [dhis2-prod]      # optional, exposed to this project
    collections:                   # optional
      - name: my-collection
    workflows:                     # workflow-spec documents, see below
      - name: my-workflow
        jobs:
          transform:
            name: transform
            adaptor: "@openfn/language-common@latest"
            body: "fn(state => state);"
            credential: dhis2-prod   # optional, a credential above
        triggers:
          webhook:
            type: webhook
            enabled: true
        edges:
          webhook->transform:
            source_trigger: webhook
            target_job: transform
            condition_type: always
            enabled: true

Each entry under workflows is a workflow spec — the same hand-writable format the collaborative editor imports and exports and that workflow templates are written in, validated against the same JSON Schema and converted by Lightning.Workflows.Spec. There is no kickstart-specific workflow dialect: anything you can paste into the editor's YAML import works here, and vice versa. The one extra key kickstart resolves is a job's credential, which names a credential declared at the top level (the schema already allows it; the editor ignores it).

Every other key — the scenario itself, and each user, credential, member and project — is checked against an explicit allow-list, so a typo (usres:) or an unsupported field (e.g. channels, not yet handled here) raises instead of being silently ignored.

Keys are strings, as produced by the YAML/JSON parsers.

Env interpolation is explicit: only ${env:VAR} references are replaced (and it is an error for VAR to be unset). Plain ${...} is left alone, so JS template literals in job bodies (${id}, ${HOME}, ${state.data}) can never collide with interpolation.

Summary

Types

Per-user result: the persisted user and any generated API token.

Functions

Parse a scenario file into a map. Supports YAML and JSON.

Structured, JSON-encodable manifest of a run/1 result — everything an external harness needs to drive the instance: user emails and API tokens, record ids, and webhook paths.

Create or update everything described by scenario, atomically.

Load a scenario file (.yaml, .yml or .json) and run/1 it.

Human-readable one-line-per-record summary of a run/1 result.

Types

result()

@type result() :: %{
  users: %{required(String.t()) => user_result()},
  credentials: %{required(String.t()) => Lightning.Credentials.Credential.t()},
  projects: [map()]
}

user_result()

@type user_result() :: %{
  user: Lightning.Accounts.User.t(),
  api_token: String.t() | nil
}

Per-user result: the persisted user and any generated API token.

Functions

load_file!(path)

@spec load_file!(Path.t()) :: map()

Parse a scenario file into a map. Supports YAML and JSON.

manifest(map)

@spec manifest(result()) :: map()

Structured, JSON-encodable manifest of a run/1 result — everything an external harness needs to drive the instance: user emails and API tokens, record ids, and webhook paths.

run(scenario)

@spec run(map()) :: result()

Create or update everything described by scenario, atomically.

Returns a result map describing the records; pass it to manifest/1 for a JSON-encodable summary or summary/1 for a human-readable one.

Raises unless kickstarting is enabled (see the module docs), and rolls the whole run back on any error.

run_file(path, opts \\ [])

@spec run_file(
  Path.t(),
  keyword()
) :: result()

Load a scenario file (.yaml, .yml or .json) and run/1 it.

Options:

  • :manifest - path to write the JSON manifest to.

summary(map)

@spec summary(result()) :: String.t()

Human-readable one-line-per-record summary of a run/1 result.