# Deployment

Head to the [Deploy](https://docs.openfn.org/documentation/deploy/options)
section of our docs site to get started.

See below for technical considerations and instructions.

## Encryption

Lightning enforces encryption at rest for credentials, TOTP backup codes, and
webhook trigger authentication methods, for which an encryption key must be
provided when running in production.

The key is expected to be a randomized set of bytes, 32 long; and Base64 encoded
when setting the environment variable.

There is a mix task that can generate keys in the correct shape for use as an
environment variable:

```sh
mix lightning.gen_encryption_key
0bJ9w+hn4ebQrsCaWXuA9JY49fP9kbHmywGd5K7k+/s=
```

Copy your key _(NOT THIS ONE)_ and set it as `PRIMARY_ENCRYPTION_KEY` in your
environment.

## Workers

Lightning uses external worker processes for executing Runs. There are three
settings required to configure worker authentication.

- `WORKER_RUNS_PRIVATE_KEY`: an RSA private key, as the PEM file's contents or
  that file base64-encoded
- `WORKER_LIGHTNING_PUBLIC_KEY`: base64 encoded RSA public key
- `WORKER_SECRET`: 256bit long shared secret used by the worker to sign JWTs

You can use the `mix lightning.gen_worker_keys` task to generate these for
convenience. It prints each key as one line of base64 in double quotes, ready
for a `.env` file. For other ways to set the private key, see
[PEM keys in environment variables](#pem-keys-in-environment-variables).

For more information see the [Workers](WORKERS.md) documentation.

## Environment Variables

Note that for secure deployments, it's recommended to use a combination of
`secrets` and `configMaps` to generate secure environment variables.

### PEM keys in environment variables

Set the variable to the contents of the PEM file. If your tooling keeps one
variable per line, base64-encode the file on one line instead.

This applies to `SERVICE_ACCOUNT_PUBLIC_KEY`, `WORKER_RUNS_PRIVATE_KEY` and
`GITHUB_CERT`. `WORKER_LIGHTNING_PUBLIC_KEY` is read by the worker, not
Lightning, and stays one line of base64. Use `openssl base64 -A -in FILE`, which
behaves the same on macOS and Linux.
[Service accounts](SERVICE_ACCOUNTS.md#set-up-a-new-instance) step 2 has an
example for Kubernetes, Docker, Docker Compose, a `.env` file and systemd.

Keep the base64 on one line. A file that holds one variable per line, such as a
`docker run --env-file` file or an unquoted value in a Compose `env_file:`,
keeps only the first line of wrapped base64 and says nothing. Lightning then
stops at boot because the key has a BEGIN line but no END line.

In a `docker run --env-file` file, or a Compose `env_file:` with `format: raw`,
don't put quotes around the value. They become part of it.

A raw PEM in a file that holds one variable per line shows up as an error before
Lightning starts:

- If you see
  `docker: --env-file: invalid env file (…): variable '-----END PUBLIC KEY-----' contains whitespaces`,
  or from Compose
  `failed to read …: line 3: unexpected character "/" in variable name …` or
  `variable '-----END PUBLIC KEY-----' contains whitespaces`, the file holds a
  multi-line PEM without quotes. In a plain Compose `env_file:`, put it in
  double quotes. With `docker run --env-file` or `format: raw`, use one line of
  base64.
- If you see
  `** (RuntimeError) Could not load .env: it is not a valid env file. The parser's own message is left out because it quotes the line it stopped at, which can be part of a secret. The usual cause is a value that runs over several lines without double quotes around it, such as a PEM key.`,
  Lightning's own env file holds a multi-line PEM without quotes. Put it in
  double quotes. The message names whichever of `.env`, `.<env>.env` or
  `.<env>.override.env` failed. It doesn't show the line, because the line could
  be part of a key, so look in that file for an unquoted value that spans
  several lines.

Lightning's own boot errors name the variable and say what to set.
[Service accounts](SERVICE_ACCOUNTS.md#set-up-a-new-instance) lists them for
`SERVICE_ACCOUNT_PUBLIC_KEY`. `WORKER_RUNS_PRIVATE_KEY` and `GITHUB_CERT` give
the same messages for a value that isn't a PEM key, is quoted, is cut short, is
damaged or holds more than one PEM block. Each names its own variable and key
file. The other cases differ:

- `WORKER_RUNS_PRIVATE_KEY` has its own messages for an empty or unset value, a
  public key, a key that isn't RSA, and a PEM block that isn't a key. Set it to
  the RSA private key. Its public key goes in the worker's
  `WORKER_LIGHTNING_PUBLIC_KEY`. Each message ends with
  `You can generate new worker keys using: mix lightning.gen_worker_keys`.
- An empty `GITHUB_CERT` counts as unset. Lightning starts with GitHub sync off
  and gives no message.

If a message says the key
`has a BEGIN line that names no key type Lightning reads`, the BEGIN line lost
some of its dashes, or the key is in a format Lightning doesn't read, such as an
OpenSSH key. Copy the PEM file again, or make the key with `openssl`.

### Limits

- `WORKER_MAX_RUN_MEMORY_MB` - how much memory (in MB) can a single run use?
- `RUN_GRACE_PERIOD_SECONDS` - how long _after_ the `MAX_RUN_DURATION_SECONDS`
  should the server wait for the worker to send back data on a run.
- `WORKER_MAX_RUN_DURATION_SECONDS` - the maximum duration (in seconds) that
  workflows are allowed to run (keep this plus `RUN_GRACE_PERIOD_SECONDS` below
  your termination_grace_period if using kubernetes)
- `WORKER_CAPACITY` - the number of runs a ws-worker instance will take on
  concurrently.
- `MAX_DATACLIP_SIZE_MB` - the maximum size (in MB) of a dataclip created via
  the webhook trigger URL for a job. This limits the max request size via the
  JSON plug and may (in future) limit the size of dataclips that can be stored
  as run_results via the websocket connection from a worker.
- `MAX_CREDENTIAL_SENSITIVE_VALUES` - the maximum number of sensitive values
  that can be stored in a credential. Defaults to 50.
- `MAX_SANDBOX_NESTING_DEPTH` - the maximum depth at which sandboxes can be
  nested. A direct child sandbox is depth 1, a sandbox of a sandbox is depth 2,
  etc. Provision attempts beyond this depth return
  `{:error, :nesting_too_deep}`, and the **Create Sandbox** button on the
  Sandboxes page is disabled with a tooltip when the current project is at the
  cap. Defaults to 5. Set to `0` to disable sandbox creation entirely.

### GitHub

Lightning enables connection to GitHub via GitHub Apps. The following GitHub
repository permissions are needed for the GitHub app:

| **Resource** | **Access**     |
| ------------ | -------------- |
| Actions      | Read and Write |
| Contents     | Read and Write |
| Metadata     | Read only      |
| Secrets      | Read and Write |
| Workflows    | Read and Write |

Ensure you set the following URLs:

- **Homepage URL:** `<app_url_here>`
- **Callback URL for authorizing users:** `<app_url_here>/oauth/github/callback`
  (Do NOT check the two checkboxes in this section requesting Device Flow and
  OAuth.)
- **Setup URL for Post installation:** `<app_url_here>/setup_vcs` (Check the box
  for **Redirect on update**)

These environment variables will need to be set in order to configure the github
app:

| **Variable**               | **Description**                                                                                                           |
| -------------------------- | ------------------------------------------------------------------------------------------------------------------------- |
| `GITHUB_APP_ID`            | the github app ID.                                                                                                        |
| `GITHUB_APP_NAME`          | the github app name. This is the name used in the public link. It is the downcased name with spaces replaced with hyphens |
| `GITHUB_APP_CLIENT_ID`     | the github app Client ID                                                                                                  |
| `GITHUB_APP_CLIENT_SECRET` | the github app Client Secret                                                                                              |
| `GITHUB_CERT`              | The GitHub App's private key, as the PEM file's contents or that file base64-encoded.                                     |

You can access these from your github app settings menu. Also needed for the
configuration is:

- `REPO_CONNECTION_SIGNING_SECRET` - secret used to sign access tokens. This
  access token is used to authenticate requests made from the github actions.
  You can generate this using `mix lightning.gen_encryption_key`

### Storage

Lightning can use a storage backend to store exports.

| **Variable**      | Description                                      |
| ----------------- | ------------------------------------------------ |
| `STORAGE_BACKEND` | the storage backend to use. (default is `local`) |
| `STORAGE_PATH`    | the path to store files in. (default is `.`)     |

#### Supported backends:

- `local` - local file storage
- `gcs` - Google Cloud Storage

#### Google Cloud Storage

For Google Cloud Storage, the following environment variables are required:

| **Variable**                          | Description                                                                      |
| ------------------------------------- | -------------------------------------------------------------------------------- |
| `GCS_BUCKET`                          | the name of the bucket to store files in                                         |
| `GOOGLE_APPLICATION_CREDENTIALS_JSON` | A base64 encoded JSON keyfile for the service account with access to the bucket. |

> ℹ️ Note: The `GOOGLE_APPLICATION_CREDENTIALS_JSON` should be base64 encoded,
> currently Workload Identity is not supported.

### Mail

Lightning can send emails for various reasons, such as password resets and
alerts for failed runs.

In order to send emails, you need to set the `MAIL_PROVIDER` environment
variable to one of the following:

- `local` (the default)
- `mailgun`
- `smtp`

You will also want to set the `EMAIL_ADMIN` environment variable to the email
address that will be used as the sender for system emails.

If you are planning on using the `smtp` provider, with `TLS` enabled, the
current implementation has the following contraints:

- Only TLS 1.3 is supported.
- For the purposes of Server Name Indication (SNI), the hostname provided as the
  `SMTP_RELAY` is used. This means that the `SMTP_RELAY` value must be present
  in a SAN `dNSName` on the cert. Practically, this means that TLS is unlikely
  to work if an IP address, or an internal-only hostname is provided as the
  `SMTP_RELAY` value.

#### Mailgun

For mailgun, the following environment variables are required:

| **Variable**      | Description              |
| ----------------- | ------------------------ |
| `MAIL_PROVIDER`   | Must be set to `mailgun` |
| `MAILGUN_API_KEY` | the mail gun api key     |
| `MAILGUN_DOMAIN`  | the mail gun domain      |

#### SMTP

For SMTP, the following environment variables are required:

| **Variable**    | Description                                                              |
| --------------- | ------------------------------------------------------------------------ |
| `MAIL_PROVIDER` | Must be set to `smtp`                                                    |
| `SMTP_USERNAME` | Username for your server                                                 |
| `SMTP_PASSWORD` | Password for the user                                                    |
| `SMTP_RELAY`    | IP address or hostname                                                   |
| `SMTP_TLS`      | Use TLS, defaults to `true`, options are `true`, `false`, `if_available` |
| `SMTP_PORT`     | Which port to use, defaults to `587`                                     |

### Other config

| **Variable**                                      | Description                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   |
| ------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `ADAPTORS_ICONS_PATH`                             | Directory the on-disk adaptor icon cache is written to. The official image sets this to `/app/priv/adaptor_icons`; outside the image it defaults to a subdirectory of the system temp directory. Either way, mount a persistent volume at this path so cached icons survive a restart.                                                                                                                                                                                                                                                                                                                                                                                        |
| `ADAPTORS_LOCAL_REPO`                             | Path to an OpenFn adaptors checkout, used when `ADAPTORS_STRATEGY` is `local`. Comma-separate several. The first checkout that has a package wins. See [Adaptors](ADAPTORS.md).                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               |
| `ADAPTORS_NPM_GITHUB_REF`                         | Git ref of `OpenFn/adaptors` that adaptor icons are read from. Defaults to `main`.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            |
| `ADAPTORS_NPM_GITHUB_URL`                         | Raw GitHub host that adaptor icons are fetched from. Defaults to `https://raw.githubusercontent.com`.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |
| `ADAPTORS_NPM_HTTP_TIMEOUT`                       | Receive timeout in milliseconds for npm registry, schema and icon requests. Defaults to `30000`.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              |
| `ADAPTORS_NPM_JSDELIVR_URL`                       | CDN that adaptor credential schemas are fetched from. Defaults to `https://cdn.jsdelivr.net`.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 |
| `ADAPTORS_NPM_REGISTRY_URL`                       | npm registry the adaptor catalogue is read from. Defaults to `https://registry.npmjs.org`. Set this and the two URLs above to use an internal mirror. See [Adaptors](ADAPTORS.md).                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            |
| `ADAPTORS_PATH`                                   | Where you store your locally installed adaptors                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               |
| `ADAPTORS_REFRESH_INTERVAL_SECONDS`               | How often, in seconds, the adaptor catalogue refreshes from npm. Defaults to one hour. Set to `0` to disable scheduled refreshes. See [Adaptors](ADAPTORS.md).                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                |
| `ADAPTORS_STRATEGY`                               | Where the adaptor catalogue comes from: `npm` (default) or `local`. See [Adaptors](ADAPTORS.md).                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              |
| `ALLOW_SIGNUP`                                    | Set to `true` to enable user access to the registration page. Set to `false` to disable new user registrations and block access to the registration page.<br>Default is `false`.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              |
| `CORS_ORIGIN`                                     | A list of acceptable hosts for browser/cors requests (',' separated)                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          |
| `DATABASE_TCP_USER_TIMEOUT`                       | How long, in milliseconds, a database connection may go without its peer acknowledging data before the OS drops it. Lightning also enables TCP keepalive on these connections, so a connection whose network path has died is dropped within roughly this time even while it waits on a reply, including queries run with no timeout. A live query is unaffected. **Linux-only**: Lightning does not apply it on other platforms. Defaults to `DATABASE_TIMEOUT` + 5000 (`20000` with the default 15s query timeout). `0` disables it (and keepalive); other values must be between 1 and 2147483647, or boot fails. A value below `DATABASE_TIMEOUT` logs a warning at boot. |
| `DISABLE_DB_SSL`                                  | In production, the use of an SSL connection to Postgres is required by default.<br>Setting this to `"true"` allows unencrypted connections to the database. This is strongly discouraged in a real production environment.                                                                                                                                                                                                                                                                                                                                                                                                                                                    |
| `DISABLE_DB_SSL_CERT_VERIFY`                      | When a SSL connection is used to connect to Postgres, the server's certificate will be verified by default.<br> Setting this to `"true"` disables certificate verification. This is strongly discouraged in a real production environment.                                                                                                                                                                                                                                                                                                                                                                                                                                    |
| `EMAIL_ADMIN`                                     | This is used as the sender email address for system emails. It is also displayed in the menu as the support email.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            |
| `EMAIL_SENDER_NAME`                               | This is displayed in the email client as the sender name for emails sent by the application.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |
| `ERLANG_NODE_DISCOVERY_VIA_POSTGRES_CHANNEL_NAME` | The name of the Postgresql channel that is used when Erlang node discovery via Postgres is enabled. Defaults to `lightning-cluster` if not set.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               |
| `ERLANG_NODE_DISCOVERY_VIA_POSTGRES_ENABLED`      | If set to `true`, Lightning will use Postgres to discover Erlang nodes. This strategy will be used in addition to other strategies that are in use. Default value is `false`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |
| `IDLE_TIMEOUT`                                    | The number of seconds that must pass without data being received before the Lightning web server kills the connection.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        |
| `IS_RESETTABLE_DEMO`                              | If set to `yes`, it allows this instance to be reset to the initial "Lightning Demo" state. Note that this will destroy _most_ of what you have in your database!                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             |
| `K8S_HEADLESS_SERVICE`                            | This environment variable is automatically set if you're running on GKE and it is used to establish an Erlang node cluster. Note that if you're _not_ using Kubernetes, the "gossip" strategy is used to establish clusters.                                                                                                                                                                                                                                                                                                                                                                                                                                                  |
| `LISTEN_ADDRESS`                                  | The address the web server should bind to. Defaults to `127.0.0.1` to block access from other machines.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       |
| `LOG_LEVEL`                                       | How noisy you want the logs to be (e.g., `debug`, `info`)                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     |
| `METRICS_RUN_PERFORMANCE_AGE_SECONDS`             | The oldest a run can be to be included in Run performance metrics.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            |
| `METRICS_RUN_QUEUE_AGE_SECONDS`                   | The polling period for run queue metrics.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     |
| `METRICS_STALLED_RUN_THRESHOLD_SECONDS`           | The length of time a Run must be in the `available` state before it is considered stalled.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    |
| `METRICS_UNCLAIMED_RUN_THRESHOLD_SECONDS`         | The length of time a Run must be in the `available` state before it counts towards an impeded project.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        |
| `MIX_ENV`                                         | Your mix env, likely `prod` for deployment                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    |
| `NODE_ENV`                                        | Node env, likely `production` for deployment                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |
| `ORIGINS`                                         | The allowed origins for web traffic to the backend                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            |
| `PER_WORKFLOW_CLAIM_LIMIT`                        | The maximum number of runs per workflow to consider during run claiming. This prevents any single workflow from dominating the processing queue while ensuring fairness across workflows.<br>Default is `50`.                                                                                                                                                                                                                                                                                                                                                                                                                                                                 |
| `CLAIM_WORK_MEM`                                  | PostgreSQL `work_mem` setting for the run claim query. Helps optimize complex sorting operations. Set to a valid PostgreSQL memory value (e.g., `32MB`, `64MB`, `1GB`). Set to empty string to disable.<br>Default: disabled in dev/test, `32MB` in production.                                                                                                                                                                                                                                                                                                                                                                                                               |
| `PORT`                                            | The port your Phoenix app runs on                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             |
| `PROMEX_DATASOURCE_ID`                            | The datasource that PromEx will use if configured to push initial dashboards to Grafana. Defaults to an empty string.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |
| `PROMEX_ENABLED`                                  | Enables PromEx tracking and publishing of metrics if set to 'true' or 'yes'. Defaults to false.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               |
| `PROMEX_ENDPOINT_SCHEME`                          | The scheme needed when connecting to the Promex Endpoint. Defaults to https.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |
| `PROMEX_EXPENSIVE_METRICS_ENABLED`                | Certain metrics may be expensive to generate if Lightning is under load. If set to 'true', or 'yes' these metrics will be enabled. Defaults to 'false'.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       |
| `PROMEX_GRAFANA_HOST`                             | This is used when PromEx is required to push data to a Grafana instance, e.g. when PromEx sets up initial dashboards.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |
| `PROMEX_GRAFANA_PASSWORD`                         | This is used when PromEx is required to push data to a Grafana instance, e.g. when PromEx sets up initial dashboards.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |
| `PROMEX_GRAFANA_USER`                             | This is used when PromEx is required to push data to a Grafana instance, e.g. when PromEx sets up initial dashboards.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |
| `PROMEX_METRICS_ENDPOINT_AUTHORIZATION_REQUIRED`  | If set to 'true' or 'yes', the PromEx endpoint on Lightning will require consumers to provide credentials for authorization. Defaults to 'true'.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              |
| `PROMEX_METRICS_ENDPOINT_TOKEN`                   | A Bearer token that the consumer of the promEx endpoint must provide in the Authorization header. Defaults to a random series of bytes.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       |
| `PROMEX_UPLOAD_GRAFANA_DASHBOARDS_ON_START`       | Instructs PromEx to upload iniital dashboards to a Grafana instance if set to 'true' or 'yes'. Defaults to false.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             |
| `PRIMARY_ENCRYPTION_KEY`                          | A base64 encoded 32 character long string.<br>See [Encryption](#encryption).                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |
| `QUEUE_RESULT_RETENTION_PERIOD_MINUTES`           | The number of minutes to keep completed (successful) `ObanJobs` in the queue (not to be confused with runs and/or history)                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    |
| `SECRET_KEY_BASE`                                 | A secret key used as a base to generate secrets for encrypting and signing data.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              |
| `SENTRY_DSN`                                      | If using Sentry for error monitoring, your DSN                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                |
| `SERVICE_ACCOUNT_PUBLIC_KEY`                      | An RSA public key of at least 2048 bits, as the PEM file's contents or that file base64-encoded. It sets up a service account that configures this instance through its API. While it is set, first setup is turned off. Lightning will not start if the value is wrong. See [Service accounts](SERVICE_ACCOUNTS.md) and [PEM keys in environment variables](#pem-keys-in-environment-variables).                                                                                                                                                                                                                                                                             |
| `URL_HOST`                                        | The host used for writing URLs (e.g., `demo.openfn.org`)                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      |
| `URL_PORT`                                        | The port, usually `443` for production                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        |
| `URL_SCHEME`                                      | The scheme for writing URLs (e.g., `https`)                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   |
| `USAGE_TRACKER_HOST`                              | The host that receives usage tracking submissions<br>(defaults to https://impact.openfn.org)                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |
| `USAGE_TRACKING_DAILY_BATCH_SIZE`                 | The number of days that will be reported on with each run of `UsageTracking.DayWorker`. This will only have a noticeable effect in cases where there is a backlog or where reports are being generated retroactively (defaults to 10).                                                                                                                                                                                                                                                                                                                                                                                                                                        |
| `USAGE_TRACKING_ENABLED`                          | Enables the submission of anonymized usage data to OpenFn (defaults to `true`)                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                |
| `USAGE_TRACKING_RESUBMISSION_BATCH_SIZE`          | The number of failed reports that will be submitted on each resubmission run (defaults to 10)                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 |
| `USAGE_TRACKING_RUN_CHUNK_SIZE`                   | The size of each batch of runs that is streamed from the database when generating UsageTracking reports (default 100). Decreasing this may decrease memory consumption when generating reports.                                                                                                                                                                                                                                                                                                                                                                                                                                                                               |
| `USAGE_TRACKING_UUIDS`                            | Indicates whether submissions should include cleartext UUIDs or not. Options are `cleartext` or `hashed_only`, with the default being `hashed_only`.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          |
| `REQUIRE_EMAIL_VERIFICATION`                      | Indicates whether user email addresses should be verified. Defaults to `false`.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               |

### AI Chat

🧪 **Experimental**

Lightning can be configured to use an AI chatbot for user interactions.

See [openfn/apollo](https://github.com/OpenFn/apollo) for more information on
the Apollo AI service.

The following environment variables are required:

- `AI_ASSISTANT_API_KEY` - API key to use the assistant. This currently requires
  an Anthropic key.
- `APOLLO_ENDPOINT` - the endpoint for the OpenFn Apollo AI service.

Three optional variables control how long Lightning waits on Apollo. Each one
has a default, so set them only if those defaults do not suit your deployment.

- `APOLLO_CONNECT_TIMEOUT_MS` - how long to wait to reach Apollo at all.
  Defaults to 5000. Reaching Apollo happens inside the idle budget, so a value
  above `APOLLO_IDLE_TIMEOUT_MS` never takes effect; Lightning warns at boot if
  you set one.
- `APOLLO_IDLE_TIMEOUT_MS` - the longest acceptable silence, both before the
  first byte of an answer and between the chunks that follow. Defaults to 30000.
- `APOLLO_REQUEST_TIMEOUT_MS` - the longest one whole request may take, however
  steadily it is streaming. Defaults to 300000.

The idle default assumes Apollo v3.1.1 or later, which sends a keepalive every
15 seconds. On an older Apollo a stream can go quiet for minutes while a model
is thinking, and 30 seconds will cut it off, so raise `APOLLO_IDLE_TIMEOUT_MS`
or upgrade Apollo.

The three added together, plus a ten-second buffer, bound how long one AI job
may run, and that has to stay under Oban's shutdown grace period of six minutes.
Raising them past it means a deploy landing on a running answer kills it with
nothing left to report the failure, so Lightning warns at boot if the sum gets
too close. Note that the longer grace period also makes rolling restarts slower,
since Oban now waits up to six minutes for a running job rather than two.

`APOLLO_TIMEOUT` is the old name for `APOLLO_IDLE_TIMEOUT_MS`. On the wire it
only ever covered the silence, never the other two. It is no longer read, and
Lightning logs a warning at boot if it is still set.

### OAuth credential connections (Google, Salesforce, etc.)

OAuth clients that **jobs** use to connect to external systems (Google Sheets,
Salesforce, and similar) are no longer configured via environment variables.
They are registered in the UI under **Credentials → OAuth clients** and scoped
to the projects that use them; the client id, secret, and redirect/callback URL
are entered in that form.

> The older `GOOGLE_CLIENT_ID` / `GOOGLE_CLIENT_SECRET` and
> `SALESFORCE_CLIENT_ID` / `SALESFORCE_CLIENT_SECRET` environment variables are
> no longer read by Lightning and can be removed from your deployment.

### OAuth Provider Egress

When a user configures an OAuth client, they supply the provider's endpoint URLs
(authorization, token, userinfo, revocation, introspection). Lightning makes
server-side requests to those URLs during credential setup and automatic token
refresh, which makes them a Server-Side Request Forgery (SSRF) sink: a
low-privilege user could point an endpoint at an internal service or a cloud
metadata address (`169.254.169.254`) and use the server as a proxy.

To prevent this, all outbound OAuth requests are routed through an egress guard
that resolves the endpoint hostname, rejects any address in an internal or
reserved range, and pins the connection to the validated IP (so DNS cannot swap
in an internal address after the check). This is **on by default** and requires
no configuration; internal endpoints are blocked and the request fails with a
generic network error.

| **Variable**                   | **Description**                                                                                                                                         | **Default** |
| ------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------: |
| `OAUTH_PROVIDER_ALLOWED_HOSTS` | Comma-separated list of hostnames that bypass the internal-range block. Only set this if your OAuth provider legitimately lives on an internal network. |   _(empty)_ |

Only add a host here if you are self-hosting an OAuth provider (or a proxy to
one) on an address the guard would otherwise block, and you trust it. Matching
is by **hostname string**, case-insensitive — e.g.
`OAUTH_PROVIDER_ALLOWED_HOSTS=idp.internal.example,localhost`. An allow-listed
host is still resolved, but its resolved IP is not checked against the block
list, so keep the list as small as possible.

### Channel Egress

Lightning can proxy requests through to a channel's configured upstream via the
`/channels/:id/...` reverse proxy (powered by the `philter` dependency). Because
the upstream URL is operator-supplied, that proxy is a Server-Side Request
Forgery (SSRF) sink: a request could be pointed at an internal service or a
cloud metadata address (`169.254.169.254`) and use the server as a proxy.

To prevent this, the channel proxy runs an egress guard that resolves the
upstream hostname and rejects any address in a private, loopback, link-local, or
otherwise reserved range. This is **on by default** and requires no
configuration; with neither variable set, blocking is on and nothing is
allow-listed.

| **Variable**                     | **Description**                                                                                                                                          | **Default** |
| -------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------: |
| `CHANNEL_BLOCK_PRIVATE_NETWORKS` | Whether to reject channel upstreams that resolve to private/loopback/link-local/reserved IP ranges (`true`/`false`/`yes`/`no`). Set false to allow them. |      `true` |
| `CHANNEL_ALLOWED_HOSTS`          | Comma-separated list of hostnames that bypass the block entirely — the escape hatch.                                                                     |   _(empty)_ |

The allow-list is checked **first**. A host in `CHANNEL_ALLOWED_HOSTS` is
allowed even if it resolves to a private IP, and even while
`CHANNEL_BLOCK_PRIVATE_NETWORKS=true`. So the common safe setup is to leave
blocking on (the default) and allow-list just the specific internal host you
need. With `CHANNEL_BLOCK_PRIVATE_NETWORKS=false` the allow-list is moot —
everything is allowed.

Matching is by **exact hostname only**, case-insensitive, with a single trailing
dot ignored. There is **no CIDR range and no wildcard support**; a literal IP
entry matches only if the channel URL uses that exact IP string. Entries are
bare hostnames — an entry containing a URL scheme (`://`), a path (`/`),
internal whitespace, or `@` is rejected at boot with a clear error naming the
bad entry. For example: `CHANNEL_ALLOWED_HOSTS=api.internal.example,localhost`.

### Webhook Retry Configuration

Lightning automatically retries webhook processing on **transient database
connection errors** using exponential backoff. This helps prevent data loss
during brief database outages.

The following environment variables control webhook retry behavior:

| **Variable**                     | **Description**                                                                                  | **Default** |
| -------------------------------- | ------------------------------------------------------------------------------------------------ | ----------: |
| `WEBHOOK_RETRY_MAX_ATTEMPTS`     | Maximum number of attempts (the first attempt runs immediately; backoffs occur between retries). |         `5` |
| `WEBHOOK_RETRY_INITIAL_DELAY_MS` | Initial backoff delay in milliseconds.                                                           |       `100` |
| `WEBHOOK_RETRY_MAX_DELAY_MS`     | Maximum backoff delay in milliseconds.                                                           |     `10000` |
| `WEBHOOK_RETRY_BACKOFF_FACTOR`   | Multiplier for exponential backoff (each delay × this factor, up to the max delay).              |         `2` |
| `WEBHOOK_RETRY_TIMEOUT_MS`       | Total time budget for all attempts (including sleeps) in milliseconds.                           |     `60000` |
| `WEBHOOK_RETRY_JITTER`           | Whether to add \~0–25% randomization to each delay to avoid thundering herd (`true`/`false`).    |      `true` |

**How backoff works (defaults, jitter off):** attempt 1 runs immediately; on
failure we sleep and retry up to `WEBHOOK_RETRY_MAX_ATTEMPTS - 1` times with
exponential delays starting at `WEBHOOK_RETRY_INITIAL_DELAY_MS` and multiplying
by `WEBHOOK_RETRY_BACKOFF_FACTOR`, capped by `WEBHOOK_RETRY_MAX_DELAY_MS`.
Example sequence: `100ms → 200ms → 400ms → 800ms` (four sleeps for five total
attempts), stopping sooner if `WEBHOOK_RETRY_TIMEOUT_MS` elapses.

**Server timeout alignment:** Set your Phoenix `IDLE_TIMEOUT` to be **at least**
`WEBHOOK_RETRY_TIMEOUT_MS + 15000` (in milliseconds) so long retries finish
before the connection is closed.
