# Deployment Head to the [Deploy](https://docs.openfn.org/documentation/deploy/options) section of our docs site to get started. See below for technical considerations and instructions. ## Encryption Lightning enforces encryption at rest for credentials, TOTP backup codes, and webhook trigger authentication methods, for which an encryption key must be provided when running in production. The key is expected to be a randomized set of bytes, 32 long; and Base64 encoded when setting the environment variable. There is a mix task that can generate keys in the correct shape for use as an environment variable: ```sh mix lightning.gen_encryption_key 0bJ9w+hn4ebQrsCaWXuA9JY49fP9kbHmywGd5K7k+/s= ``` Copy your key _(NOT THIS ONE)_ and set it as `PRIMARY_ENCRYPTION_KEY` in your environment. ## Workers Lightning uses external worker processes for executing Runs. There are three settings required to configure worker authentication. - `WORKER_RUNS_PRIVATE_KEY`: base64 encoded RSA private key - `WORKER_LIGHTNING_PUBLIC_KEY`: base64 encoded RSA public key - `WORKER_SECRET`: 256bit long shared secret used by the worker to sign JWTs You can use the `mix lightning.gen_worker_keys` task to generate these for convenience. For more information see the [Workers](WORKERS.md) documentation. ## Environment Variables Note that for secure deployments, it's recommended to use a combination of `secrets` and `configMaps` to generate secure environment variables. ### Limits - `WORKER_MAX_RUN_MEMORY_MB` - how much memory (in MB) can a single run use? - `RUN_GRACE_PERIOD_SECONDS` - how long _after_ the `MAX_RUN_DURATION_SECONDS` should the server wait for the worker to send back data on a run. - `WORKER_MAX_RUN_DURATION_SECONDS` - the maximum duration (in seconds) that workflows are allowed to run (keep this plus `RUN_GRACE_PERIOD_SECONDS` below your termination_grace_period if using kubernetes) - `WORKER_CAPACITY` - the number of runs a ws-worker instance will take on concurrently. - `MAX_DATACLIP_SIZE_MB` - the maximum size (in MB) of a dataclip created via the webhook trigger URL for a job. This limits the max request size via the JSON plug and may (in future) limit the size of dataclips that can be stored as run_results via the websocket connection from a worker. - `MAX_CREDENTIAL_SENSITIVE_VALUES` - the maximum number of sensitive values that can be stored in a credential. Defaults to 50. - `MAX_SANDBOX_NESTING_DEPTH` - the maximum depth at which sandboxes can be nested. A direct child sandbox is depth 1, a sandbox of a sandbox is depth 2, etc. Provision attempts beyond this depth return `{:error, :nesting_too_deep}`, and the **Create Sandbox** button on the Sandboxes page is disabled with a tooltip when the current project is at the cap. Defaults to 5. Set to `0` to disable sandbox creation entirely. ### GitHub Lightning enables connection to GitHub via GitHub Apps. The following GitHub repository permissions are needed for the GitHub app: | **Resource** | **Access** | | ------------ | -------------- | | Actions | Read and Write | | Contents | Read and Write | | Metadata | Read only | | Secrets | Read and Write | | Workflows | Read and Write | Ensure you set the following URLs: - **Homepage URL:** `` - **Callback URL for authorizing users:** `/oauth/github/callback` (Do NOT check the two checkboxes in this section requesting Device Flow and OAuth.) - **Setup URL for Post installation:** `/setup_vcs` (Check the box for **Redirect on update**) These environment variables will need to be set in order to configure the github app: | **Variable** | **Description** | | -------------------------- | ------------------------------------------------------------------------------------------------------------------------- | | `GITHUB_APP_ID` | the github app ID. | | `GITHUB_APP_NAME` | the github app name. This is the name used in the public link. It is the downcased name with spaces replaced with hyphens | | `GITHUB_APP_CLIENT_ID` | the github app Client ID | | `GITHUB_APP_CLIENT_SECRET` | the github app Client Secret | | `GITHUB_CERT` | the github app private key (Base 64 encoded) | You can access these from your github app settings menu. Also needed for the configuration is: - `REPO_CONNECTION_SIGNING_SECRET` - secret used to sign access tokens. This access token is used to authenticate requests made from the github actions. You can generate this using `mix lightning.gen_encryption_key` ### Storage Lightning can use a storage backend to store exports. | **Variable** | Description | | ----------------- | ------------------------------------------------ | | `STORAGE_BACKEND` | the storage backend to use. (default is `local`) | | `STORAGE_PATH` | the path to store files in. (default is `.`) | #### Supported backends: - `local` - local file storage - `gcs` - Google Cloud Storage #### Google Cloud Storage For Google Cloud Storage, the following environment variables are required: | **Variable** | Description | | ------------------------------------- | -------------------------------------------------------------------------------- | | `GCS_BUCKET` | the name of the bucket to store files in | | `GOOGLE_APPLICATION_CREDENTIALS_JSON` | A base64 encoded JSON keyfile for the service account with access to the bucket. | > โ„น๏ธ Note: The `GOOGLE_APPLICATION_CREDENTIALS_JSON` should be base64 encoded, > currently Workload Identity is not supported. ### Mail Lightning can send emails for various reasons, such as password resets and alerts for failed runs. In order to send emails, you need to set the `MAIL_PROVIDER` environment variable to one of the following: - `local` (the default) - `mailgun` - `smtp` You will also want to set the `EMAIL_ADMIN` environment variable to the email address that will be used as the sender for system emails. If you are planning on using the `smtp` provider, with `TLS` enabled, the current implementation has the following contraints: - Only TLS 1.3 is supported. - For the purposes of Server Name Indication (SNI), the hostname provided as the `SMTP_RELAY` is used. This means that the `SMTP_RELAY` value must be present in a SAN `dNSName` on the cert. Practically, this means that TLS is unlikely to work if an IP address, or an internal-only hostname is provided as the `SMTP_RELAY` value. #### Mailgun For mailgun, the following environment variables are required: | **Variable** | Description | | ----------------- | ------------------------ | | `MAIL_PROVIDER` | Must be set to `mailgun` | | `MAILGUN_API_KEY` | the mail gun api key | | `MAILGUN_DOMAIN` | the mail gun domain | #### SMTP For SMTP, the following environment variables are required: | **Variable** | Description | | --------------- | ------------------------------------------------------------------------ | | `MAIL_PROVIDER` | Must be set to `smtp` | | `SMTP_USERNAME` | Username for your server | | `SMTP_PASSWORD` | Password for the user | | `SMTP_RELAY` | IP address or hostname | | `SMTP_TLS` | Use TLS, defaults to `true`, options are `true`, `false`, `if_available` | | `SMTP_PORT` | Which port to use, defaults to `587` | ### Other config | **Variable** | Description | | ------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `ADAPTORS_ICONS_PATH` | Directory the on-disk adaptor icon cache is written to. The official image sets this to `/app/priv/adaptor_icons`; outside the image it defaults to a subdirectory of the system temp directory. Either way, mount a persistent volume at this path so cached icons survive a restart. | | `ADAPTORS_LOCAL_REPO` | Path to an OpenFn adaptors checkout, used when `ADAPTORS_STRATEGY` is `local`. Comma-separate several. The first checkout that has a package wins. See [Adaptors](ADAPTORS.md). | | `ADAPTORS_NPM_GITHUB_REF` | Git ref of `OpenFn/adaptors` that adaptor icons are read from. Defaults to `main`. | | `ADAPTORS_NPM_GITHUB_URL` | Raw GitHub host that adaptor icons are fetched from. Defaults to `https://raw.githubusercontent.com`. | | `ADAPTORS_NPM_HTTP_TIMEOUT` | Receive timeout in milliseconds for npm registry, schema and icon requests. Defaults to `30000`. | | `ADAPTORS_NPM_JSDELIVR_URL` | CDN that adaptor credential schemas are fetched from. Defaults to `https://cdn.jsdelivr.net`. | | `ADAPTORS_NPM_REGISTRY_URL` | npm registry the adaptor catalogue is read from. Defaults to `https://registry.npmjs.org`. Set this and the two URLs above to use an internal mirror. See [Adaptors](ADAPTORS.md). | | `ADAPTORS_PATH` | Where you store your locally installed adaptors | | `ADAPTORS_REFRESH_INTERVAL_SECONDS` | How often, in seconds, the adaptor catalogue refreshes from npm. Defaults to one hour. Set to `0` to disable scheduled refreshes. See [Adaptors](ADAPTORS.md). | | `ADAPTORS_STRATEGY` | Where the adaptor catalogue comes from: `npm` (default) or `local`. See [Adaptors](ADAPTORS.md). | | `ALLOW_SIGNUP` | Set to `true` to enable user access to the registration page. Set to `false` to disable new user registrations and block access to the registration page.
Default is `false`. | | `CORS_ORIGIN` | A list of acceptable hosts for browser/cors requests (',' separated) | | `DATABASE_TCP_USER_TIMEOUT` | How long, in milliseconds, a database connection may go without its peer acknowledging data before the OS drops it. Lightning also enables TCP keepalive on these connections, so a connection whose network path has died is dropped within roughly this time even while it waits on a reply, including queries run with no timeout. A live query is unaffected. **Linux-only**: Lightning does not apply it on other platforms. Defaults to `DATABASE_TIMEOUT` + 5000 (`20000` with the default 15s query timeout). `0` disables it (and keepalive); other values must be between 1 and 2147483647, or boot fails. A value below `DATABASE_TIMEOUT` logs a warning at boot. | | `DISABLE_DB_SSL` | In production, the use of an SSL connection to Postgres is required by default.
Setting this to `"true"` allows unencrypted connections to the database. This is strongly discouraged in a real production environment. | | `DISABLE_DB_SSL_CERT_VERIFY` | When a SSL connection is used to connect to Postgres, the server's certificate will be verified by default.
Setting this to `"true"` disables certificate verification. This is strongly discouraged in a real production environment. | | `EMAIL_ADMIN` | This is used as the sender email address for system emails. It is also displayed in the menu as the support email. | | `EMAIL_SENDER_NAME` | This is displayed in the email client as the sender name for emails sent by the application. | | `ERLANG_NODE_DISCOVERY_VIA_POSTGRES_CHANNEL_NAME` | The name of the Postgresql channel that is used when Erlang node discovery via Postgres is enabled. Defaults to `lightning-cluster` if not set. | | `ERLANG_NODE_DISCOVERY_VIA_POSTGRES_ENABLED` | If set to `true`, Lightning will use Postgres to discover Erlang nodes. This strategy will be used in addition to other strategies that are in use. Default value is `false` | | `IDLE_TIMEOUT` | The number of seconds that must pass without data being received before the Lightning web server kills the connection. | | `IS_RESETTABLE_DEMO` | If set to `yes`, it allows this instance to be reset to the initial "Lightning Demo" state. Note that this will destroy _most_ of what you have in your database! | | `K8S_HEADLESS_SERVICE` | This environment variable is automatically set if you're running on GKE and it is used to establish an Erlang node cluster. Note that if you're _not_ using Kubernetes, the "gossip" strategy is used to establish clusters. | | `LISTEN_ADDRESS` | The address the web server should bind to. Defaults to `127.0.0.1` to block access from other machines. | | `LOG_LEVEL` | How noisy you want the logs to be (e.g., `debug`, `info`) | | `METRICS_RUN_PERFORMANCE_AGE_SECONDS` | The oldest a run can be to be included in Run performance metrics. | | `METRICS_RUN_QUEUE_AGE_SECONDS` | The polling period for run queue metrics. | | `METRICS_STALLED_RUN_THRESHOLD_SECONDS` | The length of time a Run must be in the `available` state before it is considered stalled. | | `METRICS_UNCLAIMED_RUN_THRESHOLD_SECONDS` | The length of time a Run must be in the `available` state before it counts towards an impeded project. | | `MIX_ENV` | Your mix env, likely `prod` for deployment | | `NODE_ENV` | Node env, likely `production` for deployment | | `ORIGINS` | The allowed origins for web traffic to the backend | | `PER_WORKFLOW_CLAIM_LIMIT` | The maximum number of runs per workflow to consider during run claiming. This prevents any single workflow from dominating the processing queue while ensuring fairness across workflows.
Default is `50`. | | `CLAIM_WORK_MEM` | PostgreSQL `work_mem` setting for the run claim query. Helps optimize complex sorting operations. Set to a valid PostgreSQL memory value (e.g., `32MB`, `64MB`, `1GB`). Set to empty string to disable.
Default: disabled in dev/test, `32MB` in production. | | `PORT` | The port your Phoenix app runs on | | `PROMEX_DATASOURCE_ID` | The datasource that PromEx will use if configured to push initial dashboards to Grafana. Defaults to an empty string. | | `PROMEX_ENABLED` | Enables PromEx tracking and publishing of metrics if set to 'true' or 'yes'. Defaults to false. | | `PROMEX_ENDPOINT_SCHEME` | The scheme needed when connecting to the Promex Endpoint. Defaults to https. | | `PROMEX_EXPENSIVE_METRICS_ENABLED` | Certain metrics may be expensive to generate if Lightning is under load. If set to 'true', or 'yes' these metrics will be enabled. Defaults to 'false'. | | `PROMEX_GRAFANA_HOST` | This is used when PromEx is required to push data to a Grafana instance, e.g. when PromEx sets up initial dashboards. | | `PROMEX_GRAFANA_PASSWORD` | This is used when PromEx is required to push data to a Grafana instance, e.g. when PromEx sets up initial dashboards. | | `PROMEX_GRAFANA_USER` | This is used when PromEx is required to push data to a Grafana instance, e.g. when PromEx sets up initial dashboards. | | `PROMEX_METRICS_ENDPOINT_AUTHORIZATION_REQUIRED` | If set to 'true' or 'yes', the PromEx endpoint on Lightning will require consumers to provide credentials for authorization. Defaults to 'true'. | | `PROMEX_METRICS_ENDPOINT_TOKEN` | A Bearer token that the consumer of the promEx endpoint must provide in the Authorization header. Defaults to a random series of bytes. | | `PROMEX_UPLOAD_GRAFANA_DASHBOARDS_ON_START` | Instructs PromEx to upload iniital dashboards to a Grafana instance if set to 'true' or 'yes'. Defaults to false. | | `PRIMARY_ENCRYPTION_KEY` | A base64 encoded 32 character long string.
See [Encryption](#encryption). | | `QUEUE_RESULT_RETENTION_PERIOD_MINUTES` | The number of minutes to keep completed (successful) `ObanJobs` in the queue (not to be confused with runs and/or history) | | `SECRET_KEY_BASE` | A secret key used as a base to generate secrets for encrypting and signing data. | | `SENTRY_DSN` | If using Sentry for error monitoring, your DSN | | `SERVICE_ACCOUNT_PUBLIC_KEY` | An RSA public key, as PEM encoded in unpadded base64, for a service account that configures this instance through its API. While it is set, first setup is turned off. Lightning will not start if the value is not an RSA public key of at least 2048 bits. See [Service accounts](SERVICE_ACCOUNTS.md). | | `URL_HOST` | The host used for writing URLs (e.g., `demo.openfn.org`) | | `URL_PORT` | The port, usually `443` for production | | `URL_SCHEME` | The scheme for writing URLs (e.g., `https`) | | `USAGE_TRACKER_HOST` | The host that receives usage tracking submissions
(defaults to https://impact.openfn.org) | | `USAGE_TRACKING_DAILY_BATCH_SIZE` | The number of days that will be reported on with each run of `UsageTracking.DayWorker`. This will only have a noticeable effect in cases where there is a backlog or where reports are being generated retroactively (defaults to 10). | | `USAGE_TRACKING_ENABLED` | Enables the submission of anonymized usage data to OpenFn (defaults to `true`) | | `USAGE_TRACKING_RESUBMISSION_BATCH_SIZE` | The number of failed reports that will be submitted on each resubmission run (defaults to 10) | | `USAGE_TRACKING_RUN_CHUNK_SIZE` | The size of each batch of runs that is streamed from the database when generating UsageTracking reports (default 100). Decreasing this may decrease memory consumption when generating reports. | | `USAGE_TRACKING_UUIDS` | Indicates whether submissions should include cleartext UUIDs or not. Options are `cleartext` or `hashed_only`, with the default being `hashed_only`. | | `REQUIRE_EMAIL_VERIFICATION` | Indicates whether user email addresses should be verified. Defaults to `false`. | ### AI Chat ๐Ÿงช **Experimental** Lightning can be configured to use an AI chatbot for user interactions. See [openfn/apollo](https://github.com/OpenFn/apollo) for more information on the Apollo AI service. The following environment variables are required: - `AI_ASSISTANT_API_KEY` - API key to use the assistant. This currently requires an Anthropic key. - `APOLLO_ENDPOINT` - the endpoint for the OpenFn Apollo AI service. Three optional variables control how long Lightning waits on Apollo. Each one has a default, so set them only if those defaults do not suit your deployment. - `APOLLO_CONNECT_TIMEOUT_MS` - how long to wait to reach Apollo at all. Defaults to 5000. Reaching Apollo happens inside the idle budget, so a value above `APOLLO_IDLE_TIMEOUT_MS` never takes effect; Lightning warns at boot if you set one. - `APOLLO_IDLE_TIMEOUT_MS` - the longest acceptable silence, both before the first byte of an answer and between the chunks that follow. Defaults to 30000. - `APOLLO_REQUEST_TIMEOUT_MS` - the longest one whole request may take, however steadily it is streaming. Defaults to 300000. The idle default assumes Apollo v3.1.1 or later, which sends a keepalive every 15 seconds. On an older Apollo a stream can go quiet for minutes while a model is thinking, and 30 seconds will cut it off, so raise `APOLLO_IDLE_TIMEOUT_MS` or upgrade Apollo. The three added together, plus a ten-second buffer, bound how long one AI job may run, and that has to stay under Oban's shutdown grace period of six minutes. Raising them past it means a deploy landing on a running answer kills it with nothing left to report the failure, so Lightning warns at boot if the sum gets too close. Note that the longer grace period also makes rolling restarts slower, since Oban now waits up to six minutes for a running job rather than two. `APOLLO_TIMEOUT` is the old name for `APOLLO_IDLE_TIMEOUT_MS`. On the wire it only ever covered the silence, never the other two. It is no longer read, and Lightning logs a warning at boot if it is still set. ### OAuth credential connections (Google, Salesforce, etc.) OAuth clients that **jobs** use to connect to external systems (Google Sheets, Salesforce, and similar) are no longer configured via environment variables. They are registered in the UI under **Credentials โ†’ OAuth clients** and scoped to the projects that use them; the client id, secret, and redirect/callback URL are entered in that form. > The older `GOOGLE_CLIENT_ID` / `GOOGLE_CLIENT_SECRET` and > `SALESFORCE_CLIENT_ID` / `SALESFORCE_CLIENT_SECRET` environment variables are > no longer read by Lightning and can be removed from your deployment. ### OAuth Provider Egress When a user configures an OAuth client, they supply the provider's endpoint URLs (authorization, token, userinfo, revocation, introspection). Lightning makes server-side requests to those URLs during credential setup and automatic token refresh, which makes them a Server-Side Request Forgery (SSRF) sink: a low-privilege user could point an endpoint at an internal service or a cloud metadata address (`169.254.169.254`) and use the server as a proxy. To prevent this, all outbound OAuth requests are routed through an egress guard that resolves the endpoint hostname, rejects any address in an internal or reserved range, and pins the connection to the validated IP (so DNS cannot swap in an internal address after the check). This is **on by default** and requires no configuration; internal endpoints are blocked and the request fails with a generic network error. | **Variable** | **Description** | **Default** | | ------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------: | | `OAUTH_PROVIDER_ALLOWED_HOSTS` | Comma-separated list of hostnames that bypass the internal-range block. Only set this if your OAuth provider legitimately lives on an internal network. | _(empty)_ | Only add a host here if you are self-hosting an OAuth provider (or a proxy to one) on an address the guard would otherwise block, and you trust it. Matching is by **hostname string**, case-insensitive โ€” e.g. `OAUTH_PROVIDER_ALLOWED_HOSTS=idp.internal.example,localhost`. An allow-listed host is still resolved, but its resolved IP is not checked against the block list, so keep the list as small as possible. ### Channel Egress Lightning can proxy requests through to a channel's configured upstream via the `/channels/:id/...` reverse proxy (powered by the `philter` dependency). Because the upstream URL is operator-supplied, that proxy is a Server-Side Request Forgery (SSRF) sink: a request could be pointed at an internal service or a cloud metadata address (`169.254.169.254`) and use the server as a proxy. To prevent this, the channel proxy runs an egress guard that resolves the upstream hostname and rejects any address in a private, loopback, link-local, or otherwise reserved range. This is **on by default** and requires no configuration; with neither variable set, blocking is on and nothing is allow-listed. | **Variable** | **Description** | **Default** | | -------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------: | | `CHANNEL_BLOCK_PRIVATE_NETWORKS` | Whether to reject channel upstreams that resolve to private/loopback/link-local/reserved IP ranges (`true`/`false`/`yes`/`no`). Set false to allow them. | `true` | | `CHANNEL_ALLOWED_HOSTS` | Comma-separated list of hostnames that bypass the block entirely โ€” the escape hatch. | _(empty)_ | The allow-list is checked **first**. A host in `CHANNEL_ALLOWED_HOSTS` is allowed even if it resolves to a private IP, and even while `CHANNEL_BLOCK_PRIVATE_NETWORKS=true`. So the common safe setup is to leave blocking on (the default) and allow-list just the specific internal host you need. With `CHANNEL_BLOCK_PRIVATE_NETWORKS=false` the allow-list is moot โ€” everything is allowed. Matching is by **exact hostname only**, case-insensitive, with a single trailing dot ignored. There is **no CIDR range and no wildcard support**; a literal IP entry matches only if the channel URL uses that exact IP string. Entries are bare hostnames โ€” an entry containing a URL scheme (`://`), a path (`/`), internal whitespace, or `@` is rejected at boot with a clear error naming the bad entry. For example: `CHANNEL_ALLOWED_HOSTS=api.internal.example,localhost`. ### Webhook Retry Configuration Lightning automatically retries webhook processing on **transient database connection errors** using exponential backoff. This helps prevent data loss during brief database outages. The following environment variables control webhook retry behavior: | **Variable** | **Description** | **Default** | | -------------------------------- | ------------------------------------------------------------------------------------------------ | ----------: | | `WEBHOOK_RETRY_MAX_ATTEMPTS` | Maximum number of attempts (the first attempt runs immediately; backoffs occur between retries). | `5` | | `WEBHOOK_RETRY_INITIAL_DELAY_MS` | Initial backoff delay in milliseconds. | `100` | | `WEBHOOK_RETRY_MAX_DELAY_MS` | Maximum backoff delay in milliseconds. | `10000` | | `WEBHOOK_RETRY_BACKOFF_FACTOR` | Multiplier for exponential backoff (each delay ร— this factor, up to the max delay). | `2` | | `WEBHOOK_RETRY_TIMEOUT_MS` | Total time budget for all attempts (including sleeps) in milliseconds. | `60000` | | `WEBHOOK_RETRY_JITTER` | Whether to add \~0โ€“25% randomization to each delay to avoid thundering herd (`true`/`false`). | `true` | **How backoff works (defaults, jitter off):** attempt 1 runs immediately; on failure we sleep and retry up to `WEBHOOK_RETRY_MAX_ATTEMPTS - 1` times with exponential delays starting at `WEBHOOK_RETRY_INITIAL_DELAY_MS` and multiplying by `WEBHOOK_RETRY_BACKOFF_FACTOR`, capped by `WEBHOOK_RETRY_MAX_DELAY_MS`. Example sequence: `100ms โ†’ 200ms โ†’ 400ms โ†’ 800ms` (four sleeps for five total attempts), stopping sooner if `WEBHOOK_RETRY_TIMEOUT_MS` elapses. **Server timeout alignment:** Set your Phoenix `IDLE_TIMEOUT` to be **at least** `WEBHOOK_RETRY_TIMEOUT_MS + 15000` (in milliseconds) so long retries finish before the connection is closed.