Deployment

Head to the Deploy section of our docs site to get started.

See below for technical considerations and instructions.

Encryption

Lightning enforces encryption at rest for credentials, TOTP backup codes, and webhook trigger authentication methods, for which an encryption key must be provided when running in production.

The key is expected to be a randomized set of bytes, 32 long; and Base64 encoded when setting the environment variable.

There is a mix task that can generate keys in the correct shape for use as an environment variable:

mix lightning.gen_encryption_key
0bJ9w+hn4ebQrsCaWXuA9JY49fP9kbHmywGd5K7k+/s=

Copy your key (NOT THIS ONE) and set it as PRIMARY_ENCRYPTION_KEY in your environment.

Workers

Lightning uses external worker processes for executing Runs. There are three settings required to configure worker authentication.

WORKER_RUNS_PRIVATE_KEY: base64 encoded RSA private key
WORKER_LIGHTNING_PUBLIC_KEY: base64 encoded RSA public key
WORKER_SECRET: 256bit long shared secret used by the worker to sign JWTs

You can use the mix lightning.gen_worker_keys task to generate these for convenience.

For more information see the Workers documentation.

Environment Variables

Note that for secure deployments, it's recommended to use a combination of secrets and configMaps to generate secure environment variables.

Limits

WORKER_MAX_RUN_MEMORY_MB - how much memory (in MB) can a single run use?
RUN_GRACE_PERIOD_SECONDS - how long after the MAX_RUN_DURATION_SECONDS should the server wait for the worker to send back data on a run.
WORKER_MAX_RUN_DURATION_SECONDS - the maximum duration (in seconds) that workflows are allowed to run (keep this plus RUN_GRACE_PERIOD_SECONDS below your termination_grace_period if using kubernetes)
WORKER_CAPACITY - the number of runs a ws-worker instance will take on concurrently.
MAX_DATACLIP_SIZE_MB - the maximum size (in MB) of a dataclip created via the webhook trigger URL for a job. This limits the max request size via the JSON plug and may (in future) limit the size of dataclips that can be stored as run_results via the websocket connection from a worker.
MAX_CREDENTIAL_SENSITIVE_VALUES - the maximum number of sensitive values that can be stored in a credential. Defaults to 50.
MAX_SANDBOX_NESTING_DEPTH - the maximum depth at which sandboxes can be nested. A direct child sandbox is depth 1, a sandbox of a sandbox is depth 2, etc. Provision attempts beyond this depth return {:error, :nesting_too_deep}, and the Create Sandbox button on the Sandboxes page is disabled with a tooltip when the current project is at the cap. Defaults to 5. Set to 0 to disable sandbox creation entirely.

GitHub

Lightning enables connection to GitHub via GitHub Apps. The following GitHub repository permissions are needed for the GitHub app:

Resource	Access
Actions	Read and Write
Contents	Read and Write
Metadata	Read only
Secrets	Read and Write
Workflows	Read and Write

Ensure you set the following URLs:

Homepage URL: <app_url_here>
Callback URL for authorizing users: <app_url_here>/oauth/github/callback (Do NOT check the two checkboxes in this section requesting Device Flow and OAuth.)
Setup URL for Post installation: <app_url_here>/setup_vcs (Check the box for Redirect on update)

These environment variables will need to be set in order to configure the github app:

Variable	Description
`GITHUB_APP_ID`	the github app ID.
`GITHUB_APP_NAME`	the github app name. This is the name used in the public link. It is the downcased name with spaces replaced with hyphens
`GITHUB_APP_CLIENT_ID`	the github app Client ID
`GITHUB_APP_CLIENT_SECRET`	the github app Client Secret
`GITHUB_CERT`	the github app private key (Base 64 encoded)

You can access these from your github app settings menu. Also needed for the configuration is:

REPO_CONNECTION_SIGNING_SECRET - secret used to sign access tokens. This access token is used to authenticate requests made from the github actions. You can generate this using mix lightning.gen_encryption_key

Storage

Lightning can use a storage backend to store exports.

Variable	Description
`STORAGE_BACKEND`	the storage backend to use. (default is `local`)
`STORAGE_PATH`	the path to store files in. (default is `.`)

Supported backends:

local - local file storage
gcs - Google Cloud Storage

Google Cloud Storage

For Google Cloud Storage, the following environment variables are required:

Variable	Description
`GCS_BUCKET`	the name of the bucket to store files in
`GOOGLE_APPLICATION_CREDENTIALS_JSON`	A base64 encoded JSON keyfile for the service account with access to the bucket.

ℹ️ Note: The GOOGLE_APPLICATION_CREDENTIALS_JSON should be base64 encoded, currently Workload Identity is not supported.

Mail

Lightning can send emails for various reasons, such as password resets and alerts for failed runs.

In order to send emails, you need to set the MAIL_PROVIDER environment variable to one of the following:

local (the default)
mailgun
smtp

You will also want to set the EMAIL_ADMIN environment variable to the email address that will be used as the sender for system emails.

If you are planning on using the smtp provider, with TLS enabled, the current implementation has the following contraints:

Only TLS 1.3 is supported.
For the purposes of Server Name Indication (SNI), the hostname provided as the SMTP_RELAY is used. This means that the SMTP_RELAY value must be present in a SAN dNSName on the cert. Practically, this means that TLS is unlikely to work if an IP address, or an internal-only hostname is provided as the SMTP_RELAY value.

Mailgun

For mailgun, the following environment variables are required:

Variable	Description
`MAIL_PROVIDER`	Must be set to `mailgun`
`MAILGUN_API_KEY`	the mail gun api key
`MAILGUN_DOMAIN`	the mail gun domain

SMTP

For SMTP, the following environment variables are required:

Variable	Description
`MAIL_PROVIDER`	Must be set to `smtp`
`SMTP_USERNAME`	Username for your server
`SMTP_PASSWORD`	Password for the user
`SMTP_RELAY`	IP address or hostname
`SMTP_TLS`	Use TLS, defaults to `true`, options are `true`, `false`, `if_available`
`SMTP_PORT`	Which port to use, defaults to `587`

Other config

Variable	Description
`ADAPTORS_PATH`	Where you store your locally installed adaptors
`ALLOW_SIGNUP`	Set to `true` to enable user access to the registration page. Set to `false` to disable new user registrations and block access to the registration page.<br>Default is `false`.
`CORS_ORIGIN`	A list of acceptable hosts for browser/cors requests (',' separated)
`DISABLE_DB_SSL`	In production, the use of an SSL connection to Postgres is required by default.<br>Setting this to `"true"` allows unencrypted connections to the database. This is strongly discouraged in a real production environment.
`DISABLE_DB_SSL_CERT_VERIFY`	When a SSL connection is used to connect to Postgres, the server's certificate will be verified by default.<br> Setting this to `"true"` disables certificate verification. This is strongly discouraged in a real production environment.
`EMAIL_ADMIN`	This is used as the sender email address for system emails. It is also displayed in the menu as the support email.
`EMAIL_SENDER_NAME`	This is displayed in the email client as the sender name for emails sent by the application.
`ERLANG_NODE_DISCOVERY_VIA_POSTGRES_CHANNEL_NAME`	The name of the Postgresql channel that is used when Erlang node discovery via Postgres is enabled. Defaults to `lightning-cluster` if not set.
`ERLANG_NODE_DISCOVERY_VIA_POSTGRES_ENABLED`	If set to `true`, Lightning will use Postgres to discover Erlang nodes. This strategy will be used in addition to other strategies that are in use. Default value is `false`
`IDLE_TIMEOUT`	The number of seconds that must pass without data being received before the Lightning web server kills the connection.
`IS_RESETTABLE_DEMO`	If set to `yes`, it allows this instance to be reset to the initial "Lightning Demo" state. Note that this will destroy most of what you have in your database!
`K8S_HEADLESS_SERVICE`	This environment variable is automatically set if you're running on GKE and it is used to establish an Erlang node cluster. Note that if you're not using Kubernetes, the "gossip" strategy is used to establish clusters.
`LISTEN_ADDRESS`	The address the web server should bind to. Defaults to `127.0.0.1` to block access from other machines.
`LOG_LEVEL`	How noisy you want the logs to be (e.g., `debug`, `info`)
`METRICS_RUN_PERFORMANCE_AGE_SECONDS`	The oldest a run can be to be included in Run performance metrics.
`METRICS_RUN_QUEUE_AGE_SECONDS`	The polling period for run queue metrics.
`METRICS_STALLED_RUN_THRESHOLD_SECONDS`	The length of time a Run must be in the `available` state before it is considered stalled.
`METRICS_UNCLAIMED_RUN_THRESHOLD_SECONDS`	The length of time a Run must be in the `available` state before it counts towards an impeded project.
`MIX_ENV`	Your mix env, likely `prod` for deployment
`NODE_ENV`	Node env, likely `production` for deployment
`ORIGINS`	The allowed origins for web traffic to the backend
`PER_WORKFLOW_CLAIM_LIMIT`	The maximum number of runs per workflow to consider during run claiming. This prevents any single workflow from dominating the processing queue while ensuring fairness across workflows.<br>Default is `50`.
`CLAIM_WORK_MEM`	PostgreSQL `work_mem` setting for the run claim query. Helps optimize complex sorting operations. Set to a valid PostgreSQL memory value (e.g., `32MB`, `64MB`, `1GB`). Set to empty string to disable.<br>Default: disabled in dev/test, `32MB` in production.
`PORT`	The port your Phoenix app runs on
`PROMEX_DATASOURCE_ID`	The datasource that PromEx will use if configured to push initial dashboards to Grafana. Defaults to an empty string.
`PROMEX_ENABLED`	Enables PromEx tracking and publishing of metrics if set to 'true' or 'yes'. Defaults to false.
`PROMEX_ENDPOINT_SCHEME`	The scheme needed when connecting to the Promex Endpoint. Defaults to https.
`PROMEX_EXPENSIVE_METRICS_ENABLED`	Certain metrics may be expensive to generate if Lightning is under load. If set to 'true', or 'yes' these metrics will be enabled. Defaults to 'false'.
`PROMEX_GRAFANA_HOST`	This is used when PromEx is required to push data to a Grafana instance, e.g. when PromEx sets up initial dashboards.
`PROMEX_GRAFANA_PASSWORD`	This is used when PromEx is required to push data to a Grafana instance, e.g. when PromEx sets up initial dashboards.
`PROMEX_GRAFANA_USER`	This is used when PromEx is required to push data to a Grafana instance, e.g. when PromEx sets up initial dashboards.
`PROMEX_METRICS_ENDPOINT_AUTHORIZATION_REQUIRED`	If set to 'true' or 'yes', the PromEx endpoint on Lightning will require consumers to provide credentials for authorization. Defaults to 'true'.
`PROMEX_METRICS_ENDPOINT_TOKEN`	A Bearer token that the consumer of the promEx endpoint must provide in the Authorization header. Defaults to a random series of bytes.
`PROMEX_UPLOAD_GRAFANA_DASHBOARDS_ON_START`	Instructs PromEx to upload iniital dashboards to a Grafana instance if set to 'true' or 'yes'. Defaults to false.
`PRIMARY_ENCRYPTION_KEY`	A base64 encoded 32 character long string.<br>See Encryption.
`QUEUE_RESULT_RETENTION_PERIOD_MINUTES`	The number of minutes to keep completed (successful) `ObanJobs` in the queue (not to be confused with runs and/or history)
`SCHEMAS_PATH`	Path to the credential schemas that provide forms for different adaptors
`ADAPTORS_REGISTRY_JSON_PATH`	Path to adaptor registry file. When provided, the app will attempt to read from it then later fallback to the internet
`SECRET_KEY_BASE`	A secret key used as a base to generate secrets for encrypting and signing data.
`SENTRY_DSN`	If using Sentry for error monitoring, your DSN
`URL_HOST`	The host used for writing URLs (e.g., `demo.openfn.org`)
`URL_PORT`	The port, usually `443` for production
`URL_SCHEME`	The scheme for writing URLs (e.g., `https`)
`USAGE_TRACKER_HOST`	The host that receives usage tracking submissions<br>(defaults to https://impact.openfn.org)
`USAGE_TRACKING_DAILY_BATCH_SIZE`	The number of days that will be reported on with each run of `UsageTracking.DayWorker`. This will only have a noticeable effect in cases where there is a backlog or where reports are being generated retroactively (defaults to 10).
`USAGE_TRACKING_ENABLED`	Enables the submission of anonymized usage data to OpenFn (defaults to `true`)
`USAGE_TRACKING_RESUBMISSION_BATCH_SIZE`	The number of failed reports that will be submitted on each resubmission run (defaults to 10)
`USAGE_TRACKING_RUN_CHUNK_SIZE`	The size of each batch of runs that is streamed from the database when generating UsageTracking reports (default 100). Decreasing this may decrease memory consumption when generating reports.
`USAGE_TRACKING_UUIDS`	Indicates whether submissions should include cleartext UUIDs or not. Options are `cleartext` or `hashed_only`, with the default being `hashed_only`.
`REQUIRE_EMAIL_VERIFICATION`	Indicates whether user email addresses should be verified. Defaults to `false`.

AI Chat

🧪 Experimental

Lightning can be configured to use an AI chatbot for user interactions.

See openfn/apollo for more information on the Apollo AI service.

The following environment variables are required:

AI_ASSISTANT_API_KEY - API key to use the assistant. This currently requires an Anthropic key.
APOLLO_ENDPOINT - the endpoint for the OpenFn Apollo AI service.

Three optional variables control how long Lightning waits on Apollo. Each one has a default, so set them only if those defaults do not suit your deployment.

APOLLO_CONNECT_TIMEOUT_MS - how long to wait to reach Apollo at all. Defaults to 5000. Reaching Apollo happens inside the idle budget, so a value above APOLLO_IDLE_TIMEOUT_MS never takes effect; Lightning warns at boot if you set one.
APOLLO_IDLE_TIMEOUT_MS - the longest acceptable silence, both before the first byte of an answer and between the chunks that follow. Defaults to 30000.
APOLLO_REQUEST_TIMEOUT_MS - the longest one whole request may take, however steadily it is streaming. Defaults to 300000.

The idle default assumes Apollo v3.1.1 or later, which sends a keepalive every 15 seconds. On an older Apollo a stream can go quiet for minutes while a model is thinking, and 30 seconds will cut it off, so raise APOLLO_IDLE_TIMEOUT_MS or upgrade Apollo.

The three added together, plus a ten-second buffer, bound how long one AI job may run, and that has to stay under Oban's shutdown grace period of six minutes. Raising them past it means a deploy landing on a running answer kills it with nothing left to report the failure, so Lightning warns at boot if the sum gets too close. Note that the longer grace period also makes rolling restarts slower, since Oban now waits up to six minutes for a running job rather than two.

APOLLO_TIMEOUT is the old name for APOLLO_IDLE_TIMEOUT_MS. On the wire it only ever covered the silence, never the other two. It is no longer read, and Lightning logs a warning at boot if it is still set.

OAuth credential connections (Google, Salesforce, etc.)

OAuth clients that jobs use to connect to external systems (Google Sheets, Salesforce, and similar) are no longer configured via environment variables. They are registered in the UI under Credentials → OAuth clients and scoped to the projects that use them; the client id, secret, and redirect/callback URL are entered in that form.

The older GOOGLE_CLIENT_ID / GOOGLE_CLIENT_SECRET and SALESFORCE_CLIENT_ID / SALESFORCE_CLIENT_SECRET environment variables are no longer read by Lightning and can be removed from your deployment.

OAuth Provider Egress

When a user configures an OAuth client, they supply the provider's endpoint URLs (authorization, token, userinfo, revocation, introspection). Lightning makes server-side requests to those URLs during credential setup and automatic token refresh, which makes them a Server-Side Request Forgery (SSRF) sink: a low-privilege user could point an endpoint at an internal service or a cloud metadata address (169.254.169.254) and use the server as a proxy.

To prevent this, all outbound OAuth requests are routed through an egress guard that resolves the endpoint hostname, rejects any address in an internal or reserved range, and pins the connection to the validated IP (so DNS cannot swap in an internal address after the check). This is on by default and requires no configuration; internal endpoints are blocked and the request fails with a generic network error.

Variable	Description	Default
`OAUTH_PROVIDER_ALLOWED_HOSTS`	Comma-separated list of hostnames that bypass the internal-range block. Only set this if your OAuth provider legitimately lives on an internal network.	(empty)

Only add a host here if you are self-hosting an OAuth provider (or a proxy to one) on an address the guard would otherwise block, and you trust it. Matching is by hostname string, case-insensitive — e.g. OAUTH_PROVIDER_ALLOWED_HOSTS=idp.internal.example,localhost. An allow-listed host is still resolved, but its resolved IP is not checked against the block list, so keep the list as small as possible.

Channel Egress

Lightning can proxy requests through to a channel's configured upstream via the /channels/:id/... reverse proxy (powered by the philter dependency). Because the upstream URL is operator-supplied, that proxy is a Server-Side Request Forgery (SSRF) sink: a request could be pointed at an internal service or a cloud metadata address (169.254.169.254) and use the server as a proxy.

To prevent this, the channel proxy runs an egress guard that resolves the upstream hostname and rejects any address in a private, loopback, link-local, or otherwise reserved range. This is on by default and requires no configuration; with neither variable set, blocking is on and nothing is allow-listed.

Variable	Description	Default
`CHANNEL_BLOCK_PRIVATE_NETWORKS`	Whether to reject channel upstreams that resolve to private/loopback/link-local/reserved IP ranges (`true`/`false`/`yes`/`no`). Set false to allow them.	`true`
`CHANNEL_ALLOWED_HOSTS`	Comma-separated list of hostnames that bypass the block entirely — the escape hatch.	(empty)

The allow-list is checked first. A host in CHANNEL_ALLOWED_HOSTS is allowed even if it resolves to a private IP, and even while CHANNEL_BLOCK_PRIVATE_NETWORKS=true. So the common safe setup is to leave blocking on (the default) and allow-list just the specific internal host you need. With CHANNEL_BLOCK_PRIVATE_NETWORKS=false the allow-list is moot — everything is allowed.

Matching is by exact hostname only, case-insensitive, with a single trailing dot ignored. There is no CIDR range and no wildcard support; a literal IP entry matches only if the channel URL uses that exact IP string. Entries are bare hostnames — an entry containing a URL scheme (://), a path (/), internal whitespace, or @ is rejected at boot with a clear error naming the bad entry. For example: CHANNEL_ALLOWED_HOSTS=api.internal.example,localhost.

Webhook Retry Configuration

Lightning automatically retries webhook processing on transient database connection errors using exponential backoff. This helps prevent data loss during brief database outages.

The following environment variables control webhook retry behavior:

Variable	Description	Default
`WEBHOOK_RETRY_MAX_ATTEMPTS`	Maximum number of attempts (the first attempt runs immediately; backoffs occur between retries).	`5`
`WEBHOOK_RETRY_INITIAL_DELAY_MS`	Initial backoff delay in milliseconds.	`100`
`WEBHOOK_RETRY_MAX_DELAY_MS`	Maximum backoff delay in milliseconds.	`10000`
`WEBHOOK_RETRY_BACKOFF_FACTOR`	Multiplier for exponential backoff (each delay × this factor, up to the max delay).	`2`
`WEBHOOK_RETRY_TIMEOUT_MS`	Total time budget for all attempts (including sleeps) in milliseconds.	`60000`
`WEBHOOK_RETRY_JITTER`	Whether to add \~0–25% randomization to each delay to avoid thundering herd (`true`/`false`).	`true`

How backoff works (defaults, jitter off): attempt 1 runs immediately; on failure we sleep and retry up to WEBHOOK_RETRY_MAX_ATTEMPTS - 1 times with exponential delays starting at WEBHOOK_RETRY_INITIAL_DELAY_MS and multiplying by WEBHOOK_RETRY_BACKOFF_FACTOR, capped by WEBHOOK_RETRY_MAX_DELAY_MS. Example sequence: 100ms → 200ms → 400ms → 800ms (four sleeps for five total attempts), stopping sooner if WEBHOOK_RETRY_TIMEOUT_MS elapses.

Server timeout alignment: Set your Phoenix IDLE_TIMEOUT to be at least WEBHOOK_RETRY_TIMEOUT_MS + 15000 (in milliseconds) so long retries finish before the connection is closed.

← Previous Page Running Locally

Next Page → Benchmarking