View Source Workers

The OpenFn Runtime used by Lightning is a Node.js application that runs on a server. It is responsible for executing the code that you write in your OpenFn jobs.

A Runtime Manager is a module or application that processes Runs, which contains reference to the Workflow, the starting point and the initial data.

Runs are enqueued, and Runtime Managers request work to be performed when they are ready.

History

In previous versions of OpenFn products, the server would invoke a NodeJS child process for each Run that needs to be executed. Deciding which Job to run in a workflow is decided after each run is completed.

The current approach to executing Runs, is that a worker checks out the entire run and executes it and all the jobs required.

The advantage of this approach is a significant reduction in latency of launching new workers for every Run to be processed.

Connecting Workers to Lightning

Our standard Runtime Manager uses WebSockets to communicate with Lightning.

However, since the workers are treated no differently from any other incoming traffic; there are two layers of security applied to anything that can access your data or process work.

We use two different JWTs to facilitate the follow:

Worker Token

This token is created by the worker and signed with a shared HS256 secret that Lightning uses to verify.

Run Token

This token is created by Lightning for every Run that a worker receives. The token is signed with a private key, and therefore cannot be generated by 3rd parties.

The token is scoped for a single Run and cannot be used to access anything outside of the scope of that Run.

Generating Keys

Lightning comes with a mix task to help generate keys:

mix lightning.gen_worker_keys

Configuring Lightning and Workers

There are three environment variables required to make everything work.

WORKER_RUNS_PRIVATE_KEY

Lightning only

A Base64 encoded private key in PEM format, used to generate JWTs granting workers permission to access a specific Run, the related Workflow and credentials.

[!IMPORTANT] This key should never be shared. If you suspect it has been compromised, generate a new one and reconfigure both Lightning and your workers.

WORKER_SECRET

Lightning and Workers

A 256bit long shared secret used by the worker to sign JWTs for authentication with the Lightning API.

WORKER_LIGHTNING_PUBLIC_KEY

Workers only

A Base64 encoded public key in PEM format derived from the private key. This is used by workers to verify Run Tokens coming from Lightning.

Troubleshooting

Problems with NodeJS DNS name resolution

NodeJS v17 changed how DNS name resolution is handled. In earlier versions, Node would reorder DNS lookup results so IPv4 addresses came before IPv6 addresses. In v17, Node started returning addresses in the same order provided by the resolver. The net effect is that your workers might fail to connect to Lightning if the native DNS resolver on your system is returning the loopback IPv6 address (::1) before the IPv4 address (127.0.0.1). In these cases, your worker might report an error similar to the following:

CRITICAL ERROR: could not connect to lightning at ws://localhost:4000/worker

If you experience this, it can be helpful to set the NODE_OPTIONS environment variable as shown below. This will force pre-v17 behavior of returning IPv4 addresses first.

NODE_OPTIONS=--dns-result-order=ipv4first

You can directly inspect what IP address Node returns for localhost by running the following command:

node -e 'dns.lookup("localhost", console.log)'