View Source OpenFn/Lightning (beta)
OpenFn/Lightning is a fully open source, workflow automation platform that's used to automate critical business processes and integrate information systems. From last-mile services to national-level reporting, it boosts efficiency & effectiveness while enabling secure, stable, scalable interoperability at all levels. Read more about Lightning on OpenFn/Docs.
Lightning is the latest "v2" of OpenFn, the Digital Public Good for workflow automation. Use Lightning to visually build, execute and manage workflows.
- The latest version of the OpenFn technology - first launched in 2014, now tried and tested by NGOs in over 40 countries
- Fully open source (no premium features or community edition, you get the same product whether using SaaS or self-hosted)
- Recognised as a Digital Public Good by the DPGA and a Global Good for Health by Digital Square
- Sign up for the BETA
- Getting started
- Generate the documentation
- Security and standards
- Server specs for self-hosting
- Questions or feedback?
Watch a quick demo here: https://www.youtube.com/watch?v=BNaxlHAWb5I
*Note that the demo app refreshes daily, so do not configure workflows you want to save.
Plan and build workflows using Lightning's visual interface to quickly define when, where and what you want your automation to do.
Use our CLI to quickly build, edit and deploy projects from the comfort of your own code editor.
Monitor all workflow activity in one place.
- Filter and search runs to identify issues that need addressing and follow how a specific request has been processed
- Configure alerts to be notified on run failures
- Receive a project digest for a daily/weekly/monthly summary of your project activity
Manage users and access by project
Authorization is a central part of Lightning. As such, users are given different roles which determine what level of access they have for resources in the application. For more details about roles and permissions in Lightning, please refer to our documentation.
View our public roadmap here.
- If you only want to RUN Lightning on your own server, we recommend using Docker.
- If you want to DEPLOY Lightning, we recommend Docker builds and Kubernetes.
- If you want to CONTRIBUTE to the project, we recommend running Lightning on your local machine.
- Install the latest version of Docker
- Clone this repo using git
- Copy the
docker compose run --rm web mix ecto.migrate
By default the application will be running at localhost:4000.
You can then rebuild and run with
docker compose build and
docker compose up. See "Problems with Docker" for
additional troubleshooting help. Note that you can also create your own
docker-compose.yml file, configuring a postgres database and using a
Head to the Deploy section of our docs site to get started.
For technical guidelines, see deployment considerations for more detailed information.
git clone firstname.lastname@example.org:OpenFn/Lightning.git # or from YOUR fork! cd Lightning cp .env.example .env # and adjust as necessary!
Take note of database names and ports in particular—they've got to match across your Postgres setup and your ENVs. You can run lightning without any ENVs assuming a vanilla postgres setup (see below), but you may want to make adjustments.
If you're already using Postgres locally, create a new database called
lightning_dev, for example.
If you'd rather use Docker to set up a Postgres DB, create a new volume and image:
docker volume create lightning-postgres-data docker create \ --name lightning-postgres \ --mount source=lightning-postgres-data,target=/var/lib/postgresql/data \ --publish 5432:5432 \ -e POSTGRES_PASSWORD=postgres \ postgres:15.3-alpine docker start lightning-postgres
We use asdf to configure our local
environments. Included in the repo is a
.tool-versions file that is read by
asdf in order to dynamically make the specified versions of Elixir and Erlang
available. You'll need asdf plugins for
asdf install # Install language versions mix local.hex mix deps.get mix local.rebar --force mix ecto.create # Create a development database in Postgres mix ecto.migrate [[ $(uname -m) == 'arm64' ]] && mix compile.rambo # Force compile rambo if on M1 mix lightning.install_runtime mix lightning.install_schemas mix lightning.install_adaptor_icons npm install --prefix assets
Lightning is a web app. To run it in interactive Elixir mode, start the development server by running with your environment variables by running:
iex -S mix phx.server
or if you have set up custom environment variables, run:
env $(cat .env | grep -v "#" | xargs ) iex -S mix phx.server
Once the server has started, head to
in your browser.
Before the first time running the tests, you need a test database setup.
MIX_ENV=test mix ecto.create
And then after that run the tests using:
MIX_ENV=test mix test
We also have
test.watch installed which can be used to rerun the tests on file
We are using k6 to benchmark Lightning. Under
folder you can find a script for benchmarking Webhook Workflows.
See Benchmarking for more detailed information.
For troubleshooting custom environment variable configuration it's important to know how an Elixir app loads and modifies configuration. The order is as follows:
- Stuff in
- That is then modified (think: overwritten) by stuff your ENV-specific
- That is then modified by
runtime.exswhich is where you are allowed to use
init/2(if present in a child application) gets called (which takes the config which has been set in steps 1-3) when that child application is started during the parent app startup defined in
If you're having connecting issues with Postgres, check the database section of
.env to ensure the DB url is correctly set for your environment — note
that composing a DB url out of other, earlier declared variables, does not work
If you're getting this error on debian
==> earmark_parser Compiling 1 file (.yrl) /usr/lib/erlang/lib/parsetools-2.3.1/include/yeccpre.hrl: no such file or directory could not compile dependency :earmark_parser, "mix compile" failed. You can recompile this dependency with "mix deps.compile earmark_parser", update it with "mix deps.update earmark_parser" or clean it with "mix deps.clean earmark_parser"
You need to install erlang development environment
sudo apt install erlang-dev
refer to this issue
The build may not work on old versions of Docker and Docker compose. It has been tested against:
Docker version 20.10.17, build 100c701 Docker Compose version v2.6.0
mix compile.rambo on Apple Silicon (an Apple M1/M2,
aarch64-apple-darwin) and encountering the following error:
** (RuntimeError) Rambo does not ship with binaries for your environment. aarch64-apple-darwin22.3.0 detected Install the Rust compiler so a binary can be prepared for you. lib/mix/tasks/compile.rambo.ex:89: Mix.Tasks.Compile.Rambo.compile!/0 lib/mix/tasks/compile.rambo.ex:51: Mix.Tasks.Compile.Rambo.run/1 (mix 1.14.2) lib/mix/task.ex:421: anonymous fn/3 in Mix.Task.run_task/4 (mix 1.14.2) lib/mix/cli.ex:84: Mix.CLI.run_task/2
You can resolve this error by installing the Rust compiler using Homebrew. Run
the following command in your terminal:
brew install rust
If you have already compiled Rambo explicitly via
mix compile.rambo, and you
are still seeing the following error:
sh: /path_to_directory/Lightning/_build/dev/lib/rambo/priv/rambo: No such file or directory sh: line 0: exec: /path_to_directory/Lightning/_build/dev/lib/rambo/priv/rambo: cannot execute: No such file or directory
You can try renaming
If neither of the approaches above work, please raise an issue.
Starting from scratch
If you're actively working with docker, you start experiencing issues, and you would like to start from scratch you can clean up everything and start over like this:
# To remove any ignored files and reset your .env to it's example git clean -fdx && cp .env.example .env # You can skip the line below if you want to keep your database docker compose down --rmi all --volumes docker compose build --no-cache web && \ docker compose create --force-recreate docker compose run --rm web mix ecto.migrate docker compose up
For help getting started with your OpenFn/Lightning Project and Workflows configuration, check out the Lightning Quickstart Guidance on OpenFn/Docs.
First, thanks for being here! You're contributing to a digital public good that will always be free and open source and aimed at serving innovative NGOs, governments, and social impact organizations the world over! You rock. ❤️
If you'd like to contribute to this projects, follow the steps below:
Read through the existing issues, assign yourself to the issue you have chosen. Leave a comment on the issue to let us know you'll be working on it, and if you have any questions of clarifications that would help you get started ask them there - we will get back to you as soon as possible.
If there isn't already an issue for the feature you would like to contribute, please start a discussion in our community forum.
Clone the Lightning repository, then fork it.
Run through setting up your environment and make your changes.
Make sure you have written your tests and updated /CHANGELOG.md (in the 'Unreleased' section, add a short description of the changes you are making, along with a link to your issue).
Open a draft pull request by clicking "Contribute > Open Pull Request" from your forked repository. Fill out the pull request template (this will be added automatically for you), then make sure to self-review your code and go through the 'Review checklist'. Don't worry about the QA checkbox, our product manager Amber will tick that once she has reviewed your PR. You can leave any notes for the reviewer in a comment.
Once you're ready to submit a pull request, you can mark your draft PR as 'Ready for review' and assign @stuartc or @taylordowns2000.
You can generate the HTML and EPUB documentation locally using:
mix docs and opening
doc/index.html in your browser.
We use a host of common Elixir static analysis tools to help us avoid common pitfalls and make sure we keep everything clean and consistent.
In addition to our test suite, you can run the following commands:
mix format --check-formatted
Code formatting checker, run again without the
--check-formattedflag to have your code automatically changed.
Static analysis for type mismatches and other common warnings. See dialyxir.
Static analysis for consistency, and coding standards. See Credo.
Check for commonly known security exploits. See Sobelow.
MIX_ENV=test mix coveralls
Test coverage reporter. This command also runs the test suite, and can be used in place of
mix testwhen checking everything before pushing your code. See excoveralls.
For convenience there is a
verifymix task that runs all of the above and defaults the
For more guidance on security best practices for workflow automation implementations, check out OpenFn Docs: docs.openfn.org/documentation/getting-started/security
For recommend server specifications for self-hosting of Lightning, check out this Community topic: community.openfn.org/t/specs-for-self-hosting-lightning/292
If you have any questions, feedback, or issues, please: