View Source API Reference Lightning v2.10.4
Modules
Lightning keeps the contexts that define your domain and business logic.
Behaviour for implementing the Lightning interface.
The Accounts context.
Publishes and subscribe to events related to user accounts.
The User model.
User backup codes schema
The UserNotifier module.
User Time based OTPs schema
The UserToken model.
Registry process to query and maintain a list of adaptors available for writing jobs.
NPM API functions
The Adaptor Service is use to query and install adaptors in order to run jobs.
The AI assistant module.
The AI assistant limiter to check for AI query quota.
Client for communicating with the Apollo service.
Context for working with Audit records.
Macro module to add common model behaviour to a given Ecto model
Context module for dealing with external Auth Providers.
AuthProvider model
Dummy warmer which caches database rows every 30s.
Provides common functionality for handling OAuth authentication across different providers.
Defines a schema for OAuth token information.
Handles the specifics of the Google OAuth authentication process.
Module which wraps Oauth configuration and a WellKnown document into a convenient struct that can be used to authenticate users against any OIDC compliant provider.
Defines a behaviour for OAuth providers within the Lightning application, specifying a common interface for OAuth operations. This interface ensures consistency and interoperability among different authentication providers (e.g., Google, Salesforce) by defining a set of required functions that each provider must implement.
Handles OAuth interactions for generic providers, including token fetching, refreshing, and user information retrieval. This module uses Tesla to make HTTP requests configured with middleware appropriate for OAuth specific tasks.
Handles the specifics of the Salesforce OAuth authentication process.
Store module for caching Handlers.
A datastructure to fetch and hold information about a given OIDC/OAuth provider
Macros for building Lightning.
Module providing facilities to make calls to the OpenFn CLI.
Struct that wraps the output of an OpenFn CLI call.
Extra functions for Ecto.Changeset.
Access to collections of unique key-value pairs shared across multiple workflows.
Collection referenced by name associated to a project.
A key value entry of a collection bound to a project.
Centralised runtime configuration for Lightning.
Bootstrap the application environment.
Utility functions for working with the application environment.
The Credentials context.
Model for storing changes to Credentials
The Credential model.
Defines the Ecto schema for an OAuth client. This schema is responsible for representing OAuth client data in the database, including details such as client ID, client secret, and endpoints necessary for OAuth operations. It also links to associated users and projects through relational fields.
Model for storing changes to Oauth clients
Structure that can parse JsonSchemas (using ExJsonSchema) and validate
changesets for a given schema.
Provides facilities to dynamically create and validate a changeset for a given Schema
Functions to pull out sensitive values inside a credential.
Dashboard stats for a project and its workflows.
Aggregated metrics for a project.
Stats for each workflow.
Handles scrubbing of dataclips
Demo encapsulates logic for setting up a demonstration site.
Module that expose a function generating a complete and valid yaml string from a project and its workflows.
Allows handling user creation or registration atomically without relying on async events.
Allows adding, removing or claiming work to be executed by the Runtime.
Message for the limiters to communicate with the client.
Allows handling user creation or registration atomically without relying on async events.
Allows handling project creation atomically without relying on async events.
Rate limiting stub for Lightning.
Rate limiting for Lightning API endpoints.
Which user is making the request for a certain project.
Extension to customize the scheduling of workloads on Lightning Runtime.
Runtime limiting stub for Lightning.
Rate limiting for Lightning workloads that depends on Runtime.
A graph model for workflows transversal.
Common functions for the context
The Invocation context.
Ecto model for Dataclips.
Ecto model for run logs.
Query functions for working with Steps and Dataclips
Ecto model for Steps.
The Janitor is responsible for detecting runs that have been "lost" due to communication issues with the worker.
The Jobs context.
Contains the logic to manage kafka trigger and their associated pipelines.
Responsible for cleaning up stale TriggerKafkaMessageRecords entries. TriggerKafkaMessageRecords are used to deduplicate incoming messages from a Kafka cluster.
Listens for events related to Kafka triggers and updates the affected pipeline process by enabling, reloading or disabling it.
Contains the logic to persist a Kafka messages as a WorkOrder, Run and Dataclip.
This module contains functionality to recover Kafka messages that have been persisted to the file system. It should only be used to process files that originate from a trusted source (i.e. Lightning's Kafka pipeline.)
Broadway pipeline that processes messages from Kafka clusters and persists the received messages if they are not duplicating a previous message.
Supervisor of the BroadwayKafka pipelines that are created for each enabled Kafka trigger.
Starts all the processes needed to pull data from Kafka clusters and then generate work orders based on the messages received.
Persists a lightweight represenattion of a message that has been recevied from a Kafka cluster, for the purpose of deduplication.
A custom type to handle JSON log messages.
Retrieves metadata for a given credential and adaptor using the OpenFn CLI.
Generates a random names.
Manages operations for OAuth clients within the Lightning application, providing functions to create, retrieve, update, and delete OAuth clients, as well as managing their associations with projects and handling audit trails for changes.
The Oban Manager
The Bodyguard Policy module for Collections.
The Bodyguard Policy module for dataclips actions
The Bodyguard Policy module for Exports
This module defines a unique interface managing authorizations in Lightning.
The Bodyguard Policy module for projects members roles.
The Bodyguard Policy module for users roles.
The Bodyguard Policy module for users roles.
The Projects context.
Generate Audit changesets for selected changes to project settings.
Events for Projects changes.
The project files module
Project model
Join table to assign credentials to a project
Manages the relationship between OAuth clients and projects, acting as a join table.
This module defines the schema and validations needed for creating and managing
associations between OauthClient and Project. It ensures that an OAuth client
can be associated with a project, facilitating the management of access permissions
and settings specific to a project.
Represents a summarized view of a project for a user, used in the project overview table.
Join table to assign users to a project
Provides functions for importing projects. This module is used by the provisioning HTTP API.
Be sure to add the following to finish setting up PromEx
Used for executing DB release tasks when run in production without Mix installed.
Ecto model for Runs.
Ecto model for an the Steps in a Run.
Gathers operations to create, update and delete Runs.
Handler modules for working with runs.
Schema to validate the input attributes of a completed run.
Schema to validate the input attributes of a completed step.
Schema to validate the input attributes of a started run.
Schema to validate the input attributes of a started step.
Metrics callbacks implementation for the PromEx plugin.
Query functions for working with Runs
Allows adding, removing or claiming work to be executed by the Runtime.
Options that are passed to the worker to control configurable limits and behaviors during run execution and reporting.
Agent facility to consume STDOUT/STDERR byte by byte.
Internal datastructure to hold and process new bytes for a list of characters.
Locates and runs the Runtime server. Added in order to ease development and default installations of Lightning
Behaviour for runtime clients to improve testability.
Utilities for securely encoding serializable structs, lists and strings into URL-safe strings.
Defines the database schema and primary key type for Thunderbolt schemas.
Process used to scrub strings of sensitive information.
Allows handling user creation or registration atomically without relying on async events.
Maps the module to be used for the extension service.
Allows handling project creation atomically without relying on async events.
Adapter to call the extension for rate limiting.
Adapter to call the extension for selecting Runtime workloads.
Adapter to call the extension for limiting Runtime workloads.
Demo encapsulates logic for setting up a demonstration site.
SetupUtils encapsulates logic for setting up initial data for various sites.
Time ticker to assure time progress/sequence specially for multiple logs.
The external storage module.
The behaviour for storage backends.
A storage backend module for handling file storage in Google Cloud Storage (GCS).
A storage backend module for handling local file storage.
This module provides functionality for managing the storage and retrieval of project files.
A TaskWorker with concurrency limits.
Token generation, verification and validation.
A custom DateTime type for Ecto that uses millisecond Unix timestamps.
The UsageTracking context.
Client for Usage Tracker service
Configuration for the creation of daily reports
Worker to manage per-day report generation
A github client to make unauthenticated HTTP requests to Github.
Builds project-related metrics.
Report submitted to Usage Tracker
Builds data set for submission to Usage Tracker
Worker to generate report for given day
Utility module to abstract dealing with some of the Tesla plumbing
Worker to find reports that have failed submissions and enqueue jobs to reprocess them.
Worker to resubmit report that has failed submission.
Supports the generation of metrics related to runs and steps.
Contains queries used to determine user-related metrics.
Returns counts for the various user-related metrics.
Query module to support workflow metrics while allowing for fewer test permutations.
Builds workflow-related metrics.
Utility functions for cryptographic operations.
Extra validators for Ecto.Changeset.
Module for handling the encryption and decryption of database fields.
Boundary module for handling Version control activities for project, jobs workflows etc Use this module to create, modify and delete connections as well as running any associated sync jobs
Tesla github http client we use this to make any network requests to github from Lightning
GitHub Error exception
A module that uses Joken to handle building and signing application
tokens for communicating with github
Ecto model for project repo connections
Provides functionality for managing webhook authentication methods.
Ecto model for WorkOrders.
Context for creating WorkOrders.
Module to log history export actions as audit events.
This module handles the export of work orders for a given project. The export process is performed asynchronously using the Oban background job system.
A model is used to build WorkOrders with custom input data.
Query functions for the Lightning.WorkOrders module.
Enqueue multiple work orders for retry.
This module is used to parse search parameters for work orders and provide a query to the database.
Lightning uses external worker processes to execute workflow jobs.
JWT token configuration to verify if workers work is legit.
JWT token configuration to authenticate workers.
The Workflows context.
Generate Audit changesets for selected changes to workflows.
Ecto model for Workflow Edges.
Ecto model for Jobs.
Represents a node in a workflow graph.
Handles user presence tracking within the Workflow canvas page.
Query module for finding Jobs.
The Scheduler is responsible for finding jobs that are ready to run based on their cron schedule, and then running them.
Ecto model for Workflow Snapshots.
Ecto model for Triggers.
Responsible for the publishing of and subscription to trigger-related events.
Configuration of Kafka Triggers.
The Lightning.Workflows.WebhookAuthMethod module defines the schema for webhook authentication methods and provides functionalities to handle them.
Model for storing changes to WebhookAuthMethod
Ecto model for Workflows.
The entrypoint for defining your web interface, such as controllers, views, channels and so on.
Helpers for the API views
A LiveView component for displaying an account confirmation modal.
LiveView for listing Audit events
Form Component for working with a single Job
LiveView for listing and managing Projects
LiveView for user backup codes.
Renders changesets as JSON.
Helper functions for channels
UI component to render a pill to create tags.
Menu components to render menu items for project and user/profile pages.
A modal component that can be used to display a modal on the page.
A temporary module that will serve as a place to put new inputs that conform with the newer CoreComponents conventions introduced in Phoenix 1.7.
UI component to render a pill to create tags.
Components for rendering Logs and Dataclips.
Form Component for working with a single Credential
This module provides helper functions for managing project associations within credential live views, allowing dynamic assignment and update of projects associations to credentials and / or oauth clients.
LiveView for listing and managing credentials
Form Component for working with a single Credential
Form Component for working with a single dataclip
LiveView for showing a single dataclip.
Translates controller action results into valid Plug.Conn responses.
Superuser setup liveview
Conveniences for building forms.
A module providing Internationalization with a gettext-based API.
LiveView Hooks
Ensures common assigns are applied to all LiveViews attaching this hook.
Component allowing selecting an adaptor and it's version
Component allowing selecting a credential or creating a new one via a modal.
A live component for managing cron setup in a form.
General purpose LiveView helper functions
Component for rendering content inside layout without full DOM patch.
A set of helper functions to encodes state and coordinate OAuth callbacks back to a LiveView component.
Pagination Components
Dynamically initialize Plugs that don't accept dynamic configs in :prod ENV.
Authenticates api calls based on JWT bearer token.
Plug to conditionally block specified routes based on configuration flags and custom messages.
Plug to redirect HTTP requests to /first_setup if there are no
superusers in the system yet.
A plug for redirecting requests to a specified URL.
A Plug to authenticate and authorize requests based on paths starting with '/i/'. It verifies the presence of correct API keys or Basic Authentication credentials.
LiveView for user profile page.
Form component update profile email and password
Component to enable MFA on a User's account
This schema is used for building the changeset for adding new collaborators to a project.
It is mirroring the Project -> ProjectUser relationship.
Form Component for working with a single Job
LiveView for listing and managing Projects
This schema is used for building the changeset for adding new collaborators to a project.
It is mirroring the Project -> ProjectUser relationship.
Liveview for project access denied error messages
Index Liveview for project settings
Implement custom authorization for PromEx metrics endpoint as per https://hexdocs.pm/prom_ex/1.1.1/PromEx.Plug.html
LiveView for re-authentication page.
Convenience functions for generating paths.
The router for Lightning.
Phoenix channel to interact with Runs.
Index Liveview for Runs
Rerun job component
Work Order component
Assorted metrics to collect during runtime.
LiveView for listing and managing tokens
The UserAuth controller.
LiveView for editing a single job, which inturn uses LightningWeb.JobLive.BigFormComponent
for common functionality.
Form component for creating and editing users
Index page for listing users
Helper functions to deal with forms and query params.
Websocket channel to handle when workers join or claim something to run.
Helper functions for the Workflow LiveViews.
Various function for reconciling changes to a workflow params map.
The Oban Pruner removes completed Oban jobs. It leaves everything else for manual inspection.
Mix Tasks
Helper to generate a unique encryption key for Vault
Helper to generate the private and public keys for worker authentication
Installs the adaptor icons
Installs the following NodeJS packages
Install the credential json schemas Use --exclude language-package1, language-package2 to exclude specific packages