Lightning.Adaptors (Lightning v2.19.0)
View SourceThe adaptor catalogue: which npm adaptors exist, their versions, schemas and icons.
A scheduler fetches the catalogue from the configured source and
persists it. Reads check an in-memory cache first and the database
second, and answer immediately: a read that finds nothing in a catalogue
which has never loaded is {:error, :not_ready}. Waiting for that first
load is opt-in, through ensure_loaded/2 and fetch_adaptor/3.
Adaptor specs
A spec is the name@version string a job stores, where the version is
a semver or one of two sentinels: latest resolves to the catalogue's
current version when the job runs, and local points the worker at an
adaptor on its own filesystem. The version may also be omitted.
Configuration
Under config :lightning, Lightning.Adaptors:
:strategy- the module that fetches from the source:refresh_interval- how often the scheduler re-checks the source, in milliseconds;0disables the periodic tick:first_load_timeout- how long a read waits for the initial load:cache_timeout_ms- how long a read waits for a cache fill:icon_path- where fetched icons are written
Defaults are in Lightning.Adaptors.Config.
Testing
Every function that talks to a running process takes the supervisor
name as an optional first argument, defaulting to
Lightning.Adaptors.Config.default_instance/0. Start a
Lightning.Adaptors.Supervisor under another name and pass that name
to run a test against its own catalogue and cache, or stub
default_instance/0 to make it the default for the test. The
recommended way to get a private instance is setup :isolated_adaptors
from Lightning.AdaptorTestHelpers, which does both.
Summary
Types
Why a catalogue read could not answer.
What an icon read can fail with on top of error/0: the fetched bytes
disagreeing with the row that described them, or the strategy failing to
fetch them at all, which it reports in its own terms.
Functions
Returns {:ok, {{latest_updated_at, count}, entries}}: every adaptor
with its full version list and icon URLs, rendered once per change
rather than per request, alongside the ETag basis for it.
Writes the catalogue to a JSON snapshot file. See
Lightning.Adaptors.Dump.dump_to_file/2.
Waits until the catalogue has loaded at least once, triggering the
first load if needed, bounded by :timeout (default
Lightning.Adaptors.Config.first_load_timeout/0).
Returns {:ok, adaptor} for the adaptor named name, waiting for the
catalogue's first load if it has never loaded, bounded by :timeout
(default Lightning.Adaptors.Config.first_load_timeout/0).
Returns the on-disk path of the adaptor's :square or :rectangle
icon, fetching it on the first request. {:error, :not_ready} against a
catalogue that has never loaded.
Returns the stored extension and sha256 of each icon shape for the
adaptor named name, without touching disk. See
Lightning.Adaptors.Store.icon_meta/0.
Returns every adaptor in the catalogue as Package structs, or
{:error, :not_ready} against a catalogue that has never loaded.
Splits an adaptor spec into {:ok, {name, version}}, with version nil
when the spec carries none, or {:error, :invalid_format} for a spec that
is not a package name plus an optional @version.
Starts a catalogue refresh, or joins one already running.
Refetches every adaptor's icons and updates those whose bytes changed.
Returns {:ok, %{updated: n, unchanged: m}}.
Refetches the adaptor named name from the source and persists it,
whether or not its version changed.
Resolves a legacy short adaptor name such as "http" to its full npm
package name, "@openfn/language-http", when the full name is in the
catalogue. A name the catalogue already knows comes back unchanged. So
does a name it knows in neither form, because a loaded catalogue that
has never heard of it is a real answer.
Returns the credential schema of the adaptor named pkg, as a JSON
binary. An adaptor with no schema yields "{}" and an unknown name
is {:error, :not_found}, or {:error, :not_ready} against a catalogue
that has never loaded.
Populates the catalogue from a JSON snapshot file. See
Lightning.Adaptors.Seed.seed_from_file/2.
Subscribes the calling process to catalogue update broadcasts.
Renders an adaptor spec for the worker: latest becomes the
catalogue's current version, local is kept, and a :local source
forces name@local. A nil spec renders as "".
Returns whether spec is a well-formed adaptor spec: a package name
plus an optional @version, with no newlines or shell metacharacters.
Types
@type error() :: :not_found | :not_ready | :timeout | :unavailable
Why a catalogue read could not answer.
:not_found- the catalogue has loaded and holds no such adaptor:not_ready- the catalogue has never loaded:timeout- the first load, or a cache fill, did not finish in time:unavailable- no Scheduler process is reachable
What an icon read can fail with on top of error/0: the fetched bytes
disagreeing with the row that described them, or the strategy failing to
fetch them at all, which it reports in its own terms.
Functions
@spec catalogue(atom()) :: {:ok, {{DateTime.t() | nil, non_neg_integer()}, [Lightning.Adaptors.Store.rendered_entry()]}} | {:error, error()}
Returns {:ok, {{latest_updated_at, count}, entries}}: every adaptor
with its full version list and icon URLs, rendered once per change
rather than per request, alongside the ETag basis for it.
One read, so the stamp always describes the entries it comes with.
Returns {:error, reason} on a backing-store failure; see error/0.
Writes the catalogue to a JSON snapshot file. See
Lightning.Adaptors.Dump.dump_to_file/2.
Waits until the catalogue has loaded at least once, triggering the
first load if needed, bounded by :timeout (default
Lightning.Adaptors.Config.first_load_timeout/0).
Returns :ok, or one of the fetch_adaptor/3 errors other than
:not_found.
@spec fetch_adaptor(atom(), String.t(), keyword()) :: {:ok, Lightning.Adaptors.Package.t()} | {:error, error()}
Returns {:ok, adaptor} for the adaptor named name, waiting for the
catalogue's first load if it has never loaded, bounded by :timeout
(default Lightning.Adaptors.Config.first_load_timeout/0).
Takes a bare package name, not a spec; see parse_spec/1.
Errors:
{:error, :not_found}- the loaded catalogue has no such adaptor{:error, :timeout}- the first load did not finish in time{:error, :unavailable}- no Scheduler process is reachable{:error, :not_ready}- the load ran but left the catalogue empty
@spec icon(atom(), String.t(), :square | :rectangle) :: {:ok, Path.t()} | {:error, error() | icon_error()}
Returns the on-disk path of the adaptor's :square or :rectangle
icon, fetching it on the first request. {:error, :not_ready} against a
catalogue that has never loaded.
@spec icon_meta(atom(), String.t()) :: {:ok, Lightning.Adaptors.Store.icon_meta()} | {:error, :not_found | :not_ready}
Returns the stored extension and sha256 of each icon shape for the
adaptor named name, without touching disk. See
Lightning.Adaptors.Store.icon_meta/0.
@spec packages(atom()) :: {:ok, [Lightning.Adaptors.Package.t()]} | {:error, error()}
Returns every adaptor in the catalogue as Package structs, or
{:error, :not_ready} against a catalogue that has never loaded.
Splits an adaptor spec into {:ok, {name, version}}, with version nil
when the spec carries none, or {:error, :invalid_format} for a spec that
is not a package name plus an optional @version.
Starts a catalogue refresh, or joins one already running.
Returns :ok as soon as the refresh is underway. With await: true,
blocks until the cycle completes, bounded by :timeout (default
Lightning.Adaptors.Config.first_load_timeout/0), and returns
{:ok, counts} or {:error, reason}; see
Lightning.Adaptors.Scheduler.await_refresh/2 for the counts.
@spec refresh( atom(), keyword() ) :: :ok | {:ok, Lightning.Adaptors.Scheduler.refresh_counts()} | {:error, term()}
@spec refresh_icons(atom()) :: {:ok, %{updated: non_neg_integer(), unchanged: non_neg_integer()}} | {:error, term()}
Refetches every adaptor's icons and updates those whose bytes changed.
Returns {:ok, %{updated: n, unchanged: m}}.
Refetches the adaptor named name from the source and persists it,
whether or not its version changed.
Resolves a legacy short adaptor name such as "http" to its full npm
package name, "@openfn/language-http", when the full name is in the
catalogue. A name the catalogue already knows comes back unchanged. So
does a name it knows in neither form, because a loaded catalogue that
has never heard of it is a real answer.
"raw" and "oauth" are sentinels, not adaptor names, and are returned
unchanged without consulting the catalogue.
Does not wait for a first load. A catalogue that has never loaded is
{:error, :not_ready}.
Returns the credential schema of the adaptor named pkg, as a JSON
binary. An adaptor with no schema yields "{}" and an unknown name
is {:error, :not_found}, or {:error, :not_ready} against a catalogue
that has never loaded.
Populates the catalogue from a JSON snapshot file. See
Lightning.Adaptors.Seed.seed_from_file/2.
Subscribes the calling process to catalogue update broadcasts.
Updates arrive as %{event: "adaptors_updated", payload: %{names: [...]}}
messages.
@spec to_wire(atom(), String.t() | nil) :: {:ok, String.t()} | {:error, :not_found | :timeout | :unavailable | :not_ready}
Renders an adaptor spec for the worker: latest becomes the
catalogue's current version, local is kept, and a :local source
forces name@local. A nil spec renders as "".
Returns whether spec is a well-formed adaptor spec: a package name
plus an optional @version, with no newlines or shell metacharacters.