Lightning.Adaptors (Lightning v2.19.0)

View Source

The adaptor catalogue: which npm adaptors exist, their versions, schemas and icons.

A scheduler fetches the catalogue from the configured source and persists it. Reads check an in-memory cache first and the database second, and answer immediately: a read that finds nothing in a catalogue which has never loaded is {:error, :not_ready}. Waiting for that first load is opt-in, through ensure_loaded/2 and fetch_adaptor/3.

Adaptor specs

A spec is the name@version string a job stores, where the version is a semver or one of two sentinels: latest resolves to the catalogue's current version when the job runs, and local points the worker at an adaptor on its own filesystem. The version may also be omitted.

Configuration

Under config :lightning, Lightning.Adaptors:

  • :strategy - the module that fetches from the source
  • :refresh_interval - how often the scheduler re-checks the source, in milliseconds; 0 disables the periodic tick
  • :first_load_timeout - how long a read waits for the initial load
  • :cache_timeout_ms - how long a read waits for a cache fill
  • :icon_path - where fetched icons are written

Defaults are in Lightning.Adaptors.Config.

Testing

Every function that talks to a running process takes the supervisor name as an optional first argument, defaulting to Lightning.Adaptors.Config.default_instance/0. Start a Lightning.Adaptors.Supervisor under another name and pass that name to run a test against its own catalogue and cache, or stub default_instance/0 to make it the default for the test. The recommended way to get a private instance is setup :isolated_adaptors from Lightning.AdaptorTestHelpers, which does both.

Summary

Types

Why a catalogue read could not answer.

What an icon read can fail with on top of error/0: the fetched bytes disagreeing with the row that described them, or the strategy failing to fetch them at all, which it reports in its own terms.

Functions

Returns {:ok, {{latest_updated_at, count}, entries}}: every adaptor with its full version list and icon URLs, rendered once per change rather than per request, alongside the ETag basis for it.

Writes the catalogue to a JSON snapshot file. See Lightning.Adaptors.Dump.dump_to_file/2.

Waits until the catalogue has loaded at least once, triggering the first load if needed, bounded by :timeout (default Lightning.Adaptors.Config.first_load_timeout/0).

Returns {:ok, adaptor} for the adaptor named name, waiting for the catalogue's first load if it has never loaded, bounded by :timeout (default Lightning.Adaptors.Config.first_load_timeout/0).

Returns the on-disk path of the adaptor's :square or :rectangle icon, fetching it on the first request. {:error, :not_ready} against a catalogue that has never loaded.

Returns the stored extension and sha256 of each icon shape for the adaptor named name, without touching disk. See Lightning.Adaptors.Store.icon_meta/0.

Returns every adaptor in the catalogue as Package structs, or {:error, :not_ready} against a catalogue that has never loaded.

Splits an adaptor spec into {:ok, {name, version}}, with version nil when the spec carries none, or {:error, :invalid_format} for a spec that is not a package name plus an optional @version.

Starts a catalogue refresh, or joins one already running.

Refetches every adaptor's icons and updates those whose bytes changed. Returns {:ok, %{updated: n, unchanged: m}}.

Refetches the adaptor named name from the source and persists it, whether or not its version changed.

Resolves a legacy short adaptor name such as "http" to its full npm package name, "@openfn/language-http", when the full name is in the catalogue. A name the catalogue already knows comes back unchanged. So does a name it knows in neither form, because a loaded catalogue that has never heard of it is a real answer.

Returns the credential schema of the adaptor named pkg, as a JSON binary. An adaptor with no schema yields "{}" and an unknown name is {:error, :not_found}, or {:error, :not_ready} against a catalogue that has never loaded.

Populates the catalogue from a JSON snapshot file. See Lightning.Adaptors.Seed.seed_from_file/2.

Subscribes the calling process to catalogue update broadcasts.

Renders an adaptor spec for the worker: latest becomes the catalogue's current version, local is kept, and a :local source forces name@local. A nil spec renders as "".

Returns whether spec is a well-formed adaptor spec: a package name plus an optional @version, with no newlines or shell metacharacters.

Types

error()

@type error() :: :not_found | :not_ready | :timeout | :unavailable

Why a catalogue read could not answer.

  • :not_found - the catalogue has loaded and holds no such adaptor
  • :not_ready - the catalogue has never loaded
  • :timeout - the first load, or a cache fill, did not finish in time
  • :unavailable - no Scheduler process is reachable

icon_error()

@type icon_error() ::
  {:icon_sha_mismatch, keyword()} | {:ext_mismatch, keyword()} | term()

What an icon read can fail with on top of error/0: the fetched bytes disagreeing with the row that described them, or the strategy failing to fetch them at all, which it reports in its own terms.

Functions

catalogue(sup \\ Config.default_instance())

@spec catalogue(atom()) ::
  {:ok,
   {{DateTime.t() | nil, non_neg_integer()},
    [Lightning.Adaptors.Store.rendered_entry()]}}
  | {:error, error()}

Returns {:ok, {{latest_updated_at, count}, entries}}: every adaptor with its full version list and icon URLs, rendered once per change rather than per request, alongside the ETag basis for it.

One read, so the stamp always describes the entries it comes with.

Returns {:error, reason} on a backing-store failure; see error/0.

dump_to_file(path, opts \\ [])

Writes the catalogue to a JSON snapshot file. See Lightning.Adaptors.Dump.dump_to_file/2.

ensure_loaded(sup \\ Config.default_instance(), opts \\ [])

@spec ensure_loaded(
  atom(),
  keyword()
) :: :ok | {:error, error()}

Waits until the catalogue has loaded at least once, triggering the first load if needed, bounded by :timeout (default Lightning.Adaptors.Config.first_load_timeout/0).

Returns :ok, or one of the fetch_adaptor/3 errors other than :not_found.

fetch_adaptor(sup \\ Config.default_instance(), name, opts \\ [])

@spec fetch_adaptor(atom(), String.t(), keyword()) ::
  {:ok, Lightning.Adaptors.Package.t()} | {:error, error()}

Returns {:ok, adaptor} for the adaptor named name, waiting for the catalogue's first load if it has never loaded, bounded by :timeout (default Lightning.Adaptors.Config.first_load_timeout/0).

Takes a bare package name, not a spec; see parse_spec/1.

Errors:

  • {:error, :not_found} - the loaded catalogue has no such adaptor
  • {:error, :timeout} - the first load did not finish in time
  • {:error, :unavailable} - no Scheduler process is reachable
  • {:error, :not_ready} - the load ran but left the catalogue empty

icon(sup \\ Config.default_instance(), pkg, shape)

@spec icon(atom(), String.t(), :square | :rectangle) ::
  {:ok, Path.t()} | {:error, error() | icon_error()}

Returns the on-disk path of the adaptor's :square or :rectangle icon, fetching it on the first request. {:error, :not_ready} against a catalogue that has never loaded.

icon_meta(sup \\ Config.default_instance(), name)

@spec icon_meta(atom(), String.t()) ::
  {:ok, Lightning.Adaptors.Store.icon_meta()}
  | {:error, :not_found | :not_ready}

Returns the stored extension and sha256 of each icon shape for the adaptor named name, without touching disk. See Lightning.Adaptors.Store.icon_meta/0.

packages(sup \\ Config.default_instance())

@spec packages(atom()) :: {:ok, [Lightning.Adaptors.Package.t()]} | {:error, error()}

Returns every adaptor in the catalogue as Package structs, or {:error, :not_ready} against a catalogue that has never loaded.

parse_spec(spec)

@spec parse_spec(String.t()) ::
  {:ok, {String.t(), String.t() | nil}} | {:error, :invalid_format}

Splits an adaptor spec into {:ok, {name, version}}, with version nil when the spec carries none, or {:error, :invalid_format} for a spec that is not a package name plus an optional @version.

refresh(opts)

Starts a catalogue refresh, or joins one already running.

Returns :ok as soon as the refresh is underway. With await: true, blocks until the cycle completes, bounded by :timeout (default Lightning.Adaptors.Config.first_load_timeout/0), and returns {:ok, counts} or {:error, reason}; see Lightning.Adaptors.Scheduler.await_refresh/2 for the counts.

refresh(sup \\ Config.default_instance(), opts \\ [])

@spec refresh(
  atom(),
  keyword()
) ::
  :ok | {:ok, Lightning.Adaptors.Scheduler.refresh_counts()} | {:error, term()}

refresh_icons(sup \\ Config.default_instance())

@spec refresh_icons(atom()) ::
  {:ok, %{updated: non_neg_integer(), unchanged: non_neg_integer()}}
  | {:error, term()}

Refetches every adaptor's icons and updates those whose bytes changed. Returns {:ok, %{updated: n, unchanged: m}}.

refresh_package(sup \\ Config.default_instance(), name)

@spec refresh_package(atom(), String.t()) :: :ok | {:error, :not_found | term()}

Refetches the adaptor named name from the source and persists it, whether or not its version changed.

resolve_name(sup \\ Config.default_instance(), name)

@spec resolve_name(atom(), String.t()) :: {:ok, String.t()} | {:error, error()}

Resolves a legacy short adaptor name such as "http" to its full npm package name, "@openfn/language-http", when the full name is in the catalogue. A name the catalogue already knows comes back unchanged. So does a name it knows in neither form, because a loaded catalogue that has never heard of it is a real answer.

"raw" and "oauth" are sentinels, not adaptor names, and are returned unchanged without consulting the catalogue.

Does not wait for a first load. A catalogue that has never loaded is {:error, :not_ready}.

schema(sup \\ Config.default_instance(), pkg)

@spec schema(atom(), String.t()) :: {:ok, String.t()} | {:error, error()}

Returns the credential schema of the adaptor named pkg, as a JSON binary. An adaptor with no schema yields "{}" and an unknown name is {:error, :not_found}, or {:error, :not_ready} against a catalogue that has never loaded.

seed_from_file(path, opts \\ [])

Populates the catalogue from a JSON snapshot file. See Lightning.Adaptors.Seed.seed_from_file/2.

subscribe_to_updates(sup \\ Config.default_instance())

@spec subscribe_to_updates(atom()) :: :ok | {:error, term()}

Subscribes the calling process to catalogue update broadcasts.

Updates arrive as %{event: "adaptors_updated", payload: %{names: [...]}} messages.

to_wire(sup \\ Config.default_instance(), spec)

@spec to_wire(atom(), String.t() | nil) ::
  {:ok, String.t()}
  | {:error, :not_found | :timeout | :unavailable | :not_ready}

Renders an adaptor spec for the worker: latest becomes the catalogue's current version, local is kept, and a :local source forces name@local. A nil spec renders as "".

valid_format?(spec)

@spec valid_format?(String.t()) :: boolean()

Returns whether spec is a well-formed adaptor spec: a package name plus an optional @version, with no newlines or shell metacharacters.