Project 3: Wildmeat
See the Wildmeat repository for the job code. These jobs run on the
Wildmeat OpenFn.org project. https://github.com/OpenFn/wcs-wildmeat
Wildmeat Kobo Integration 2020
EU SWM uses Kobo Toolbox to collect data on Rural Consumption across sites (see the Kobo form template here). OpenFn automates data integration between Kobo Toolbox and a Postgresql transitional database.
(1) Data Flows & OpenFn Jobs
See this data flow diagram. The following jobs are configured on OpenFn.org to run automatically.
- fetch-rc-submissions.js: On a timer-basis, OpenFn fetches all Kobo survey submissions where form
namecontains “Rural Consumption”. This job sends the Kobo data to OpenFn.org and automatically triggers the next job.
- rural-consumption-to-postgres.js: OpenFn automatically cleans, maps, & loads the Kobo survey data into structured tables in a Postgres Wildmeat database.
See here for the integration mapping specifications. These jobs leverage language-postgresql to perform
upsert() operations in the Postgres database.
- The jobs and mapping design are based on this DRC version of the Rural Consumption Kobo survey.
- All data cleaning will be done in Kobo Toolbox. Every time Kobo data is synced with the DB, it will overwrite the records saved there and use the above uuid to upsert existing records.
- Kobo forms only capture species
name. OpenFn was unable to determine a way to match this species
codeto the relevant species
taxonId, so OpenFn is only mapping the
codeuntil a master list of species codes and corresponding names can be provided.
- The jobs currently use hard-coded values for
site_id(‘1001’) as this information is currently not captured in the Kobo forms. All surveys will be mapped to these default
- See Wildmeat Map for a list of fields that were intentionally not mapped in these jobs (i.e.,
- All Kobo surveys to be fetched by OpenFn will contain “Rural Consumption” in the form name. If this criteria should change, job #1 should be updated.
- All surveys will have a default
sample unit(e.g., kilograms).
- All surveys will have the default type
consumptionto start (but downstream other types like
marketmay be added).
Wildmeat Kobo Integration
2021 (Phase 2)
(1) Solution Overview
OpenFn has configured an automated data integration solution between Kobo Toolbox and a PostgresSQL database to sync Kobo submissions and enable real-time monitoring of field data collection.
See here for the data model of the destionation database.
(2) Integration Flow
The solution is a one-way Kobo Toolbox-to-PostgresSQL integration that connects the following Kobo form types:
See here for the full list of Kobo forms which were used to design these integrations.
See here for the data element mapping specification for exchange data between Kobo and the database.
Trigger Type: Message Filter
A message filter trigger has been configured for each of the forms above. The corresponding job will run when a form with the matching message filter is recieved in the project inbox. These can be adjusted in the OpenFn project.
The Kobo forms map to the following database tables:
|DB Table||External Uuid||Source data||Form Type|
|tbl_site||site_id||hardcoded default (e.g., ||All|
|tbl_sample||sample_id||_id + _xform_id_string||Rural Consumption|
|tbl_study||study_id||hardcoded default (e.g., ||All|
|swm_transaction||uuid||_id + _xform_id_string||Rural Consumption|
|tbl_market||external_id||market (e.g., “djazzi”)||Market|
|tbl_sample_market||sample_id||_id + _xform_id_string||Market|
|tbl_sample_hunter||sample_id||_id + _xform_id_string||Offtake|
|tbl_sample_urban||sample_id||_id + _xform_id_string||Urban Consumption|
(3) Assumptions & Considerations for Change Management
site_idare hardcoded values.
- This integration assumes that the master list of species used across forms has already been added to the PostgresSQL database. The master list can be found here.
(4) Administration & Support
Provisioning, Hosting, & Maintenance
This integration is hosted on OpenFn.org with hosted SaaS. The Postgres DB is managed by WCS/EU SWM Wildmeat partners (email: U.Muchlish@cgiar.org).
Questions or support needed?