Architecture#

The OCDS Data Review tool comprises two main parts, which are documented here.

  • cove-ocds (this repository): a web application which makes light use of Django components.

  • lib-cove-ocds (open-contracting/lib-cove-ocds): a library containing most of the functions for performing validation and transformation on OCDS data.

cove-ocds#

cove_ocds/lib contains OCDS data specific exceptions (errors generated by invalid data or input), as well as additional functions for OCDS Show (the JavaScript data explorer).

tests/ also contains fixtures for testing, and the tests themselves; templates and related static files; code for the CLI version of the DRT; and locale files for translations.

cove_ocds/views.py does most of the heavy lifting of taking an input file from the web interface and carrying out the various validation checks and conversions, then piping the output back to the right templates.

cove_project/ contains the Django components (settings, URL paths, server).

lib-cove-ocds#

lib-cove-ocds contains OCDS data specific functions, but in a way that means they can be reused by other software and scripts without having to import the whole of the DRT to do so. It includes the OCDSSchema object. Most of the validation checks for OCDS data are here.

lib-cove-ocds produces results in the form of dictionaries with meaningful keys. cove-ocds is responsible for mapping these keys onto a suitable UI. It’s worth noting that cove-ocds is not the only tool that consumes the output of lib-cove-ocds. Different tools can produce the best UI for their particular context, so any UI or display work is done in the tool and not in the underlying library. This also means that the respective UIs can be translated or localized independently, according to the needs or usage of the tools. Having machine readable output from the library also means that we can take a database of output and easily parse it to create statistics about results or errors, etc.

External libraries#

The OCDS Data Review Tool is just one manifestation of software historically known as ‘CoVE’ (Convert, Validate, Explore). Instances of CoVE exist for other standards as well as OCDS. We modularize and reuse code where possible, so as a result the DRT has dependencies on external libraries related to CoVE:

  • lib-cove (opendataservices/lib-cove): contains functions and helpers that may be useful for data validation and review, regardless of the particular data standard.

  • lib-cove-web (opendataservices/lib-cove-web): provides a barebones Django configuration, and baseline CSS, JS and templates that are common for all CoVE instances. It is also a place for common functions relating to presentation or display of data or output. Any templates edited here typically affect all CoVE instances. Sometimes this is useful, but for OCDS-only changes, templates can be overridden in cove-ocds. This and cove-ocds are the only places where frontend output and translatable strings should be.

  • flatten-tool (opendataservices/flatten-tool): a general purpose library for converting data between JSON and CSV/XLS formats. While not CoVE-specific, it is listed here because it is a specialized tool of which the DRT makes heavy use.

Config#

Some configuration variables are set in COVE_CONFIG, found in cove_project/settings.py.

  • app_name, app_verbose_name, app_strapline, support_email: set human readable strings for the DRT that can be reused in templates etc.

  • app_base_template, input_template, input_methods: set the templates for the landing page.

  • schema_name, schema_item_name, schema_host, schema_version_choices ((version: (display, url))), schema_codelists: point to JSON schema files that the DRT uses for validating and converting data. Since there is more than one version of the Open Contracting Data Standard, this lets the user choose which version of the scheme they are validating against.

  • root_list_path, root_id: set so the DRT knows which key to use as the root of the data, and which key to use as the main identifier when parsing the data. These, along with convert_titles are passed to the flattentool library.

Path through the code#

  1. The input form on the landing page can be found in the input template (cove_ocds/templates). The OCDS DRT overrides the default input template from lib-cove-web so it can be customized compared to other instances of CoVE. This override is set using the input_template setting in COVE_CONFIG (see above).

  2. Submitting the form calls the explore_ocds function (cove_ocds/views.py).

  • This checks for basic problems with the input file, and adds metadata, using explore_data_context from lib-cove-web.

  • It then performs OCDS specific JSON checks, if the input data is JSON.

  • And then schema-specific OCDS checks, depending on the version specified, using SchemaOCDS and common_checks from lib-cove-ocds. Functions from lib-cove-ocds also takes care of handling any extension data included in the input. lib-cove-ocds calls on lib-cove to perform schema validation that is not specific to OCDS, using JSON Schema and JSONRef, as well as common things like checking for duplicate identifiers.

  • lib-cove-ocds runs additional checks, which are basic data quality checks outside of the OCDS schema.

  • The results of the various stages of validation are added to the context so they can be displayed on the frontend. The JSON schema error messages are currently set in lib-cove and OCDS schema specific messages are set in lib-cove-ocds or in this repo (cove_ocds/lib/exceptions.py).

  • It uses flattentool to convert the input JSON data into XLSX, or vice versa.

  1. The results of the validation, as well as some basic statistics on the data, are output to the explore_record and explore_release html templates in cove_ocds/templates.