Quickstart

This quickstart guide is designed to illustrate how to begin using the D-TRO API. It covers:

  • Authenticating your application to receive an access token

  • Making requests to publish D-TROs to the service

  • Making requests to retrieve published D-TROs

Note

This guide assumes you have already created a publisher/consumer application, and know your API key and secret. You can create applications and retrieve application credentials in the D-TRO User Portal.

A Note on Environments

The D-TRO service exposes two environments; integration and production.

  • The integration environment is designed to allow users of the service to test their application’s ability to publish to/consume from the service, without live data

  • Once happy with the process for publishing/consuming with their application, live data can be published/consumed from the production environment

Note

The integration environment includes upcoming features for testing and adapting applications to future API or data model changes. It may differ from the production environment. DfT is considering adding a pre-production environment that mirrors production, allowing users to test with confidence. In this setup, integration becomes a sandbox for new features. DfT will provide updates if this new environment is introduced.

The integration and production environments are two isolated services, and as such have different URLs.

  • Integration: https://dtro-integration.dft.gov.uk/v1

  • Production: https://dtro.dft.gov.uk/v1

Because of this, separate applications are required for communicating with each environment.

Authenticating Your Application

The D-TRO service makes use of the OAuth 2.0 client credentials flow for authentication. In this flow, a client ID and client secret are exchanged with the service for an access token. This access token permits a user to interact with the service for a fixed period of time, until the token expires. Token expiry is configured to 30 minutes. After this time a new token will need to be issued through the same flow.

Exchanging credentials for an access token is achieved by making a POST request to the OAuth 2.0 token generation endpoint, providing the client ID and secret as HTTP basic authentication credentials, and the client credentials grant type in the request body.

curl -X POST https://dtro-integration.dft.gov.uk/v1/oauth-generator
    -u <client_id>:<client_secret>
    -d "grant_type=client_credentials"

The response payload contains access_token, which is the token you will use when making requests to the API to authenticate.

Danger

Do not share your access token with anybody. A valid access token allows your application to interface with the service. If somebody knows your access token they can use the D-TRO service on your behalf.

Publishing to the D-TRO Service

There are three main ways to publish a D-TRO to the service. These are:

  • Sending a JSON payload in the body of a POST request

  • Sending a JSON file in a POST request

  • Sending a gzip-compressed JSON file in a POST request

Note

Only applications with the publisher scope can publish D-TROs to the service. If you try to publish a D-TRO using a consumer account, you will receive the following response:

{
    "fault": {
        "faultstring": "Required scope(s): [cso, dsp]",
        "detail": {
            "errorcode": "steps.oauth.v2.InsufficientScope"
        }
    }
}

Submitting JSON in the body

A D-TRO can be published to the service by making a POST request to the /dtros/createFromBody endpoint, passing the access token as a header and the payload in the body.

JSON=$(<file.json)

curl -X POST https://dtro-integration.dft.gov.uk/v1/dtros/createFromBody
    -H "Authorization: Bearer <access_token>"
    -H "Content-Type: application/json"
    -d "$JSON"

The response payload contains a single field, id, which is the ID of the created D-TRO record.

Note

There is a 10MB request payload limit imposed by the API proxy. If the request body exceeds this limit, the submission will be rejected. Consider minifying your JSON payload, or uploading as a gzip-compressed file (see Submitting a gzip-compressed file).

Submitting a JSON file

This endpoint is similar to the endpoint that handles publishing D-TROs using a JSON body, but instead allows a file to be uploaded. This is achieved by making a POST request to the /dtros/createFromFile endpoint, passing the access token as a header and attaching a JSON file.

curl -X POST https://dtro-integration.dft.gov.uk/v1/dtros/createFromFile
    -H "Authorization: Bearer <access_token>"
    -F "file=@file.json"

The response payload again contains a single field, id, which is the ID of the created D-TRO record. The same file size limits are imposed here as when submitting JSON in the request body.

Submitting a gzip-compressed file

gzip is an excellent compression algorithm for JSON data, as it excels at compressing large, text-based, repetitive data, exactly like JSON. There are various tools available for compressing JSON with gzip. The following command uses the gzip CLI utility, installed on many Unix-like systems, to compress file.json and output it to a file named file.json.gz.

gzip -c file.json file.json.gz

The process for submitting a gzip-compressed JSON file is identical to that of submitting a JSON file.

curl -X POST https://dtro-integration.dft.gov.uk/v1/dtros/createFromFile
    -H "Authorization: Bearer <access_token>"
    -F "file=@file.json.gz"

Note

As gzip can compress very large raw files into very small gzip-compressed files, there is a hard limit on the server when decompressing gzip files. To prevent zip bomb attacks gzip files are processed in 8-byte chunks up to a maximum of 25MB. If the uncompressed file exceeds this 25MB limit, the API will return an error.

Consuming Data Within the D-TRO Service

Both publisher and consumer applications have the necessary scopes to consume data from the D-TRO service. The API allows retrieval of individual D-TROs by their ID, searching for D-TROs that match some filter criteria, and retrieveing all D-TROS published to the service.

Retrieving a D-TRO by ID

A single D-TRO can be retrieved by making a GET request to the /dtros/<id> endpoint, passing the D-TRO ID as a path parameter, and a valid access token as a header.

curl https://dtro-integration.dft.gov.uk/v1/dtros/146375a5-7cc6-4092-b515-0c5c105820ff
    -H "Authorization: Bearer <access_token>"

Note

Retrieving a D-TRO by ID relies on knowing the ID of the D-TRO.

Retrieving all D-TROs

The D-TRO service provides the capability to retrieve all published D-TROs. Due to the large volumes of data contained within D-TROs, retriveing all D-TRO records is not achieved via the API. Instead, the API is used to request a signed URL, which enables users to download a .csv file containing all D-TRO data.

This URL can be requested by making a GET request to the /dtros/all endpoint, passing a valid access token as a header.

curl https://dtro-integration.dft.gov.uk/v1/dtros/all
    -H "Authorization: Bearer <access_token>"

Note

The signed URL is valid for 60 minutes only, and once expired can no longer be used to retrieve the dataset.