A framework and processes for opening data

What we’re up to at NZTA

Aimee Whitcroft – Open Data Establishment Lead, New Zealand Transport Agency (NZTA)

While many people know about the value of releasing and using open data, many organisations are unsure how to go about it: where to start, what to bear in mind, and how to keep going.

This means that organisations open data in scattershot or ad-hoc ways, or poorly (for example, not looking after privacy properly), or don’t open (appropriate) data at all. None of this allows us to leverage the enormous (positive) potential value of the data that organisations collect.

To help address this situation, we at NZTA have been working on developing a simple open data framework and processes. As part of working in the open, and our ‘iterate and optimise’ philosophy, we’re releasing them openly for comment and feedback.

After all, they’re resources devoted to opening things :D 

Also, after reading about the first version of our open data framework and processes, check out the follow up blog post we wrote about implementation.

A Framework and process for opening data, part 2

The framework and processes were developed after reviewing other publicly available open data frameworks from around the world* – including New Zealand – and are still very much a first rough version. We’re working to flesh out the governance layers more, and these (especially) will differ between different organisations.

The idea is that the framework and processes form a living resource, and that people continue to iterate, adapt, and optimise them for their own uses.

You can use them as a resource for a dataset in any part of the lifecycle, including datasets which you’re considering opening, or datasets you’ve already opened.

And we’re developing the more detailed processes underlying them (you’ll see references to toolkit worksheets etc). The prioritisation process(es) especially are a fascinating challenge :)

We’d love to know what you think!

You can get in touch with me at aimee.whitcroft (at) nzta.govt.nz

Alternatively, come along to an open data doers’ catchup :)

Open data doers’ monthly catchup

The work above – and people’s interest in it – led to a thought, now validated with some fab peeps.

If you're working on building such frameworks / processes, for example, or are an open data practitioner, you're invited to a monthly lunchtime session we’re starting, as of early August. It’s an opportunity to share thoughts, challenges, ideas, and knowledge, and fill gaps rather than reinvent wheels :) All welcome!

Let me know if you’d like to come along – email me at aimee.whitcroft (at) nzta.govt.nz

The framework – an overview

NZTA’s open data framework – overview. Image explained in text below. Credit: NZTA. Licence: CC-BY 4.0 International.

NZTA’s open data framework – details. Image explained in text below. Credit: NZTA. Licence: CC-BY 4.0 International.

Open data framework – primary layer

Step 1: Identify

Identify the dataset(s) you’re considering opening using a range of internal and external sources.

Components:

• requests log
• reports log. 

 Step 2: Assess

Assess the dataset(s) for appropriateness to publish (eg risk assessments, privacy assessments, etc), and gain approval to continue the opening process.

Components:

• request log
• assessments
• open data toolkit
• approval.

Step 3: Prioritise

Prioritise the dataset(s) against other open dataset candidates, using a range of prioritisation metrics.

Components:

• open data toolkit
• prioritisation considerations
• request log.

Step 4: Prepare

Prepare the dataset(s) for publishing, including preparing the relevant metadata and documentation.

Components:

• open data toolkit
• metadata / documentation
• master open data directory
• dataset(s)
• information asset register (?).

Step 5: Approve

Gain final approval to publish the dataset(s) and accompanying metadata.

• open data toolkit
• final approval.

Step 6: Release and maintain

Publish the dataset(s) and accompanying metadata on the open data portal, list them on other relevant sites like data.govt.nz, and engage with user groups.

Components:

• open data toolkit
• organisation open data portal
• data.govt.nz
• user groups.

Step 7: Measure and report

Measure and report against the use of and feedback about the datasets, feeding that info into data quality and change processes, as well as steps 1, 3, 6 and 7 (informing what to measure / report against).

Components:

• regular + ad-hoc reports
• user groups.

Open data framework – governance layer

The data governance layer is made up, essentially, of:

• 8: data stewards
• 9: subject matter experts
• 10: data custodians.

We’re working through the details of the functions associated with these.

Note: the data custodian role may be filled, at least in part, by an organisational open data champion.

High-level processes

Step 1: Identify

Inputs:

• data.govt.nz requests
• Official Information Act (OIA) requests
• other internal / external requests
• requests from user groups
• ad-hoc and regular reporting on open data use.

Outputs:

• requests / suggestions logged in appropriate document.

Step 2: Assess

Inputs:

• requests log
• privacy assessment (if required)
• risk assessment (if required)
• information and data assessment (if required).

Outputs:

• data discovery worksheet
• first check-in: written permission to continue
• list of any organisational (ie NZTA) considerations for prioritisation, from approving manager(s).

Step 3: Prioritise

Inputs:

• completed assessments (as necessary)
• NZTA considerations
• All-of-Government / Ministry of Transport / etc considerations.

Outputs:

• prioritisation worksheet(s)
• requests log updated to reflect dataset priority.

Step 4: Prepare

Inputs:

• data discovery worksheet
• data.govt.nz listing template.

Outputs:

• ready-to-publish dataset (any necessary transformations are complete)
• data.govt.nz directory listing information
• updated NZTA data directory
• any other metadata and documentation (eg methodology, caveats, licence detail) required
• updated information asset register (?).

Step 5: Approve

Inputs:

• data directory listing.

Outputs:

• approved and signed permission to publish.

Step 6: Release and maintain

Inputs:

• implementation worksheet
• master open data directory.

Outputs:

• dataset published on NZTA open data portal
• dataset listing published on data.govt.nz
• user groups alerted to publishing.

Step 7: Measure and report

Inputs:

• requests (internal / external) on open data use
• NZTA reporting schedule (internal and also external ideally) on open data use.

Outputs:

• ad-hoc and regular reports on open data use. Metrics could include number of downloads / number of queries relating to dataset / stories of usage (ie case studies), etc
• improved and increasing knowledge of how people want to use our data, and which data they want, which feed back into the identify / assess and prioritise steps, as well as the maintain steps (for datasets which need bug-fixing).
• feedback on open data use, quality etc (user groups, etc)
• feedback sent to data quality subject matter experts for incorporation into their work.

We’ve built some very basic automation for these, to mark where outputs automatically mean the inputs for another step are completed – the idea is to help prevent accidental duplication, and save time.

Interested in the more detailed processes, or the information management side of all this (ie the documents we’ll be setting up and running)? Get in touch :)

Check out part 2 on the open data framework journey

A Framework and process for opening data, part 2

 let us know if you’d like a list of reading materials / references :)