Formats for open data

To maximise the re-use and value of your open data, you need to ensure that you release it in machine and human readable formats, such as PDF, CSV, and XLS. This also ensures that you comply with the NZ Government Open Access and Licensing framework (NZGOAL), as required by the 2011 Declaration on Open and Transparent Government.

Formats

The format of a file refers to how the file is represented on a computer disk. The format usually corresponds to the last part of the file name (‘extension’), e.g. a file in CSV format might be called "schools-list.csv".

CSV

An image which demonstrates what a CSV file would look like from inside the files application of a windows computer

These internal formats do not relate directly to how the file looks when opened on a computer. Some internal formats, such as CSV and XLS, are structured different but may look similar or identical when opened on a computer. While others, such as PDF and XLS files, are structured and look different when opened on a computer.

XLS

PDF

In relation to open data, the internal format of a file determines what a computer can do with it and whether it is machine readable. The look of the file when opened by the appropriate computer application determines whether the format is human readable.

Human readable formats

Data in a human readable format can be conveniently read by a human. They often contain extra formating, colours and styles that help people read them. Examples include PDF files published for people to read and understand some data.

Human readable

The image below contains an exmaple of open data in a PDF format. A human can easily read and understand the data (some statistical knowledge may be required, results may vary). Unfortunately, a machine cannot automatically read or process this data as PDFs are not structured data. If someone wanted to use this data in an analysis or for a computer application, they must manually copy the data into a machine readable format first - not cool.

An example of open data in a PDF format

Business price indexes: Revisions for the June 2017 and September 2017 quarters [PDF 335KB]

Business price indexes' catalogue listing on data.govt.nz [link]

Note: Stats NZ no longer publishes this data in PDF format. The data is now available in XLS and CSV formats, human and machine readable.

Structured data

Machine readable formats

Data in a machine readable format can be automatically read and processed by a computer. Examples include CSV, JSON, XML, etc. Machine-readable data must be structured data.

JSON

XML

We can make data in a PDF format, such as the data mentioned above, machine readable by publishing it in CSV. However, as you can see in the images below, although the data is now machine readable, it is less human readable.

An image which demonstrates what a CSV file would look like from inside the files application of a windows computer

Business Price Indexes - June 2020 Quarter [CSV 11KB]

Business price indexes' catalogue listing on data.govt.nz [link]

Note: The appropriate machine readable format may vary by type of data - so, for example, machine readable formats for geographic data may differ from those for tabular data. Furthermore, formats may vary in their level of human-readability when opened in different computer applications.

NZGOAL and the declaration

The NZ Government open access and licensing framework (NZGOAL) provides further guidance on formats in accordance with the Declaration on open and transparent government.

NZGOAL Guidance Note 2: File formats – Formats for Open Data

NZ Government Open Access and Licensing framework

Declaration on Open and Transparent Government