Scotland’s Open Data and the EU27 – a comparison

Introduction

Since the Brexit vote, Scotland’s First Minister has often talked about Scotland’s place in Europe. There have been many articles written about that ambition.

Last week the EU Data Portal’s published the eighth annual assessment of the Open Data Maturity of Europe’s 27 member states plus eight others including Ukraine. So, it provides a timely opportunity to assess Scotland’s open data maturity alongside those countries’ assessments.

When I resigned as civic lead for the Access to Data / Information strand of the Scottish Government’s Open Government Partnership group in 2021 I had collated from a broad civic society community a list of what they wanted in relation to open data in Scotland. I submitted that to the Scottish Government. Point four was “Scoring, comparisons and benchmarking of Scotland’s open data against other nations (à la EU Open Data Portal / Open Data Barometer) with regular reporting from SG on performance.”

This continuing lack of performance measurement or of benchmarking is something that has come up repeatedly since the launch of the Scottish Government’s Open Data Strategy in 2015.

With the publication of the EU report and its methodology we have the chance to perform the same assessment of Scotland as has been done with the 35 other countries and see how Scotland performs.

Methodology

Firstly the methodology, which is tweaked from year to year. This year’s approach is described in the Method Paper. Three steps were followed to complete the questionnaires once the format was standardised:

• Step 2: Coordinate and assist national teams in filling out the landscaping questionnaire
• Step 3: Analyse and validate the data together with the national teams
• Step 4: Complement the results with additional desk research.

Country-level questionnaire responses have been published (zip).

In the original, many questions and sub-questions were assigned specific scores, meaning that questions had individual weightings. We’ve jumped straight to a desk research phase, with three individual assessors from civic society, and have assigned uniform weightings to all individual questions. We accept that it is very slightly different to the EU approach. Each marker selected 1 or 0 per question. Sections (e.g. the 10 questions under Policy Framework ) were summed up, divided by 10 and multiplied by 275 to align with the EU mark out of 275. Our marking sheet can be checked. After doing this, the scores of the three markers at a question level were looked at 2 ways. The first was to take the majority (eg A scored 1, B scored 1, C scored 0, so majority is 1), The second was to average them (eg A scored 1, B scored 1, C scored 0, so average is 0.6666).

As the markers differed slightly on what they scored up or down, the first approach (majority) produced lower scores since there was less consensus. The latter scored more since and 1s in the minority are not discounted. We erred on the side of generosity and opted to go with the average scores. All of that said, the overall outcomes – choosing any one scorer individually, choosing the average, or choosing the majority actually didn’t make much material difference.

I’ve requested a blank questionnaire from the Eu Data Portal. If someone at the Scottish Government would take on its completion (see step 2 above) we’d be delighted to go through steps 3 and 4 with them and revise our scoring accordingly. However, having studied several completed questionnaires (zip), we are confident that the final outcome would not be significantly different to the outcome of the evaluation which we have carried out.

The Four Sections

There are four sections to the questionnaire. These can be seen in the methodology paper, above. Our assessment of each is covered individually below.

Open Data Policy

Such are the requirements of this section that Scotland was almost inevitably going to come out of it badly. In fact Scotland scored, by some considerable measure, the poorest of the 36 countries assessed. The median score for EU countries was 562 and the mean 548. The second poorest scoring, Bosnia & Herzegovina, achieved a score of 200. Scotland scored 57.

• We have a 7 year-old national strategy. While we welcome SG’s intention to review it soon, it has not been updated since its publication.
• There is no policy for open data in Scotland.
• Open Data reuse by either the public or private sector is not incentivised by government.
• There is no governance structure to oversee publication of open data at all government levels (local, regional and national).
• While there have been some high-quality ODS publishing programmes such as https://statistics.gov.scot, the 2015 OD strategy has been seen not as a national strategy nor even as a strategy for all of the Scottish Government itself.
• It is not clearly published who is responsible nationally for open data. And we are not sure that such a role exists.
• No documents exist to identify responsibility for OD locally, regionally or nationally.
• Open Data officers have not been appointed at every public body.
• There aren’t plans for OD publication at every public body, progress against which is monitored nationally. As far as we can tell, national monitoring – if it happens – is against https://statistics.gov.scot and probably nothing else.

Much of these requirements – and more below – were, of course, contained as requests from civic society in the Statement released after SODU2022 which we sent to Scottish Government.

Only the work of The Spatial Hub, as a place for local authorities to publish geospatial open data, scored significantly in this section.

Open Data Impact

In this section the questions seek to elicit how each country measures and monitors the impacts that open data use has on various aspects of society. Again Scotland performs poorly with just five countries performing worse: Bosnia and Herzegovina, Albania, Malta, Iceland, Greece. The EU 27 median score was 440, and mean 424. Scotland achieved 117 points.

Some of the key factors in this are:

• There is no monitoring of reuse of open data at national, regional, or local level as far as we can tell.
• While only In a very limited number of cases are activities are in place at public body level to boost and monitor the reuse of own published data, this is by no means uniform.
• A methodology is in place to measure the impact of open data, nor are we aware of first steps in this direction having been taken.
• Similarly, we are not aware of activities in place to better understand re-users ́ needs.
• We are not aware of a process being in place to systematically gather reuse cases. In fact, government appears to be reliant on civic society to do this.
• A significant amount of marking is concentrated on whether reuse cases are classified according to categories (e.g., environmental, social, economic reuse cases etc.) and specific aspects of those impacts. The three markers were not aware of any [significant] effort being put into this at all.

Open Data Portal

This section assesses the national open data portal; its features, management and operation; and data provision and usage. It will come as no surprise to anyone who has followed the story of civic society’s calls for such a portal leading ultimately to the emergence of https://opendata.scot, the community -built and -maintained portal, that there still is no Government-provided national open data portal in Scotland. This alone sets Scotland apart from the 35 other countries assessed.

With a maximum possible score of 650 for this section, the mean score of the EU countries was 540, and the median score 571. Under our agreed average scoring method Scotland scored 47. Had we gone with majority scoring it would have been 26. The next poorest score was 185 (Bosnia and Herzegovina). The poorest score from the core EU27 was 300 from Slovakia.

Given that there is no national portal most of the subsidiary questions in this section cannot be answered positively. One scorer gave credit for the work on https://statistics.gov.scot while accepting that this is not a national portal, and nor does it perform functions such as the harvesting of metadata from local or regional portals.

It is worth noting that 89 countries are listed on Wikidata as having a national open data portal, many times more than the EU27+. In Sweden alone we can find over 200 open data portals, all aggregated to a national level, and in France over 100.

Open Data Quality

This section assesses the quality of data provided via the national portal. The maximum points achievable in this section were 650. The median score for the EU27+ was 517, and the mean 467. Only Bosnia and Herzegovina, with 30 points scored more poorly than Scotland which achieved 78 (with some of those marks based on the performance of https://statistics.gov.scot in the absence of a national portal. The minimum score of any core EU country was 313.

Again, in the absence of a national portal it is difficult to score many of these questions positively.

Overall Outcome

When we add up the final scores we see that Scotland has by far the poorest outcome of all countries marked, with 299 of a possible 2,540 (or 11%). See fig.1 below. Only five other countries scored under 1,270 (or 50%), For comparison, the mean score of the core EU countries was 2,019 or 79%, and the median score was 2,056 or 81%. France, Ukraine, Poland, Ireland, Cyprus, Estonia, Spain, Italy, and Slovenia all scored 90% or more.

Had we gone with the majority voting approach to combining the three markers’ input we would have arrived at a total of 216 or 8%. See fig.1 above.

It should be noted that in 2021, the last year in which the UK was included in the official benchmarking, it scored 1,617 out of 2,600 or 62%. I’ve long argued that Scotland’s performance in the Open Data Barometer was a drag on the performance of the UK as a whole, which this would again appear to back up.

According to the report’s clustering, to which we have added Scotland (see fig.2, above) Scotland is at the very bottom of the Beginner’s cluster. After seven years of having a national open data strategy this is an exceedingly embarrassing place to find our country.

 

 

In conclusion

In this exercise we have attempted to show just how far behind European countries Scotland lies in respect of its work on governmental Open Data. This has been claimed, and written about, for a number of years but at last we have a measure of that gap.

Why has this come to be? While that would best be answered by the Scottish Government itself, we can offer some suggestions.

• There is no open data policy in Scotland. The 2015 Strategy set out some good intentions but it created no mandate to publish and there was no person or team appointed to lead the delivery of OD across Scotland’s public sector.
• While some great work has been done, and continues to be done by those in the Data Division of the Digital Directorate that is mainly focused on what they themselves can deliver – one example being the high-standard https://statistics.gov.scot platform.
• There is no lead for Open Data for Scotland, nor are there named individuals in public sector bodies with a responsibility to deliver OD for their own organisation.
• There is no coordination between similar organisations (local councils, health boards etc). COSLA, while an attendee at the Open Government Partnership for Scotland has ignored calls to join up work on OD in local authorities.
• When metadata and other standards are developed organisations are free to disregard them.
• There is no government-provided national portal for open data. And when we have requested that the government provide one (as each of the EU27+ do, and many more countries worldwide) that has been forcefully rejected. Instead, the Government has invested in a Civtech programme to ‘find’ its own data. We shouldn’t need to find our country’s data. We should have coordinated platforms, metadata standards, roles and responsibilities, and monitoring and measuring of what is published. We shouldn’t have to rely on Civic Society to step in to build and maintain a portal in the place of government. This lumps us with Myanmar, Somalia and other developing countries.

Having carried out this exercise once, we are committed to re-running it each year that the results are published.

Recently, the Scottish Government announced that it is about to revise its open data strategy after 7 years. We suggest that the questions raised in the questionnaire above should be used to frame how the next phase of open data in Scotland is approached. That would set a framework such as civic society has called for, with clear responsibilities, measurements, reporting mechanisms, aligned approaches and mandated standards. If these are put in place we are confident that Scotland could be re-assessed against the EU27+ in the next and future years, and not find itself in such a lamentable position.