Census 2018 independent review – Expert reaction

The head of Statistics New Zealand has stepped down following an independent review into the 2018 Census.

Chief Statistician Liz MacPherson resigned this morning, saying she took full responsibility for the “shortfalls identified in the report”.

The independent review concluded that Census 2018 was adequately funded, but problems arose with IT systems and staff displacement following the November 2016 Kaikōura earthquake.

The official results were initially slated to be released in August 2018, but they will now be released in September 2019.

The SMC gathered expert comments on the review.

Associate Professor Polly Atatoa Carr, National Institute of Demographic and Economic Analysis, University of Waikato, comments:

“The release of the review into New Zealand’s 2018 Census released today highlights the concerns about the governance of the Census processes, and the coverage and quality of Census data. Censuses form a critical component in the planning for and responding to issues that are fundamental to political participation and social and economic wellbeing: electoral boundaries, the planning and funding of health, education and other social services, and the work of researchers seeking to address social, economic and environmental challenges.

“The 2018 Census has not achieved comprehensive population coverage, with response rates of 68% for Māori and 65% for Pacific people. This has significant equity implications, and is likely to result in new inequities in the development and monitoring of our policy and programmes. Those most likely to be undercounted are those experiencing the worst outcomes in the context of a government that has made a priority commitment to achieving equity and addressing wellbeing. The sectors who have the greatest responsibilities to achieve equity—health, education, social support—will not have the information they need to assess and respond to current and future needs.

“Further, some gaps in Census information are being filled by the extensive use of government administrative data. However, such data is collected for different purposes, under different circumstances with different governance arrangements and substantial inaccuracies. This has further implications for the appropriate interpretation and use of the 2018 Census.

“Finally, the Census transformation required to manage the failings of the 2018 data collection has created a new data environment for New Zealand. Important community conversations are now required about how our data is collected and used, and who has control over these processes. The review points particularly to failings in the lack of partnership with iwi, Māori and Pacific communities from governance, through design and decision-making phases and throughout the process and the review. These lessons must be learnt so that we can move forward. Accurate, up to date and meaningful data is fundamental to our work as social scientists.”

No conflict of interest.

Professor Richard Arnold, Victoria University of Wellington, comments:

“Statistics New Zealand has just received its independent review of the Census 2018.

“The census was a ‘digital-first’ collection, in which 70% of respondents completed an online census form. The goal of this new version of the census was to save money for the government, in what is an extremely costly five yearly exercise to count the New Zealand population. It was also hoped that the online census would reverse the (slow) decline in response rates.

“However, the census had a catastrophically low response rate. Instead of an undercount of 2-3% (the trend of recent censuses), the undercount was 9% for partial completion of the census, and 14% for full completion.

“Statistics NZ has tried to repair the data set – by imputing data from administrative and other data sources. They have used other data sources, including the 2013 Census, to estimate the properties of the people who did not respond to the census, and claim that in doing so they have reduced the undercount to only 1.4%.

“For key outcomes – the drawing of the electoral boundaries and DHB funding – the Census has met its requirements. However, they admit that for a host of other variables, such as iwi affiliation, the results will be insufficiently accurate. A separate External Data Quality review of the adjustments to the Census will report its findings in October.

“This has also led to a delay of more than a year in the release of the first census results: they were planned to be released in August 2018, and they will now finally be released on 23 September 2019.

“What went wrong?

“The review found that ‘many aspects of the census went well’, and that:

  • The goal of the digital-first census was laudable
  • The approach that Stats NZ took to it was sound
  • The resources Stats NZ had were sufficient.

“But:

  • There was lack of ‘overall governance and strategic leadership’;
  • There was an over-optimistic attitude that didn’t recognise the complexity of the task;
  • There was an overemphasis on the online dimension of the census, and insufficient attention paid to the respondents who would find the online census difficult to complete;
  • Testing was insufficient;
  • Some poor decisions were made;
  • The field force was too small for the task that it faced;
  • Decisions were made late, and in particular remedial actions were made too late to fix the response rate when it was clear it was too low.

“Key among the decisions that put the response rate at risk were the decisions not to leave a paper form and not to make personal contact with households that had individuals who had not responded and who needed an access code to begin the census.

“What seems clear is that the lack of personal contact with a census field operative, which was the backbone of the previous census model, was a barrier to participation for too many people. Moreover a decision was made not to follow up partially responding households, where only some of the residents had completed their personal forms.

“The review also found accounts of long waiting times for helpline calls, and insufficient staff for community liaison, especially for Māori people.

“From the outside, it looks as if the appropriate technical and statistical expertise was lacking at the highest levels of decision making. The Review concludes that Statistics NZ should continue to follow this model for the next census in 2023, but that there are a number of serious lessons it needs to learn from this experience.”

No conflict of interest.