We Risk Losing a Generation's Worth of Archives

The Australian National Audit Office has released its latest audit into records management in the Federal Government, contained some fairly shocking statistics, highlighting major black holes in the way records are managed. The document – which focused on the implementation of the National Archives’ (NAA) ‘Building Trust in the Public Record’ policy and considered two case studies in its effectiveness (Prime Minister and Cabinet, and the Australian National Maritime Museum) – found compliance to be partly effective in the case of PM&C, with its effectiveness overall lacking at ANMM. 

What were the key findings?

The worrying statistics revealed in the report:

• More than half of agencies have never or rarely transferred Retain as National Archives (RNA) records to the NAA
• The NAA do not have any performance measures in place to ensure records are transferred to them (or for other mandatory measures of the Policy)
• 93% of records in agencies are now unsentenced, up from 63% in 2019 (meaning they will never be identified for transfer to Archives)
• The NAA does not include a risk for agencies failing to sentence records in its risk register.

This means that Australian society is at risk of losing a generation's worth of archives. The impact on culture and society of this will be profound. Exacerbating this, 'the lowest maturity scores were amongst agencies with cultural or heritage... functions' (which manage a high proportion of archival data).  

As well as the risk to heritage, failing to sentence records means they may be under-retained (and destroyed while there is still a need for them) or over-retained. Having less than ten percent of records in a Department sentenced is an incredible risk to privacy, cybersecurity, reputation and trust, effectiveness, regulatory compliance, and efficiency.

As a reminder, under-retained records have affected reparations for Indigenous Peoples and victims of institutional abuse, led to the unlawful deportation of dozens of members of the Windrush Generation in the UK, and affected tens of thousands of active cases at the Dallas Police, for just some examples. Over-retained records have clogged the systems of Policing and Immigration agencies, contributing to the Manchester Bombing and the unlawful deportation and detention of Vivian Solon and Cornelia Rau. They have also magnified major data spills like the Australian National University, Optus, Medibank -- records have been breached in those attacks that should simply not have been in the network any more.

What were the other findings?

There are also some other findings that help show that overall governance is insufficient:

• PM&C are still developing new systems without considering recordkeeping requirements
• Check-up Reports provided by agencies to the NAA often have inconsistencies and errors
• Accountability for Information Management is not consistently assigned to the Executive
• The NAA has no process to analyse agency reporting and identify or target those at risk of non-compliance.

What does this mean for the rest of government?

Government has never had effective management of its information, since the beginning of the digital age. There is too much volume, variety, and velocity of information. Traditional models have been proven to be ineffective time and time again (although the ANAO may have actually contributed to this, by continuing to recommend implementation of traditional models whenever they have encountered information governance failings in their audits). The reasons agencies are not sentencing records include too much:

• User impact: Requiring staff to know and understand thousands of retention rules, and manually apply them, is not practical
• System impact: Needing to customise productivity systems, or install agents or connectors on them for compliance, is not sustainable
• Data impact: Needing to label, tag, move, modify, or duplicate records for compliance purposes is not supportable
• Governance impact: Requiring the (small) records team to manage file plans, rules engines, or supervise machine learning is not possible.

The volume of digital records held by entities has grown on average by 328 per cent a year, from 51 Terabytes in 2013, to over 314,000 Terabytes in 2022. There is no way for this amount of records to be compliantly and effectively managed using any model or system that causes one or more of these detrimental impacts.

Traditional models became outmoded five years ago when we released a new way to actually manage all information in a network, in-place, in every system and format, with explainable, ethical Artificial Intelligence. (The ethical consideration is very important: it's clear that decisions about records can impact citizens -- like Vivian and Cornelia -- so any automated management of them is potentially high-risk. Laws are now coming into force that don't allow use of black-box AI, such as Machine Learning or LLMs, for this purpose.)

How does Castlepoint solve this challenge?

The good news is, Castlepoint's ethical AI is now in use across most Federal Government portfolios in Australia, including Defence, Agriculture, Education, Infrastructure, Attorney-Generals, Employment and Workplace Relations, Treasury, and Health. The agencies using Castlepoint are able to sentence 100% of classifiable records in all the systems under Castlepoint's management, with accuracy consistently benchmarked up to 98%. And they don't have any of the user, system, data, or governance team impacts preventing organisations like the Maritime Museum from succeeding, who noted in their response to the ANAO audit that funding and limited resources had hampered them from ever implementing effective recordkeeping systems.

It is simple and fast to get full control of records, including RNA records, across the whole enterprise. If you would like to know more about how to achieve this, and protect cultural heritage as well as manage risk and value, please contact us for a demonstration!

This blog was also repurposed and published in Canberra Times on 13 July 2023 titled 'Australia at risk of losing its own history due to government attitudes to records, archives'