NSW gov AI projects to face new risk regime

NSW government agencies will need to assess all projects that use bespoke artificial intelligence systems before deployment from March 2022 under a new framework aimed at identifying potential risks.

The AI assurance framework has been developed by the Department of Customer Service to assist agencies to design, build and use the technology “safely and securely”.

It comes just weeks after the NSW Ombudsman revealed that Revenue NSW unlawfully used an automated system to recover unpaid debts between 2016 and 2019, without human oversight.

A lack of human oversight meant that, in some cases, the Revenue NSW system emptied the bank accounts of vulnerable people, leading to calls for greater visibility of AI in government decision-making.

The new AI assurance framework contains questions that agencies will need to self-assess against at “all stages of an AI project from inception to handover” and then periodically after a system is deployed.

Questions align with the five mandatory AI ethics principles covering community benefit, fairness, privacy and security, transparency and accountability that agencies are already expected to adhere.

For AI projects that are funded from the state’s $2.1 billion digital restart fund or cost in excess of $5 million, agencies will need to submit their self-assessment to a newly-created AI review body.

The AI review body will review the self-assessment and make recommendations to mitigate any risks, though these recommendations will not be binding on the agency that undertook the review.

“Any decision to not meet them should be documented with accompanying reasons,” the assurance framework states.

Agencies will also need to submit other projects to the AI review body where “residual risks (after mitigations) which are midrange or higher” have been identified in the self-assessment.

Only projects that use an AI system that is a “widely available commercial application” or solutions “not being customised in any way” are considered exempt from the framework.

Chief data scientist Ian Oppermann said the framework would be vital for agencies as AI technology continues to evolve.

"The framework lays the foundation for appropriate use of AI systems, which in turn means new ways of delivering government services in NSW," he said.

Oppermann said the framework is “designed to support government agencies at every stage as projects are delivered with AI technology”, providing “clear guidance on the safe use of AI”.

"With the framework, the government will be able to support more projects which provide essential services to everyone across NSW while also managing the risks associated with the technology,” he said.