The speed with which many organisations have embraced AI has quickly led some to a reckoning – that being that the promised gains of AI will also be accompanied by significant pain unless they step back and seriously reconsider their actions.

Numerous examples exist of AI producing incorrect, racist, and outright baffling outputs, including Google's AI enhanced search engine suggesting that astronauts have played with cats on the moon.

Much of this market opportunity is in education, with numerous tertiary providers and industry groups such as the Governance Institute of Australia and the Australian Institute of Company Directors creating courses and making other resources available to bring executives up to speed with these emerging liabilities.

The principles of AI governance

According to Vasant Prabhu, global data protection lead at Toll Group, the key principles of AI governance can be broadly broken down into four categories.

"AI should be designed to avoid unfair biases. It should treat all users equitably and not discriminate based on race, gender, or other personal characteristics."

Thirdly, he says it is critical to ensure that personal data used by AI is handled securely and privately, complying with data protection laws and respecting user consent.

And finally, he says there needs to be accountability and oversight, and clear responsibilities and control mechanisms for AI systems.

For Sarah Dods, advanced analytics and AI leader for the Southern Hemisphere at GHD Digital, understanding how governance should be applied to AI systems is a critical requirement in a world where use of AI is both ubiquitous and almost completely uncontrolled.

Applying AI governance is easier said than done however, in part thanks to the rapid pace at which AI capabilities are evolving. Additionally, the seemingly inscrutable nature in which AI systems make decisions makes their true functioning difficult to determine.

"There is a lot of unrealised risk in the way that AI gets used at the moment, and because it is buried in algorithms that produce outputs that are not necessarily explainable," Dods said.

AI governance in the real world

In a practical sense, the activities required for AI governance can be broadly broken down into those tasks that must occur before an AI system produces an output (such as ensuring that it is trained using appropriate and unbiased data), and those tasks that must occur once an output is produced (such as evaluating its output against known expectations, and checking for bias and errors).

These have been key considerations for Claire Mason, principle research scientist at CSIRO's Data61 division, in her work in creating AI models that use job ads to predict skills requirements in the Australian workforce.

Governance principles have played a critical role in understanding the appropriateness of her project's use of content from job ads as training data.

"We acknowledge for instance that AI is not so good at picking up job postings in the community care sector, because those jobs tend to be advertised through more informal networks and therefore, they are not getting captured," Mason said.

"Generative AI will not have the same understanding of each type of job ad, based on the length of the information, that is put into them. So it is going to do a poorer job for certain types of job postings."

Understanding this potential for discrepancies provides Mason with the insight to then check outputs against established and trustworthy alternative information sources to correlate the accuracy of the systems outputs.

All too often however it seems that AI projects proceed without this level of governance.

"You need to understand how the algorithm was trained, and that means there will have been some kind of dataset that has been tagged in some way for the AI," Mason said.

"The question becomes who tagged it and did they have the expertise to describe the construct or the phenomenon you are interested in?

A human intelligence crisis

The extensive promise of AI and the criticality of getting it right has led to significant investments in the human aspects of AI implementations, including the formation of AI ethics and governance committees to drive the adoption of responsible AI.

In some ways however the challenge of AI governance is not unrelated to many of the challenges that organisations have always faced when working with sensitive data and using technology to generate business outcomes.

At ANZ, chief technology officer Tim Hogarth said numerous AI use cases have been considered, and the bank is yet to see any one that is not already covered by its existing policies on concepts such as the design of computer systems and adherence to privacy commitments.

"But at the same time, we do have an elevated level of attention to AI risks," Hogarth said.

"Increasingly, it has started to task us with things like capabilities we're going to need in the future that we haven't needed in the past, such as the kind of statistical testing methods we're going to need to prove confidence levels and tolerance levels for model accuracy."

"I think you'll see it in many institutions … there are many new topics that we still need to discover that are not strictly speaking just about the data."