CBA requires layers of data to mount business case for more AI tools

CBA is placing a greater emphasis on quantitatively assessing the impact of AI pilots on participants, in a bid to work out which to advance and seek additional funding for.

Gavin Munroe from CBA.

Group executive of Technology and group CIO Gavin Munroe told a Microsoft AI Tour event in Sydney that the bank is purposely being very “data-driven” in its assessments, seeking appropriate detail it sees as necessary to building a business case for the technology.

Munroe illustrated this by discussing the bank’s early access to Copilot for Microsoft 365, which it was granted in “late 2023”.

CBA gave the Copilot add-on to 300 of its “associates” - newer staff - who used the tool “for tasks like accelerating market research and summarising text”, according to a statement released by the bank and Microsoft.

Munroe said that initial feedback from the cohort was positive, with some 85 percent liking using the tool.

“But we wanted to make it data-driven - I couldn't take that stat back to my boss and say, ‘Hey, go fund Copilot [for M365],” Munroe said.

“We have to have a business case that makes sense. 

“So we actually wanted to quantify what this 85 percent was experiencing so we could put it in [the]  front of the business case.”

Munroe asked the cohort if they would prefer a “$50 coupon for lunch every month” or a Copilot license, and said 75 percent selected the latter; of that 75 percent, nine out of 10 said it made them “more productive”.

“But again in the spirit of quantifying ‘what does it mean to be more productive’, we actually asked them to explain what that is, and the term they used - 50 percent of them said it made them more creative,” he said.

“I really liked that because it covers a lot of things.”

Munroe said the bank then collected data on what creativity meant in the context of Copilot.

“70 percent said they spend less time looking for content or browsing for information, and 50 percent said they use Copilot to analyse data in their role,” he said of the findings.

“It’s taken mundane or ‘busy’ work away. I think that’s key because it brings more value to the role of the associates and actually makes their job more satisfying. 

“So we’re quite pleased with what that’s showing us, and this will actually help us drive the rollout.”

It’s a similar story with the bank’s use of GitHub Copilot, a so-called AI pair-programming tool that is marketed as a way to improve developer productivity.

This initially went to between 100 and 200 engineers, whose feedback was assessed to determine a path forward.

Again, the initial feedback from the cohort suggested they found the tool “helpful”.

“It’s all nice to say it’s very helpful, but I have to put a business case behind this,” Munroe said.

“I think you have to be data-driven if you really want a good business case to understand how [an AI tool is] being used in your company. 

The bank ran its experiments across 12 weeks - “two sprints for us” - and looked at both the amount of code suggested by the tool that wound up being used, and what that code related to.

“We ‘double-clicked’ on the lines [of code], they basically [fell into] three categories - 30-50 percent of those were repeat work/basic code, 15-20 percent fitted into the category of testing or documenting code, and then 15-20 percent [was] new stuff - new patterns, new frameworks, difficult stuff,” Munroe said.

“The reason I call out those stats is because it covers the entire spectrum. 

“On the front end, 30-50 percent increases your velocity. Engineers are doing more value-add, not the stuff that they don’t want to be doing. 

“And then on the other side, we’ve improved the quality and the engineering quality. I’m happy with the full spectrum. 

“We’re in the process of now rolling it out across all of engineering.”

Ry Crozier attended the Microsoft AI on Tour event as a guest of Microsoft.