Services Australia finally talks about its private cloud environment

Services Australia has provided the first real behind-the-scenes look at the OpenStack private cloud it first stood up on IBM servers back in 2018, and how the infrastructure is likely to evolve.

The agency first revealed the private cloud effort at an IBM conference in the United States in March 2018, though offered scant detail at the time, limited to the basic layered architecture and IBM hardware underpinning it.

The private cloud resurfaced again in May this year when Services Australia re-platformed part of its national vaccine register to run in containers on the private cloud.

A month later in mid-June, Services Australia presented to an OpenShift Commons Gathering as part of this year’s virtual Red Hat Summit.

“Charged with delivering all services digitally by 2025, a large government department set out to create a self-service platform treading the fine line between governance, security, and developer productivity,” the session description read.

The result is a “federated Kubernetes platform, built on Red Hat OpenShift and Red Hat OpenStack Platform, extensible to public cloud, and automated through Red Hat Ansible Automation Platform (Ansible Tower) and GitLab, providing product teams with the tools to build applications quickly, get to production faster, and scale elastically as required.”

Cloud engineer Robert Powell - who left Services Australia shortly after this presentation - said the agency dealt with “a number of constraints” and considerations when standing up the private cloud and a platform-as-a-service (PaaS), and associated infrastructure-as-code tooling.

“We have multiple data centres and within each of these data centres, we have our own OpenStack,” Powell said.

“Being their own cloud, our OpenStacks are not reliant on any other data centre. They're not ‘stretched’ OpenStack clouds; they're tied to the data centre that they reside in, and we deploy our applications across them.”

The structure means that applications with high availability requirements need to be deployed to multiple sites.

“Where we have two or more data centres, and we have our OpenStack clouds deployed across that, they're not connected in any way aside from Layer 3 networking, so that means that applications need to deploy themselves across both sites to get high availability, or [across] other sites as required,” Powell said.

The agency has “both x86 and IBM Power compute nodes within [its] OpenStack clouds”, and has a “standard rack design” that it uses to house its OpenStack infrastructure.

“One of the principles we really wanted to take with OpenStack is we wanted to simplify infrastructure and the way we were working,” Powell said.

“We didn't want to build complex infrastructure solutions. We wanted really simple, easy to manage infrastructure. 

“We had a standard rack design, which helped us for simplicity with our physical hardware. 

“Our racks come in pairs. We have compute nodes in the top and then we have our switching and then we have our Ceph [software-defined storage] nodes, and then below that we have our Swift [object storage] nodes. 

“We have IBM and x86 hardware in these racks, and the racks are cross-cabled for redundancy."

Powell said that the agency also had security constraints around standing up the environment, as well as requirements for performance and scalability.

“We obviously run some of the country's largest applications and most public-facing applications, certainly for the government, if not wholly for Australia,” he noted.

OpenShift environment

On top of the OpenStack foundation, Services Australia has deployed “multicluster OpenShift”.

“We essentially have multiple clusters per OpenStack cloud,” Powell said.

“We have eight production OpenShift clusters - four in each of our primary data centres.

“They're all functionally the same, but they have different labels attached and that's really to support how our development community expects things to work. 

“So we have a dev OpenShift cluster, we have a test OpenShift cluster, we have a staging or pre-production OpenShift cluster, and then we have a production OpenShift cluster."

Powell said that all OpenShift clusters at present are x86-based. 

“We don't have any Power clusters,” he said. “We're hoping to evolve that in the near future.”

Some of the challenges

Powell made some general observations of Services Australia’s experiences with OpenStack and OpenShift, noting the agency started the program of work with “a small team” and “lots of new technology”.

“That was quite difficult, really, in the beginning, where we had a small team of five or six people that were building and deploying OpenStack and OpenShift,” he said.

“There was a lot of upskilling and cross-skilling required.”

Powell said that things had taken longer than originally expected.

“It always takes longer than you expect - I don't think this would be a great surprise to anybody,” he said. 

“We were implementing lots of new things here - [there was] lots of technology change, but lots of ways of working changes [as well] and it did take us longer than we first thought. 

“So, spend as much time as you can in design and thinking about what you're going to do, but inevitably, you've got to deliver at some point, so there's a balance to strike there.”

He said that having a good internal lab to test the technology is also “critical”.

“Aside from whatever the vendor is testing, it's really good to be able to test it in your environment with your networking and other integrations that you have that are specific to your landscape,” he said.

“We've had an evolving lab story where we started out with a small lab and we've been able to grow that, so we've had good support here to be able to do that. 

“But I think if people aren't investing in a lab, then you're going to get poor outcomes in production. It's kind of a no-brainer.”

As for where the agency is headed next, Powell said it is looking to move up to OpenStack 16 - with features including active-active load balancing upgraded virtual network - and OpenShift 4.

Updated, 7.52am: An eagle-eyed reader has spotted that Powell has moved on from Services Australia since this presentation. The story has been briefly updated to reflect this.