Nine hones its incident response

Nine is using PagerDuty software to help locate the best person internally to fix a problem that may arise with one of its media products.

Nine media brands.

Nine’s product engineering technology director Andre Lachmann told the PagerDuty Summit 2022 that “with such a disparate amount of services, products and systems, knowing what’s happening in the environment and bringing it together is what we rely on PagerDuty for”.

He said that identifying a problem in the first place, but then who is the correct person to deal with it, was a challenge, particularly in an organisation the size of Nine.

“It’s not always obvious who the right responder should be,” Lachmann said.

“It’s not always the case that the first person that gets the alert can fix the problem either, and with a fairly large organisation, and particularly one that is a merged company and has people that have come together from different backgrounds or different parts of the business, they dont always know each other as well as maybe a smaller organisation would.”

The tool also helped product engineering’s 250 staff locate the owner of a dependency that their own product or system used.

While product engineering teams have responsibility for both building and running products - that is, not just for creating the software and deploying it, but also “making sure that your services are online and are available … the challenge with that is those teams are only responsible for their part of the universe.”

“If and when they rely on other parts of the organisation or other services, and there’s other dependencies they have to rely upon, we have to knit that together in some way. We look at PagerDuty to help us do that,” Lachmann said.

“When we do get an alert, it lands mostly with one of the product engineering teams. They’ll look at the issue, and try and resolve it if they can, but in the case where they do have a dependency that they can’t manage themselves, they will have to reach out into the rest of the business. 

“It’s a large company. Just in software engineering in our group, there’s about 250 people. In the wider technology group it’s 400-plus, and then there’s a larger business beyond that that’s producing content and involved in the business itself. 

“So when it comes to reaching out and connecting to the right person at the right time, we need a system to do that, and that system is PagerDuty.”

Lachmann said the need for good incident response systems was highighted by changing customer expectations in media, particularly video-on-demand where viewers were demanded high levels of reliability and performance.

“The scale of our responsibility and the breadth of what we need to respond to, if and when there’s issues, is much higher, and you’re knitting together different vendors and services and applications as well, which ups that complexity quite a lot,” he said.

Lachmann added that Nine is in the process of reintegrating PagerDuty with ServiceNow - the ticketing solution used by Group IT at Nine. 

While the two systems are connected already, the integration is custom, and this will now be redone using official integrations between the two platforms.