Whereas once the cloud was seen as an inevitable destination for much of the world's data, rising costs and the emergence of edge use cases are forcing a rethink.

While on-premises storage vendors have lost the limelight of late, battles between on- and off-premises providers continue to be fought along lines of reducing costs and increasing functionality.

Indeed, the on-premises storage market continues to show healthy growth. According to Statista, the storage market (incorporating storage-intensive servers, external storage systems, and storage expansion systems, but not consumer grade products or network attached storage) is projected to reach a value of US$50.6 billion ($76.32 billion) this year, with an expected annual growth rate of 5.49 percent that will take its worth to US$62.6 billion ($94.42 billion) by 2028.

The research for Australia also shows growth, from US$800 million ($1.21 billion) in 2024 to US$1 billion ($1.51 billion) in 2028.

However, this finding is dwarfed by revenue reported for the global cloud storage market, which Statista estimates as being worth US$90.2 billion ($136.05 billion) in 2022, propelled by a compound annual growth rate of 23 percent that will see it valued at US$472.5 billion ($712.66 billion) in 2030.

Adapting to real world needs

A look inside data storage trends shows that the factors driving sales of on-premises solutions are numerous.

While security concerns regarding cloud solutions appear to have mostly faded, they have been replaced by worries relating to overall cost and data processing speeds. The former of these concerns arise from cloud storage being a service, and as other cloud services have shown, those costs can rise suddenly at the discretion of the provider, creating concerns about long term costs.

The need of some use cases to ensure that data collection, storage, and processing are located within close proximity is also increasing the attractiveness of on-premises solutions. Cloud providers are responding with edge services, but this requirement is also causing something of a renaissance for local storage-as-a-service (StaaS) providers.

According to Allied Market Research, over the next ten years the global use of StaaS will grow by at a compound rate of 28.2 percent from its 2022 value of US$34 billion ($51.28 billion). This will be fueled in part by the increasing demand of digital solutions and rise in integration of industry 4.0.

On-premises providers are also increasingly offering pricing models that mirror the offerings of cloud services, most specifically through the use of consumption-based pricing that alleviate the need for large upfront investments in infrastructure that may not be utilised for some time.

Keeping tabs on data storage requirements

One organisation that has come down firmly on the side of cloud data storage is Tabcorp. When Matt McKenzie joined the company as general manager of technology in October 2022 he took on responsibility for data (amongst other roles), inheriting seven different data platforms across four different technologies.

He set about consolidating those platforms, and in the third quarter of the calendar year expects to have all data sources consolidated onto a cloud-based platform, and he has a very solid reason for doing so.

"I am 100 percent of the belief that an organisation's ability to maximise its use of machine learning, AI, and by extension, GenAI, is by being able to have all the data in one place and have confidence in that data," McKenzie said.

He added that while he will initially gain a cost advantage through the consolidation, cost cutting was never the primary motivator, and he is aware of the potential for a cloud-based solution to become more expensive over time.

"It was very much a value creation program," McKenzie said.

"We have predicted how much data science and AI we will do, so if the costs do increase commensurately over time, we will only continue if we realise the value from it.

A consolidated approach

The consolidated approach McKenzie described is another key driver behind storage investment decisions. The desire for centralisation of data resources however has created something of a philosophical battle between solutions that bring all data together into a single location (such as a data warehouse, data lake, or more recently, a lakehouse configuration) and those that advocate virtual centralisation using data mesh or data fabric techniques.

The lakehouse concept specifically has won favour due to its ability to store both structured data (as in a data warehouse) and unstructured data (as in a data lake). According to Orbis Research, the global data lakehouse market will grow from US$4.6 billion ($6.94 billion) in 2024 to US$9 billion ($13.57 billion) in 2030.

On the flipside, data mesh architectures provide a data architecture that brings together different data sources by managing data sharing and governance centrally, enabling data to be owned and organised by specific domains or functions. This market is predicted by MarketsandMarkets to have been worth US$1.2 billion ($1.81 billion) in 2023, rising to US$2.5 billion ($3.77 billion) in 2028.

AI fuelling the future of storage

At least part of the reason why data models are so popular now is the important role they play in AI projects, and the criticality of having data stored close to where the AI model needs it.

However, AI is also proving useful when it comes to managing storage, due to its capability to readily assess the many variables associated with data storage and make recommendations that can improve performance and cost. AI powered features are expected to appear in tasks such as storage provisioning, capacity planning, and data protection.

The role of AI in data storage is also taking on a physical component, as highlighted by research from Gartner, which found that half of cloud data centres will deploy robots with AI capabilities by 2025, resulting in 30 percent higher operating efficiencies.

And with AI mimicking many of the processes of the human brain, perhaps it is not surprising then that another previously human attribute may play a greater role in data storage.

The ever-increasing growth in data volumes is pushing researchers towards more esoteric solutions, including looking at ways of encoding data in DNA sequences, taking advantage of the inherent efficiency of DNA's four letter code.

What is certain however is that with global data volumes growing at greater than 20 percent each year, the market for data storage in all its forms is only set to increase.