For storage managers, the coming years will be challenging, to say the least.
According to a recent survey by the 1105 Public Sector Media Group, agency data now is increasing approximately 40 percent each year. And some agencies have it worse than others: 11 percent of respondents said the volume of data at their agencies was increasing 75 percent annually.
Unfortunately, the increase in data volume often is not matched by an increase in storage spending, noted Henry Baltazar, an infrastructure and operations analyst at Forrester Research, in a recent report.
And if they do have budgets, organizations sometimes simply add new platforms without a lot of forethought, “creating even more silos within data center environments and further complicating the storage landscape,” Baltazar wrote.
Storage managers also are dealing with a wide variety of data types. Structured data remains the most common, but now it only accounts for about a third of all data. And the volume of real-time data (25 percent overall) and semi-structured (23 percent) is expected to continue rising.
As might be expected, video and audio are among the leading sources of unstructured data, having a measurable impact at 68 percent of agencies. But geospatial data, at 59 percent, is not far behind. Geospatial data could be more and more of a challenge for storage managers in the future.
In a report issued last fall, the National Digital Stewardship Alliance, a consortium of organizations committed to the long-term preservation of digital information, noted that when it comes to archiving data, many organizations often have a “keep everything” policy. That is because the cost of adding storage often is lower than the costs associated with constantly reviewing data for possible deletion.
But the dynamics might be different with geospatial data. “The sheer size, in terms of file size and number of files, of many geospatial resources —especially in the case of raster data —in combination with the limited nature of storage capacity…may affect decisions with regard to disposition and archival acquisition of data,” the report stated.
The increasing interest in big data across government likely will create even more storage challenges. Big data initiatives typically involve both high volumes and multiple types of data. Consequently, agencies are considering a number of possible storage solutions.
For example, 53 percent of agencies that have implemented big data initiatives will invest more in data warehousing during the next 12 months, while 37 percent of those agencies will invest in a scale-out or clustered network-attached storage architecture.
Eventually, storage managers might get additional relief with the emergence of software-defined storage. SDS, like server virtualization, makes it possible to pool storage resources, giving managers much more flexibility in how they manage those resources.
With SDS, “clients or applications can request specific performance and capacity requirements and have this storage resource delivered without the intervention of traditional storage administrators,” wrote Forrester's Baltazar.