Moving agricultural data from silos to the cloud
Connecting state and local government leaders
Built on a centralized cloud-based database, the Agricultural Research Service's Data Innovations project provides access to observational and real-time data from multiple sources.
The Agriculture Department’s Agricultural Research Service (ARS) has launched a program to make data more useful and effective by getting it out of silos.
Introduced earlier this month, the Data Innovations project involves deployment of sensor networks and data-collection and sharing systems. It has many moving parts that all point back to a centralized cloud-based database, where observational and real-time data from multiple sources will be housed. The agency has about 2,000 scientists and 6,000 support staff across more than 90 locations nationwide, each of which collects its own data.
“We are just a huge community of data silos,” said Michael Buser, national program leader for engineering at ARS. “How do we create a culture to where we can more openly share this data to really gain more value from it? We’re spending a lot of time and a lot of effort and a lot of money collecting this data, but I don’t believe that we’re really capturing all the value.”
He envisions, for example, modeling with weather data by pulling up an interactive GIS map of networked sensors in, say, Oklahoma, dragging a box across the state and having the system highlight all the weather stations. A box would pop up asking what specific data is necessary: wind speed and direction or soil moisture temperature. Another would let users set a time frame, from days to months to years, and another would ask what format the data should be in.
Currently, to get that information ARS scientists have to call their counterparts at weather stations in the state to request the data and then wait for someone to gather it and email it or ship a flash drive.
Data Innovations would eliminate kludgy, error-prone and time-consuming processes like that. The project comprises several efforts. In one, ARS stood up a centralized data repository called the Agricultural Collaborative Research Outcomes System, or AgCROS, in July 2018. ARS scientists and others can publish any data from their research projects into the cloud-based network-of networks system.
“The system has to be able to accommodate more than just ARS employees, and the big reason why is because we interact" with university faculty and other agencies, he said. "If I create an environment where I restrict access to this just to our ARS scientists, then it’s not very effective for our scientists because it doesn’t help them in that collaborative process,” he added.
Over the summer, ARS took advantage of its ability to quickly create customized apps for researchers through Esri tools with AgVoice, a mobile voice-interaction service for food and agriculture professionals. “Now I can collect the data hands-free, eyes-free,” Buser said. “Really, the key in any of these apps, it’s got to be functional for the people entering the data, otherwise it’s never going to get used.”
ARS employees also collect real-time or continuous monitoring data from weather station sensors that send readings as often as 10 times per second. Buser wants to develop, through Data Innovations, automated quality assurance/quality control (QA/QC) plans to reduce the amount of time scientists spend on those datasets.
He also wants to ease the process of taking measurements. “We weigh a lot of stuff,” Buser said. Labs are full of scales with forms on clipboards nearby. "We’ll take the sample, we’ll set it on the scale, we’ll write down the sample ID. Once that scale stabilizes, we’ll write down the weight. Once we finish weighing all those samples, we’ll take that sheet back to a laptop or a computer and enter that information into Excel -- not very efficient and room for lots of errors.”
Off-the-shelf software exists to automate that process, but it’s expensive, he said. Instead, ARS hired a programmer to develop software that meets scientists’ specifications.
Still, much of this new data work is happening through public/private partnerships like the one with AgVoice. At a demonstration at the Beltsville Area Research Center in Maryland on Oct. 7, ARS showed how its partnerships with companies such as Esri and Microsoft are advancing data-driven agricultural technology. For instance, it demoed how data coming into AgCROS integrates with Esri’s ArcGIS.
Buser sees potential beyond data collection, however. ARS develops models and decision-support tools, an exercise that a centralized repository will facilitate, he said.
“The other thing we’re looking at doing is developing a cloud-based platform for our models and our decision-support tools,” Buser said. By housing them in the cloud, ARS can link them back to ongoing science and link that to AgCROS to create a larger, more scientifically sound data repository that can give those tools more power.
NEXT STORY: JEDI award signals rise of hybrid cloud