Howe big science is cutting big data down to size

How science is cutting big data down to size

A group of scientific researchers, who work  with datasets in the terabyte range, wants to develop a set of tools for data sharing, analysis and access for data management challenges across the scientific community.

Under the auspices of the National Science Foundation, the group has begun a project called SciServer, whose mission is to, “build a long term, flexible ecosystem” to provide access to datasets generated by astronomy and space science projects.

"By building a common infrastructure, we can create data access and analysis tools useful to all areas of science,” Alex Szalay of Johns Hopkins University, the leader of the NSF-funded project told

SciServer grew out of work with the Sloan Digital Sky Survey (SDSS), an ongoing project to map the entire universe. SDSS,  begun 15 years ago, now has over 70 terabytes in its database covering 220 million galaxies and 260 million stars.

But though the datasets are big, the researchers’ data management problems  are common to those found in many agency or business offices today, including:

  • Preserving data as file formats change and scientists retire.
  • Consistently storing and applying metadata for how the data should be used.
  • Providing equal access to data and expertise among researchers.
  • Encouraging opportunities for new insights enabled by combining data for joint analysis.

The SciServer team, which began working on the solutions to the problems in 2013, said they would launch the project in phases over the next four years. 

The tactics they will bring to the project include:

Bring the analysis to the data. “This means scientists can search and analyze big data without downloading terabytes of data, resulting in much faster processing times, Szalay said in a statement.

Specifiy real-world use cases. The SciServer team is collaborating to ensure the system will be most helpful to working scientists.

Develop new tools. To help ease the burden on researchers, the team developed "SciDrive," a cloud data storage system that allows scientists to upload and share data using a Dropbox-like interface.

Adapt existing working tools.  The strategy of building tools by adapting existing, successful tools is a key factor in ensuring the success of the project.

"The tools we build will create a fully-functional, user-driven system from the beginning, making SciServer an indispensable tool for doing science in the 21st century,” Szalay said.

As SciServer becomes more mature, the team will expand to other areas of science including genomics and connectomics , which explores cellular connections across the structure of the brain, according to the researchers. 

About the Author

Connect with the GCN staff on Twitter @GCNtech.


  • business meeting (Monkey Business Images/

    Civic tech volunteers help states with legacy systems

    As COVID-19 exposed vulnerabilities in state and local government IT systems, the newly formed U.S. Digital Response stepped in to help. Its successes offer insight into existing barriers and the future of the civic tech movement.

  • data analytics (

    More visible data helps drive DOD decision-making

    CDOs in the Defense Department are opening up their data to take advantage of artificial intelligence and machine learning tools that help surface insights and improve decision-making.

Stay Connected