Why data drills should be part of emergency planning

It’s understood that data sharing across cities departments is useful -- particularly now as the local governments continue to deal with the coronavirus pandemic. What’s less clear, however, is how to go about it. A data expert at Georgetown University’s Beeck Center says data drills can help by uncovering challenges related to key stakeholders, governance structures, datasets and technology.

Data drills use real-world scenarios to surface those details through exercises. Although they work best before a crisis, they can still help during one, too.

Take COVID-19, for instance, said Amen Ra Mashariki, a fellow at the Beeck Center who created data drills when he was an adjunct professor at New York University’s Center for Urban Science and Progress. Many U.S. businesses and residents are looking to data on virus caseloads to determine reopening plans and safe returns to school. In this situation, a city could use data that it and the Centers for Disease Control and Prevention collected since March to predict where COVID-19 infection numbers will be in two, four and six months.

“You could actually begin to identify the most likely scenario,” Mashariki said. “If these things are this way two months from now, we should still see a downward trend. If these things shift, if we go back to indoor restaurants, it may trend up.”

Data drills help identify key players, avoiding situations like the one Mashariki saw in New York, where the mayor would ask a senior adviser a data-related question, the adviser would ask a commissioner at a related agency and the commissioner would ask the agency CIO, who would then ask the database manager.

“But in that line that I just walked you through, none of those people were likely to be the expert on data at that agency,” he said. That’s the person officials must identify. “It could be a general counsel. It could be an analyst,” he added.

For the COVID-19 scenario, cities must determine who will be their contact for CDC data and who will manage coronavirus-related data for the city. That might be an individual, a team or a collection of teams from several agencies. This point person or team would also determine whether data from other federal agencies -- in this case, others under the Health and Human Services Department -- would be helpful.

Another result of a data drill is identifying the needed datasets. “The biggest mistake around data analysis and reporting is when you use all sorts of noise,” said Mashariki, who is currently global director of the World Resources Institute’s Data Lab.  In an emergency like the pandemic, when the people at the top say they need answers now, our first reaction is “not to be surgical,” he said. We’re likely “to take everything we’ve got, throw it all in the bucket and throw it at this reporting tool. You lose accuracy and precision when you’re adding all of this data.”

In addition to fine-tuning what they need, agencies can identify stumbling blocks to obtaining the data, such as regulations and legal restrictions like the Health Insurance Portability and Accountability Act that prevents sharing certain kinds of data. 

“The data drills would uncover those things now so when you walk into a real-world scenario, you know  … key people to call when something goes bad,” Mashariki said. “Not only that, what are the key datasets we need to surface? Not only that, what datasets do we need to now go to Office of the Inspector General, Congress, wherever and get an OK ... and get access to the data we know is going to be useful?” 

Other challenges that will surface during data drills include reporting and technology needs. Reporting is one of the biggest mistakes cities make in that they often use one set of metrics, but different stakeholders -- scientists, government leaders, health practitioners, business owners and residents, for instance -- need different types, Mashariki said.

A benefit of data drills, he added, is that they don’t require a large technology investment. Mashariki said cities can start sharing data using Microsoft Excel spreadsheets and email and then build from there.

“Use it as a mechanism to understand what tools [are needed] to share data more effectively,” he said.

Cities that have conducted data drills include Baton Rouge, La.; Birmingham, Ala.; Chicago; Los Angeles; New York; and San Francisco. One in New York in 2016 involved a blackout scenario in Brooklyn, during which elevators shut down and people needed rescuing. City departments synthesized data from multiple agencies to find all the elevators in the region, predict which ones likely had injured riders inside and develop a strategy to dispatch emergency services to them.

“You’re entitled to your own opinion, but you’re not entitled to your own fact,” Mashariki said. “What data drills begin to move toward is forcing everyone to think about all sorts of different things, but basing it on the same data.”