AI-enabled drone maps disaster victims' location, need
Connecting state and local government leaders
DroneAid uses visual recognition to detect and count SOS icons on the ground from drone video streams and automatically plots the emergency needs on a map for first responders.
An open-source disaster response tool that uses visual recognition and learns through artificial intelligence and cloud tools began as an idea that a self-taught developer had at IBM’s Call for Code hackathon in Puerto Rico last year.
IBM announced DroneAid on Oct. 2 as an open-source project through Code and Response, the company’s $25 million program dedicated to the creation and deployment of open-source solutions tackling real-world problems.
DroneAid uses visual recognition technology to detect and count SOS icons on the ground gleaned from drone video streams and automatically plots the emergency needs on a map for first responders. Developer Pedro Cruz had planned to use optical character recognition to detect messages, but reading different handwriting and languages complicated that approach. Instead, the tool relies on a subset of the U.N. Office for the Coordination of Humanitarian Affairs’ 500 humanitarian icons – symbols that DroneAid can learn and first responders can quickly understand. For example, an icon of a drop of water represents a need for water and a medical kit a need for medicine.
Cruz put the symbols on eye-catching physical mats, though disaster victims can also draw the icons with spray paint or chalk on asphalt or rooftops. “My vision was that if these nonprofit organizations or the government could distribute these mats that have the symbols," so people could indicate what they need to drones flying overhead, said Cruz, now an IBM developer advocate.
To recognize the images -- even when they are faded or distorted -- DroneAid uses visual recognition and AI to learn the symbols. The company’s cloud annotations tool, which is also open source, was used to train the model to find and/or label images with help from IBM Cloud Object Storage.
Next, Cruz applied the solution to a live stream of images coming from a drone as it surveyed an area. DroneAid analyzes the video frame by frame, and when it finds an icon, it pulls the location and time from the video frame’s metadata and then automatically plots the information on a map for first responders.
“Once we have the machine, or the model, ready, we deploy it to the browser,” Cruz said. “Right now we are using a library called TensorFlow.js, which allows to do all the analysis on the browser. You wouldn’t even have to download an application.”
The tool works with any drone that can capture video. In fact, it’s evolved to be something of a misnomer. “The good thing about DroneAid is that it does not require a drone, actually,” Cruz said. “It can work with satellite imagery, it can work with images from social media. It’s really drone-agnostic.”
Cruz made DroneAid open source so that contributors worldwide could add images -- such as floods in India or earthquakes in California -- that would make the tool more robust. “Making it open source is very powerful because of that,” he said.
Cruz, who’s from Puerto Rico, came up with the idea after surviving the havoc Hurricane Maria wrought on the island in 2017. He worked with nonprofits to deliver food and other supplies to people in need, but the process was inefficient, he said, because sometimes they’d arrive with water only to find out the Red Cross had been by with some the previous day.
He also flew his personal drone and captured images of the damage -- plus handwritten messages asking for help or letting the world know the population of a given town was unharmed.
He combined those two experiences to build DroneAid at the hackathon in Bayamón, Puerto Rico, in August 2018. It won first place.
“I took the inspiration from the images that I saw, and I thought about a way that we could automate” assistance, Cruz said.
Currently, a demo with a toy-like Tello drone is available, but Cruz said the plan is to connect to other consumer drones in the future. He said expects to see the first pilot tests of DroneAid take shape in the next couple months.