Utah shares hard lessons on privacy, bias
The state auditor’s software procurement principles are designed to help agencies review applications for privacy violations and potential algorithmic bias.
Last year, it was revealed that the CEO of Banjo, the company Utah’s state government had hired to deliver public safety analytics for several agencies, had previous associations with KKK groups and participated in a drive-by terrorist attack on a synagogue. Shortly after that revelation, the Banjo CEO resigned, the contracts were cancelled and Utah State Auditor John Dougall formed a commission to review the contracts and the application for privacy violations and potential algorithmic bias.
Banjo, a Utah-based surveillance analytics company, scrapes public data from billions of social media accounts, as well as data from government traffic and surveillance cameras, 911 calls, audio sensors, location data for state-owned vehicles and other sources. It uses artificial intelligence to analyze the data streams and detect anomalies, giving public safety agencies near real-time information on incidents like car crashes, medical emergencies, shootings, fires and dangerous road conditions. Banjo secured several Utah government contracts with support from Attorney General Sean Reyes, according to the Salt Lake Tribune.
Utah’s Commission on Protecting Privacy and Preventing Discrimination was formed in June 2020. It was composed of technical, public policy, business and public safety experts and charged with reviewing Banjo’s contract with Utah and identifying “actions that governmental entities should take to ensure their software systems protect Utahns’ privacy and prevent discrimination against them,” Dougall said.
On Feb. 1, the commission released key recommendations to help state and local governmental entities make thoughtful decisions about emerging technologies, such as artificial intelligence or machine learning, that collect personally identifiable information or other sensitive data and could impact the privacy of Utah residents or lead to discrimination, the State Auditor’s Office said in its announcement.
The 12 software application procurement principles are briefly described in one document, with a companion document providing questions agencies should ask when applying the principles for evaluating software that may impact privacy or cause discrimination.
The 12 principles recommend that agencies:
- Limit sharing of sensitive data.
- Minimize sensitive data collection and accumulation.
- Validate technology claims with a capability review.
- Rely on objective, repeatable metrics.
- Assess the vendors’ threat models.
- Perform in-depth review of artificial intelligence and machine learning software.
- Verify the vendors’ privacy compliance.
- Review the steps vendors take to mitigate discrimination.
- Determine the vendors’ privacy and non-discrimination procedures for ongoing validation.
- Require vendors to obtain consent of individuals contained within training datasets.
- Vet key vendor personnel.
- Evaluate vendor corporate management and vendor solvency.
As the commission began its work, “it became clear that the availability of some clear, thoughtful principles aimed at these emerging technologies would be a powerful resource for Utah’s government agencies,” Dougall said.
“I expect that the report that we have produced will provide very effective guidance for state agencies,” said Suresh Venkatasubramanian, a professor at the University of Utah’s School of Computing. “[It] places Utah at the forefront of efforts around transparency, privacy protection and bias reduction, especially when using new AI-based technologies.”