Ghosts in the machine

Error-riddled databases haunt Justice


The misguided use of technology
by Justice Department officials is
partially responsible for flaws in the statistics
that track federal counterterrorism
activities, department auditors
found during an extensive analysis of
the numbers.

Auditors with the Office of the Inspector
General evaluated the methods used
by three major Justice agencies to gather,
process and report critical statistics
about their counterterrorism work.

The indexes included terrorism-related
convictions, the number of prison
sentences resulting from terrorism
prosecutions, the production of terrorism
intelligence reports, the number of
terrorism threats tracked and the output
of threat assessments. The IG's staff
found that some of the statistics were
understated and others overstated, with
no consistent pattern.

Meanwhile, officials of three Justice
agencies that gather and report counterterrorism
statistics varied in their responses
to the IG's analysis of their data
and statistics practices.

The FBI generally concurred with the
auditors' findings and described system
upgrades and procedure changes that
officials will use to improve the bureau's
internal controls.

In contrast, officials in the Executive
Office of U.S. Attorneys flatly rejected
many of the auditors' findings,
calling the findings inaccurate or

The department's Criminal Division
also found fault with the report and
called for substantial corrections.

The report, titled 'The Department
of Justice's Internal Controls Over Terrorism
Reporting,' took as
its starting point a group
of 209 terrorism-related
statistics that Justice and
its components reported
602 times between October
2000 and September
2005. Some of the report's
analysis continued into

From the initial group of
209 statistics, the auditors
eliminated 17 indexes that
agencies outside Justice
had generated.

In reviewing the remaining
192 statistics by interviewing
Justice officials,
the auditors found that
'the collection and reporting
of terrorism-related
statistics within the department
is decentralized
and haphazard.'

The auditors found several
causes for the inaccuracy
of Justice's counterterrorism
statistics, including
faulty technology,
internal controls and documentation.

Some of the technology
problems resulted from
improper use of spreadsheet
applications that
process terrorism statistics.

The auditors found that 'the statistics
reported by the Criminal Division
were inaccurate primarily because the
database used to track the statistics
was incomplete and not kept up-todate.'
The Criminal Division had failed
to properly train the database users
and neglected to validate the information
generated by the system, the report

In its response, the FBI cited its project
to create a new central-management
information system called Compass,
which is designed to centralize
statistical controls and furnish management
data to senior officials.

The bureau said its massive systems
upgrade project, now called Sentinel,
which consists largely of an investigative
case management system, has
been tweaked so it will provide additional
management information and
controls, partly to track the FBI's
counterterrorism work.

The FBI described a system called
Guardian that it launched in July
2004 to centralize management controls.
The bureau started using an upgrade
called Guardian Version 2.0 in autumn
2006, with improved metrics and better
terrorism threat tracking, the FBI
told the IG.

Stay Connected

Sign up for our newsletter.

I agree to this site's Privacy Policy.