NRC to OMB: Rethink evaluation tool
- By Trudy Walsh
- Feb 01, 2008
The National Research Council is recommending that the Office of Management and Budget adjust its evaluation tool for measuring the efficiency of research programs.
OMB uses a set of questions called the Program Assessment Rating Tool (PART) to evaluate research programs at the Environmental Protection Agency and other agencies.
OMB uses PART to find out whether agencies can measure and demonstrate annual improvements in efficiency. OMB then uses the answers to rate the research programs as effective, ineffective or somewhere in the middle.
Agencies who get an 'ineffective' rating can face serious consequences, according to an NRC report
EPA had been having trouble meeting OMB's requirements for efficiency, so the agency asked an NRC committee for help.
In the report the committee recommended four changes in how the federal government assesses the efficiency level of research at EPA and other agencies.
- Assessing efficiency should be considered only one part of evaluating a program's quality, relevance and effectiveness.
- There's a difference between 'investment efficiency' and 'process efficiency,' and the two kinds of efficiencies need to be evaluated in different ways. Investment efficiency should be assessed by examining whether an agency's R&D portfolio matches its strategic plan. The evaluations should be made by panels of experts, made up of scientists and other stakeholders. Process efficiency should be evaluated in terms of the inputs ' the people, funds and facilities dedicated to research ' and outputs ' the services, grants, publications, monitoring and new techniques produced by research.
- Likewise, there's a difference between outputs and outcomes. EPA's difficulties in answering PART's questions about efficiency have grown out of OMB's insistence that the agency find ways to measure the efficiency of its research based on outcomes rather than outputs. Measuring research efficiency based on 'ultimate outcomes,' such as a program that results in cleaner air or fewer deaths, is neither achievable nor valid. These sort of outcomes occur far in the future and are highly dependent upon actions take by many other people. The report does, however, endorse evaluating efficiency based on 'intermediate outcomes, such as disseminating newly developed tools and models.'
- EPA's research programs should be evaluated according to the same standards used at other agencies. OMB rejected some methods for measuring research efficiency when proposed by EPA but accepted them when proposed by other agencies, the report said.
The study was sponsored by EPA. The National Research Council functions under the auspices of the National Academy of Sciences, the National Academy of Engineering and the Institute of Medicine.
Trudy Walsh is a senior writer for GCN.