December 5, 2021

News

News Network

Program Evaluation: Key Terms and Concepts

9 min read
<div>Both the executive branch and congressional committees need evaluative information to help them make decisions about the programs they oversee—information that tells them whether, and why, a program is working well or not. The Government Performance and Results Act of 1993 (GPRA) and GPRA Modernization Act of 2010 (GPRAMA) established a framework for performance management and accountability within the federal government. Building on that foundation, Congress has since passed, among other laws, the Foundations for Evidence-Based Policymaking Act of 2018 (Evidence Act) to strengthen the evidence-building efforts of executive branch agencies. This product updates our previous glossary (GAO-11-646SP) to highlight different types of evaluations for answering questions about program performance, as well as relevant issues to ensure study quality. This glossary can help agency officials better understand fundamental concepts related to evaluation and enhance their evidence-building capacity. For more information, contact Lawrance Evans, Jr. at 202-512-2700 or EvansL@gao.gov.</div>

Both the executive branch and congressional committees need evaluative information to help them make decisions about the programs they oversee—information that tells them whether, and why, a program is working well or not. The Government Performance and Results Act of 1993 (GPRA) and GPRA Modernization Act of 2010 (GPRAMA) established a framework for performance management and accountability within the federal government. Building on that foundation, Congress has since passed, among other laws, the Foundations for Evidence-Based Policymaking Act of 2018 (Evidence Act) to strengthen the evidence-building efforts of executive branch agencies. This product updates our previous glossary (GAO-11-646SP) to highlight different types of evaluations for answering questions about program performance, as well as relevant issues to ensure study quality. This glossary can help agency officials better understand fundamental concepts related to evaluation and enhance their evidence-building capacity.

For more information, contact Lawrance Evans, Jr. at 202-512-2700 or EvansL@gao.gov.

More from:

News Network

  • U.S. Participation in the Fall 2021 Ad Hoc Liaison Committee Meeting for the Palestinians
    In Crime Control and Security News
    Office of the [Read More…]
  • F-35 Sustainment: DOD Needs to Address Key Uncertainties as It Re-Designs the Aircraft’s Logistics System
    In U.S GAO News
    The Autonomic Logistics Information System (ALIS) is integral to supporting F-35 aircraft operations and maintenance. However, F-35 personnel at 5 locations GAO visited for its March 2020 report cited several challenges. For example, users at all 5 locations we visited stated that electronic records of F-35 parts in ALIS are frequently incorrect, corrupt, or missing, resulting in the system signaling that an aircraft should be grounded in cases where personnel know that parts have been correctly installed and are safe for flight. At times, F-35 squadron leaders have decided to fly an aircraft when ALIS has signaled not to, thus assuming operational risk to meet mission requirements. GAO found that DOD had not (1) developed a performance-measurement process for ALIS to define how the system should perform or (2) determined how ALIS issues were affecting overall F-35 fleet readiness, which remains below warfighter requirements. DOD recognizes that ALIS needs improvement and plans to leverage ongoing re-design efforts to eventually replace ALIS with a new logistics system. However, as DOD embarks on this effort, it faces key technical and programmatic uncertainties (see figure). Uncertainties about the Future F-35 Logistics Information System These uncertainties are complicated and will require significant planning and coordination with the F-35 program office, military services, international partners, and the prime contractor. For example, GAO reported in March 2020 that DOD had not determined the roles of DOD and the prime contractor in future system development and management. DOD had also not made decisions about the extent to which the new system will be hosted in the cloud as opposed to onsite servers at the squadron level. More broadly, DOD has experienced significant challenges sustaining a growing F-35 fleet. GAO has made over 20 recommendations to address problems associated with ALIS, spare parts shortages, limited repair capabilities, and inadequate planning. DOD has an opportunity to re-imagine the F-35's logistics system and improve operations, but it must approach this planning deliberately and thoroughly. Continued attention to these challenges will help ensure that DOD can effectively sustain the F-35 and meet warfighter requirements. The F-35 Lightning II is DOD's most ambitious and costly weapon system in history, with total acquisition and sustainment costs for the three U.S. military services who fly the aircraft estimated at over $1.6 trillion. Central to F-35 sustainment is ALIS—a complex system that supports operations, mission planning, supply-chain management, maintenance, and other processes. A fully functional ALIS is critical to the more than 3,300 F-35 aircraft that the U.S. military services and foreign nations plan to purchase. Earlier this year, DOD stated that it intends to replace ALIS with a new logistics system. This statement highlights (1) current user challenges with ALIS and (2) key technical and programmatic uncertainties facing DOD as it re-designs the F-35's logistics system. This statement is largely based on GAO's March 2020 report on ALIS ( GAO-20-316 ), as well as previous F-35 sustainment work. GAO previously recommended that DOD develop a performance-measurement process for ALIS, track how ALIS is affecting F-35 fleet readiness, and develop a strategy for re-designing the F-35's logistics system. GAO also suggested that Congress consider requiring DOD to develop a performance-measurement process for its logistics system. DOD concurred with GAO's recommendations and is taking actions to address them. For more information, contact Diana C. Maurer at (202) 512-9627 or maurerd@gao.gov.
    [Read More…]
  • Japan Travel Advisory
    In Travel
    Reconsider travel to [Read More…]
  • Justice Department and FTC File Suit to Stop Deceptive Marketing of Nasal Spray Product Advertised as Purported COVID-19 Treatment
    In Crime News
    The Department of Justice, together with the Federal Trade Commission (FTC), Thursday announced a civil enforcement action against defendants Xlear Inc. and Nathan Jones for alleged violations of the COVID-19 Consumer Protection Act and the FTC Act.
    [Read More…]
  • The U.S.’ Action Against Belarusian Individuals Involved in Efforts To Undermine Belarusian Democracy
    In Crime Control and Security News
    Michael R. Pompeo, [Read More…]
  • Priority Open Recommendations: Board of Governors of the Federal Reserve System
    In U.S GAO News
    What GAO Found In April 2020, GAO identified eight priority recommendations for the Federal Reserve. Since then, the Federal Reserve has implemented five of those recommendations. As of April 2021, the remaining open three priority recommendations for the Federal Reserve involve the following areas: Collaborating with other financial regulators to communicate with banks that have third-party relationships with fintech lenders about using alternative data in underwriting. Communicating uncertainties surrounding stress testing, including capital ratio estimates. Communicating uncertainties surrounding stress testing, including tolerance levels for key risks, and the degree of uncertainty in projected estimates. The Federal Reserve's continued attention to these issues could improve its ability to more effectively oversee risks to consumers and the safety and soundness of the U.S. banking system. Why GAO Did This Study Priority open recommendations are GAO recommendations that warrant priority attention from heads of key departments or agencies because their implementation could save large amounts of money; improve congressional or executive branch decision-making on major issues; eliminate mismanagement, fraud, and abuse; or ensure that programs comply with laws and funds are legally spent, among other benefits. Since 2015 GAO has sent letters to selected agencies to highlight the importance of implementing such recommendations. For more information, contact Daniel Garcia-Diaz at (202) 512-8678 or garciadiazd@gao.gov.
    [Read More…]
  • Close Air Support: Actions Needed to Enhance Friendly Force Tracking Capabilities and Fully Evaluate Training
    In U.S GAO News
    The Department of Defense (DOD) has made progress implementing initiatives to enhance capabilities that are used to identify friendly force locations during close air support (CAS) missions, but GAO identified additional actions that are needed to strengthen these efforts. Specifically, DOD has made limited progress in implementing 10 changes the department approved to address gaps in the interoperability of digital communications systems used to conduct CAS, hindering efforts to improve the speed and accuracy of information exchanges. DOD's efforts to assess the interoperability of digital systems used to perform CAS have been limited in scope. GAO found that DOD had formally assessed two out of 10 approved changes during joint service and multinational events, and these assessments were not conducted in a training environment that replicated capabilities of near-peer adversaries. DOD implemented a new capability in the U.S. Central Command area of responsibility to help identify the positions of friendly forces during CAS missions. However, GAO found that DOD did not provide adequate training for personnel who operate it or conduct an evaluation to resolve implementation challenges that have hampered its performance. DOD conducts evaluations of training programs for forces that participate in CAS missions, but GAO identified two areas where DOD can improve its efforts. First, the Army and Marine Corps have not systematically evaluated the effectiveness of periodic training for ground observers providing targeting information due to a lack of centralized systems for tracking training data and the absence of designated entities to monitor service-wide training. Second, the use of contract aircraft for training increased substantially between 2017 and 2019, but DOD has not fully evaluated the use of non-military contract aircraft to train air controllers for CAS (see fig.). GAO found that differences between U.S. military aircraft and contract aircraft (e.g., airspeed) can result in a misalignment of aircraft capabilities for certain types of training events. Without evaluating CAS training fully, DOD cannot have assurance that its forces are prepared to conduct CAS missions safely and effectively. Number of Hours Non-Military Aircraft Were Used to Train for Close Air Support for Fiscal Years 2017 through 2019 The use of ordnance delivered by aircraft to support U.S. military forces that are in close proximity to enemy forces on the ground requires detailed planning, seamless communications, and effective training. Mistakes in communications or procedures used to identify and maintain an awareness of the positions of friendly forces on the battlefield during CAS can result in the loss of U.S. military personnel. Senate Report 116-48 and House Report 116-120, accompanying bills for the National Defense Authorization Act for Fiscal Year 2020, included provisions for GAO to evaluate issues related to friendly-force identification capabilities in CAS missions. Among other things, this report evaluates the extent to which DOD has (1) implemented initiatives to enhance friendly-force identification capabilities during CAS, and (2) evaluated training for forces that participate in CAS. GAO analyzed documentation and interviewed officials regarding DOD efforts to develop and implement friendly force tracking capabilities for CAS; reviewed CAS training programs; and analyzed training data, including the number of hours that DOD used non-military contract aircraft for CAS training from 2017 through 2019. GAO is making 11 recommendations to DOD, including that DOD implement and assess initiatives to improve the interoperability of digital systems used in CAS and take additional steps to evaluate the training for certain forces that participate in CAS missions. DOD concurred with the recommendations. For more information, contact Cary Russell at (202) 512-5431 or RussellC@gao.gov.
    [Read More…]
  • Secretary Antony J. Blinken at Top of Meeting with German Foreign Minister Heiko Maas
    In Crime Control and Security News
    Antony J. Blinken, [Read More…]
  • Six Additional Individuals Indicted On Antitrust Charges In Ongoing Broiler Chicken Investigation
    In Crime News
    A federal grand jury in the U.S. District Court in Denver, Colorado, returned a superseding indictment charging six additional defendants for their roles in a previously indicted conspiracy to fix prices and rig bids for broiler chicken products, and containing additional allegations against the previously charged defendants in the same conspiracy, the Department of Justice announced today.  The superseding indictment also charges one defendant with making false statements and obstruction of justice. 
    [Read More…]
  • Chinese National Sentenced for Laundering Millions for Mexican Drug Cartels
    In Crime News
    A Chinese national was sentenced today to five years in prison and ordered to forfeit more than $4.2 million for laundering drug proceeds generated by large-scale cocaine trafficking in the United States.
    [Read More…]
  • Netherlands Travel Advisory
    In Travel
    Reconsider travel to the [Read More…]
  • Even During COVID, Courts Find Ways to Welcome New Americans
    In U.S Courts
    When the coronavirus (COVID-19) pandemic first forced courthouses to limit access to the general public, one of the first events to be canceled was an especially joyous rite: the naturalization of new U.S. citizens.
    [Read More…]
  • Vanuatu Travel Advisory
    In Travel
    Reconsider travel to [Read More…]
  • Facial Recognition Technology: Federal Law Enforcement Agencies Should Have Better Awareness of Systems Used By Employees
    In U.S GAO News
    What GAO Found In June 2021, GAO reported the results of its survey of 42 federal agencies that employ law enforcement officers about their use of facial recognition technology. Twenty reported owning systems with the technology or using systems owned by other entities, such as state, local, and non-government entities (see figure). Ownership and Use of Facial Recognition Technology Reported by Federal Agencies that Employ Law Enforcement Officers Note: For more details, see figure 1 in GAO-21-105309. Agencies reported using the technology to support several activities (e.g., criminal investigations) and in response to COVID-19 (e.g., verify an individual's identity remotely). Six agencies reported using the technology on images of the unrest, riots, or protests following the killing of Mr. George Floyd in May 2020. Three agencies reported using it on images of the U.S. Capitol attack on January 6, 2021. Agencies said the searches used images of suspected criminal activity. Fourteen of the 42 agencies reported using the technology to support criminal investigations. However, only one had a mechanism to track what non-federal systems were used by employees. By having a mechanism to track use of these systems and assessing the related risks (e.g., privacy and accuracy-related risks), agencies can better mitigate risks to themselves and the public. Why GAO Did This Study Federal agencies that employ law enforcement officers use facial recognition technology to assist criminal investigations, among other activities. For example, the technology can help identify an unknown individual in a photo or video surveillance. This statement describes (1) the ownership and use of facial recognition technology by federal agencies that employ law enforcement officers, (2) the types of activities these agencies use the technology to support, and (3) the extent that these agencies track employee use of facial recognition technology owned by non-federal entities, including the potential privacy and accuracy implications. This statement is based on GAO's June 2021 report on federal law enforcement's use of facial recognition technology (GAO-21-518). To conduct that prior work, GAO administered a survey questionnaire to 42 federal agencies that employ law enforcement officers regarding their use of the technology. GAO also reviewed relevant documents and interviewed agency officials. The June 2021 report was a public version of a sensitive report that GAO issued in April 2021. Information that agencies deemed sensitive was omitted from the June 2021 report and this statement.
    [Read More…]
  • Medicare Part B: Payments and Use for Selected New, High-Cost Drugs
    In U.S GAO News
    Hospital outpatient departments perform a wide range of procedures, including diagnostic and surgical procedures, which may use drugs that Medicare considers to function as supplies. If the drug is new, and its cost is high relative to Medicare's payment for the procedure, then hospitals can receive a separate “pass-through” payment for the drug in addition to Medicare's payment for the procedure. These pass-through payments are in effect for 2 to 3 years. When the pass-through payments expire, Medicare no longer pays separately for the drug, and payment for the drug is “packaged” with the payment for the related procedure. The payment rate for the procedure does not vary by whether or not the drug is used. Medicare intends this payment rate to be an incentive for hospitals to furnish services efficiently, such as using the most cost-efficient items that meet the patient's needs. Examples of Types of Drugs that Medicare Considers to Function as Supplies GAO's analysis of Medicare data showed that higher payments were associated with six of seven selected drugs when they were eligible for pass-through payments versus when their payments were packaged. For example, one drug used in cataract removal procedures was eligible for pass-through payments in 2017. That year, Medicare paid $1,824 for the procedure and $463 for the drug pass-through payment—a total payment of $2,287. If a hospital performed the same cataract removal procedure when the drug was packaged the following year, there was no longer a separate payment for the drug. Instead, Medicare paid $1,921 for the procedure whether or not the hospital used the drug. Of the seven selected drugs, GAO also reviewed differences in use for four of them that did not have limitations on Medicare coverage during the time frame of GAO's analysis, such as coverage that was limited to certain clinical trials. GAO found that hospitals' use of three of the four drugs was lower when payments for the drugs were packaged. This was consistent with the financial incentives created by the payment system. In particular, given the lower total payment for the drug and procedure when the drug is packaged, hospitals may have a greater incentive to use a lower-cost alternative for the procedure. Hospitals' use of a fourth drug increased regardless of payment status. The financial incentives for that drug appeared minimal because the total payment for it and its related procedure was about the same when it was eligible for pass-through payments and when packaged. Other factors that can affect use of the drugs include the use of the drugs for certain populations and whether hospitals put the drugs on their formularies, which guide, in part, whether the drug is used at that hospital. The Department of Health and Human Services reviewed a draft of this report and provided technical comments, which GAO incorporated as appropriate. Medicare makes “pass-through” payments under Medicare Part B when hospital outpatient departments use certain new, high-cost drugs. These temporary payments are in addition to Medicare's payments for the procedures using the drugs. They may help make the new drugs accessible for beneficiaries and also allow Medicare to collect information on the drugs' use and costs. The Consolidated Appropriations Act, 2018 included a provision for GAO to review the effect of Medicare's policy for packaging high-cost drugs after their pass-through payments have expired. This report describes (1) the payments associated with selected high-cost drugs when eligible for pass-through payments versus when packaged, and (2) hospitals' use of those drugs when eligible for pass-through payments versus when packaged. GAO reviewed federal regulations on pass-through payments and Medicare payment files for all seven drugs whose pass-through payments expired in 2017 or 2018 and that were subsequently packaged. All of these drugs met Medicare's definition for having a high cost relative to Medicare's payment rate for the procedure using the drug. GAO also reviewed Medicare claims data on the use of the drugs for 2017 through 2019 (the most recent available). To supplement this information, GAO also interviewed Medicare officials, as well officials from 11 organizations representing hospitals, physicians, and drug manufacturers, about payment rates, use, reporting, and clinical context for the drugs. For more information, contact James Cosgrove at (202) 512-7114 or cosgrovej@gao.gov.
    [Read More…]
  • Local tax preparer charged with fraudulently filing tax returns
    In Justice News
    A local man who had [Read More…]
  • Biogen Agrees To Pay $22 Million To Resolve Alleged False Claims Act Liability For Paying Kickbacks
    In Crime News
    Pharmaceutical company Biogen, Inc. (Biogen), based in Cambridge, Massachusetts, has agreed to pay $22 million to resolve claims that it violated the False Claims Act by illegally using  foundations as a conduit to pay the copays of Medicare patients taking Biogen’s multiple sclerosis drugs, Avonex and Tysabri, the Justice Department announced today. 
    [Read More…]
  • Three Former U.S. Intelligence Community and Military Personnel Agree to Pay More Than $1.68 Million to Resolve Criminal Charges Arising from Their Provision of Hacking-Related Services to a Foreign Government
    In Crime News
    On Sept. 7, U.S. citizens, Marc Baier, 49, and Ryan Adams, 34, and a former U.S. citizen, Daniel Gericke, 40, all former employees of the U.S. Intelligence Community (USIC) or the U.S. military, entered into a deferred prosecution agreement (DPA) that restricts their future activities and employment and requires the payment of $1,685,000 in penalties to resolve a Department of Justice investigation regarding violations of U.S. export control, computer fraud and access device fraud laws.
    [Read More…]
  • Secretary Blinken’s Call with Japanese Foreign Minister Motegi
    In Crime Control and Security News
    Office of the [Read More…]
  • Border Security Metrics: Progress Made, but DHS Should Take Additional Steps to Improve Information Quality
    In U.S GAO News
    What GAO Found In its fiscal year 2019 Border Security Metrics Report , the Department of Homeland Security (DHS) reported information on 37 of the 43 metrics required by the National Defense Authorization Act for Fiscal Year 2017 (NDAA). As shown below, GAO found that 18 of the 37 metrics in the fiscal year 2019 report generally corresponded with their descriptions in the NDAA, while 19 metrics differed, such as in scope or calculation. How the Department of Homeland Security (DHS) Reported on the 43 Metrics Required by the National Defense Authorization Act for Fiscal Year 2017 (NDAA) DHS improved the quality of information in several metrics in the fiscal year 2019 report compared with earlier versions. However, data reliability issues remain—such as DHS not fully disclosing limitations of some metrics—because DHS does not have a process to systematically review the data. Implementing GAO's prior recommendations to develop and implement a process to systematically review the reliability of data—and communicate limitations identified through this process—would position DHS to maximize the quality of information and provide Congress and the public with contextual information needed to evaluate the metrics. DHS used a statistical model to estimate four metrics, including the estimated number of undetected unlawful entries between ports of entry. In response to one of GAO's prior recommendations, DHS, in its fiscal year 2019 Border Security Metrics Report , showed how the sensitivity of three assumptions affected the precision of the model's estimates. For example, the model assumed that DHS apprehends all families unlawfully crossing the border, and the fiscal year 2019 report outlined how this assumption affected the model-based metrics in 2018. This sensitivity analysis will allow Congress and the public to better understand the limitations of DHS's model and better evaluate the validity of border security metrics derived from it. In response to another GAO recommendation, DHS conveyed statistical uncertainty for one of the unlawful entry metrics in its fiscal year 2019 report. In particular, DHS outlined how this uncertainty might affect the model-based apprehension rate. However, DHS did not report how uncertainty would affect the other three metrics that rely on its statistical model. As GAO previously recommended, including measures of statistical uncertainty for all relevant metrics in future reports would allow Congress, policy makers, and the public to more fully evaluate the extent to which the metrics are valid. Why GAO Did This Study The United States has approximately 6,000 miles of land borders, 95,000 miles of coastline, and more than 300 ports of entry where travelers and cargo are inspected. Securing U.S. border areas is a key part of DHS's mission, and its ability to measure border security activities is essential to managing its responsibilities effectively and efficiently. The 2017 NDAA requires DHS to report annually on 43 border security metrics. The act also includes a provision for GAO, within 270 days of receipt of the first report and biennially for the following 10 years, to review and report on the data and methodology in DHS's report. GAO issued its initial report in March 2019. This second report evaluates the report DHS issued in August 2020, which is known as the fiscal year 2019 report and contains data through fiscal year 2018. This report addresses the extent to which DHS (1) reported metrics outlined in the 2017 NDAA using quality information and (2) has taken steps to determine and convey the sensitivity of key assumptions and the statistical uncertainty of its unlawful entry metrics. GAO assessed the methodology and data in DHS's report, analyzed DHS's use of statistical modeling, and interviewed officials from DHS offices and components.
    [Read More…]
Network News © 2005 Area.Control.Network™ All rights reserved.