Updated Guidelines

for Evaluating Public Health

Surveillance Systems

Recommendations from

the Guidelines Working Group

U.S. DEPARTMENT OF HEALTH AND HUMAN SERVICES

Centers for Disease Control and Prevention (CDC)

Atlanta, GA 30333

July 27, 2001 / Vol. 50 / No. RR-13

Updated Guidelines

for Evaluating Public Health Surveillance Systems

Recommendations from the Guidelines Working Group

Summary

The purpose of evaluating public health surveillance systems is to ensure that problems of public health importance are being monitored efficiently and effectively. CDC’s Guidelines for Evaluating Surveillance Systems are being updated to address the need for a) the integration of surveillance and health information systems, b) the establishment of data standards, c) the electronic exchange of health data, and d) changes in the objectives of public health surveillance to facilitate the response of public health to emerging health threats (e.g., new diseases). This report provides updated guidelines for evaluating surveillance systems based on CDC’s Framework for Program Evaluation in Public Health , research and discussion of concerns related to public health surveillance systems, and comments received from the public health community. The guidelines in this report describe many tasks and related activities that can be applied to public health surveillance systems.

INTRODUCTION

In 1988, CDC published Guidelines for Evaluating Surveillance Systems ( 1) to pro-mote the best use of public health resources through the development of efficient and effective public health surveillance systems. CDC’s Guidelines for Evaluating Surveillance Systems are being updated to address the need for a) the integration of surveillance and health information systems, b) the establishment of data standards, c) the electronic exchange of health data, and d) changes in the objectives of public health surveillance to facilitate the response of public health to emerging health threats (e.g., new diseases). For example, CDC, with the collaboration of state and local health

departments, is implementing the National Electronic Disease Surveillance System (NEDSS) to better manage and enhance the large number of current surveillance systems and allow the public health community to respond more quickly to public health threats (e.g., outbreaks of emerging infectious diseases and bioterrorism) ( 2). When NEDSS is completed, it will electronically integrate and link together several types of surveillance systems with the use of standard data formats; a communications infra-structure built on principles of public health informatics; and agreements on data access, sharing, and confidentiality. In addition, the Health Insurance Portability and Accountability Act of 1996 (HIPAA) mandates that the United States adopt national uniform standards for electronic transactions related to health insurance enrollment and eligibility, health-care encounters, and health insurance claims; for identifiers for health-care providers, payers and individuals, as well as code sets and classification systems used in these transactions; and for security of these transactions ( 3). The electronic exchange of health data inherently involves the protection of patient privacy.

Based on CDC’s Framework for Program Evaluation in Public Health ( 4), research and discussion of concerns related to public health surveillance systems, and comments received from the public health community, this report provides updated guide-lines for evaluating public health surveillance systems.

BACKGROUND

Public health surveillance is the ongoing, systematic collection, analysis, interpretation, and dissemination of data regarding a health-related event for use in public health action to reduce morbidity and mortality and to improve health ( 5–7). Data disseminated by a public health surveillance system can be used for immediate public health action, program planning and evaluation, and formulating research hypotheses. For example, data from a public health surveillance system can be used to

• guide immediate action for cases of public health importance;

• measure the burden of a disease (or other health-related event), including changes in related   factors, the identification of populations at high risk, and theidentification of new or emerging health concerns;

• monitor trends in the burden of a disease (or other health-related event), including

the detection of epidemics (outbreaks) and pandemics;

• guide the planning, implementation, and evaluation of programs to prevent and

control disease, injury, or adverse exposure;

• evaluate public policy;

• detect changes in health practices and the effects of these changes;

• prioritize the allocation of health resources;

• describe the clinical course of disease; and

• provide a basis for epidemiologic research.

Public health surveillance activities are generally authorized by legislators and carried out by public health officials. Public health surveillance systems have been developed to address a range of public health needs. In addition, public health information systems have been defined to include a variety of data sources essential to public health action and are often used for surveillance ( 8). These systems vary from a simple system collecting data from a single source, to electronic systems that receive data from many sources in multiple formats, to complex surveys. The number and variety of systems will likely increase with advances in electronic data interchange and integration of data, which will also heighten the importance of patient privacy, data confidentiality, and system security. Appropriate institutions/agencies/scientific officials should be consulted with any projects regarding pubic health surveillance.

Variety might also increase with the range of health-related events under surveillance. In these guidelines, the term “health-related event” refers to any subject related to a public health surveillance system. For example, a health-related event could include infectious, chronic, or zoonotic diseases; injuries; exposures to toxic substances; health promoting or damaging behaviors; and other surveilled events associated with public health action.

The purpose of evaluating public health surveillance systems is to ensure that problems of public health importance are being monitored efficiently and effectively. Public health surveillance systems should be evaluated periodically, and the evaluation should include recommendations for improving quality, efficiency, and usefulness. The goal of these guidelines is to organize the evaluation of a public health surveillance system. Broad topics are outlined into which program-specific qualities can be integrated. Evaluation of a public health surveillance system focuses on how well the system operates to meet its purpose and objectives.

The evaluation of public health surveillance systems should involve an assessment of system attributes, including simplicity, flexibility, data quality, acceptability, sensitivity, predictive value positive, representativeness, timeliness, and stability. With the continuing advancement of technology and the importance of information architecture and related concerns, inherent in these attributes are certain public health informatics concerns for public health surveillance systems. These concerns include comparable hardware and software, standard user interface, standard data format and coding, appropriate quality checks, and adherence to confidentiality and security standards ( 9). Because public health surveillance systems vary in methods, scope, purpose, and objectives, attributes that are important to one system might be less important to another. A public health surveillance system should emphasize those attributes that are most important for the objectives of the system. Efforts to improve certain attributes (e.g., the ability of a public health surveillance system to detect a health-related event [sensitivity]) might detract from other attributes (e.g., simplicity or timeliness). An evaluation of the public health surveillance system must therefore consider those attributes that are of the highest priority for a given system and its objectives. Considering the attributes that are of the highest priority, the guidelines in this report describe many tasks and related activities that can be applied in the evaluation of public health surveillance systems, with the understanding that all activities under the tasks might not be appropriate for all systems.

Organization of This Report

This report begins with descriptions of each of the tasks involved in evaluating a public health surveillance system. These tasks are adapted from the steps in program evaluation in the Framework for Program Evaluation in Public Health ( 4) as well as from the elements in the original guidelines for evaluating surveillance systems ( 1). The report concludes with a summary statement regarding evaluating surveillance systems. A checklist that can be detached or photocopied and used when the evaluation is implemented is also included (Appendix A).

To assess the quality of the evaluation activities, relevant standards are provided for each of the tasks for evaluating a public health surveillance system (Appendix B). These standards are adapted from the standards for effective evaluation (i.e., utility, feasibility, propriety, and accuracy) in the Framework for Program Evaluation in Public Health ( 4). Because all activities under the evaluation tasks might not be appropriate for all systems, only those standards that are appropriate to an evaluation should be used.

Task A. Engage the Stakeholders in the Evaluation

Stakeholders can provide input to ensure that the evaluation of a public health surveillance system addresses appropriate questions and assesses pertinent attributes and that its findings will be acceptable and useful. In that context, we define stakeholders as those persons or organizations who use data for the promotion of healthy lifestyles and the prevention and control of disease, injury, or adverse exposure. Those stakeholders who might be interested in defining questions to be addressed by the surveillance system evaluation and subsequently using the findings from it are public health practitioners; health-care providers; data providers and users; representatives of affected communities; governments at the local, state, and federal levels; and professional and private nonprofit organizations.

Task B. Describe the Surveillance System to be Evaluated

Activities

• Describe the public health importance of the health-related event under

  surveillance.

• Describe the purpose and operation of the system.

• Describe the resources used to operate the system.  

Discussion

To construct a balanced and reliable description of the system, multiple sources of information might be needed. The description of the system can be improved by consulting with a variety of persons involved with the system and by checking reported descriptions of the system against direct observation. 

B.1. Describe the Public Health Importance of the Health-Related Event

Under Surveillance

Definition. The public health importance of a health-related event and the need to have that event under surveillance can be described in several ways. Health-related events that affect many persons or that require large expenditures of resources are of public health importance. However, health-related events that affect few persons might also be important, especially if the events cluster in time and place (e.g., a limited out-break of a severe disease). In other instances, public concerns might focus attention on a particular health-related event, creating or heightening the importance of an evaluation.

Diseases that are now rare because of successful control measures might be perceived as unimportant, but their level of importance should be assessed as a possible sentinel health-related event or for their potential to reemerge. Finally, the public health importance of a health-related event is influenced by its level of preventability ( 10).

Measures. Parameters for measuring the importance of a health-related event—

and therefore the public health surveillance system with which it is monitored—can

include ( 7)

• indices of frequency (e.g., the total number of cases and/or deaths; incidence

  rates, prevalence, and/or mortality rates); and summary measures of population

              health status (e.g., quality-adjusted life years [QALYS]);

• indices of severity (e.g., bed-disability days, case-fatality ratio, and hospitalization

              rates and/or disability rates);

• disparities or inequities associated with the health-related event;

• costs associated with the health-related event;

• preventability ( 10);

• potential clinical course in the absence of an intervention (e.g., vaccinations)  ( 11,12);

• public interest.

Efforts have been made to provide summary measures of population health status that can be used to make comparative assessments of the health needs of populations ( 13). Perhaps the best known of these measures are QALYs, years of healthy life (YHLs), and disability-adjusted life years (DALYs). Based on attributes that represent health status and life expectancy, QALYs, YHLs, and DALYs provide one-dimensional measures of overall health. In addition, attempts have been made to quantify the public health importance of various diseases and other health-related events. In a study that describes such an approach, a score was used that takes into account age-specific morbidity and mortality rates as well as health-care costs ( 14). Another study used a model that ranks public health concerns according to size, urgency, severity of the problem, economic loss, effect on others, effectiveness, propriety, economics, acceptability, legality of solutions, and availability of resources ( 15).

Preventability can be defined at several levels, including primary prevention (preventing the occurrence of disease or other health-related event), secondary prevention (early detection and intervention with the aim of reversing, halting, or at least retarding the progress of a condition), and tertiary prevention (minimizing the effects of disease and disability among persons already ill). For infectious diseases, preventability can also be described as reducing the secondary attack rate or the number of cases transmitted to contacts of the primary case. From the perspective of surveillance, prevent-ability reflects the potential for effective public health intervention at any of these levels.

B.2. Describe the Purpose and Operation of the Surveillance System

Methods. Methods for describing the operation of the public health surveillance system include

• List the purpose and objectives of the system.

• Describe the planned uses of the data from the system.

• Describe the health-related event under surveillance, including the case definition

  for each specific condition.

• Cite any legal authority for the data collection.

• Describe where in the organization(s) the system resides, including the context

  (e.g., the political, administrative, geographic, or social climate) in which the

  system evaluation will be done.

• Describe the level of integration with other systems, if appropriate.

• Draw a flow chart of the system.

• Describe the components of the system. For example

  —What is the population under surveillance?

  —What is the period of time of the data collection?

  —What data are collected and how are they collected?

  —What are the reporting sources of data for the system?

  —How are the system’s data managed (e.g., the transfer, entry, editing, storage,

      and back up of data)? Does the system comply with applicable standards for

                  data formats and coding schemes? If not, why?

  — How are the system’s data analyzed and disseminated?

  — What policies and procedures are in place to ensure patient privacy, data

      confidentiality, and system security? What is the policy and procedure for

      releasing data? Do these procedures comply with applicable federal and state

      statutes and regulations? If not, why?

  —Does the system comply with an applicable records management program?

  For example, are the system’s records properly archived and/or disposed of?  

Discussion. The purpose of the system indicates why the system exists, whereas its objectives relate to how the data are used for public health action. The objectives of a public health surveillance system, for example, might address immediate public health action, program planning and evaluation, and formation of research hypotheses (see Background). The purpose and objectives of the system, including the planned uses of its data, establish a frame of reference for evaluating specific components.

A public health surveillance system is dependent on a clear case definition for the health-related event under surveillance ( 7). The case definition of a health-related event can include clinical manifestations (i.e., symptoms), laboratory results, epidemiologic information (e.g., person, place, and time), and/or specified behaviors, as well as levels of certainty (e.g., confirmed/definite, probable/presumptive, or possible/suspected). The use of a standard case definition increases the specificity of reporting and improves the comparability of the health-related event reported from different sources of data, including geographic areas. Case definitions might exist for a variety of health-related events under surveillance, including diseases, injuries, adverse exposures, and risk factor or protective behaviors. For example, in the United States, CDC and the Council of State and Territorial Epidemiologists (CSTE) have agreed on standard case definitions for selected infectious diseases ( 16). In addition, CSTE publishes Position Papers that discuss and define a variety of health-related events ( 17). When possible, a public health surveillance system should use an established case definition, and if it does not, an explanation should be provided.

The evaluation should assess how well the public health surveillance system is integrated with other surveillance and health information systems (e.g., data exchange and sharing in multiple formats, and transformation of data). Streamlining related systems into an integrated public health surveillance network enables individual systems to meet specific data collection needs while avoiding the duplication of effort and lack of standardization that can arise from independent systems ( 18). An integrated system can address comorbidity concerns (e.g., persons infected with human immunodeficiency virus and Mycobacterium tuberculosis); identify previously unrecognized risk factors; and provide the means for monitoring additional outcomes from a health-related event. When CDC’s NEDSS is completed, it will electronically integrate and link together several types of surveillance activities and facilitate more accurate and timely reporting of disease information to CDC and state and local health departments ( 2). CSTE has organized professional discussion among practicing public health epidemiologists at state and federal public health agencies. CSTE has also proposed a national public health surveillance system to serve as a basis for local and state public health agencies to a) prioritize surveillance and health information activities and b) advocate for necessary resources for public health agencies at all levels ( 19). This national public health system would be a conceptual framework and virtual surveillance system that incorporates both existing and new surveillance systems for health-related events and their determinants.

Listing the discrete steps that are taken in processing the health-event reports by the system and then depicting these steps in a flow chart is often useful. An example of a simplified flow chart for a generic public health surveillance system is included in this report (Figure 1). The mandates and business processes of the lead agency that operates the system and the participation of other agencies could be included in this chart. The architecture and data flow of the system can also be depicted in the chart ( 20,21). A chart of architecture and data flow should be sufficiently detailed to explain all of the functions of the system, including average times between steps and data transfers.

The description of the components of the public health surveillance system could include discussions related to public health informatics concerns, including comparable hardware and software, standard user interface, standard data format and coding, appropriate quality checks, and adherence to confidentiality and security standards ( 9). For example, comparable hardware and software, standard user interface, and standard data format and coding facilitate efficient data exchange, and a set of common data elements are important for effectively matching data within the system or to other systems.

To document the information needs of public health, CDC, in collaboration with state and local health departments, is developing the Public Health Conceptual Data Model to a) establish data standards for public health, including data definitions, component structures (e.g., for complex data types), code values, and data use; b) collaborate with national health informatics standard-setting bodies to define standards for the exchange of information among public health agencies and health-care providers; and c) construct computerized information systems that conform to established data and data interchange standards for use in the management of data relevant to public health ( 22). In addition, the description of the system’s data management might address who is editing the data, how and at what levels the data are edited, and what checks are in place to ensure data quality.

In response to HIPAA mandates, various standard development organizations and terminology and coding groups are working collaboratively to harmonize their separate systems ( 23). For example, both the Accredited Standards Committee X12 ( 24), which has dealt principally with standards for health insurance transactions, and Health Level Seven (HL7) ( 25), which has dealt with standards for clinical messaging and exchange of clinical information with health-care organizations (e.g., hospitals), have collaborated on a standardized approach for providing supplementary information to support health-care claims ( 26). In the area of classification and coding of diseases and other medical terms, the National Library of Medicine has traditionally provided the Unified Medical Language System, a metathesaurus for clinical coding systems that allows terms in one coding system to be mapped to another ( 27). The passage of HIPAA and the anticipated adoption of standards for electronic medical records have increased efforts directed toward the integration of clinical terminologies ( 23) (e.g., the merge of the College of American Pathologists’ Systematized Nomenclature of Medicine [SNOMED ® ] [ 28] and the British Read Codes, the National Health Service thesaurus of health-care terms in Great Britain).  

FIGURE 1. Simplified flow chart for a generic surveillance system

Occurrence of health-related event

(An infectious, chronic, or zoonotic disease;

injury; adverse exposure; risk factor or

protective behavior; or other surveilled event

associated with public health action)  

Case

Confirmation

( Identification by whom and how)

Reporting sources

(Reporting process

 • Data entry and editing possible

• Assurance of confidentiality  )

Physicians

Health-care providers

Veterinarians

Survey respondents

Laboratories

Hospitals

Health-care organizations

Schools

Vital records

Other

Data recipients

(Data management

• Collection

• Entry

• Editing

• Storage

• Analysis

• Report generation

• Report dissemination

• Assurance of confidentiality)

Primary level

(e.g., county health

department)

Secondary level

(e.g., state health

department)

Tertiary level

(e.g., Federal agency)

Data recipients

Audiences

The data analysis description might indicate who analyzes the data, how they are analyzed, and how often. This description could also address how the system ensures that appropriate scientific methods are used to analyze the data.

The public health surveillance system should operate in a manner that allows effective dissemination of health data so that decision makers at all levels can readily under-stand the implications of the information ( 7). Options for disseminating data and/or information from the system include electronic data interchange; public-use data files; the Internet; press releases; newsletters; bulletins; annual and other types of reports; publication in scientific, peer-reviewed journals; and poster and oral presentations, including those at individual, community, and professional meetings. The audiences for health data and information can include public health practitioners, health-care providers, members of affected communities, professional and voluntary organizations, policymakers, the press, and the general public.

In conducting surveillance, public health agencies are authorized to collect personal health data about persons and thus have an obligation to protect against inappropriate use or release of that data. The protection of patient privacy (recognition of a person’s right not to share information about him or herself), data confidentiality (assurance of authorized data sharing), and system security (assurance of authorized system access) is essential to maintaining the credibility of any surveillance system. This protection must ensure that data in a surveillance system regarding a person’s health status are shared only with authorized persons. Physical, administrative, operational, and computer safeguards for securing the system and protecting its data must allow authorized access while denying access by unauthorized users.

A related concern in protecting health data is data release, including procedures for releasing record-level data; aggregate tabular data; and data in computer-based, inter-active query systems. Even though personal identifiers are removed before data are released, the removal of these identifiers might not be a sufficient safeguard for sharing health data. For example, the inclusion of demographic information in a line-listed data file for a small number of cases could lead to indirect identification of a person even though personal identifiers were not provided. In the United States, CDC and CSTE have negotiated a policy for the release of data from the National Notifiable Disease Surveillance System ( 29) to facilitate its use for public health while preserving the confidentiality of the data ( 30). The policy is being evaluated for revision by CDC and CSTE.

Standards for the privacy of individually identifiable health data have been pro-posed in response to HIPAA ( 3). A model state law has been composed to address privacy, confidentiality, and security concerns arising from the acquisition, use, disclosure, and storage of health information by public health agencies at the state and local levels ( 31). In addition, the Federal Committee on Statistical Methodology’s series of Statistical Policy Working Papers includes reviews of statistical methods used by federal agencies and their contractors that release statistical tables or microdata files that are collected from persons, businesses, or other units under a pledge of confidentiality. These working papers contain basic statistical methods to limit disclosure (e.g., rules for data suppression to protect privacy and to minimize mistaken inferences from small numbers) and provide recommendations for improving disclosure limitation practices ( 32).

A public health surveillance system might be legally required to participate in a records management program. Records can consist of a variety of materials (e.g., completed forms, electronic files, documents, and reports) that are connected with operating the surveillance system. The proper management of these records prevents a “loss of memory” or “cluttered memory” for the agency that operates the system, and enhances the system’s ability to meet its objectives.  

B.3. Describe the Resources Used to Operate the Surveillance System

Definition. In this report, the methods for assessing resources cover only those resources directly required to operate a public health surveillance system. These resources are sometimes referred to as “direct costs” and include the personnel and financial resources expended in operating the system.

Methods. In describing these resources consider the following:

• Funding source(s): Specify the source of funding for the surveillance system. In the

United States, public health surveillance often results from a collaboration among

federal, state, and local governments.

• Personnel requirements: Estimate the time it takes to operate the system, including

the collection, editing, analysis, and dissemination of data (e.g., person-time

expended per year of operation). These measures can be converted to dollar

estimates by multiplying the person-time by appropriate salary and benefit costs.

• Other resources: Determine the cost of other resources, including travel, training,

supplies, computer and other equipment, and related services (e.g., mail, telephone,

computer support, Internet connections, laboratory support, and hardware and

software maintenance).

When appropriate, the description of the system’s resources should consider all levels of the public health system, from the local health-care provider to municipal, county, state, and federal health agencies. Resource estimation for public health surveillance systems have been implemented in Vermont (Table 1) and Kentucky (Table 2).

Resource Estimation in Vermont. Two methods of collecting public health surveillance data in Vermont were compared ( 33). The passive system was already in place and consisted of unsolicited reports of notifiable diseases to the district offices or state health department. The active system was implemented in a probability sample of physician practices. Each week, a health department employee called these practitioners to solicit reports of selected notifiable diseases. In comparing the two systems, an attempt was made to estimate their costs. The estimates of direct expenses were computed for the public health surveillance systems (Table 1).

Resource Estimation in Kentucky. Another example of resource estimation was provided by an assessment of the costs of a public health surveillance system involving the active solicitation of case reports of type A hepatitis in Kentucky (Table 2) ( 34). The resources that were invested into the direct operation of the system in 1983 were for personnel and telephone expenses and were estimated at $3,764 and $535, respectively. Nine more cases were found through this system than would have been found through the passive surveillance system, and an estimated seven hepatitis cases were prevented through administering prophylaxis to the contacts of the nine case-patients

 TABLE 1. Comparison of estimated expenses for health department active and passive surveillance systems — Vermont, June 1, 1980–May 31, 1981*

*Vogt RL, LaRue D, Klaucke DN, Jillson DA. Comparison of an active and passive surveillance system of primary care providers for hepatitis, measles, rubella, and salmonellosis in Vermont. Am J Public Health 1983;73:795–7.

† Active surveillance — weekly calls were made from health departments requesting reports. § Passive surveillance — provider-initiated reporting.

Discussion. This approach to assessing resources includes only those personnel and material resources required for the operation of surveillance and excludes a broader definition of costs that might be considered in a more comprehensive evaluation. For example, the assessment of resources could include the estimation of indirect costs (e.g., follow-up laboratory tests) and costs of secondary data sources (e.g., vital statistics or survey data).

The assessment of the system’s operational resources should not be done in isolation of the program or initiative that relies on the public health surveillance system. A more formal economic evaluation of the system (i.e., judging costs relative to benefits) could be included with the resource description. Estimating the effect of the system on decision making, treatment, care, prevention, education, and/or research might be possible ( 35,36). For some surveillance systems, however, a more realistic approach would be to judge costs based on the objectives and usefulness of the system.

Task C. Focus the Evaluation Design

Definition

The direction and process of the evaluation must be focused to ensure that time and resources are used as efficiently as possible.

Methods

Focusing the evaluation design for a public health surveillance system involves

• determining the specific purpose of the evaluation (e.g., a change in practice);

• identifying stakeholders (Task A) who will receive the findings and recommendations

  of the evaluation (i.e., the intended users);

• considering what will be done with the information generated from the evaluation

  (i.e., the intended uses);

• specifying the questions that will be answered by the evaluation; and

• determining standards for assessing the performance of the system.

Discussion

Depending on the specific purpose of the evaluation, its design could be straightforward or complex. An effective evaluation design is contingent upon a) its specific purpose being understood by all of the stakeholders in the evaluation and b) persons who need to know the findings and recommendations of the design being committed to using the information generated from it. In addition, when multiple stakeholders are involved, agreements that clarify roles and responsibilities might need to be established among those who are implementing the evaluation.

Standards for assessing how the public health surveillance system performs establish what the system must accomplish to be considered successful in meeting its objectives. These standards specify, for example, what levels of usefulness and simplicity are relevant for the system, given its objectives. Approaches to setting useful standards for assessing the system’s performance include a review of current scientific literature on the health-related event under surveillance and/or consultation with appropriate specialists, including users of the data.

TABLE 2. Costs of a 22-week active surveillance program for hepatitis A —

Kentucky, 1983*