Evaluating HIV and Development

Evaluating HIV and development: guidelines and suggestions
Evaluating HIV and AIDS: Why capacity development is central to assessing performance

Almost since its inception, the HIV and Development Programme (HDP) recognised the potential contribution, which could be made by evaluation to improved programme and project effectiveness. In the early nineties, HDP worked in partnership with the Salvation Army in designing and facilitating a workshop intended to build the capacity of programme implementers to identify, document and disseminate lessons learned. In 1996, HDP took on the role of co-ordinating the donor group response to the draft report of the team of consultants tasked with evaluating the International HIV/AIDS Alliance, an initiative in which UNDP had been involved almost since the beginning.

Taking advantage of the opportunity presented by an evaluation of its own UNDP Global Programme-funded activities, HDP worked with colleagues inside and outside UNDP, particularly those working in the area of evaluation, in designing an evaluation methodology consistent with the capacity-building emphasis of the Programme.

A number of events occurring more or less simultaneously led HDP towards a more consistent involvement in the issue of the evaluation of HIV-related initiatives. First among these was HDP’s own history in relation to involvement with evaluation and the specific challenges posed by evaluating initiatives relating to HIV and development. Second, the experience of co-ordinating the donor response to the Alliance evaluation highlighted some of the challenges of working simultaneously on evaluation with large groups of stakeholders and beneficiaries. Third, the experience and process of the HDP programme evaluation pointed to the possibility of evaluation methodologies that could respond to the needs of a broader range of stakeholders instead of focusing narrowly upon the needs of donors. Finally, in addition to the above mentioned evaluation reports, a number of other significant evaluations of HIV-related projects, programmes and organisations had been conducted and reported on in the course of the previous eighteen months.

It was decided by HDP to form a ‘virtual’ working group which would include HDP personnel and a group of consultants with experience of working with HDP, UNDP, UNAIDS or other relevant agencies, such as NGOs. Group members were tasked with reviewing (according to a set of key questions) one or more of the above evaluation reports. These reviews and the ensuing deliberations formed the basis of a meeting of the working group in September 1999. In response to the presentation of the reviews and the ensuing deliberations, it was decided to form two smaller groups focusing respectively upon issues of evaluation methodology and capacity building.

The papers enclosed in this publication are the results of the initiative. Bruce Parnell and Des Cohen, the respective focal points for the two ‘theme’ groups, are the authors of the papers. Cate Hankins, Peter Gordon and Tang Tuong (members of the working group) together with colleagues from the UNDP Evaluation Office, reviewed drafts of the papers and made a number of suggestions which (to the extent that it has been possible to do so) are reflected in these final versions.

Time constraints prevented the kind of exhaustive consultation process that might have led to institutional ownership of the papers. Therefore, they should be understood as reflecting the views of the individual authors rather than the views of the entire working group, HDP or UNDP. The purpose of these papers is not to present the definitive opinions of ‘experts’: instead, it is to share more widely current thinking on evaluation as it relates to HIV and development and capacity building approaches, together with existing constraints and possibilities, in recognition of the potential of evaluation to enhance learning among donors, and stakeholders and beneficiaries.

Addressing HIV as a development issue requires responses which are broader than the more ‘traditional’ focus on health promotion and service provision which is characteristic of a ‘public health response’. Epidemiology, clinical and behavioural research, in isolation, are seldom adequate to the task of measuring subtle, complex changes in culture and socio-economic conditions that play such an integral role in the HIV epidemic. Clearly, HIV surveillance is a useful activity for tracking the epidemic, and behavioural research can make important contributions to our understanding of the extent and role of factors associated with specific sexual or drug using behaviour as they relate to the epidemic. However, neither can account for the varying levels of susceptibility among different groups living in similar circumstances: the very factors that a ‘development’ approach to the epidemic seeks to address.

Many countries and regions are implementing responses to the challenges of HIV and development in order to:

Processes and mechanisms adopted toward these ends have included broad programmes and targeted projects, policy development and strategic planning. But how can these best be evaluated? In other words, how can we maximise the learning from these initiatives? This is an important question, because effective evaluation can provide the information necessary to improve the design, scope and methodology of many policy and programming responses to the HIV epidemic. Asking the right questions is a first, important step in ensuring that the required information and lessons are captured through evaluation.

The purpose of this paper is to articulate the kinds of questions which might enhance our capacity to understand and mitigate the impact of the HIV epidemic upon sustainable human development and to demonstrate the importance of evaluation as a way of generating learning in the field of HIV and development.

As the UN agency with the specific mandate of promoting sustainable human development, UNDP, with some foresight, initiated organisational responses to the HIV epidemic in the late 1980s in anticipation of the devastating impact of the epidemic upon the social and economic fabric of developing countries. In the early 1990s this response was institutionalised in the form of the HIV and Development Programme, based in New York, which was charged with working throughout the UNDP system in support of co-ordination of its HIV-related activities. By then, UNDP had a significant body of relevant human and technical resources, for example, in relation to poverty, gender and governance - all of which were recognised as critical dimensions of the epidemic. Moreover, UNDP had an established team (the Evaluation Office) working on the monitoring and evaluation of development projects and programmes

Drawing on a number of sources, this paper suggests some issues to consider in relation to evaluating initiatives on HIV and development. These include:

This paper is intended to articulate some of the questions that might most usefully be explored through the evaluation of HIV and development initiatives and to identify potentially appropriate methods for this work. The paper does not offer a set of "how to" instructions for the evaluation of HIV-initiatives. This kind of advice is already available from a variety of sources. For instance, information relating to precorders for the evaluation of UNDP supported programmes and projects can be found in the UNDP programming manual (Chapter 7) and further guidance on organising and conducting evaluations is to be found in the UNDP Evaluation Office’s Results-Oriented Monitoring and Evaluation: A Handbook for Programme Managers.

These guidelines focus more specifically on some of the key issues that should be considered in evaluating initiatives on HIV and development. It is hoped that the use of these guidelines will lead to the identification of a broader range of issues to be addressed by future evaluations and that these, in turn, will clarify a more extensive set of programming and policy options. The ultimate goal is to achieve more effective, extensive and sustainable responses to the challenges of HIV and development.

Evaluation is a process used to enhance understanding. Within the context of HIV and development, there is no ‘single issue’ that should be understood. Rather, evaluation processes have to be responsive to specific needs as they arise during particular initiatives. So, what makes a suitable starting point?

A primary consideration is that an evaluation framework should be developed which leads to processes that are useful:

· Programme evaluation is the systematic collection of information about the activities, characteristics, and outcomes of programmes to make judgements about the programme, improve programme effectiveness, and/or inform decisions about future programming. Utilisation-focused programme evaluation (as opposed to programme evaluation in general) is evaluation done for and with specific, intended primary users for specific, intended uses³.

Evaluation findings are used to make decisions. Evaluation is different from monitoring or auditing of programme implementation in that it asks questions that go beyond those that might have been the focus of programme designers. Monitoring and auditing are processes that measure the extent to which a project has done what its designers said it would do. In contrast, evaluation involves reflection on what has been done, and learning about the value of what has been done, in order to feed into the next stage of planning for what should happen next.

UNDP’s Results-Oriented Monitoring and Evaluation: A Handbook for Programme Managers notes that monitoring and evaluation are closely related, and that they are mutually supportive. It also notes the distinction between these two functions and other functions of auditing and research, summarising the relationships as follows⁴:

Another useful way of considering the same relationships is to focus upon the purpose of the three functions:

Given these distinctions, an emphasis on learning has been stressed for development programmes:

· Traditionally, monitoring and evaluation have been perceived as forms of control mainly because their objectives were not clearly articulated and understood. Thus, the learning aspect of monitoring and evaluation needs to be stressed along with the role that these functions play in decision-making and accountability. In the context of (development), the contribution of learning to the building of ………..... capacity to manage development should be emphasised⁵.

Thus, evaluation can be considered as part of an ongoing learning process, through which people will find the solutions to specific problems relating to HIV and development. This is summarised in the diagram below. Evaluation occurs when people:

Within the learning cycle, what is the particular role of evaluation? This depends on what is considered most important to learn at any stage of the process. UNDP (1997b) suggests that three interrelated dimensions of programmes and projects must be assessed as the substantive focus of monitoring and evaluation: relevance, performance and success⁶. The handbook suggests the following considerations for each of these areas of focus:

Given this range of options, decisions must be made about the focus of any particular evaluation. These choices will depend on the purpose of the evaluation: i.e. what it is intended to be used for. Patton (1997)⁷ suggests three distinct categories and recommends that evaluation should have a primary purpose, based on choosing just one of these categories:

Ideally the time to begin planning an evaluation is during the project design phase. In this way, realistic monitoring and evaluation mechanisms can be identified and put in place from the outset. This will assist project management and staff and volunteers to assess the extent to which they are achieving what they set out to do, and if necessary recognise when and what changes need to be introduced. This approach is preferable to the usual practice of planning the evaluation at the end of the project since this timing can seriously constrain the options of the evaluation exercise. With early planning and the use of project staff, volunteers and beneficiaries as key ‘researchers’, evaluation can become a critical and valuable aspect of project culture, instead of an unwelcome, externally imposed requirement.

While the integration of evaluation from the outset may be an ideal, nonetheless there are certain conditions that need to be in place for an evaluation to have significant impact on programming and policy development. For example, those who are in a position to implement recommended changes and those who will be directly affected by any recommendations must have a minimum level of ‘buy-in’ to the evaluation process and willingness to implement its recommendations. There is a requisite level of relevant capacity (or commitment to building it) in order for people and institutions to implement or respond to any necessary changes. Adequate financial, logistical, facilitation, and managerial support are also essential if an evaluation is to proceed in a timely and effective fashion.

Once the purpose of the evaluation has been clarified together with the specific aspects of programme quality that have been defined as its focus, the next step is to consider how the actual evaluation process will be conducted. Two sets of issues are important at this point:

a) Which issues will be considered? Deciding upon the primary purpose of the evaluation, is only part of the process of determining the specific issues to be considered. The next section lists a number of key issues that might usefully be included in the evaluation.

b) Who will conduct the evaluation? Who will design the process, articulate the questions to be answered, collect and analyse the data, and apply the lessons learned through the evaluation?

Ideally evaluation should be conducted in ways that are designed to build the capacity of people and institutions by supporting them in undertaking their own data collection and analysis, drawing conclusions and making changes. In this way, sustainability can be promoted together with the ability to reflect and adapt in response to changes in local and national circumstances. This process can facilitate meaningful participation in the identification of the critical questions to be answered by an evaluation by those whose lives are most affected by a programme.

The following tables contain questions that might usefully be explored in relation to the evaluation of initiatives on HIV and development. Some tables are considerably more detailed than others and simply reflect the extent of relevant discussion to date.

CAPACITY DEVELOPMENT

Key Evaluation Question

Examples of Questions to Explore⁸

Comments/Notes

What would indicate that useful capacities have been developed?

What kinds of capacity matter most for this initiative?

In what ways, and with what effects, has gender been considered and addressed?

Was an assessment done of the organisation/ partners’ capacity to conduct and apply gender analysis? If so, how was it able to respond to the identified gaps? With what results?

Did the project consider the issue of complementary capacity in other organisations or groups etc?

Did it see improving complementary capacity as relevant to its own effectiveness? If so how was this done? What constraints were faced? How effective was what was done?

What capacities were considered important by project participants at the start of the project?

What indicators were perceived by the participants as indicative of capacity having been built or strengthened?

How would we know if such capacities are being developed?

How can the evaluation ITSELF be designed in such a way that it also builds capacity?

What new relationships or partnerships have resulted from the project?

What do people now understand about the nature of the HIV epidemic? How have they learned this?

How has the project enhanced their ability to learn from and with one another, about issues that affect their lives, their communities and nation?

What concerns have emerged in relation to the underlying factors that create the conditions in which the epidemic thrives?

What evidence exists that people now understand these concerns? How are people talking about the epidemic and what issues do they discuss?

What are the indicators that the project has been successful in facilitating problem solving by people in their own ways?

What new policies, projects and skills have been developed (and within which sectors) about HIV since the project’s inception?

What sectors have produced policies and projects or initiated skill development?

What action has been taken by managers to explore and address the impact of HIV upon their own workforce?

Fundamentally, capacity development is concerned with developing and enhancing the capacity of individuals, families, communities and nations to respond to the issues that concern them. The capacity to respond is made up of the combination of a range of individual human efforts. These endeavours depend upon peoples’ abilities to:

form and enhance relationships
learn from one another
identify common concerns
collectively solve problems
develop policies
implement various programmes
conduct research
enact laws and encourage observance of those laws
build consensus
form networks to address specific concerns
work collaboratively within networks
create, and respond to, cultural and social change
improve quality of life of the whole community
build and sustain a sense of hope.

These kinds of questions could usefully be explored through the use of qualitative methods such as focus group discussions and participatory learning and action processes designed to explore issues related to the development determinants of the epidemic. Qualitative methods could be used to answer these questions stimulating project participants to explore and understand the kinds of capacity they have developed, and the contribution of the initiative to this process. Quantitative methods could also be employed: for example, questionnaires could be used to measure the number of people who have developed intended capacities to a satisfactory level.

SUSTAINABILITY
Key Evaluation Question	Examples of Questions to Explore		Comments/Notes
*What would indicate that the project is likely to lead to sustainable responses to the HIV epidemic?*	*How is sustainability being facilitated?* *How will we know?* *In what specific ways have critical gender dimensions of sustainability been addressed?* How did the project address sustainability issues relating to human resources – within the project and among collaborating partners? How, and to what degree, did the project address the sustainability of programme activities amongst collaborating partners? With what success? *Did the project address the issue of sustaining funding in the short and medium-term?* *What was done to secure ongoing financial and other support for the project and related activities?* *Did the project implement internal processes of reflection and learning?* If so, did this lead to revision of objectives and implementation processes to ensure continuing relevance to the needs of project beneficiaries? *Pilot Projects* *What criteria should be used to assess the sustainability of a future project?* *What specifically is a pilot project to deliver in terms of learning?* What does the evaluation of a pilot project need to concentrate on in order to gain the necessary insights/lessons for moving forward? *What needs to be learned from a pilot project to make subsequent decisions about ‘scaling-up’?*		Because they deal specifically with HIV/AIDS (and the associated morbidity and mortality) there are likely to be particular problems for some projects and programmes in terms of sustainability.
BUILDING NATIONAL CAPACITY FOR SUSTAINABLE HUMAN DEVELOPMENT
Key Evaluation Question		Examples of Questions to Explore		Comments/Notes
What can we ask about specific sectors (e.g. education, health, agriculture) to ascertain how capacity in those spheres will be affected by the HIV epidemic, and thus clarify the types of response that might be most useful?		In what ways, and with what effectiveness, has the project/programme addressed the impact of the epidemic on poverty and gender? *How is the project addressing development factors?* In what specific ways does the national response to the HIV epidemic analyse and address relevant development issues such as governance, links between poverty and HIV, and underlying socio-economic factors which influence the spread and impact of the epidemic? *How have poverty and gender equality considerations been integrated within HIV projects?* *How has the issue of HIV/AIDS been considered in the design and implementation of poverty and gender projects?* *How was this done and with what results? And with what implications for future projects?* How have poverty and gender equality been considered in relation to -relevant to the design and implementation of future poverty and gender projects and how were these to be integrated in subsequent projects and programmes?
INTEGRATING DEVELOPMENT CONCERNS INTO CURRENT RESPONSES TO HIV
Key Evaluation Question	Examples of Questions to Explore		Comments/Notes
	In what ways does a programme increase understanding of the relationship between HIV and development? For example through: *identifying determinants of the risk situations and behaviours that enable transmission in certain populations* *analysis of the socio-economic factors that facilitate the spread of the epidemic* piloting of different approaches to meeting the need for increased care and support of people living with HIV and their dependants, in situations where present institutional arrangements are unable to meet those needs the development of capacities required to cope with a range of socio-economic impacts of the epidemic, including enhanced capacity to analyse and understand the nature of current and likely future impacts of the epidemic. *Does the program address these issues?* *With what effect?*		Indicators might include new understanding, specific factors which people now understand but which were previously unseen or misunderstood by policy makers, different sectors or communities before the project started.
INTEGRATING CONCERNS ABOUT HIV INTO DEVELOPMENT PROJECTS
Key Evaluation Question	Examples of Questions to Explore		Comments/Notes
How can we ascertain the extent to which a development project is influencing the spread or, consequences of, or responses to the HIV epidemic?	What evidence is there that project designers and implementers have given serious consideration to possible interactions between development and the HIV epidemic, and vice versa? *What action has been taken to anticipate HIV-related impact among the workforce?* *What action has been taken in response to gender-specific dimensions of development and the HIV epidemic?* How did the project influence susceptibility to HIV of beneficiaries and other population groups affected by it, e.g. the displacement of producers and their families by major infrastructure investments in dams? What actions were taken to reduce these adverse consequences on affected populations and with what results? Did the investment project address the needs of a mobile labour force for HIV prevention and support? With what success? Did the project collaborate with and consult communities affected by development projects with the intention of minimising the transmission of HIV through the implementation of effective prevention activities? *Did it assist the communities to support those affected by HIV and AIDS?* *How successful were these activities and how could they be strengthened?*
GUIDELINES FOR STRATEGIC PLANNING (based upon UNAIDS model)
Key Evaluation Question	Examples of Questions to Explore		Comments/Notes
*Situation assessments:* *What further questions could be asked about development?* *Response reviews:* *What further questions could be asked about development?* *Capacity assessments:* *What further questions could be asked to ascertain the capacity of people and institutions to:* *analyse and act on the socio-economic factors that enable the spread of the epidemic,* *to develop new approaches to community based care,* to work across sectors to respond to development needs in situations where human and institutional capacity has been diminished by the HIV epidemic?	Is what UNDP is doing consistent with its role within UNAIDS (i.e. is it addressing issues of development and the epidemic – both the causal and consequential aspects? *How does the project/ programme relate to the core concerns of UNDP within the UNAIDS framework?* *Is it consistent with the National Strategic Plan for HIV?* In terms of the project’s design and implementation, what have been the roles of the UNAIDS Theme Group on HIV and other national partners? What effective, mechanisms exist at country level to ensure that the project is consistent with both the National Strategic Plan for HIV and the National Development Plan? *What did the project designers and implementers do to ensure that what was done was relevant to national needs? With what success?* *What was done to ensure that HIV/ AIDS were integrated in general development activities for gender and poverty, and with what success?* In what ways did the project strengthen national capacity to understand and respond to developmental aspects of the epidemic and with what success?

EVALUATING HIV AND AIDS
WHY CAPACITY DEVELOPMENT IS CENTRAL TO ASSESSING PERFORMANCE

The core of UNDP’s mandate is the achievement of sustainable human development, and the instrument for doing this is capacity development. The term capacity development can be defined as follows:

Capacity is the ability of individuals and organisations to perform functions effectively, efficiently and sustainably. The term "capacity development" is preferred to "capacity building"; while capacity strengthening is important so are the retention of existing capacity, improvements in the way in which existing capacity is being utilised, and the retrieval of capacity which has been eroded or lost. Thus capacity development does NOT take place only through the training of additional staff or the creation of new organisations, but requires an enabling environment to ensure that people are used effectively, are retained within organisations and structures that need their inputs, and are motivated to perform their tasks.

All projects and programmes should develop a capacity development strategy and be evaluated on the basis of whether they address the key issues that this involves. Central to evaluation of the project or programme is whether in the formulation of a capacity development strategy a capacity assessment was first undertaken and whether the results of this assessment informed the strategies for capacity enhancement that would ensure that the project or programme was both feasible and sustainable.

On the basis of a capacity assessment it is possible to examine the strengths and weaknesses of the existing arrangements for undertaking the project or programme, the reasons for weaknesses in existing capacity and why these have persisted over time, and the factors which are undermining existing capacity such as losses of skilled labour due to HIV-related mortality. Such an assessment would also generate insights into past failures and successes in regard to capacity strengthening and how to build on the latter in implementing activities.

What would be the main areas of focus of capacity development strategies in the light of the capacity assessment?

It is self evident that donors have a clear interest in supporting the capacity development strategies of national governments through the full integration of donor programme activities with those already underway or planned at country level. This makes it essential that donors formulate their activities for capacity development within coherent frameworks that ensure consistency with what each other is planning to do and consistency with national needs and capacities. It may be necessary for donors to support the establishment of co-ordination machinery and the strengthening of national capacity to co-ordinate donor activities effectively.

Donors should avoid activities that are destructive of national capacity, such as the recruitment of key staff away from national uses and the distortion of national pay scales through excessive payments to some nationals. The objectives of donors should be to support national capacity development through human resource investment and to strengthen the policy environment for a more effective delivery of projects and programmes that are relevant for sustainable development. Achieving these objectives may in some cases require that donors initially seek through their own activities to strengthen their own capacity and motivation to undertake capacity development as a prior first step towards more effective general performance.

How does one ensure that capacity development is integral to projects and programmes?

It is clear from research to date that putting in place a capacity development strategy and activities for its implementation at the design stage of projects is a sine qua non for more effective performance. It follows that unless the capacity development issues noted above are addressed at the design stage, and relevant activities identified and implemented as integral to projects and programmes, then only very partial and weak overall performance will result.

Monitoring and evaluation benchmarks should be developed at the design stage of projects and programmes. These benchmarks should reflect the priorities that have been selected for interventions based on an analysis of critical constraints and should take into account the fact that project and programme objectives and modalities will require adjustment during the life of the project and should not be left to be modified as a result of an end of project evaluation. The review and evaluation processes should be iterative and should examine the quality, timeliness and quantity of services provided in relation to organisational goals and client needs. These should be examined within the broader context of capacity assessment and follow-up, organisational management, structures and performance, and in terms of the availability and utilisation of human resources.

It follows that capacity development for undertaking relevant project and programme evaluation may itself have to be supported through specific activities in order to ensure effective monitoring and evaluation. This is a familiar dilemma: capacity may have to be strengthened first so that it will be adequate to achieve the iterative monitoring and evaluation required for better development outcomes. It also follows that once such capacity has been developed, commitment and mechanisms to ensure effective utilisation are essential.

Section 1 sets out the reasons why capacity development is the critical instrument for achieving development objectives and reviews those factors relevant to its full integration in projects and programmes. The important question is not whether the principles and criteria for effective capacity development, as described earlier are changed in a world of HIV and AIDS. If anything these guiding principles remain pertinent and the need for capacity development in the context of HIV and AIDS is enhanced rather than diminished. Rather, the epidemic changes the context within which capacity development strategies are expected to function and magnifies the difficulties of sustaining capacity.

It is useful to review the factors that change the context and feasibility of capacity development and the ways in which these affect the criteria that are important in evaluation of projects and programmes. The issues are complex and the particular situations are very diverse, therefore the following discussion should be seen as illustrative rather than definitive. The aim is to increase understanding of the effects that the HIV epidemic has on capacity development strategies, on programme implementation and on evaluation of projects and programmes.

HIV prevalence is concentrated amongst those in the key social and working age groups – between the ages of 15-45 years. In general, young women become infected at earlier ages than do young men and thus lose more years of healthy life. There is evidence that in mature epidemics in many countries of sub-Saharan Africa more women are infected than men, with an overall ratio of 6:5. Often in the early stages of the epidemic, rates of HIV acquisition are greater for those in higher educational and occupational groups whose members have greater mobility and spending power. This has important implications for the maintenance of both the stock and the flow of those human resources that require substantial social investment in education and training. While HIV prevalence may be highest in urban areas it is nevertheless the case that in absolute terms the largest numbers of those infected reside in rural areas in sub-Saharan Africa.

Rates of HIV prevalence amongst adults in many countries in sub-Saharan Africa are now in the range of 15-35%, with even higher rates in some locations and cities. The severity of the epidemic affects all social and occupational groups, including both men and women who have higher level skills and experience and those who are supposedly "unskilled". Amongst the latter category is most of the rural population which, in fact, has very valuable and hard to replace task-specific experience and skills.

Since HIV is concentrated amongst the core of the working population who have important social roles, and in particular the support and socialisation of children, there are bound to be effects both on the current generation and on subsequent generations. It follows that not only is the structure within families, including gender roles, changed by the experience of the epidemic but that there are also important issues to address with respect to maintaining households as productive enterprises.

What can be concluded from the losses of human resources that are critical for capacity development?

Most obvious are the losses of human capital due to the epidemic – skilled, educated, and unskilled men and women, in both urban and rural locations. One important issue is how to sustain production in circumstances of high morbidity and mortality across wide swathes of the active labour force. All programmes and projects have to deal with this fact: how can production be maintained in the face of ongoing and often severe losses of labour? Losses may be disruptive precisely because they are not confined to categories of labour that may be "easily" replaceable, but also affect many categories of more specialised labour including supervisory and managerial components.

But the effects on capacity are unfortunately not limited to the above and there are at least three other main channels through which capacity is undermined by the epidemic. These are as follows: