The following work examines possible methodologies for measuring China's development advances.
Development management often uses techniques based on measurement or estimation of performance and impacts. To what extent do such techniques answer the problems and requirements of development as public action involving a multiplicity of agencies but with government agencies maintaining an important role?
This essay sets out to relate development practice and theory. It is set in the context of the 1997 World Development Report, ‘The State in a Changing World’, which outlines what are the two main themes of this work; the improvement of state performance and the reduction in the role of the state. I intend to explain to what extent the techniques of performance management respond to the ideas of state-role reduction. The work begins with definitions of the terms performance and impact and then discusses what is understood by ‘development as public action involving a multiplicity of agencies but with government agencies maintaining an important role’. Following, there is a description of the techniques used by development managers in the measurement or estimation of performance and impact. The subsequent section details the problems and requirements of development as public action involving multiple agencies and assesses to what extent the outlined techniques answer them. It concludes that the techniques of performance management, if applied by the state and external agencies with accuracy and transparency, can answer the requirements of development as public action involving a multiplicity of agencies but with government agencies maintaining an important role.
Performance as understood by this work is “the degree to which a development intervention or a development partner operates according to specific criteria/standards/guidelines or achieves results in accordance with stated goals or plans” (International Fund for Agricultural Development, 2002: A9). Impact is characterized as “changes in the well-being of individuals that can be attributed to a particular project, program or policy” (World Bank, 2004: para. 1). Changes in well-being are not by implication positive and intended but can be negative and unintended. The phrase ‘public action involving multiple agencies but with government agencies maintaining an important role’ is taken to mean “the provision of a good or service [which] is transferred from public to private sector, while government retains ultimate responsibility”. (Cook and Kirkpatrick, 1988: 4). This state lead can be expressed in restructuring arrangements such as Public-Private Partnerships, contracting out and concessions and may involve NGOs, private social enterprises or even private foundations. The Public-Private arrangements regarding the London underground system are good example of this kind of partnership.
Before characterizing the techniques used by development managers to measure performance and impact, I intend to discuss what exactly these techniques seek to measure. The development management literature suggests that efficiency, effectiveness, economy and equity (The Four E’s) are the criteria performance measurement techniques should use to assess policies and projects. Efficiency is described as “the rate and cost at which inputs result in outputs” and effectiveness as “performance in relation to targets set in the original plan.” (Robinson and Thin, 1993: 6). Thomas adds that economy is “how cheaply inputs can be obtained to achieve a given output” and equity as “how evenly distributed [the] benefits [are] resulting from the outputs.” (Thomas, 2004:3).
The following discussion of the techniques used to measure the four E’s of performance and impact, while not exhaustive and exclusive to development management, is provided to illustrate the scope and complementarily of tools at the disposal of development managers. The techniques measure performance and impact before, during and after policy and project decisions are made. Estimation of performance allows the state to assess how the four E’s might be achieved and on-going and retrospective analysis shows how it can be improved. The discussion begins with techniques used to estimate performance and impact.
Multiple Criteria Analysis (MCA) is “an approach for choosing from among a set of alternatives when there are multiple objectives” (UNESCAP, 2003: para. 2). In effect, Multiple Criteria Analysis illustrates to what extent different policies or projects fulfill specified indicators. The result of such an analysis aids decision-makers in choosing the best alternative. MCA uses both quantitative and qualitative data so that the four E’s are represented in its outcomes. An application of MCA in the context of multiple agency development led by the state can be seen in the granting of water concessions to social enterprises. Initially the state sets the indicators of what it considers the nature of water provision services, such as the quality of drinking water, reliable supply and broad distribution then social enterprises bid for the concession and the state matches these bids against the indicators it has set down.
The UNDP describes Cost-effectiveness Analysis (CEA) as “a type of analysis that involves comparing the relative costs of operating a programme with the extent to which the program [meets] its aims.” (UNDP Evaluation Office, 2005: para. 18). Cost-effectiveness Analysis assists development managers to assess the economy, the value-for-money, of alternative policies or projects by comparing the costs of each one against a stated aim. For example, the state is looking into partnering a NGO to organize activities which raise reading abilities in primary schools and a number of methods are under discussion. “The costs associated with each alternative [project] can be calculated… [and] the effectiveness of each alternative may be measured through setting reading tests for groups of students subject to the different methods,” by comparing the cost against the effectiveness of each alternative the most economic and effective method can be gleaned. (Asian Development Bank, 1997: 24).
Cost-benefit Analysis (CBA) “is a practical way of assessing the desirability of projects, where it is important to take a long view (in the sense of looking at repercussions in the further as well as in the near future) and a wide view (in the sense of allowing for side effects of many kinds on many persons, industries, regions, etc.), i.e. it implies the enumeration and evaluation of all the relevant costs and benefits.” (Prest and Turvey, 1965, p. 683). In essence, CBA is the subtraction of costs from benefits. “A simple cost-benefit analysis of a road scheme would measure the cost of building the road, and subtract this from the economic benefit of improving transport links.” However, as “it would not measure the cost of environmental damage”, (Mind Tools: para 4), CBA along with CEA can be viewed solely as an economic analysis of intended policies or projects.
On-going and retrospective measurement of policy and project performance and is embedded within a system of management information and looks to improve the performance and impact of policy and projects. Heeks suggests a Monitoring and Control System which stresses that the results of on-going and retrospective measurement should feed into not only policy and project inputs but also their processes and targets. Heeks adds that ascribing a pivotal role to control actions entails performance improvement throughout the policy or project.
The Monitoring and Control System as presented by Heeks gives managers a context in which to apply on-going and retrospective performance measurement, i.e. the “assessment of performance against objectives” (Robinson and Thin 1993, p.5). Paton’s eight instruments of performance management adapted to the needs of the state assessing performance by external agencies “illustrate the elements out of which performance management frameworks [such as the Monitoring and Control System] are constructed” (Paton, 2003: 22). These instruments feed into a paradigm as proposed by Heeks at the target/objective, inputs and process or activities stage. Briefly outlined the instruments are:
- Specifying government arrangements; this instrument structures the relationships between the state and social enterprises to guarantee the accountability of social enterprises to the state.
- Requiring public reports; social enterprises are obliged to produce open reports on performance which forces transparency to the state and the public.
- Financial leverage; “is used to induce social enterprises to undertake programmes, join collaborations or make decisions that will help achieve performance as the government construes it.”
- Setting and inspecting standards; the state is responsible for checking external agency performance to examine if objectives are being met.
- Giving rights to service users; the state ensures that beneficiaries have a voice in the provision of services and goods so providers are made accountable to consumers.
- Promoting new and better methods; the state promotes ‘good practice’, such as the “support for education and training initiatives concerning governance and management” in the UK education and health services.
- Adjusting sector boundaries; the state threatens to bring in the private sector if performance is below expectations in the state half of partnerships.
- Intervention; in extreme cases if performance is well-below expectations the state “reserves power to impose changes.”
(Paton, 2003; 21-30)
This work now deals with how far the techniques of performance measurement can answer firstly, the requirements and secondly, the problems of development as public action involving a multiplicity of agencies but with government agencies maintaining an important role.
The World Development Report of 1997 stressed the need for the state to improve its performance. Tools such as the economic analyses of CBA and CEA as well as the instruments suggested by Paton not only increase state efficiency through dealing with multiple agencies but also two of the other E’s-economy and effectiveness. The introduction of private sector practice into partnerships looks to evaluate them with an eye to “getting exactly what you want at the lowest possible overall cost.” (DFID, 2000: 3). However, the state needs to exercise caution over equity performance due to distributional problems. The provision of services and goods to remote and by implication more expensive-to-reach areas should not be compromised, “value for money is not the only factor to justify an intervention”. (Asian Development Bank, 1997, 24).
A competitive environment is a requirement of multiple agency involvement in state-private sector partnership development. Tools which are used to estimate the performance and impact of policies or projects aid the state in its choice of partner agency. The lack of this requirement can result in decisions which have been made because no other suitable alternative exists. In cases such as this the state should maintain its monopoly. Transparency in policy and project decision making remains important in the state. The awarding of concessions or contracts through merit rather than personal relationships is important to improving the four E’s, the tools which measure performance in advance, if accuracy is maintained, help this process.
The techniques of performance measurement guarantee that the state retains political control, as final decisions using them are made by the state themselves. Although participation of social enterprises and the public is possible in order to receive an overview of a development environment, the state is the key decision maker. Lastly, and this can be taken as a starting point too, the techniques help governments think about going into partnerships, they do not compel the state into partnership and act as tools of information.
The problems associated with multiple agency roles in state-private sector partnership development illustrate how the techniques cannot answer all the questions it raises. La Fond discusses the contradiction between performance measurement and sustainability. She asks if performance measurement is a technique in which policies and projects are sustained rather than their objectives. La Fond suggests that the sustainability of local capacity and self-reliance provide a more accurate picture of development performance. (La Fond, 1995: 23-37). A tool such as Multiple Criteria Analysis lends itself to this kind of adaptation wherein the nature of the sustainability of a project can form a performance indicator.
There is an attribution problem linked to the management information system as proposed by Heeks. While the paradigm is successful in using performance measurement to address problems with process it cannot attribute the causes of change. The question to be asked is whether the outputs a policy or project involving multiple agencies achieves would have happened anyway, outside factors may have had an influential part in the change. “A major difficulty in attempting to measure benefits is to determine the extent to which it is possible to attribute improvements to the project as opposed to factors external to the project such as economic growth, increasing demand for labour, investment in public services… and infrastructure, all of which have a positive impact on well-being”. (Robinson and Thin, 1993). The paradigm has been extended to include impact assessment and external factors, (Thomas, 2004: 5) but the measurement of these criteria remain uncertain. An example of this attribution problem can be found in the operations of the British Council in Venezuela. A clear performance management system was in place and regular reviews ensured high performance levels from the English teachers. The teaching centre experienced high enrollment in 2003 during what were politically uncertain times in Venezuela. The problem for management was where to attribute this success; to improved teacher performance or the need for Venezuelan professionals to secure English language accreditation so that they could leave the country.
Paton suggests that problems associated with state-private sector partnership development can be found within the instruments proposed to deal with performance management at this level. Paton warns against selective inspections by the state of external agencies and to be wary of the “self-congratulatory reports they produce to cover inefficient performance. The implication is that deception in both halves of the partnership can have an effect on how well policies and projects perform. The domestic political and economic need for success can have detrimental impacts on transparency. Paton adds that public deference to the state and social enterprises may not subject them to the scrutiny which exposes such ‘fudges’. In fact the desire to provide evidence that multiple agency state led development is being carried out to donors in order to secure loans leads to the formulation of policies with performance difficulties. (Paton, 2003:22-28).
In conclusion, the techniques which have been outlined in this essay are merely an aid to decision making regarding policies and projects and their performance or estimated performance. They suit many of the requirements of ‘development as public action involving a multiplicity of agencies but with government agencies maintaining an important role’, in that they answer the needs of a competitive market and the improvement of state performance through its detailed measurement. The problems which partnership present are not all answerable by the techniques, this is because they are not immune to domestic and external politics as well as economic pressures. Therefore the techniques while they provide a forum for state and external agency partnership must operate in an environment of transparency on the part of the state and external agencies if an increase in the performance criteria of efficiency, effectiveness, economy and equity are to be achieved.
Asian Development Bank (1997) ‘Guidelines for the Economic Analysis of Projects’, [Online] Available at: http://www.adb.org/Documents/Guidelines/Eco_Analysis/least-cost.asp (accessed 7 January 2005).
Cook, P., Kirkpatrick, C. (1988) ‘Privatisation in Less Developed Countries: An Overview’ in Cook, P., Kirkpatrick, C., (eds) Privatisation in Less Developed Countries, Brighton, Wheatsheaf Books, pp. 3-48.
DFID (2000) ‘Procurement and Training Manual’, London, DFID Procurement Department.
DTLR (2001) ‘Multi-criteria Analysis: a manual’ London, Department for Transport, Local Government and the Regions.
Heeks, R. (1998) ‘Public Sector Management Information Systems’, IDPM Working Paper Series, Paper No. 5, Manchester, Institute for Development Policy and Management.
International Fund for Agricultural Development (2002) ‘Managing for Impact in Rural Development: A Guide for Project M&E’, Rome, IFAD.
LaFond, A. (1995) ‘Sustaining Primary Health Care’ London, Earthscan.
Mind Tools (2004-last update) ‘Cost/Benefit Analysis-Evaluating Quantitatively Whether to Follow a Course of Action’ [Online] Available at: http://www.mindtools.com/pages/article/newTED_08.htm (accessed 6 January 2005).
Paton, R. (2003) Managing and Measuring Social Enterprises, London, Sage.
Prest, A.R., and Turvey, R. (1965) ‘Cost-benefit Analysis: a survey’, Economic Journal, vol. 75, pp. 683-735.
Robinson, M. and Thin, N. (1993) ‘Project Evaluation: A Guide for NGOs’, Glasgow, Overseas Development Administration Joint Funding Scheme, NGO Unit.
Simister, N. (2000) ‘Laying the Foundations: the role of data collection in the monitoring systems of development NGOs’, Bath, Centre for Development Studies.
Thomas, A. (2004) ‘Lecture notes, Development Management: Issues and Trends, 2A Performance and its Management’.
World Bank (1997) ‘Overview’, World Development Report 1997: The State in a Changing World, Washington DC, World Bank, pp.1-11.
World Bank (2004) ‘Impact Evaluation, Overview: What is impact evaluation?’, [Online] Available at: http://web.worldbank.org/WBSITE/EXTERNAL/TOPICS/EXTPOVERTY/EXTISPMA/0,,menuPK:384339~pagePK:162100~piPK:159310~theSitePK:384329,00.html (accessed 3 January 2005).
United Nations Development Programme (UNDP) Evaluation Office (7 January 2005-last update) ‘Glossary’ [Online] Available at: http://www.undp.org/eo/ADR/glossary.htm (accessed 7 January 2005).
United Nations Economic and Social Commission for Asia and the Pacific (UNESCAP) (2003) ‘Methods of assessing the effectiveness of policies/measures’, [Online] Available at: http://www.unescap.org/drpad/vc/orientation/M5_10.htm (accessed 5 January 2005).