|
IRIN Evaluation Report - March 2003
by Graham Mytton & Sharon Rusu
Part 1: Background and Introduction - Continued
Purpose of the Evaluation
-
On the eve of its eighth year of operation (October, 2002), the United Nations Office for the Coordination of Humanitarian Affairs (OCHA) initiated an independent evaluation process of the Integrated Regional Information Network (IRIN). The purpose of the evaluation was to review the experience of the IRIN network, learn from the experience and provide accountability to its donors, OCHA management and its constituencies by assessing the relevance, efficiency, effectiveness, value-added, impact and sustainability of IRIN's products and services, with emphasis on current services and organisation. The evaluation sought to obtain opinions from a broad base of subscribers, including UN, NGOs, media, donors, governments, academics and civil society. 16 A team of two consultants undertook the evaluation, Graham Mytton and Sharon Rusu, selected by OCHA on the basis of their backgrounds and experience in media marketing analysis, humanitarian affairs and information management. (See Annex 1, Terms of Reference for the IRIN Evaluation).
Framework for the Evaluation
-
In developing a framework for the evaluation of IRIN as a 'humanitarian news service', broadly conforming to the characteristics of regional information, news and learning networks, but with a strong focus on knowledge, the evaluators first considered an overarching question: to what extent has the network advantage been achieved? 17 Reference to the implications of this question as to what comprised an 'effective' IRIN was helpful in the development of methods for assessing the impact and measuring the effectiveness of the IRIN model. (See Annex 2, A Framework for Analysis) 18
Evaluation Method
- The method employed for the IRIN evaluation was an iterative one, over 3 phases. (See Annex 3 for a description of the 3 phases of the IRIN Evaluation). Oral interviews and an e-survey were the major methods used for capturing the information. (See Annexes 4 and 5 for examples of the e-survey and interview questionnaires).
-
The e-survey was developed and sent out 9 December 2002 to all IRIN email subscribers and posted on the IRIN website. Between 9 December 2002 and 6 January 2002, 1, 891 responses were received from the email subscribers, while the IRIN website produced 91 responses.
| E-Survey Responses |
| Email subscribers |
1891 |
| Website |
91 |
| Total |
1982 |
-
By 31 January 2003, 137 interviews had been completed. (See Annex 6 for a complete list of interviewees).
| Interviews by Affiliation |
| United Nations |
44 |
| NGO |
24 |
| Media |
31 |
| Donors |
13 |
| Academic |
7 |
| OCHA/IRIN Management, Staff and Stringers |
16 |
| Total |
137 |
| The results of both surveys were analysed and can be found in Part 2 under Findings |
16 Office for the Coordination of Humanitarian Affairs, 'Terms of Reference', Evaluation of the Integrated Regional Network (IRIN), Revised, November, 2002
17 Heather Creech and Terry Willard, Strategic Intentions: Managing Knowledge Networks for Sustainable Development,' International Institute for Strategic Development (IISD), 2001, p.5.Creech and Willard's work clearly links the transfer of knowledge to informed decision-making.
18 Assessing the IRIN network model, which is not a single but composite entity, posed several challenges. Is it a regional humanitarian news network, one that provides the context for learning and transfers knowledge to its sources of information, sister agencies and its larger readership? Or is it simply a news service that networks solely to provide and verify sources amongst a group of 'informateurs'? If it is a functioning information network, how effective is IRIN's networking capacity in terms of its institutional role and partnerships, strategic alliances, and its influence on decisions? If not, does it matter? Another challenge was assessing IRIN's credibility and independence, which are arguably central to assessing IRIN's relevance as a reliable source of information and its capacity to produce balanced reports. Although a non-commercial venture dependent on donor support for funding, IRIN is part of the UN system under the direct administration of the Office for the Coordination of Humanitarian Affairs (OCHA) which, at the least, makes the assessment of independence, a potentially thorny one. A further challenge lay in assessing the value-added of the IRIN model, especially its cost-effectiveness. Arguably IRIN's value and the costs attached to its upkeep depend on the judgement of its readership. Assessing their opinions of IRIN's products and services was viewed as central to measuring the effectiveness of IRIN's performance as a humanitarian news service. A yet further challenge lay in assessing the tendency of organisations to develop self-protecting mechanisms (including networks) that militate against change and challenge by vigorously defending the status quo. Thus, measuring the impact of a composite information network like IRIN, within the UN system, meant more than simply evaluating IRIN's performance as a humanitarian news service against agreed performance benchmarks. Overall, it meant assessing the impact and influence of IRIN products and services on the behaviour of its users, including assessing the extent to which IRIN informed policy and decision-making in support of positive humanitarian outcomes
Continued?
|
|
