Emin Evaluation Review 2pages

Intelligence Network Sep 30th 09: Evaluation Friar Gate Studios was the venue for the latest Intelligence Network event on Evaluation. Two of the region’s leading voices on evaluation provided an overview of the terminology and issues and provided practical advice on research methods. Miles Burger from East Midlands Development Agency presented a primer in evaluation, emphasising the importance of considering evaluation at the start of any project. Data is key; details of those affected by a programme must be collected early so they can be traced at a later date. The detail required in the data also needs consideration, particularly in respect of the Data Protection Act. Data already collected for monitoring purposes forms the bedrock of evaluation programmes, although additional research is usually necessary in order to research wider outcomes for a particular population. Miles also explored the measurement of impact of an intervention on an area, and how concepts such as displacement, substitution, leakage and multiplier effect must be addressed in order to assess how much of any change in an area can be attributed to a particular policy. It’s also important to specify the type of impact being measured. For some evaluations net economic impact is appropriate, relying on monetary statistics and quantitative methods. However, in some cases this may not be appropriate; Miles gave the example of a network facilitator, something that couldn’t be easily measured in monetary terms. Here, we have a problem of measuring Strategic Added Value (SAV), which require the use of more qualitative methods.

Intelligence Network Sep 30th 09: Evaluation Caroline Boucher compared commonly used research methods and the advantages and disadvantages of using them. Focus groups and indepth interviews are two commonly used qualitative methods. They are far more labour intensive than quantitative (statistical) methods and so are greatly dependent on the skills of facilitators and interviewers for their success. Although postal and online surveys are cheaper options. which are more likely to give statistically robust results, they cannot be explored to the same depth as in qualitative methods. The concept of 'statistically robust' was touched upon. Remember that the ONS regional statisticians are happy to advise on specific projects, e.g. what confidence intervals are acceptable etc. A response rates of less than 10% from postal surveys can be expected and, like some other methods, potentially biased by extreme views. Online surveys may get slightly higher response rates, but exclude those without computer access. Caroline then took on the hot topic of how to choose external consultants to carry out evaluations. Clear communication channels are essential, both to ensure clarity in expectations and to defuse potential defensiveness from those running an evaluated programme. Purpose, aims, outputs and budget must be in any Invitation To Tender...and remove jargon! Attendees then tackled a very valuable interactive exercise, examining real tenders for evaluation projects and discussing how best to put them into practice. This provided some interesting contrasts between groups on their approaches and sparked a lively discussion, something which was a hallmark of the whole event. The IEM Resource Pack from the day is available on our website Topics page.