Effect measurement

Danida Fellowship Centre (DFC) is continuously involved in effect measurement of our activities, and in addition Danida’s Evaluation Department decided in 2011 to commission an evaluation of the support to the Danida Fellowship Programme, as considerable Danida resources have been allocated to the programme over the years.

The overall purpose of the evaluation was to document and assess the supported activities and the related results, with an aim to contribute to both accountability and learning. To this end, it assessed the different achievements of the programme, as well as the framework and processes for accomplishing the results. The evaluation was expected to have its main focus on the period 2008-2011, so as to be able to include assessment of learning outcomes. It covered the entire geographical scope of Fellowship Programme at an overall level and two countries, Ghana and Uganda, were selected for field visits and more in-depth analysis.

Please find the summary and full evaluation report here.

 

DFC’s monitoring and evaluation (M&E) of training activities

DFC is in the process of developing and implementing (per mid 2012) a structured web-based monitoring and evaluation system enabling an assessment of the effect of the DFC-organised training on the level of competence and capacity of the individual fellows.

The DFC M&E system is based on Kirkpatrick’s evaluation model and will include effect measurement at three levels:

• Response level
• Learning level based on self assessment before and after
• Behavior level, primarily based on the Action Plan prepared by the fellows during the course

 

Evaluation of result level effect beyond effects directly attributable to DFC training is considered beyond the area of influence by DFC.

Response or reaction level

The response or reaction evaluation measures how the fellows feel immediately as the training ends, and measures their personal reactions to the training and training experience. This response level evaluation of course elements focus on the quality and competence of trainers and facilitators.

Learning level

Measurement of learning effect is done as a measurement of competence level before the training and similarly after the training. The difference is attributed to the conducted training.

The evaluation will include a baseline assessment, where fellows will indicate their perceived competence level on two competence aspects: i) technical competences which are specific for any given training course, ii) general and generic competences which are common for all DFC training courses, and reflect a typical competence profile for a fellow.

Learning effect can be assessed by comparing before and after values on the competence elements. “After” information can be compiled immediately at the end of each training course and follow-up after three months or more.

The learning level evaluation report for 2015 is available here.

Behaviour level

Behaviour level evaluations comprise assessment of changes in capacity to influence development processes in the fellows’ home country as seen from the perspective of the employing organization or agency, limited to changes within the mandate of the organization/agency, and not in the home country at large.

Behaviour level effect includes transfer from learning to performance and consequently a change process, which is exactly what the Action Plan approach aims at.

The proposed M&E will be mapping changes 3+ months after completing the DFC training course. This will register changes within three dimensions: knowledge and skills, the AP perspective, and the job content and performance.

Interdisciplinary courses

DFC has implemented a post-course outcome evaluation with limitations (a cohort study) of all interdisciplinary courses (22) held in 2010 in Denmark. Data for this evaluation has been collected via a digital questionnaire.

Generally, fellows indicate that their level of competence is fairly high (apart from one course, all averages are above the median line). The same goes for how fellows rate the outcome from participating in the course in Denmark. For 20 out of 22 courses the average lies in the upper quartile. One course scores significantly lower, but based on the numbers alone it has not been possible to pinpoint why.

The survey does demonstrate a direct correlation between prior competences and course impact. The scores for the two lowest ranked courses will be discussed with the relevant places of study. On the whole, it can be concluded that DFC has had ‘value for money’, i.e. the Fellowship Programme is seen as having a positive impact of the Fellows’ competencies at individual level.

The electronic M&E system, in reality a Learning Management System with functionalities exceeding that of mere evaluation, has now been finished and will be in effect from 2012.

For more detailed information, look at the report.

Survey of the Private Sector Initiative

In a recent survey of the effect of the short courses aimed at the private sector, the majority of the respondents report:

• that the course has been useful in their daily work
• that, due to their attendance at the training course, the business turnover of the organization will increase
• the competitiveness of the organization will increase
• new jobs will be created in the organization.

The survey was carried out in October 2010, and covered 225 participants attending courses in 2008-2010. The response rate was 23%.

Please find the full report here.