Interview with Dayna Albert, Manager, Towards Evidence-Informed Practice (TEIP)
Dayna has been a part of the TEIP since March 2005, first as a Program Coordinator developing and pilot testing the TEIP Program Assessment Tool and most recently as TEIP Manager.
Towards Evidence- Informed Practice (TEIP): www.teip.hhrc.net
January 13, 2010
1. What is evidence-informed practice?
At TEIP we take a broader approach to the term. We address the complete knowledge-to-action-to-knowledge cycle where we use evidence to inform practice and then generate practice-based learning through program evaluation.
2. Why is evaluation important for those starting a program or who have one already running?
Evaluation is an important piece to continuously strengthen programs. We can always make programs stronger. The purpose of evaluation can be:
- To communicate to funders or other practitioners your projects’ accomplishments
- To identify how to further strengthen the program.
You should go into evaluation with an open mind to figure out what are the important evaluation questions and the most realistic and efficient ways to collect the information.
For those starting a program, evaluation activities focus mainly on formative evaluation (e.g., are educational materials appropriate for your intended audience?) and process evaluation (how effectively is the program being implemented). Process is an important element of program effectiveness. It has been estimated that 50% of a program’s success depends on the content of your program (i.e., activities) and 50% depends on process (i.e., how the program is delivered).
The start of a program is an ideal time to think about evaluation. Look to the future and ask yourself: "What are the outcomes that I will want to measure?” and “What indicators are available to measure those outcomes?” and “How can I collect and analyze the information?”
For those with programs already up and running, the focus would be less on formative (unless you are changing something) and more on process and outcome evaluation.
3. What are some of the barriers someone might face if they want to do an evaluation? And what are some ways around such barriers?
Time is always an issue. Confidence and skills is another – people are nervous about evaluation. However, if you have a tool, then you can work through evaluation planning systematically, and this can help.
People often feel threatened or worried that an evaluation may reflect badly on them. It is important to remember that the goal of evaluation is to make the program stronger. This requires that you ask the important questions and be open to the findings. Try to view the program from an outsider’s perspective.
Another mistake people make is thinking that they must do evaluation on their own. TEIP recommends developing an evaluation team comprised of a variety of stakeholders: both people who are familiar with the program and people with evaluation expertise (perhaps someone in your organization or among your partners).
Evaluation is a learning opportunity as well: the more you do, the better you get at it. You don’t need to feel that sophisticated tools are required. Consider what is useful to know, what is relevant to know and how you can measure these while working within realistic limits.
4. Although evaluation can take some time and effort, where can it save time and effort?
The last thing you want to do is to deliver a program that misses the mark. I have seen programs that have been running for 20 years and they have missed pieces that could have easily been picked up by an evaluation.
In evaluation, you want to get the biggest bang for your buck. Consider collecting stories about how your program has impacted specific individuals. These stories can be shared (with the agreement of participants) for program promotion purposes, with funders or as a means of inspiring others. Data collection can support reporting requirements. An evaluation report provides useful documentation which can serve to teach others about the program and can serve as archival material.
5. What is the role of the program participant in evaluation?
It depends on your participants and how engaged they want to be. The gold standard is to include participant representation on your evaluation team. Or you can seek their input. For example, when brainstorming overall evaluation questions, ask participants what they would want to know through the evaluation. Your evaluation should consider some of the issues important to them.
6. Can you describe some of the tools available on the TEIP website? Which would be most relevant to those starting a program? And those already running a program?
All three TEIP tools contribute to strengthening the quality and effectiveness of programs and policies. The choice of tool depends on your goals at the time. If a program is in the planning phase, the Program Evidence Tool provides a systematic process, including guidelines and worksheets, to look at a range of evidence to inform program development.
If an organization wants to assess overall program design: the Program Assessment Tool provides 19 quality criteria associated with effective programs (e.g., the quality of partnerships, extent of program evaluations, program content, etc.). This tool is a useful starting place if you want to build evaluation capacity but lack experience. By assessing current evaluation practices, staff learn about the different types of evaluation and how to strengthen evaluation plans. We have compiled a list of alternative uses of the assessment criteria which offers less intensive processes for applying the criteria. For example, the criteria can be used in a checklist format as an adjunct to planning or to set goals and benchmarks, rather than assessing current practice.
The Program Evaluation Tool supports evaluation planning, specifically for process and outcome evaluation. This can be done during the initial planning stages, and/or when an existing program is ready to start serious evaluation planning.
7. What is a TEIP Master Trainer Workshop and who is it geared to?
The Master Trainer Workshops aim to build evaluation capacity in an organization by building skills among organizational “champions” and to generate strategies and action plans to build local organizational supports. TEIP provides ongoing support for these champions. To find out more about these workshops, visit www.teip.hhrc.net.
8. What does a membership in TEIP Master Trainer Forum offer?
The Master Trainer Forum is designed for the Master Trainers (who have undergone the training) to be a part of a community of practice to share their struggles and successes.
9. Do you have any examples of using evaluation for programs that focus on physical activity?
There are examples on the website of Program Assessment for programs that look at pedometer lending, active transportation, and walking. We encourage people to share their findings on our website.
10. Is the Evaluation Helpline still available?
The Evaluation Helpline is available until funding runs out – March 15, 2010. It is underused at the moment, so now is a good time to access it. The helpline provides up to 4 hours of free one-on-one support from an evaluation expert.
You need to have already started the evaluation process using a recognized systematic evaluation tool (see link below). You must fill out a consultation form and send any work you have done already (logic model, evaluation questions, etc.) . The process for accessing the helpline is described at this link: http://www.teip.hhrc.net/tools/evaluation_helpline.cfm
Some people use the helpline for ‘just-in-time’ supports by setting up an initial one-hour session to get feedback or coaching, and then a few weeks later have a second consultation to get further feedback.
11. Would you be available for follow-up questions from the program leaders?
If you have a simple question, you can e-mail Dayna at dalbert [at] opha [dot] on [dot] ca.
Further descriptions of the tools and TEIP services can be seen in the recent Ontario Chronic Disease Prevention Alliance newsletter at www.ocdpa.on.ca.
