Select an Evaluator
Before beginning the evaluation, it is important to select an evaluator (i.e., someone who is responsible for leading the process).
An evaluator should be someone who:
- Understands college preparation programs
- Has the capacity to understand your program
- Has the skills and experience needed to conduct a proper evaluation
Potential options for an evaluator include:
- An external source
- A consulting firm
- College and university personnel
- An internal source – a program staff member
Determining whether to use an internal staff member or to rely on an external evaluator is a critical step. Some factors to consider when making this decision include:
- The type of data you may need
- The purpose of the evaluation
- Staff workload and expertise
- Program resources (e.g., financial, necessary computer software, etc.)
Using a program staff member may be less costly and more effective in soliciting staff input. However, it also adds to staff workload. Make sure the internal staff member has experience in evaluation, collecting and working with data, and analyzing information. Additionally, the staff member will need resources to track participant data (e.g., college entry, educational progress, applications for scholarships and financial aid, and participation and progress in the program, etc.).
Some programs may not have enough technical expertise within their organization. Thus, it may be prudent to rely primarily on an external evaluator for the evaluation. This may be more costly but less time consuming for current staff. A disadvantage to this approach is the lack of staff engagement and input in the process. However, continually working with an external evaluator can help build long-term evaluation capacity that may not exist due to staff turnover if an internal evaluator is selected.
Another alternative would be to contract with an evaluation expert to support you in the more technical aspect of the evaluation. This may be a happy medium because it may thwart unneeded costs and ensures program staff will be actively involved in the process.
The Administration of Children and Families has developed a worksheet to help grantees decide to what extent they may need to rely on external evaluation resources. Based on your answers, the worksheet recommends one of the following options for conducting evaluation tasks:
- Mostly Relying on External Evaluators – The external evaluator may be an individual, research institute, or consulting firm and serves as the team leader and is supported by in-house staff.
- Relying on Internal Evaluators and Working with an External Consultant – An internal evaluator serves as the team leader and is supported by program staff and an outside consultant.
- Relying on Internal Evaluators – An in-house evaluator serves as the team leader and is supported by program staff.
|Resources for Appropriate Team Selection (check one)||Yes||No|
|Does your program have funds designated for evaluation purposes?|
|Have you successfully conducted previous evaluations of similar programs or services?|
|Are existing program practices and information collection forms useful for evaluation purposes?|
|Can you collect evaluation information as part of your regular program operations (at intake, termination)?|
|Is there program staff that has training and experience in evaluation-related tasks?|
|Are there advisory board members who have training and experience in evaluation-related tasks?|
The checklist above can help to select an evaluation team in the following ways.
Using the Checklist
- If your √ answer to all the resource questions is “no,” you may want to consider postponing your evaluation until you can obtain funds to hire an outside evaluator, at least on a consultancy basis. You may also want to consider budgeting funds for evaluation purposes in your future program planning efforts.
- If your √ answer to question 1 is “yes,” but you answer “no” to all other questions, you will need maximum assistance in conducting your evaluation and an outside evaluator with in-house support is probably your best choice.
- If you √ answer “no” to question 1, but “yes” to most of the other resource questions, then using in-house staff only may be an appropriate choice for you. Keep in mind, however, that if you plan to use evaluation findings to seek program funding, you may want to consider using an in-house evaluation team with an outside consultant. You can try to obtain evaluation funds from other areas of your agency’s budget.
- If your √ answer to question 1 is “yes” and the remainder of your answers are mixed (some “yes” and some “no”), then you can think about the extent to which you want to rely on external evaluators.
If it is decided to solicit outside assistance for the evaluation, here are some key steps to follow:
- Determine who is qualified
- Divide responsibilities
- Create a contract
Finding a qualified evaluator who understand your outreach program can be challenging. Research organizations, consulting firms, and universities are potential places for finding a qualified evaluator. The Resources section to the left provides links to databases of evaluators.
Before interviewing prospective evaluators, you should determine which qualifications are important to you. You may need evaluators who know your program or the target population, or it may be more important that evaluators have specific technical expertise.
At the very least, evaluators should possess formal training in evaluation, professional orientation, previous performance of evaluation tasks, and personal styles that fit with your organization (Worthen and Sanders, 1987).
You can use the worksheet below as a guide to determine whether the evaluator meets those criteria that are important for you:
Assessing Evaluator Qualifications
|Well Qualified||Not Well Qualified||Cannot Determine if Qualified|
|To what extent does the formal training of the potential evaluator qualify him/her to conduct evaluation studies? (Consider major or minor degree specializations; specific courses in evaluation methodology; whether the potential evaluator has conducted applied research in a human service setting, etc.)|
|To what extent does the previous evaluation experience of the potential evaluator qualify him/her to conduct evaluation studies? (Consider items such as length of experience; relevance of experience.)|
|Acceptable Match||Unacceptable Match||Cannot Determine Match|
|To what extent is the professional orientation of the potential evaluator a good match for the evaluation approach required? (Consider items such as philosophical and methodological orientations.)|
|Well Qualified||Not Well Qualified||Cannot Determine if Qualified|
|To what extent does the previous performance of the potential evaluator qualify him/her to conduct evaluation studies for your project? What prior experience does she or he have in similar settings? (Look at work samples or contact references.)|
|Acceptable||Unacceptable||Cannot Determine Acceptability|
|To what extent are the personal styles and characteristics of the potential evaluator acceptable? (Consider items such as honesty, character, interpersonal communication skills, personal mannerisms, ability to resolve conflicts, etc.)|
|Well Qualified and Acceptable||Not Well Qualified and/or Unacceptable||Cannot Determine if Qualified or Acceptable|
|Based on the questions above, to what extent is the potential evaluator qualified and acceptable to conduct the evaluation?|
If you decide to hire an external organization or consultant, you will need to think about how evaluation tasks should be divided. Some potential responsibilities of the external evaluator include:
- Developing the evaluation plan, with the input of your staff
- Training program staff on different evaluation topics
The input of an evaluation expert is needed at the planning stage to ensure that your design is appropriate for answering the evaluation questions. We recommend that you and your staff also be heavily involved in:
- The planning stage
- The development of the logic model
- Providing regular feedback to the evaluator on the data collection instruments, methodology, and results
The involvement of you and your staff in making key decisions is essential, so that the evaluation does not become a process that only the evaluator understands. For example, one college outreach program contracted an external evaluator to develop a computerized system that tracked service use and participant outcomes, but program staff had not been given a chance to voice their opinion about which computer programs they felt comfortable using. The result was that the staff were unable to fully utilize the tracking system because they were unfamiliar with the computer application on which the system was based. The program manager acknowledged that the problem would have been prevented if the staff had input in the decision about the most appropriate computer application.
The following is a possible division of responsibilities between the evaluator and the program manager/program staff (borrowed from the Administration of Children and Families’ Guide to Program Evaluation):
Potential Responsibilities of the Evaluation Contractor
- Develop an evaluation plan, in conjunction with program staff.
- Provide monthly or quarterly progress reports to staff (written or in person).
- Train project staff. Training topics could include:
- Using evaluation instruments, information collection activities, participant/case selection for sampling purposes, and other activities
- Designing information collection instruments or selecting standardized instruments or inventories
- Implement information collection procedures such as:
- Interview project staff
- Interview coordinating/ collaborating agency staff
- Interview program participants
- Conduct focus groups
- Observe service delivery activities
- Review participant case records
- Develop database
- Code, enter, and clean data
- Analyze data
- Establish and oversee procedures ensuring confidentiality during all phases of the evaluation.
- Write interim (quarterly, biannual, yearly) evaluation reports and the final evaluation report.
- Attend project staff meetings, advisory board or interagency coordinating committee meetings, and grantee meetings sponsored by funding the agency.
Potential Responsibilities of the Program Manager
- Educate the outside evaluator about the program’s operations and objectives, characteristics of the participant population, and the benefits that program staff expect from the evaluation. This may involve alerting evaluators to sensitive situations (for example, the need to report suspected child abuse) they may encounter during the course of their evaluation activities.
- Provide feedback to the evaluator on whether instruments are appropriate for the target population and provide input during the evaluation plan phase.
- Keep the outside evaluator informed about changes in the program’s operations.
- Specify information the evaluator should include in the report.
- Assist in interpreting evaluation findings.
- Provide information to all staff about the evaluation process.
- Monitor the evaluation contract and completion of work products (such as reports).
- Ensure that program staff is fulfilling their responsibilities (such as data collection).
- Supervise in-house evaluation activities, such as completion of data collection instruments and data entry.
Below is an example of division evaluation responsibilities successfully utilized by an outreach program:
Responsibilities of the Evaluation Contractor
- Designing the evaluation process and writing up the evaluation plan
- Developing the evaluation instruments and forms
- Designing the database used for tracking ongoing information
- Analyzing the information collected
- Compiling the information into yearly reports
- Possibly continuing to support the staff to maintain and further develop the tracking system after the evaluation is completed
Responsibilities of the Program Staff
- Printing, tracking, and administrating surveys and staff evaluation forms
- Entering the survey and staff evaluation data into the tracking system
- Entering student and staff information into the tracking system
- Providing the data for analysis and reporting to the evaluation contractor
The contract spells out the division of evaluation responsibilities between the evaluator and the program staff and the level of contact between the two. Some program managers have found that outside evaluators, after they are hired, delegate many of their responsibilities to less experienced staff and have little contact with the program managers or the staff. To some extent, a contract can protect your program from this type of situation.
The following are some of the questions that should be discussed and clarified when negotiating a contract with evaluators. (Modified questions by Stake, R.E. 1976).
- What resources are available for the conduct of this study? What cost estimates can be made (e.g. in money, staff time, program disruption)?
- What is the work history and working style of the prospective evaluators? Do they have a portfolio of reports and artifacts from completed studies?
- Why would the evaluators be interested in doing this study? What is in it for them?
- What will be the primary sources of data? What arrangements would be necessary to gain access to these sources? Are rules of access needed?
- During the course of the evaluation study, where and how would the data be kept? What would be the rules of access to these data?
- What would be a suitable plan for reporting the findings? Informal feedback? Progress reports? Final presentations? Are the evaluators free to publish findings in professional journals? What checks will be made on the effectiveness of the evaluation feedback?
- How will further arrangements be negotiated after the study begins? What will be the response to unexpected changes in program? What misunderstandings may arise between the sponsors of the study and the evaluators? How will conflict be resolved?
- What more needs to be said about the purposes and expectations for the evaluation study?
The University of Michigan has developed the following evaluation contract checklist to help make sure you have discussed and included all relevant information in your contract. If any item does not apply to your program evaluation, write down “n/a” next to the item. Put a check mark next to the items that are important and are incorporated, and leaving items that are important, but not agreed to, blank.