Planning guide for evaluating relevance
This tool is designed to provide evaluators with an overview for evaluating relevance. It does not seek to be comprehensive, but focuses instead on strategic action for cost-effective and useful evaluation. Use this table to assist in your planning for the evaluation or relevance.
Evaluation element
Relevance
Objective
Assess how well proposed learning and development interventions address business needs, capability needs and individual needs within the agency.
Data collection method
- Documentary analysis
- Stakeholder interviews or surveys
- Use the evaluation quality checklist to guide your evaluation activity
Possible data sources
- Organisational business plans
- HR performance reports
- Workforce plans
- Consolidated Performance Management data
- 360/180 feedback data
- Consolidated learning plan or Learning Management System data
- Organisational reviews
- Staff climate and satisfaction surveys
- EAP data
- Insight from key stakeholders
Analytical tools
Relevance index - Refer to references and resources list for more background information.
Reporting tool
Relevance and appropriateness scorecard
Timing
Conducted after learning objectives have been formulated but before learning is designed
Stakeholders and their roles
Evaluator
Conduct and manage the evaluation
Key program sponsors
Provide background, rationale and context
Key line managers in areas where learning is to be applied
Provide understanding of how learning is to be used
Relevant management and other reference committees
Provide input, guidance and approvals
Learning designer
Incorporate findings into the design
Relevance index
Overview
Evaluating relevance is about ascertaining if the learning interaction has the potential to meet the needs or goals for which it is designed.
What this tool can help you to do
This tool is designed to provide some key indicators of relevance which can be used to evaluate the degree and nature of relevance of L&D interaction.
How to use this tool
Use this index to help ascertain the extent that the objectives and outcomes of the learning are relevant to a need or goal. Record your answers for each question in the score column, then total the scores and plot this on the key index provided below
| Question | Answers | Score |
|---|---|---|
|
Index score. A score in the 1–10 zone could indicate that the learning interaction is not fully relevant to the need or goals. Insert the index score on the Relevance and appropriateness scorecard as a means of reporting on the learning’s relevance.
|
||
|
1.Basis of the learning objective/outcomes To what extent are the learning objectives/outcomes based on valid individual/team or organisational needs/goals? |
|
|
|
2. Importance of the learning How important are the learning objectives/outcomes to the business? |
|
|
|
3. Congruence with workforce planning To what extent are learning objectives/outcomes congruent with workforce planning goals? |
|
|
|
4. Making a difference What difference will successful learning and application make to the organisation? |
|
|
|
5. Addressing systemic needs/goals To what extent do the learning objectives/outcomes address systemic (individual, group and organisational) needs/goals? |
|
|
|
6. Stability of the learning objectives Are the learning objectives/outcomes likely to remain stable in the short term? |
|
|
|
7. Stakeholder buy in What degree of buy in is there from stakeholders for the learning objectives/outcomes? |
|
|
|
8. Learning design To what extent is the learning design congruent with the learning objectives/outcomes? |
|
|
| Total score | ||
Planning guide for evaluating appropriateness
This tool is designed to provide evaluators with an overview for evaluating appropriateness. It does not seek to be comprehensive, but focuses instead on strategic action for cost-effective and useful evaluation. Use this table to assist in the planning for the evaluation of appropriateness.
Evaluation element
Appropriateness
Objective
Measure how appropriate the allocation of resources for L&D is to identify needs/goals.
Data collection method
- Documentary analysis
- Cost analysis
- Stakeholder interviews or surveys
- Use the evaluation quality checklist to guide your evaluation activity
Possible data sources
- The learning designs
- Cost/time analysis of the learning design
- Key stakeholders
- Industry/APS comparisons
- Tender and procurement submissions from service providers.
Analytical tools
- Appropriateness index
- Cost calculator (excel spreadsheet)
- Output potential index (excel spread sheet)
- Refer to references and resources list for more background information.
Reporting tool
Relevance and appropriateness scorecard
Timing
Conducted once learning is designed
Stakeholders and their roles
Evaluator
Conduct and manage the evaluation
Key line managers in areas where learning is to be applied.
Provide input on design suitability, and potential workplace support
Program sponsors
Provide insight into organisational emphasis and preferences, support and budgetary constraints
Relevant management and other reference committees.
Provide input, guidance and approvals
Learning designer
Incorporate findings into design
Appropriateness index
Overview
Evaluating appropriateness is assessing the suitability of the learning interaction to organisational culture, funding, infrastructure and to needs/goals
What this tool can help you to do
This tool is designed to provide some key indicators of appropriateness which can be used to evaluate the degree and nature of appropriateness of L&D activities.
How to use this tool
Use this index to help ascertain the extent that the learning and its related aspects are appropriate to meeting a need or goal. Record your answers for each question in the score column, then total the scores and plot this on the key index provided below.
| Question | Answers | Score |
|---|---|---|
|
Index score. A score in the 1–15 zone could indicate that the learning interaction is not fully relevant to the need or goals. Insert the index score on the Relevance and appropriateness scorecard as a means of reporting on appropriateness.
|
||
|
1. Addressing the need To what extent will learning meet the expressed need or goal |
|
|
|
2. Alternative to addressing the need Are there other ways that can better meet the expressed need or goal? |
|
|
|
3. Achievability How achievable are the learning objectives/outcomes |
|
|
|
4. Proportion How proportionate is the allocation of resources to the need/goal the learning is designed to address? |
|
|
|
5. Affordability How affordable is the learning |
|
|
|
6. Value for money To what extent does this learning represent value for money? |
|
|
|
7. Integration To what extent is the learning able to integrate with other organisational processes? |
|
|
|
8. The learning design To what extent is the learning congruent with the organisational context? |
|
|
|
9. Cultural fit To what extent will the learning fit with the desired culture of the workplace? |
|
|
|
10. Alternative learning activities Are there reasonable and viable alternatives to this form of learning |
|
|
|
11. Learner intentionality To what extent will the learning design encourage learner intentionality? |
|
|
|
12. Support for application of learning To what extent will the organisational context successfully support application of learning |
|
|
| Total score | ||
Output potential index
Download the output potential index.
Tips when evaluating relevance and appropriateness
1. Appreciate that some performance needs may not be satisfactorily met by learning activities alone. Such performance needs may relate to:
- lack of proper or sufficient physical or capital resources
- inappropriate work structure, work allocation and poor work flow management
- lack of sufficient, relevant and timely information
- poor supervision, management and leadership practices
- ambivalent work attitudes and low workforce morale
- unclear or unreasonable expectations
- inadequate feedback
- inadequate or unfair incentives and rewards
- lack of capacity due to poor recruitment, selection and promotion processes.
2. Know why learning activities fail to add value
- learning activities are not linked specifically to strategies, challenges or problems in the organisation
- learning is used as a stand alone intervention and is not supported by other action or linked to other organisational processes and systems
- learning activities are too focused on the individual and not sufficiently on teams/groups or other purposeful sub-unit workgroup
- participants take part in activities for reasons other than personal or organisational need.
- learning is not aligned with participant’s work environment
- immediate supervisors do not support or reinforce learners on the job application of skills and knowledge
3. Involve stakeholders, generate their interest and raise awareness of their role in relevant and appropriate learning interactions
- Communicate the intended business value or impact from any proposed learning and development to opinion leaders. Show what would happen if learning and development is not carried out.
- Speak the language of the business.
- Submit persuasive business cases.
- Involve relevant committees early in any design and development or consultation work.
- Get line management and/or stakeholder sign off for design, content and delivery etc before learning and development interactions begin.
- Put Service Level Agreements in place to articulate all stakeholder responsibilities, deliverables and quality standards.
- Involve management in the learning and development interaction as guest speakers, lunch time guests, panel members, program sponsors and mentors, coaches and subject matter experts etc.
- Be proactive in engaging with your stakeholders.
Relevance and appropriateness scorecard
Download the relevance and appropriateness scorecard.
References and resources—Evaluating relevance and appropriateness
Books
Becker, BE, Huselid, MA & Ulrich D 2001, The HR Scorecard, HBS Press, Boston, MASS.
Comprehensive resource on embedding HR processes and initiatives including learning and development into the business and strategy of the organisation. Very useful tools for ensuring alignment and relevance.
Carliner, S 2002, Designing E-learning, ASTD, Alexandria, VA.
Contains a chapter on appropriate e- learning designs for a variety of needs and learning goals and is a useful resource when evaluating appropriateness of e- learning products and solutions.
Driscoll, M 1998, Web-based training, Jossey Bass, San Francisco.
Provides a useful chapter on making decisions on appropriate learning designs for various needs, circumstances and goals. Provides useful reference when evaluating the appropriateness of e-learning and web based products and solutions.
Kaplan, RS & Norton, DP 2004, Strategy Maps, HBSP, Boston, MASS.
Provides a comprehensive treatment of how visual information maps can be used to plot and track alignment and line of sight of business processes including learning and development to organisational goals.
Piskurich, GM, ed, HPI Essentials, ASTD, Alexandria, VA.
A performance perspective is a useful one to adopt when evaluating the relevance and appropriateness of learning interactions as not all needs and goals are best met by learning and development. This book provides a useful performance based approach which can be used as background to any evaluation of relevance and appropriateness.
Professional associations
The Australasian Evaluation Society
The American Society for Training and Development