Go to top of page

Appendix 1b: Line of sight

Planning guide for evaluating relevance

This tool is designed to provide evaluators with an overview for evaluating relevance. It does not seek to be comprehensive, but focuses instead on strategic action for cost-effective and useful evaluation. Use this table to assist in your planning for the evaluation or relevance.

Evaluation element

Relevance

Objective

Assess how well proposed learning and development interventions address business needs, capability needs and individual needs within the agency.

Data collection method

  • Documentary analysis
  • Stakeholder interviews or surveys
  • Use the evaluation quality checklist to guide your evaluation activity

Possible data sources

  • Organisational business plans
  • HR performance reports
  • Workforce plans
  • Consolidated Performance Management data
  • 360/180 feedback data
  • Consolidated learning plan or Learning Management System data
  • Organisational reviews
  • Staff climate and satisfaction surveys
  • EAP data
  • Insight from key stakeholders

Analytical tools

Relevance index - Refer to references and resources list for more background information.

Reporting tool

Relevance and appropriateness scorecard

Timing

Conducted after learning objectives have been formulated but before learning is designed

Stakeholders and their roles

Evaluator

Conduct and manage the evaluation

Key program sponsors

Provide background, rationale and context

Key line managers in areas where learning is to be applied

Provide understanding of how learning is to be used

Relevant management and other reference committees

Provide input, guidance and approvals

Learning designer

Incorporate findings into the design

Relevance index

Overview

Evaluating relevance is about ascertaining if the learning interaction has the potential to meet the needs or goals for which it is designed.

What this tool can help you to do

This tool is designed to provide some key indicators of relevance which can be used to evaluate the degree and nature of relevance of L&D interaction.

How to use this tool

Use this index to help ascertain the extent that the objectives and outcomes of the learning are relevant to a need or goal. Record your answers for each question in the score column, then total the scores and plot this on the key index provided below

Question Answers Score

Index score. A score in the 1–10 zone could indicate that the learning interaction is not fully relevant to the need or goals. Insert the index score on the Relevance and appropriateness scorecard as a means of reporting on the learning’s relevance.

  • Some relevance: 1–10
  • Relevant: 11–17
  • Highly relevant: 18–24

1.Basis of the learning objective/outcomes

To what extent are the learning objectives/outcomes based on valid individual/team or organisational needs/goals?
  1. To some extent
  2. To a large extent
  3. To a full extent
  

2. Importance of the learning

How important are the learning objectives/outcomes to the business?
  1. Somewhat important
  2. Important
  3. Very important
  

3. Congruence with workforce planning

To what extent are learning objectives/outcomes congruent with workforce planning goals?
  1. To some extent
  2. To a large extent
  3. To a full extent
  

4. Making a difference

What difference will successful learning and application make to the organisation?
  1. Some difference
  2. Significant difference
  3. Critical difference
  

5. Addressing systemic needs/goals

To what extent do the learning objectives/outcomes address systemic (individual, group and organisational) needs/goals?
  1. Address some systemic levels
  2. Addresses most systemic levels
  3. Addresses all systemic levels
  

6. Stability of the learning objectives

Are the learning objectives/outcomes likely to remain stable in the short term?
  1. Somewhat stable
  2. Stable
  3. Very stable
  

7. Stakeholder buy in

What degree of buy in is there from stakeholders for the learning objectives/outcomes?
  1. Some degrees of buy in
  2. Good buy in
  3. Very good buy in
  

8. Learning design

To what extent is the learning design congruent with the learning objectives/outcomes?
  1. Somewhat congruent
  2. Congruent
  3. Very congruent
  
  Total score  

Planning guide for evaluating appropriateness

This tool is designed to provide evaluators with an overview for evaluating appropriateness. It does not seek to be comprehensive, but focuses instead on strategic action for cost-effective and useful evaluation. Use this table to assist in the planning for the evaluation of appropriateness.

Evaluation element

Appropriateness

Objective

Measure how appropriate the allocation of resources for L&D is to identify needs/goals.

Data collection method

  • Documentary analysis
  • Cost analysis
  • Stakeholder interviews or surveys
  • Use the evaluation quality checklist to guide your evaluation activity

Possible data sources

  • The learning designs
  • Cost/time analysis of the learning design
  • Key stakeholders
  • Industry/APS comparisons
  • Tender and procurement submissions from service providers.

Analytical tools

  • Appropriateness index
  • Cost calculator (excel spreadsheet)
  • Output potential index (excel spread sheet)
  • Refer to references and resources list for more background information.

Reporting tool

Relevance and appropriateness scorecard

Timing

Conducted once learning is designed

Stakeholders and their roles

Evaluator

Conduct and manage the evaluation

Key line managers in areas where learning is to be applied.

Provide input on design suitability, and potential workplace support

Program sponsors

Provide insight into organisational emphasis and preferences, support and budgetary constraints

Relevant management and other reference committees.

Provide input, guidance and approvals

Learning designer

Incorporate findings into design

Appropriateness index

Overview

Evaluating appropriateness is assessing the suitability of the learning interaction to organisational culture, funding, infrastructure and to needs/goals

What this tool can help you to do

This tool is designed to provide some key indicators of appropriateness which can be used to evaluate the degree and nature of appropriateness of L&D activities.

How to use this tool

Use this index to help ascertain the extent that the learning and its related aspects are appropriate to meeting a need or goal. Record your answers for each question in the score column, then total the scores and plot this on the key index provided below.

Question Answers Score

Index score. A score in the 1–15 zone could indicate that the learning interaction is not fully relevant to the need or goals. Insert the index score on the Relevance and appropriateness scorecard as a means of reporting on appropriateness.

  • Appropriate in some areas: 1–15
  • Appropriate in most areas: 16–32
  • Highly appropriate: 33–36

1. Addressing the need

To what extent will learning meet the expressed need or goal
  1. Some extent
  2. Large extent
  3. Full extent

2. Alternative to addressing the need

Are there other ways that can better meet the expressed need or goal?
  1. Some other ways
  2. A few other ways
  3. No other ways

3. Achievability

How achievable are the learning objectives/outcomes
  1. Somewhat achievable
  2. Achievable
  3. Very achievable

4. Proportion

How proportionate is the allocation of resources to the need/goal the learning is designed to address?
  1. Somewhat proportionate
  2. Proportionate
  3. Very proportionate

5. Affordability

How affordable is the learning
  1. Just affordable
  2. Affordable
  3. Very affordable

6. Value for money

To what extent does this learning represent value for money?
  1. Some value for money
  2. Good value for money
  3. Significant value for money

7. Integration

To what extent is the learning able to integrate with other organisational processes?
  1. Some integration
  2. Largely integrated
  3. Fully integrated

8. The learning design

To what extent is the learning congruent with the organisational context?
  1. Some congruence
  2. Generally congruent
  3. Fully congruent

9. Cultural fit

To what extent will the learning fit with the desired culture of the workplace?
  1. Some fit
  2. Good fit
  3. Best fit

10. Alternative learning activities

Are there reasonable and viable alternatives to this form of learning
  1. Some alternatives
  2. A few alternatives
  3. No alternatives

11. Learner intentionality

To what extent will the learning design encourage learner intentionality?
  1. Some encouragement
  2. General encouragement
  3. Significant encouragement

12. Support for application of learning

To what extent will the organisational context successfully support application of learning
  1. Support to some extent
  2. Support to a large extent
  3. Fully support
Total score

Output potential index

Download the output potential index.

Tips when evaluating relevance and appropriateness

1. Appreciate that some performance needs may not be satisfactorily met by learning activities alone. Such performance needs may relate to:

  • lack of proper or sufficient physical or capital resources
  • inappropriate work structure, work allocation and poor work flow management
  • lack of sufficient, relevant and timely information
  • poor supervision, management and leadership practices
  • ambivalent work attitudes and low workforce morale
  • unclear or unreasonable expectations
  • inadequate feedback
  • inadequate or unfair incentives and rewards
  • lack of capacity due to poor recruitment, selection and promotion processes.

2. Know why learning activities fail to add value

  • learning activities are not linked specifically to strategies, challenges or problems in the organisation
  • learning is used as a stand alone intervention and is not supported by other action or linked to other organisational processes and systems
  • learning activities are too focused on the individual and not sufficiently on teams/groups or other purposeful sub-unit workgroup
  • participants take part in activities for reasons other than personal or organisational need.
  • learning is not aligned with participant’s work environment
  • immediate supervisors do not support or reinforce learners on the job application of skills and knowledge

3. Involve stakeholders, generate their interest and raise awareness of their role in relevant and appropriate learning interactions

  • Communicate the intended business value or impact from any proposed learning and development to opinion leaders. Show what would happen if learning and development is not carried out.
  • Speak the language of the business.
  • Submit persuasive business cases.
  • Involve relevant committees early in any design and development or consultation work.
  • Get line management and/or stakeholder sign off for design, content and delivery etc before learning and development interactions begin.
  • Put Service Level Agreements in place to articulate all stakeholder responsibilities, deliverables and quality standards.
  • Involve management in the learning and development interaction as guest speakers, lunch time guests, panel members, program sponsors and mentors, coaches and subject matter experts etc.
  • Be proactive in engaging with your stakeholders.

Relevance and appropriateness scorecard

Download the relevance and appropriateness scorecard.

References and resources—Evaluating relevance and appropriateness

Books

Becker, BE, Huselid, MA & Ulrich D 2001, The HR Scorecard, HBS Press, Boston, MASS.

Comprehensive resource on embedding HR processes and initiatives including learning and development into the business and strategy of the organisation. Very useful tools for ensuring alignment and relevance.

Carliner, S 2002, Designing E-learning, ASTD, Alexandria, VA.

Contains a chapter on appropriate e- learning designs for a variety of needs and learning goals and is a useful resource when evaluating appropriateness of e- learning products and solutions.

Driscoll, M 1998, Web-based training, Jossey Bass, San Francisco.

Provides a useful chapter on making decisions on appropriate learning designs for various needs, circumstances and goals. Provides useful reference when evaluating the appropriateness of e-learning and web based products and solutions.

Kaplan, RS & Norton, DP 2004, Strategy Maps, HBSP, Boston, MASS.

Provides a comprehensive treatment of how visual information maps can be used to plot and track alignment and line of sight of business processes including learning and development to organisational goals.

Piskurich, GM, ed, HPI Essentials, ASTD, Alexandria, VA.

A performance perspective is a useful one to adopt when evaluating the relevance and appropriateness of learning interactions as not all needs and goals are best met by learning and development. This book provides a useful performance based approach which can be used as background to any evaluation of relevance and appropriateness.

Professional associations

The Australasian Evaluation Society

http://www.aes.asn.au/

The American Society for Training and Development

http://www.astd.org 

Last reviewed: 
29 March 2018