Measuring learning
This presentation is the property of its rightful owner.
Sponsored Links
1 / 17

Measuring Learning PowerPoint PPT Presentation

  • Uploaded on
  • Presentation posted in: General

Leah J Rowe Air Force Research Laboratory 711 Human Performance Wing. Measuring Learning. Overview. Continuous learning About us The end game Instructional design theory Where we are going Approach for continuous learning Research and training on all levels Competency based approach

Download Presentation

Measuring Learning

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript

Measuring learning

Leah J Rowe

Air Force Research Laboratory

711 Human Performance Wing

Measuring Learning



  • Continuous learning

  • About us

  • The end game

  • Instructional design theory

  • Where we are going

  • Approach for continuous learning

  • Research and training on all levels

  • Competency based approach

  • Training research testbeds

  • Family of complimentary trainers

  • Live, virtual, and constructive training

  • Best Practices: Putting it all together

Continuous learning 711hpw rhas

Continuous Learning (711HPW/RHAS)

  • Develop mission relevant content for learning

  • Measure training outcomes in complex mission environments

  • Accelerate development of mission critical competencies

  • Optimize training value of live, virtual, and constructive environments

Measuring learning

About us

  • CHALLENGE: Training and Readiness requirements are outpacing capacity and existing opportunities to train

  • MISSION: Research human performance methods and technologies that provide the Warfighter the necessary knowledge and skill to dominate their operating environment

Instructional design theory

Instructional Design Theory

Reigeluth & Carr-Chellman, 2009 p.9

Design theory is aimed at facilitation generative outcomes… It assists in the creation of something.1

Learning Sciences: A hybrid discipline that includes learning theory and instructional-event theory.1

Purposely multidisciplinary and not as interested in goals as in the use of certain kinds of instructional methods to shed light on certain kinds of learning processes.1

1 Reigeluth, C. M., & Carr-Chellman, A. A. (2009). Instructional-Design Theories and Models: Building a Common Knowledge Base (Vol. III). New York: Routledge.

Learning management systems

Learning Management Systems

LMS Video

Where we are going game changing learning s t

Where We Are Going: Game Changing Learning S&T



  • Metrics and scenarios

    • Tailored needs assessment/gap analyses

    • Cross-domain readiness assessment

    • Proficiency tracking and feedback

  • Proficiency-based training

    • Evidence-based fidelity assessments

    • Cross domain after-action reviews

    • Team and team-of-teams assessment

    • Family of complementary trainers

  • Adaptive, distributed content and assessment

  • Scenario authoring, scripting, and metrics definition standards

  • Common data exchange, control, management

    • Managed learning/feedback across LVC contexts

  • Globally distributed learning

  • Seamless integration of LVC environments and players

  • Adaptive training and operational environments

  • Portable, personal learning and occupational records

  • Predictive performance models anticipate gaps/refreshers


Training research the big picture

Training Research: The Big Picture

The simulators are a means to an end

Each systems is in place to execute training research

  • Capitalized on a training research model introduced by Schreiber and Bennett during the Distributed Mission Operations Within-Simulator Training Effectiveness Baseline Study2.

    • 76 teams of F-16 pilots and Air Battle Managers participated

    • Represents the largest DMO effectiveness study known to exist today

    • Measures included: objective data from the simulators, multiple participant surveys, subject matter expert ratings of performance, and knowledge structure tests

    • DMO training lead to significant improvements in warfighter performance between the pre-training and post-training assessments

2 Schreiber, B. T., Bennett, W. Jr. (2006a). Distributed Mission Operations Within-Simulator Training Effectiveness Baseline Study. Volume I: Summary Report (AFRL-HE-AZ-TR-2006-0015-Vol I). Mesa AZ: Air Force Research Laboratory, Warfighter Training Research Division.

Distribution A: Approved for public release

Approach for continuous learning

Approach for Continuous Learning


  • Warehousing

  • What was learned

  • Where

  • How much

  • How often

Adapt Conditions

Based on Performance

& Diagnosis

Readiness/Proficiency Monitoring

  • Competencies

  • Knowledge

  • Skills

  • Experiences

  • Gaps …










Match Opportunities to Objectives

ScenariosInstructional EventsStructure & TriggersStrategies & Principles Support resources



Match Measures to Objectives & Conditions

Individual/Team/Team of Teams

Objective and Subjective

What/How to Measure

Research training on all levels

Research & Training on All Levels




Facilitate the development of strategies, methods, and competency based tools to promote learning

Research can be executed by assessing training at a micro level or on a grand scale


Environment that provides unique immersive training and research opportunities


Provides the capability to demonstrate robust performance assessment within an integrated theater combat environment

Training research competency based approach

Training Research: Competency Based Approach

Use a building block or competency based approach. Training curricula, scenarios, and systems are mapped to the knowledge, skills, and experiences that they are able to provide to trainees.

  • 3Alliger, G. M., Beard, R., Bennett, W., & Colegrove, C. M. (2012). Understanding Mission Essential Competencies as a Job Analysis Method. In M. A. Wilson, W. Bennett, S. Gibson, & G. M. Alliger, The Handbook of Work Analysis Methods, Systems, Applications, and Science of Work Measurement in Organizations (pp. 603-624). New York: Routledge.

  • 4 Colegrove, C M and W Bennett. Competency-Based Training: Adapting to Warfighter Needs. Technical Report: AFRL-HE-AZ-TR-2006-0014. Mesa: Air Force Research Laboratory Human Effectiveness Directorate Warfighter Readiness Research Division, 2006.

Training research testbeds

Training Research Testbeds

  • Provide innovative and highly immersive training research to develop efficient and effective training warfighter training solutions

  • What are the supporting competencies, knowledge, and skills, for effective team and team of teams training?

  • What are the training needs, gaps, and opportunities in a networked training environment for the trainees?

  • What is the most effective training structure to effectively train a team-of-teams in a testbed that will provide enhanced training capabilities to the warfighter?

  • Testbeds built to assess ContinuousLearning

Distribution A: Approved for public release

Family of complimentary trainers

Family of Complimentary Trainers

  • Assess alternative training strategies and methods for enhancing RPA training by offering solutions that match the level of fidelity required for specific types of training

  • Effectively and appropriately using training resources

  • Leverages both commercial and government off the shelf solutions to build cost effective training gap interventions that allow us to research and develop innovative strategies for continuous learning

Continuum of Training

Distributed mission operations

Distributed Mission Operations


“Train the way we intend to fight”, enabling Air Force warfighters to maintain combat readiness and conduct mission rehearsal in an environment as operationally realistic as possible4

4 Ales, R. & Buhrow, S, (2006) “The Theater Air Ground System Synthetic Battlespace”, Proceedings of the I/ITSEC 2006 Conference, Orlando, Florida, USA, November 2006.

Lvc definitions





Live – human beings operating actual operational weapons system, in a typical combat environment or training scenario

Virtual – real people operating simulators of any kind (i.e., a tank driver in a sim, a JTAC in a dome, or simulated AOC

  • Constructive – computer generated entities, weapons, systems used to enhance the training environment

Vision continuous learning in lvc ops

The Future State is Performance-Enabled Learning and Readiness

Vision: Continuous Learning in LVC Ops

A capability where aiding and learning activities are interwoven seamlessly with live operations on-demand

  • The end state optimizes individual and team performance through the design, monitoring, support, management, and enhancement of learning and developmental/recurring experiences for current and future operations

  • The foundation is in science and technology related to:

    • Modeling learning and action

    • Enabling proficiency on demand

    • Managing the learning and experiences that occur across the career of warfighters as individuals and as members of teams in complex decision-centric environments

Best practices putting it all together

Best Practices: Putting it all together

Utilize the methods in the “Within-Simulator Training Effectiveness” study as best practices to develop a robust training research program.

  • Login