Formative Research on the Heuristic Task Analysis Process

1. Formative Research on the Heuristic Task Analysis Process Charles M. Reigeluth Ji-Yeon Lee Bruce Peterson Mike Chavez Indiana University

2. Domain vs. Task Domain Task Economics Psychology Physics Chemistry Management Counseling Engineering Medicine

3. What is Heuristic Task Analysis? • Domain vs. task expertise • Procedural vs. heuristic tasks • Heuristic tasks: using principles and guidelines to decide what to do when • Heuristics are often tacit knowledge • Heuristic Task Analysis (HTA): elicits and formalizes an expert’s heuristic knowledge

4. Why is HTA important? • Increasingly complex tasks in society • Many tasks are heuristic (continuum) • Heuristic task knowledge is often the most valuable and hardest to discover • Heuristic task analysis methodologies are underdeveloped

5. Literature on HTA • Knowledge elicitation: Extraction of domain-relevant knowledge from human experts using various techniques • Knowledge analysis: Interpretation of the data extracted from experts • Knowledge representation: Determined by the purpose of application (e.g. expert system or Artificial Intelligence system)

6. Literature on HTA • Most studies deal with procedural rather than heuristic tasks • Lack of research that provides guidelines on how to analyze and represent heuristic knowledge • Lack of research to inform instructional design

7. The Initial HTA Process The handout shows the version of the Heuristic Task Analysis Process investigated in this research.

8. Research Question “ What improvements can be made to the HTA process?”

9. Formative Research • Developmental research: improves a theory and its real-world application • Process: • Apply a theory to an instance • Collect data to improve the application • Suggest improvements to the theory • Iterations • Three cases

10. Study I: Context • Task: Group Counseling • Focused on personal growth group • Combination task with both procedural and heuristic elements • Constraints: • Due to the confidential nature of counseling, direct observation was impossible • Dealing with interpersonal dynamics which is hard to verbalize • Analyst unfamiliar with group counseling

11. Study I: Methods • Formative research: designed case • Data collection: • Interview: unstructured in-person interviews • Videotapes • Participant profile: • 3 experts experienced with personal growth group counseling • Expertise ranged from 3 to more than 20 yrs • Data analysis & interpretation: • Member checks with participants • Consultation with three other researchers

12. Study I: Results • To best elicit task experts’ heuristic process, we need to incorporate various interview and observation techniques. • The analyst’s task expertise needs to be considerable. • Working with multiple experts requires certain strategies. • Providing reference material during the interview was helpful.

13. Study II: Context • Task: Tutoring on Writing • Class of problems focused on higher-level writing concerns such as thesis, unity, and coherence • Constraints: • Experts had little time to devote to this project

14. Study II: Methods • Top-down approach to identify and analyze heuristic knowledge • Participant profile: • 2 experts experienced in tutoring college students • had passed an evaluation of their tutoring skills each term they had been tutoring • Data collection& analysis • Interview: In-person interviews (60 min. each) • Iterative interview & analysis process • Index cards used to record tutor’s responses

15. Study II: Results • Using the top-down process was effective • Using index cards was helpful • Helping the tutor identify guidelines was problematic • Helping the tutor access the context of the tutoring situation was helpful • Having the tutor list, prioritize, and choose the decisions made during the tutoring session was effective

16. Study III: Context • Task: • Selecting Artwork for Product Line • Constraints: • Analyst unfamiliar with business content • Experts had little time to devote to this project • Research focus: • Speed and effectiveness of HTA process in business setting

17. Study III: Method • Formative research: designed case • HTA guidelines converted into lay questions • 1 expert, 2 interviews, 1 1/2 hours each • Content, process analysis after interviews • Member checks w/ 1 content and 1 research expert

18. Study III: Results • Speed of HTA process: • Expert was satisfied with time frame • 50% of interview was spent eliciting heuristics • 1 hour was spent on closely associated tasks • 3x5 cards helped analyst get back on track • Effectiveness of HTA process: • Expert satisfied when heuristics were elicited • Words like feel, know, like were clues to tacit knowledge and mental models

19. Discussion • Success with the top-down approach (knowledge categories) varied in the studies • More guidance was needed for eliciting the heuristics • Using multiple experts presented challenges for the analyst • Index cards were helpful for all 3 analysts • The HTA process seemed to function well in all three fields.

20. Revisions to the HTA Process • Preparatory guidance was added to help focus the entire process • Guidance was added for multiple experts • Guidance added on how to elicit heuristics • Bottom-up approach was added (Revisions on handout are in italics)

21. Questions or Comments? For the paper, e-mail Charles Reigeluth at reigelut@indiana.edu