design based research a new research paradigm for open and distance learning feb 2007 l.
Skip this Video
Loading SlideShow in 5 Seconds..
Design- Based Research: A New Research Paradigm for Open and Distance Learning Feb, 2007 PowerPoint Presentation
Download Presentation
Design- Based Research: A New Research Paradigm for Open and Distance Learning Feb, 2007

Loading in 2 Seconds...

play fullscreen
1 / 85

Design- Based Research: A New Research Paradigm for Open and Distance Learning Feb, 2007 - PowerPoint PPT Presentation

  • Uploaded on

Design- Based Research: A New Research Paradigm for Open and Distance Learning Feb, 2007. Terry Anderson, Ph.D. Canada Research Chair in Distance Education. Presentation Overview. Traditional opening joke Need for distance education research Sorry state of current research

I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
Download Presentation

Design- Based Research: A New Research Paradigm for Open and Distance Learning Feb, 2007

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
design based research a new research paradigm for open and distance learning feb 2007

Design- Based Research: A New Research Paradigm forOpen and Distance LearningFeb, 2007

Terry Anderson, Ph.D.

Canada Research Chair

in Distance Education

presentation overview
Presentation Overview
  • Traditional opening joke
  • Need for distance education research
  • Sorry state of current research
  • Methodological Orientations
    • Quantitative
    • Qualitative
    • Critical
    • Design-based
  • Dissemination and building a research culture
why is distance education better than sex
Why is Distance Education Better Than Sex?
  • If you get tired, you can stop, save your place and pick up where you left off.
  • You can finish early without feeling guilty.
  • You can get rid of any viruses you catch with a $50 program from McAfee
  • With a little coffee you can do it all night.
  • You don’t usually get divorced if your spouse interrupts you in the middle of it.
  • And If you're not sure what you are doing, you can always ask your tutor.
athabasca university alberta canada
Athabasca University, Alberta, Canada

Fastest growing university in Canada

34,000 students

700 courses

Graduate and Undergraduate programs

Largest Master of Distance Education program

Only USA Accredited University in Canada

* Athabasca University

  • Athabasca University
distance education research
Distance Education Research
  • Can you think of at least one major contribution to practice made by distance education research in the last decade?
  • Why does this particular research result make a difference?
    • Question ;
    • context;
    • methodology;
    • clarity of presentation?
typical de research
Typical DE Research
  • “Here is what we are doing at my university. Isn’t it wonderful !!”
  • “Here is what we are doing , why don’t you come and research it?”
  • Variable quality by journal, by conference, by region, by practice.
defining research
Defining Research
  • “the systematic study of materials and sources in order to establish facts and reach new conclusions” (Oxford Compact Dictionary, 1991).
  • adjectives describing research as
    • disciplined, organized,
    • transparent, problem orientated,
    • public, creative,
    • scientific, systematic,
    • diligent, labourous
    • and accessible
why do research in distance education
Why Do Research in Distance Education?
  • Many unresolved questions of traditional distance education
    • - attrition; F2F tutorial value, paced vs unpaced;
  • new forms of distance education provision.
    • what combinations of group based learning are worth the cost and inconvenience?
    • Do face to face tutorials really make a difference or is real time video or audio conferencing just as effective?
    • How much does expensive multimedia really enhance student learning?
    • how important are real time interactions compared to asynchronous ones ?
    • Do Web 2.0 and social software tools really encourage new forms of communities of inquiry?
why educational research just don t get no respect
Why Educational Research‘Just Don’t Get No Respect’
  • Most research is not valued by funders, other academics or worse by practitioners
  • Not funded financially
    • Education 0.01 % of expenditures
    • Health 3.0 %
    • High tech companies 10.0 - 15.0 %
    • Overall (Canada, 2002) 1.88%

Rodney Dangerfield

assessment of de research
Assessment of DE Research
  • Many experimental research projects do not display rigour in their design
  • Many generalize inappropriately
  • Cultural, linguistic and environmental factors often not taken into consideration
  • Few concerned with teacher and tutor support
  • Few studies based on current learning, pedagogical or psychological theories
    • Olugbemiro Jegede (1999)
a practitioner s perception of educational research
A practitioner's perception of educational research
  • “answers are too narrow to be meaningful, too superficial to be instrumental, too artificial to be relevant, and, on top of that, they usually come too late to be of any use.” van den Akker 1999
barriers to educational research
Barriers to Educational Research
  • It’s nobody’s job
    • How many in this room have at least 50% of their job requirement to do research?
  • Negligible industry support
  • No large scale focus on particular problems
  • Nobody keeping score in meaningful terms
  • Pervasive lack of trust in research efficacy
  • In sum, lack of an effective research culture
        • (Burkhardt and Schoenfeld, 2003).
who should do research
Who Should Do Research?
  • Action Research
    • Focused on an authentic problem
    • Designed and implemented by participants
    • Includes reflection ands with dissemination
  • Students as researchers
    • Why are students excluded from this rich learning experience?
  • Professional researchers
    • DE or educational technology researchers
    • Faculty teaching DE within a different academic perspective
research paradigms
Research Paradigms
  • Quantitative~ discovery of the laws that govern behavior
  • Qualitative ~understandings from an insider perspective
  • Critical~Investigate and expose the power relationships
  • Design-based ~interventions, interactions and their effect in multiple contexts
quantitative paradigm
Quantitative Paradigm
  • Key words like “evidence based” “systematic review” “scientific research”
  • employs a scientific discourse derived from the epistemologies of positivism and realism
  • Long tradition borrowed from the natural sciences
  • Since context is so pivotal in education, a great number of studies must be done to eliminate contextual variance and combined using meta analysis.
  • Inordinate support and faith in randomized controlled studies
“those who are seeking the strict way of truth should not trouble themselves about any object concerning which they cannot have a certainty equal to arithmetic or geometrical demonstration”
    • (Rene Descartes,). 1496-1650
the challenge of meeting criteria for quantitative study
The challenge of meeting criteria for quantitative study
  • Control group assignment rarely possible
  • Blind assessment not practical
  • Contextual variables in natural contexts negate transfer and replicability
  • ‘What works’ in one context, at one time does not guarantee it will work again
  • Interventions are never controlled nor identical
  • Educational results must always be interpreted
quantitative ex meta analysis
Quantitative Ex. – Meta-Analysis
  • Ungerleider and Burns (2003)
  • Systematic review of effectiveness and efficiency of ICT
  • The type of interventions studied were extraordinary diverse –only criteria was a comparison group and use of ICT
  • “Only 10 of the 25 studies included in the in-depth review were not seriously flawed, a sobering statistic given the constraints that went into selecting them for the review.”

Ungerleider, C., & Burns, T. (2003) . A systematic review of the effectiveness and efficiency of networked ICT in education . P.38 Ottawa: Industry Canada. Retireved Jan. 24, 2004 from

USA Department of Education (2003) guidelines for Identifying and Implementing Educational Practices Supported By Rigorous Evidence:
quantitative summary
Quantitative Summary
  • Can be useful especially when testing well established and consistent practice.
  • The need to “control” context often makes results of little value to practicing professionals
  • In times of rapid change too early quantitative testing may mask beneficial positive capacity
  • Will we ever be able to afford blind reviewed, random assignment studies?
qualitative paradigm
Qualitative Paradigm
  • Many different varieties
  • Generally answer the question ‘why’ rather then ‘what’ or ‘how much’?
  • Presents special challenges in DE due to distance between participants and researchers
  • Currently most common type of DE research (Rourke and Szabo, 2002)
1 st qualitative example
1st Qualitative Example
  • Dearnley (2003) Student support in open learning: Sustaining the Process
  • Practicing Nurses, weekly F2F tutorial sessions
  • Phenomenological study using grounded theory discourse
“Support structures to facilitate personal and professional development within this context need to be in place and attention must be given to the provision of effective learner support.” (Dearnley, 2003)”

Johnson H. (2007). Dialogue and the Construction of Knowledge in E-Learning: Exploring Students' Perceptions of Their Learning While Using Blackboard's Asynchronous Discussion Board. EURODL

  • Four different ways of perceiving online learning were identified
    • Practical experience (Rosemary)
    • Interconnections (Sarah, Katherine, Cindy)
    • Expressing own thoughts (Anthony, David)
    • Flexible learning (Larry)
qualitative summary
Qualitative Summary
  • Measure of quality is “critical appraisal concerning plausibility, internal consistency and fit to prevailing wisdom”
    • Burkhardt & Schoenfeld (2003)
  • But what if the only producers and consumers are researchers not practitioners?
  • Often the only answer that makes it to the practice is “it depends”!

Critical Example Friesen, N. (submitted). The Experience of Computer Use: Expert Knowledge and User Know-How. GLIMPSE: Phenomenology and Media.

  • “I will also show how the actual experience of computer use casts into doubt the educational efficacy of computers understood as instruments of cognitive amplification, or simply ‘mindtools.’ ”
  • Did my failure to save my work or to properly address my email message arise from a mismatch in "system" and "user" models?
  • “user knowledge of the system appears as embodied, performative and emphatically provisional in nature.”
but what type of research has most effect on practice
But what type of research has most effect on practice?
  • Kennedy (1999) - teachers rate relevance and value of results from each of major paradigms.
  • No consistent results – teachers are not a homogeneous group of consumers but they do find research of value
  • “The studies that teachers found to be most persuasive, most relevant, and most influential to their thinking were all studies that addressed the relationship between teaching and learning.”
but what type of research has most effect on practice37
But what type of research has most effect on Practice?
  • “The findings from this study cast doubt on virtually every argument for the superiority of any particular research genre, whether the criterion for superiority is persuasiveness, relevance, or ability to influence practitioners’ thinking.” Kennedy, (1999)
“In any dispute the intensity of feeling

is inversely proportional to the value

of the stakes at issue -- that is why

academic politics are so bitter.”

Wallace S. Sayre, 1905-1972

quoted in,

"Issawi's Laws of Social Motion" (1973)

4 th paradigm design based research
4th ParadigmDesign-Based Research
  • Related to engineering and architectural research
  • Focuses on the design, construction, implementation and adoption of a learning initiative
  • Related to ‘Development Research’
  • Closest educators have to a “home grown” research methodology
  • Interventionist within a real educational context
  • Action Research on Steroids!
the contextual turn
The Contextual Turn
  • Postmodern assertion of variability pervasively induced by uniqueness of any particular context
  • Context of:
    • Place – virtual, home, classroom, institution etc.
    • Actors – many individual differences, temporary and long lasting
    • Culture – including intra cultural heterogeneity
  • “Learning, Cognition, Knowing and Context are irreducibly co-constituted and cannot be treated as isolated entities or processes” (Barab & Squire, 2004)
context creates content creates context
Context creates Content creates Context

deFigueiredo (2005) Learning Contexts a Blueprint for Research,

the complexity turn
The Complexity Turn
  • Increased interest in viewing educational contexts as complex environments:
    • Not possible to precisely forecast or predict behaviour – explain and promote - but not predict
    • Interventions from the outside (teacher interventions) are conditioned and recursively amplified or extinguished by contextual variables
    • Resulting in emergent behaviours of organisms
    • Complexity “at the edge of chaos” provides opportunity for creativity and necessary change
    • See Underwood (2000) complexity theory; Pascale et. al (2000) Surfing the Edge of Chaos, Bennet, (2004) Organizational Survival



Evaluation &



Pervasive effect

of assessment

4th paradigm design studies
4th Paradigm Design Studies
  • iterative,
  • process focused,
  • interventionist,
  • collaborative,
  • multileveled,
  • utility oriented,
  • theory driven and generative
    • (Shavelson et al, 2003)
design based example 1 athabasca
Tutor Model

On call 2 hours/week

Part time, problem with knowledge of institution

knowledge of a single course

Personal relationship

Subject matter expertise

Personal knowledge base

Call Centre Model

Advisor on call 40 hours/week

Full time, steeped in university environment

All business curriculum

Customer relationship

Refers to academics for subject matter expertise

Formal FAQ and data collection

Design Based Example 1 - Athabasca

Call Centres:

Answer 80% of student inquiries

Saves over $100,000 /year

stage 1 informed exploration
Stage 1: Informed Exploration
  • Literature review, theoretical extrapolation and expert and participant input
  • Often an ideal provides a vision and a guide as well as significant component of the measuring stick by which the ideal, as instantiated in actions within a real context, is measured.
stage 1 informed exploration49
Stage 1: Informed Exploration
  • Review of call centre literature
  • Interviews with current tutors and managers
  • Data collection on current processes and costs
  • Visit to other call centres, especially those in related but uncompetitive contexts
stage 2 enactment
Stage 2: Enactment
  • Production phase – highly visible
  • Need for project management, tracking and documentation
  • Prototype articulation, design and construction
  • Designs should be widely circulated and critiqued
stage 2 enactment51
Stage 2: Enactment
  • Design and coded using Lotus Notes
  • Project management, data collection on development problems and costs,
  • Pilot testing
  • Multiple iterations
stage 3 local evaluation
Stage 3 Local Evaluation
  • Interviews, focus groups with call centre staff
  • Student satisfaction surveys
  • Student interviews
  • Analysis of transaction logs and FAQ
  • Cost analysis
  • Interviews with tutors, union
stage 3 local evaluation53
Stage 3 Local Evaluation
  • Multi-methodological evaluation of the intervention
  • Iterative moving from formative to summative evaluation
stage 4 broader impact evaluation
Stage 4: Broader Impact Evaluation
  • Generate and advance a particular set of theoretical constructs that transcends the ..contexts in which they were generated, selected or refined” (Barab & Squire, 2004)
  • Use of thick description and qualitative transference
  • Work to expand and develop theory
  • Tools and conceptual models to understand and adjust both the context and the intervention
  • Value of an intervention lies in its capacity to effect positive change – not in the scientific significance of the results
  • Call for national and international clearinghouse of phase 4 evaluations (Collins et al. 2004)
stage four trials in multiple context
Stage Four - Trials in Multiple Context
  • Currently four help desks operating at Athabasca
  • Continuing evaluation showing NSD between perceptions of value by students between tutor and call centre model
  • Increased use of web services decreasing need for either tutors or call centers ie Am I ready for AU adaptive testing
  • Further research analyzing institutional resistance to change
design based study 2
Design-Based Study #2
  • A work very much in progress
  • Social software solutions for continuous enrollment courses
  • But what type of interaction meets students needs, is cost effective and is least restrictive on freedom of both learners and teachers?
stage 1 informed exploration57
Stage 1: Informed Exploration
  • Review of literature on interaction and self directed study
  • Interviews with course developers, faculty in regard to experience with social interventions
  • Telephone interviews with others around the world involved in continuous enrollment DE programming
  • Survey of students in classes with social interventions
  • Anderson, Annand and Wark 2005 Having your Cake and eating it, 2005. Austariasia Journal of Educational Technology
two solitudes of distance education
Two Solitudes of Distance Education

Collaborative, Distance education

3rd gen. video, audio

and computer conf

Type C



Independent Study

1st gen. correspondence

2nd gen. telecourses

Type I



AU Undergrad

AU Grad

AU Future ??

Anderson, in press

type s distance education
Type S Distance Education

Collaborative, Distance education

3rd gen. video, audio

and computer conf

Type C



Socially Enhanced Independent Learning

Type S

Social Technology

Independent Study

1st gen. correspondence

2nd gen. telecourses

Type I



AU Undergrad

AU Grad

AU Future ??

Anderson, 2006

learning freedom
Learning Freedom
  • Paulsen’s (1993) theory of cooperative freedom:
    • Freedom of space
    • Freedom of pace
    • Freedom of time
    • Freedom of media
    • Freedom of content
    • Freedom of access
    • + Freedom of relationship (Anderson)
But these freedoms are compromised by the interactive, paced and collaborative form of 3rd generation DE that is based on Type C technologies
  • Is the value of the social, paced and collaborative learning of sufficient magnitude to justify these restrictions?
educational social technology est
Educational Social Technology (EST)
  • “Networked tools that support and encourage learning through face-to-face and virtualsocial interactions while retaining individual control over time, space, presence, activity, identity and community.” (Anderson, 2005)
  • Social software are tools that support communication using the five ‘devices’ of identity, presence, relationships, conversations and groups. (Butterfield, 2003)
stage 1 research questions
Stage 1 Research questions
  • What tools, learning activities and incentives are needed to allow students in learner-paced, continuous enrollment courses to provide social, cognitive and teaching presence to each other?
survey results
Survey results

78% of self-paced student respondents indicated that they would interact with other students as long as they were able to proceed through the course at their own pace.

95% of student respondents reported a desire to access the work of other students either currently or previously enrolled in the courses.

Preferred Mode of that interaction:

  • 70% preferred asynchronous media like email and computer conferencing,
  • 27% preferred a combination of synchronous and asynchronous technologies
  • only 3% preferred synchronous interaction alone (for example, audio conferences or face-to-face interaction).
stage 1 summary
Stage 1 summary
  • Interviews and survey of students
  • Review of the interaction literature
  • Investigate social software tools
  • Develop research questions and methodology
stage 2 enactment66
Stage 2: Enactment
  • Select and install an instance of elgg
  • Develop support strategies and documentation
  • Pilot testing (faculty, undergraduate and graduate classes)
  • Develop new instructional designs
  • Successive iterations
stage 2 enactment67
Stage 2: Enactment
  • Social software typically supports:
    • Individual and group Wiki, Blog, Events, Tasks, Polls.
    • User profiles.
    • Friend of friend
    • Interest sharing, group formation and access control
    • RSS
    • Connections on and off line

Technologies of MDE 663 Fall 2006






Asynchronous Int.



Learning Objects




Real Time


Social Presence


Knowledge Polling

stage 3 local evaluation70
Stage 3 Local Evaluation
  • Ethics clearance and resolving privacy issues
  • Interviews, focus groups with developers and faculty
  • Student satisfaction surveys
  • Student interviews
  • Analysis of transaction logs
  • Cost analysis using completion rate data
stage 1 3 iterations
Stage 1-3 Iterations
  • Adding functionality
  • Testing new designs and learning activities
  • Reacting to Organizational Change
    • Think staff development and redefinition of learning roles
    • Think decentralization – students developing unanticipated use of system
stage 4 broader impact evaluation theorizing
Stage 4: Broader Impact Evaluation & Theorizing
  • Publishing of results
  • Trials in different contexts
    • Use in paced and unpaced courses
    • Use in different disciplines
  • Collaboration with international ELGG groups
  • Synthesis of application in different contexts
  • Adoption of Innovation framework
  • Theorizing: Does Educational Social Software allow scaleable, high quality, learning while maximizing learner freedoms?
design based research conclusion
Design-based Research: Conclusion
  • Methodology developed by educators for educators
  • Developed from American pragmatism – Dewey (Anderson, 2005)
  • Recent Theme Issues:
    • The Journal of the Instructional Sciences, (13, 1, 2004),
    • Educational Researcher (32, 1, 2003) and
    • Educational Psychologist (39, 4, 2004)
    • See bibliography at
  • My article at
building the research culture
Building the Research Culture
  • Better dissemination within a combined research/practitioner communities
  • Better tools
  • More funding; less fighting
online journals
Online Journals
  • International Review of Research on Open and Distance Learning
a tale of 3 books
A Tale of 3 books

Open Access

72,000 downloads plus

indiv. chapters

400 hardcopies sold @ $50.00

Free at

Commercial publisher

934 copies sold at $52.00

Buy at Amazon!!

E-Learning for the 21st Century

Commercial Pub.

1200 sold @ $135.00

2,000 copies in Arabic Translation @ $8.

  • DE Research is grossly under-resourced to meet the magnitude of opportunity and demand.
  • Paradigm wars are unproductive.
  • Design-based research offers a promising new research design model.
  • Web 2.0 enhanced interaction (multiformat, synch, asynch, immersive environments) may offer key to more effective knowledge growth and exchange.

“Wisdom is fed equally from practical, theoretical, and ethical knowledge and from a large quantity of reflected experience”

Otto Peters, 2003 p. 137

your comments or questions most welcomed
Your Comments or QuestionsMost Welcomed !

Terry Anderson

research philosophy for professionals ulrich 2006
Research “philosophy for professionals” Ulrich 2006
  • reflective competence is:
    • self-critical: the effort of systematically examining one’s own premises through self-reflection and dialogue, with a view to carefully qualifying the meaning and validity of one’s claims;
    • emancipatory: working actively to help others in emancipating themselves from one’s claims, as well as from theirs; and
    • ethically alert: making transparent to oneself and to others the value implications of one’s claims, and limiting these claims accordingly.