slide1 n.
Download
Skip this Video
Loading SlideShow in 5 Seconds..
Dr Chen Wenli Learning Sciences and Technologies AG Learning Sciences Lab PowerPoint Presentation
Download Presentation
Dr Chen Wenli Learning Sciences and Technologies AG Learning Sciences Lab

Loading in 2 Seconds...

play fullscreen
1 / 44

Dr Chen Wenli Learning Sciences and Technologies AG Learning Sciences Lab - PowerPoint PPT Presentation


  • 123 Views
  • Uploaded on

Quantitative Research Methods (II) . Dr Chen Wenli Learning Sciences and Technologies AG Learning Sciences Lab National Institute of Education. Outline. Logic of quantitative research Constructing hypothesis Types of quantitative research methods Survey research Experimental research

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'Dr Chen Wenli Learning Sciences and Technologies AG Learning Sciences Lab' - tara


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
slide1

Quantitative Research Methods

(II)

Dr Chen Wenli

Learning Sciences and Technologies AG

Learning Sciences Lab

National Institute of Education

outline
Outline
  • Logic of quantitative research
  • Constructing hypothesis
  • Types of quantitative research methods
    • Survey research
    • Experimental research
    • Single-subject research
    • Casual-comparative research
    • Quantitative content analysis
  • Validity and reliability in quantitative research
experimental research
Experimental Research
  • Characteristics of experimental research
  • Experimental research design
    • Experimental design
    • Quasi-experimental design
    • Factorial design
  • Validity of experimental research
  • Control of extraneous variables
experimental research1
Experimental Research
  • Researcher applies some treatments to subjects for an appropriate length of time and then observes the effect of the treatments on the subjects by measuring response variables
    • IV (experimental

or treatment variable)

      • a condition or set of conditions

applied to subjects

    • DV (response, criterion

or outcome Variable)

      • results or outcome on

the subjects

examples
Examples
  • Quality of learning with an active versus passive motivational set (Benware & Deci, 1984)
  • Comparison of computer-assisted cooperative, competitive, and individualistic learning (Johnson, Johnson, & Stanne, 1986)
  • The effect of a computer simulation activity versus a hands-on activity on product creativity in technology education (Kurt, 2001)
  • The effect of language course taught with online supplement material (Shimazu, 2005)
characteristics
Characteristics
  • The only type of research that directly attempts to influence a particular variable
  • The only type that, when used properly, can really test hypotheses about cause-and-effect relationships.
      • Enable researchers to go beyond description and the identification of relationships, to at least a partial determination of what causes them
  • 3 characteristics of experimental research
manipulation of iv
Manipulation of IV
  • Researcher manipulate the IV
    • Decide the nature of treatment/intervention (what is going to happen to the subjects of the study)
    • To whom it is to be applied
    • To what extent
    • When, where and how
comparison of g roups
Comparison of Groups
  • At least 2 conditions are compared to assess the effect(s) of particular conditions or “treatments” (IV)
    • Experimental group (receive a treatment of some sort)
    • Control group (no treatment) or comparison group (receive different treatment)
  • IV may be established in several ways:
    • Presence VS absence of a particular form
    • One form of variable VS another
    • Varying degrees of the same form
randomization
Randomization
  • Random assignment of subjects to groups
    • an important ingredient in the best kinds of experiments
    • every individual who is participating in the experiment has an equal chance of being assigned to any of the experimental or control conditions being compared
      • It takes place before the experiment begins
      • Allows the researcher to form groups that are equivalent
        • Eliminate the threat of extraneous, or additional variables that might affect the outcome of the study
commonly used n otation
Commonly Used Notation
  • X1 = treatment group
  • X2 = control/comparison group
  • O = observation (pretest, posttest, etc.)
  • R = random assignment
weak experimental designs
Weak Experimental Designs
  • One-shot case study design
    • a single group is exposed to a treatment or event, and its effects assessed.
  • One-group pretest-posttest design
    • a single group is measured or observed both before and after exposure to a treatment.

X O

Technology Attitude scale to

measure interest

O X O

Pretest Treatment Post test

true experimental designs
True Experimental Designs
  • Randomized posttest-only control group design
    • involves two groups formed by random assignment and receiving different treatments
  • Randomized pretest-posttest control group design
    • differs from the randomized posttest-only control group only in the use of a pretest

Treatment group R X1 O

Control group R X2 O

Treatment group R O X1 O

Control group R O X2O

true experimental designs1
True Experimental Designs
  • Randomized Solomon four-group design
    • involves random assignment of subjects to four groups, with two being pretested and two not.

Treatment group R O X1 O

Control group R O X2 O

Treatment group R X1 O

Control group R X2 O

  • Better control the threat to internal validity
  • Drawback—requires twice as many participants
quasi experimental designs
Quasi-Experimental Designs
  • Used in place of experimental research when random assignment to groups is not feasible
    • Posttest-only design with

nonequivalent groups

      • Pretest-posttest design with

nonequivalent groups:

Treatment group X1 O

Control group X2 O

Treatment group O X1O

Control group O X2O

quasi experimental designs1
Quasi-Experimental Designs
  • Counterbalanced design: all groups are exposed to all treatments, but in a different order
    • the order in which the groups receive the treatments should be determined randomly
    • the number of groups and treatments must be equal
    • Comparing the average scores fro all groups on the posttest for each treatment

Group I X1O X2 O X3 O

Group II X3O X1 O X2 O

Group III X2O X3 O X1 O

quasi experimental designs2
Quasi-Experimental Designs
  • Time-series design: involves repeated measurements or observations over time (until scores are stable ), both before and after treatment.

O OOO X O OOO

  • Uses a single group of participants
  • Examines possible changes
  • over time

Study B

Study A

X

factorial designs
Factorial Designs
  • Factorial designs extend the number of relationships that may be examined in an experimental study.

Treatment R O X1g1 O

Control R O X2g1 O

Treatment RO X1g2 O

Control RO X2g2 O

  • Incorporates two or more factors
    • The additional factor could be treatment variable or subject characteristics
  • Enables researcher to detect differential differences (effects apparent only on certain combinations of levels of IVs)
a 2 x 2 factorial design
A 2 X 2 factorial design…

Boy

Girl

Group 2

Group 1

Traditional

Game-based learning

Group 3

Group 4

a 2 x 2 factorial design1
A 2 X 2 factorial design

No interaction

between factors

Interacting factors

Game -based

Attitudes toward learning

Game -based

Attitudes toward learning

Traditional

Traditional

Girl

Boy

Girl

Boy

validity
Validity
  • Validity: the experiment tests the variable(s) that it purports to test
  • If threats are not controlled for, they may introduce error into the study, which will lead to misleading conclusions
  • Threats to validity…
    • Internal: factors other than the IV that affect the DV
    • External: factors that affect the generalizability of the study to groups and settings beyond those of the experiment
threats to i nternal validity
Threats to Internal Validity
  • History
    • Uncontrolled event that occur during the study that may have an influence on the observed effect other than the IV
  • Maturation
    • Factors that influence a participant's performance because of time passing rather than specific incidents (e.g., the physical, intellectual, and emotional changes that occur naturally)
  • Test practice
    • The effects of participants taking a test that influence how they score on a subsequent test
  • Instrumentation
    • Influences on scores due to calibration changes in any instrument that is used to measure participant performance
  • Statistical regression
    • Problem that occurs when participants have been assigned to particular group on the basis of atypical or incorrect scores.
threats to internal validity
Threats to Internal Validity
  • Bias in group composition
    • Systematic differences between the composition of groups in addition to the treatment under study.
  • Experimental mortality
    • A differential loss of participants
  • Hawthorne effect
    • Change in the sensitivity or performance by the participants that may occur merely as a function of being a part of the study
  • Novelty effect
    • Participant interest, motivation, or engagement increases simply because they are doing something different
  • Placebo effect
    • The participants receive no treatment but believe they are
threats to external validity
Threats to External Validity
  • Population-sample differences
    • The degree to which the participants in a study are representative of the population to which generalization is desired
  • Artificial research arrangements
    • The degree that a research setting deviates from the participant's usual routine
  • Multiple-treatment interference
    • More than one treatment is administered to the same participants and results in cumulative effects that may not be similar to the outside world and may threaten generalization of the results
  • Treatment diffusion
    • The situation when different treatment groups communicate with and learn from each other
control of extraneous variables
Control of Extraneous Variables
  • Confounding:
    • the fact that the effects of the IV may intertwine with extraneous variables, such that it is difficult to determine the unique effects of each variable
  • Common ways to control for extraneous variables
    • Randomization
    • Holding certain variables constant
    • Matching
    • Comparing homogeneous groups or subgroups
    • Analysis of covariance (ANCOVA)
single subject research
Single-Subject Research
  • Most commonly used to study the changes in behavior an individual exhibits after exposure to a treatment or intervention of some sort.
  • Can be applied in settings where group designs are difficult to put into play.
  • Involves extensive collection of data on one subject at a time.
  • Primarily use line graphs to present their data and to illustrate the effects of a particular intervention or treatment.
  • Adaptations of the basic time-series design
s ingle subject research
Single-Subject Research
  • A-B design
    • …baseline measurements (O) are repeatedly made until stability is established, then the treatment (X) is introduced and an appropriate number of measurements (O) are made during treatment implementation

O OO X O X O X O

baseline treatment phase phase

A | B

single subject research1
Single-Subject Research
  • Reversal (A-B-A) design
    • …baseline measurements (O) are repeatedly made until stability is established, then the treatment (X) is introduced and an appropriate number of measurements (O) are made during treatment implementation, followed by an appropriate number of baseline measurements (O) to determine stability of treatment (X)

O OO X O X O X O OO

baseline treatment baseline phase phasephase

A | B | A

other single subject research designs
Other Single-Subject Research Designs
  • A-B-A-B design
    • 2wo baseline periods are combined with two treatment periods
  • B-A-B design
    • Used when an individual's behavior is so severe or disturbing that a researcher cannot wait for a baseline to be established
  • A-B-C-B design:
    • "C" condition refers to a variation of the intervention in the "B" condition. The intervention is changed during the "C" phase typically to control for any extra attention the subject may have received during the "B" phase.
threats to validity in single subject research
Threats to Validity in Single Subject Research
  • Internal Validity
    • length of the baseline and intervention conditions
    • the number of variables changed when moving from one condition to another
    • the degree and speed of any change that occurs
    • whether or not the behavior returns to baseline levels
    • the independence of behaviors
    • the number of baselines
  • External Validity
    • weak when it comes to generalizability
    • It is important to replicate single-subject studies to determine whether they are worthy of generalization.
controlling threats in single subject studies
Controlling Threats in Single-Subject Studies
  • Single-subject designs are most effective in controlling for subject characteristics, mortality testing, and history threats.
  • They are less effective with location, data collector characteristics, maturation, and regression threats.
  • They are especially weak when it comes to instrument decay, data-collector bias, attitude, and implementation threats.
causal comparative research
Causal-Comparative Research
  • Explores the possibility of cause-and-effect relationships when experimental and quasi-experimental approaches are not feasible
  • Differs from experimental and quasi-experimental research
    • IV is not manipulated (not ethical or not possible)
    • Focuses first on the effect, then tries to determine possible
  • Relationships can be identified in causal-comparative study, but causation cannot be fully established.
steps in causal comparative research
Steps in Causal-Comparative Research
  • Formulating a problem
    • Identify and define the particular phenomena of interest, and then to consider possible causes for, or consequences of, these phenomena.
  • Selecting a sample
    • Define carefully the characteristic to be studied and then to select groups that differ in this characteristic.
  • Instrumentation
    • No limits to the kinds of instruments that can be used
  • Design
    • Select two groups that differ on a particular variable of interest and then comparing them on another variable or variables.
threats to internal validity in causal comparative research
Threats to Internal Validity in Causal-Comparative Research
  • Weaknesses :
    • lack of randomization
    • Inability to manipulate an IV
  • A major threat: the possibility of a subject selection bias.
    • The procedures used to reduce this threat
      • matching subjects on a related variable
      • creating homogeneous subgroups
      • the technique of statistical matching.
  • Other threats to internal validity
    • Location
    • Instrumentation
    • Loss of subjects.
data analysis in causal comparative studies
Data Analysis in Causal-Comparative Studies
  • The first step: construct frequency polygons.
  • Means and SDare usually calculated if the variables involved are quantitative.
  • The most commonly used test is a t-test for differences between means.
  • ANCOVA is particularly useful in causal-comparative studies.
  • The results of causal-comparative studies should always be interpreted with caution, because they do not prove cause and effect.
common quantitative measure in learning and education
Common quantitative measure in learning and education
  • Learning gain
    • Post-pre
    • (post-pre)/(1-pre) (Hake’s gain)
    • Adjusted post score (through ANCOVA)
  • Learning efficacy
    • Does it help reduce time spent for problem solving?
  • User’s attitude
  • Teachbacks
    • How well learner can teach back?
quantitative content analysis
Quantitative Content Analysis
  • Content analysis is a quantitative research instrument for a systematical and intersubjective description of content
    • A form of textual analysis *usually*
    • Categorizes chunks of text according to Code
  • Based on the principles of social science of “measuring and counting”
  • Reduces the complexity of content as it brings out the central patterns of the coverage
  • One objective is to examine large amounts of content with statistic methods
rough history
Rough History
  • Classical Content Analysis
    • Used as early as the 30’s in military intelligence
    • Analyzed items such as communist propaganda, and military speeches for themes
    • Created matrices searching for the number of occurrences of particular words/phrases
  • (New) Content Analysis
    • Moved into Social Science Research
    • Study trends in Media, Politics, and provides method for analyzing open ended questions
    • Can include visual documents as well as texts
    • More of a focus on phrasal/categorical entities than simple word counting
the sample
The Sample
  • The sample
    • Which types of content?
    • Which period?
    • Which characteristics?
  • Elements of the research instrument
    • Sampling units
    • Units of analysis: unit of the content on which our “measurements” are based. The categories describe the properties of the media content which is relevant to our research question
validity in quantitative research
Validity in Quantitative Research
  • Definition: the extent to which any measuring instrument measures what it is intended to measure
  • Types of validity
    • Construct Validity: examines the fit between the conceptual definitions & operational definitions of the variables
    • Content Validity : verifies that the method of measurement actually measures the expected outcomes.
    • Predictive Validity : determines the effectiveness of the instrument as a predictor of a future event
    • Statistical Conclusion Validity: concerned with whether the conclusions about relationships and/or differences drawn from statistical analysis are an accurate reflection of the real world
reliability in quantitative research
Reliability in Quantitative Research
  • Definition: refers to the accuracy and consistency of information obtained in a study; important in interpreting the results of statistical analyses; and refers to the probability that the same results would be obtained with different samples (generalizability)
  • 3 common methods to check reliability
    • test-retest method
      • administering the same instrument twice to the same group of individuals after a certain time interval has elapsed.
    • equivalent-forms method
      • administering two different, but equivalent, forms of an instrument to the same group of individuals at the same time.
    • internal-consistency method
      • comparing responses to different sets of items that are part of an instrument.
summary
Summary
  • Logic of quantitative research
  • Constructing hypothesis
  • Types of quantitative research methods
    • Survey research
    • Experimental research
    • Single-subject research
    • Casual-comparative research
    • Others
  • Validity and reliability in quantitative research