Chapter 12. Critical Appraisal of Quantitative and Qualitative Research for Nursing Practice. Intellectual Research Critique. A careful examination of all aspects of a study to judge: Merits Limitations Meaning Significance. Intellectual Critique Questions.
Critical Appraisal of Quantitative and Qualitative Research for Nursing Practice
A careful examination of all aspects of a study to judge:
Was the research problem significant?
What are the major strengths of the study?
What are the major weaknesses of the study?
Did the researchers use sound methodology?
Do the findings accurately reflect reality?
Are the findings consistent with those from previous studies?
Can the study be replicated by others?
What are the implications of the findings?
Read and critique the entire study.
Examine the organization and presentation of the research report.
Examine the significance of the probe, studied for nursing practice.
Identify strengths and weaknesses of a study.
Be objective and realistic in identifying the study’s strengths and weaknesses.
Provide specific examples of the strengths and weaknesses.
Provide a rationale for your critique.
Suggest modifications for future studies.
Discuss feasibility of replication of the study.
Discuss usefulness of findings for practice.
Read the article carefully.
Identify terms you do not understand and determine their meaning in a dictionary or the glossary of Burns and Grove.
Read the article a second time.
Highlight each step of the research process.
Requires knowledge of what each step of the research process should be like
The ideal is compared with the real.
Must examine the extent to which the researcher followed the rules for an ideal study
Involves a critique of logical links connecting one study element with another
Overall logical development of the study must be examined.
Review research text(s).
Compare the steps in the study you are analyzing with the criteria in the research text(s).
Analyze the logical links among the steps of the study.
Is the problem sufficiently delimited in scope without being trivial?
Is the problem significant to nursing?
Is there evidence of researcher biases?
Does the purpose narrow and clarify the aim of the study?
Was the study feasible in terms of funding, expertise, subjects, facility, equipment, and ethical considerations?
Does it demonstrate progressive development of ideas through previous research?
Is a theoretical knowledge base developed for the problem and purpose?
Does the literature review provide rationale and direction for the study?
Is a clear, concise summary presented of the current empirical and theoretical knowledge in the area of study?
Is the framework presented with clarity?
If a map or model is presented, is it adequate to explain the phenomenon of concern?
Is the framework linked to the research purpose?
Would another framework fit more logically with the study?
Is the framework related to nursing knowledge?
If a proposition from a theory is tested, is the proposition clearly identified and linked to the study hypotheses?
Are the objectives, questions, or hypotheses expressed clearly?
Are the objectives, questions, or hypotheses logically linked to the research purpose and framework?
Are the research objectives, questions, or hypotheses linked to concepts and relationships from the framework?
Do the variables reflect the concepts identified in the framework?
Are the variables clearly defined?
Is the conceptual definition of a variable consistent with the operational definition?
Is the design used the most appropriate to obtain the needed data?
Does the design provide a means to examine all of the objectives, questions, or hypotheses?
Have threats to design validity been minimized?
Is the design logically linked to the sampling method and statistical analyses?
Is the treatment clearly described?
Was a protocol developed to promote consistent implementation of the treatment?
Did the researcher monitor the implementation of the treatment to ensure consistency?
Is the sampling method adequate to produce a representative sample?
What are the potential biases in the sampling method?
Are any subjects excluded from the study based on age, socioeconomic status, or race, without a sound rationale?
Were the sampling criteria appropriate for the type of study conducted?
Is the sample size sufficient to avoid a type II error?
If more than one group is used, do the groups appear equivalent?
Are the rights of human subjects protected?
Is the setting used in the study typical of clinical settings?
Was sample mortality a problem? If so, how might this influence the findings?
Do the instruments adequately measure the study variables?
Are the instruments sufficiently sensitive to detect small differences?
Does the instrument have adequate validity and reliability?
Are the instruments clearly described?
Are techniques to complete and score the instruments provided?
Are validity and reliability of the instruments described?
If the instrument was developed for the study, is the instrument development process described?
Are the phenomena to be observed clearly identified and defined?
Is interrater and intrarater reliability described?
Are the techniques for recording observations described?
Do the interview questions address concerns expressed in the research problem?
Are the interview questions relevant for the research purpose and objectives, questions, or hypotheses?
Does the design of the questions tend to bias subjects’ responses?
Does the sequence of questions tend to bias subjects’ responses?
Are the physiological measures or instruments clearly described? If appropriate, are brand names of instruments identified?
Are the accuracy, selectivity, precision, sensitivity, and error of the instruments discussed?
Are the physiological measures appropriate for the research purpose and objectives, questions, or hypotheses?
Are the methods for recording data from the physiological measures clearly described?
Is the recording of data consistent?
Is the data collection process clearly described?
Is the training of data collectors clearly described and adequate?
Is the data collection process conducted in a consistent manner?
Are the data collection methods ethical?
Do the collected data address the research objectives, questions, or hypotheses?
Are data analysis procedures appropriate to the type of data collected?
Are data analysis procedures clearly described?
Are the results presented in an understandable way?
Do data analyses address each research objective, question, or hypothesis?
Are the analyses interpreted appropriately?
Are the statistical analyses logically linked to the design?
Is the sample size sufficient to detect significant differences?
Was power analysis used to determine sample size?
Are findings discussed in relation to each objective, question, or hypothesis?
Are the findings clinically significant?
Do the conclusions fit the findings from the analyses?
Are conclusions based on statistically significant and clinically significant results?
Are there limitations the researcher did not identify?
Involves determining the meaning and significance of the study by examining the links among the study process, study findings, and previous studies
Study findings are examined in light of previous study findings.
Evaluation builds on conclusions reached during the first three stages of the critique and provides the basis for the fifth step—conceptual clustering.
The steps of the study are evaluated based on previous studies.
Present hypotheses are based on previous hypotheses.
Present design is based on previous designs.
Present methods of measurement are based on previous measurement.
What rival hypotheses can be suggested for the findings?
How much confidence can be placed in the study findings?
To what populations can the findings be generalized?
What questions emerge from the findings, and are these identified by the researcher?
What future research can be envisioned?
Could the limitations of the study have been corrected?
When the findings are examined based on previous studies, what is now known and not known about the phenomenon under study?
Are the findings of previous studies used to generate the research problem and purpose?
Do the findings build on findings of previous studies?
Is the design an advance over previous designs?
Do sampling strategies show an improvement over previous studies?
Does the sample selection have the potential for adding diversity to samples previously studied?
Does the current research build on previous measurement strategies so that measurement is more precise or more reflective of the variables?
How do statistical analysis techniques compare with those used in previous studies?
Is the current knowledge in this area identified?
Does the author indicate the implication of the findings for practice?
Conceptualization, theoretical modeling, and theory analysis
Transforming ideas across levels of abstraction
Definition: the capacity to switch from one context or worldview to another, to shift perception so as to see things from a different perspective
It is not necessary to become committed to a perspective to follow or apply its logical structure.
All scholarly work requires a willingness and ability to examine and evaluate works from diverse perspectives.
For example, analysis of the internal structure of a theory requires context flexibility.
Necessary so as to follow the logic of a qualitative researcher
Used in the transformation process during data analysis
Revealed in the move from concrete descriptions to the abstract level of science
Standard 1: Descriptive vividness
Standard 2: Methodological congruence
Standard 3: Analytical and interpretative
Standard 4: Philosophical or theoretical
Standard 5: Heuristic relevance
Description of the site and subjects, the experience of collecting the data, and the thinking of the researcher during the process need to be presented so clearly that the reader has the sense of personally experiencing the event.
Because one of the assumptions of qualitative research is that all data are context specific, the evaluator of a study must understand the context of that study.
A contextual understanding of the whole is essential and prerequisite to the capability of the reviewer to evaluate the study in the light of the other four standards.
Failure to include essential descriptive information
Lack of clarity and/or depth of description
Inadequate skills in writing descriptive narrative
Reluctance to reveal self in written material
Reviewer must have knowledge of philosophy and methodological approach used by the researcher.
Rigor in documentation
The reviewer examines if the researcher clearly and concisely presents study elements.
The reviewer examines the study elements for completeness and clarity.
The reviewer identifies any threats to rigor in documentation.
Significance of phenomenon
Identification of assumptions
Identification of philosophy
Role of researcher
Sampling and subjects
Data analysis strategies
Suggestions for further study
Suggestions for practice
Failure to present all elements or steps of the study
Failure to present all elements or steps of the study accurately or clearly
Definition: the rigor of the researcher in applying selected procedures for the study
The researcher needs to make clear the steps taken to ensure data were accurately recorded and that the data obtained are representative of data as a whole.
When critiquing a qualitative study, the reviewer examines the description of the data collection process and study findings for threats to procedural rigor.
The researcher asked the wrong questions. The questions need to tap the subjects’ experiences, not their theoretical knowledge of the phenomenon.
The questions included terminology from the theoretical orientation of the researcher.
The informant might have misinformed the researcher.
Other individuals present might have inhibited the informant.
The informant might have wanted to impress the researcher by giving the most desirable response.
The informant did not observe the details requested or was not able to recall the event and substituted instead what he or she supposed happened.
The researcher placed more weight on data obtained from well-informed, articulate, high-status individuals than on data from those who were less articulate, obstinate, or of lower status.
The presence of the researcher distorted the event being observed.
The researcher’s involvement with the subject-participants distorted the data.
Atypical events were interpreted as typical
The informants lacked credibility.
An insufficient amount of data were gathered.
An insufficient length of time was spent in data gathering.
Approaches for gaining access to the site and/or subjects were inappropriate.
The researcher failed to keep in-depth field notes.
Requires recognition and discussion by researcher of ethical implications related to study
Consent is obtained from subjects and documented.
Report must indicate that the researcher ensured rights of subjects were protected.
Reviewer examines data-gathering process and identifies potential threats to ethical rigor.
Researcher failed to inform subjects of their rights.
Researcher failed to obtain consent from subjects.
Researcher failed to ensure protection of subjects’ rights.
Definition: the rigorous development of a decision trail
To achieve this, the researcher must report all decisions involved in transformation of data to theoretical schema.
A second researcher, using original data and decision trail, should be able to arrive at conclusions similar to those of original researcher.
Reviewer examines decision trail for threats to auditability.
Description of data collection process was inadequate.
Researcher failed to develop or identify decision rules for arriving at ratings or judgments.
Researcher failed to record nature of decisions, data on which they were based, and reasoning that entered into decisions.
Evidence for conclusions was not presented.
Other researchers were unable to arrive at similar conclusions after applying decision rules to the data.
Requires researcher to make intense efforts to identify and record decision-making processes through which data transformations are made
Premature patterning may occur before the researcher can logically fit all of the data within the emerging schema.
The consequence may be a poor fit between data and theoretical schema.
It is critical to test the schema by rechecking the fit between the schema and the original data.
When critiquing study, the reviewer examines decision-making processes and theoretical schema to detect threats to analytical preciseness.
Interpretive statements do not correspond with findings.
Categories, themes, or common elements are not logical.
Samples are not representative of the class of joint acts referred to by the researcher.
Set of categories, themes, or common elements fails to set forth a whole picture.
Set of categories, themes, or common elements is not inclusive of data that exist.
Data are inappropriately assigned to categories, themes, or common elements.
Inclusion and exclusion criteria for categories, themes, or common elements are not consistently followed.
Working hypotheses or propositions are not identified or cannot be verified by data.
Various sources of evidence fail to provide convergence.
There is incongruence of evidence.
Subject-participants fail to validate findings.
Conclusions are not data based or do not encompass all the data.
Data are made to appear more patterned, regular, or congruent than they are.
Requires that theoretical schema developed from the study be:
Reflective of the data
Compatible with knowledge base of nursing
Findings are trivialized.
Concepts are inadequately refined.
Concepts are not validated by data.
Set of concepts lacks commonality.
Relationships between concepts are not clearly expressed.
Proposed relationships among concepts are not validated by data.
Working propositions are not validated by data.
Data are distorted during development of the theoretical schema.
Theoretical schema fails to yield a meaningful picture of the phenomenon studied.
A conceptual framework or map is not derived from the data.
Reflects reader’s capacity to recognize phenomenon described in the study, its theoretical significance, applicability to nursing practice situations, and influence on future research activities
Intuitive recognition: when individuals are confronted with theoretical schema derived from the data, it has meaning within their personal knowledge base
Reader is unable to recognize phenomenon.
Description is not consistent with common meanings.
Theoretical connectedness is lacking.
There should be intersubjectivity with existing theoretical knowledge in nursing and previous research.
When critiquing, reviewer examines strength of the link of study findings to existing knowledge.
Researcher failed to examine existing body of knowledge.
Process studied was not related to nursing and health.
There is a lack of correspondence with existing knowledge base in nursing.
Applicability: nurses need to be able to integrate the findings into their knowledge base and apply them to nursing practice situations
Findings also need to contribute to theory development.
Findings are not significant for discipline of nursing.
There is a failure to achieve methodological congruence.
There is a failure to achieve analytical precision.
There is a failure to achieve theoretical connectedness.