Rigor in academic research

1. Rigor in academic research Dan Remenyi PhD

3. We are in the age of systematic research • Miracles or unexplained revelations are simply not allow in academic research • Archimedes, Galileo and Newton would not get a degree for at least some of their discovers • Rigor is a concept which spans a spectrum or perhaps a continuum. Research needs to be “rigorous enough”.

4. A definition of rigor • A piece of research can be said to be rigorous if there is no doubt that it has been conducted in terms of all the rules associated with the research paradigm under which it was produced. •  The degree to which research methods are scrupulously and meticulously applied • Is there a trade off between Relevance and Rigor? Some researchers suggest that there is an Heisenberg type relationship but this does not have to be the case.

5. Another definition of rigor • Academic research has to end up as an argument that something of value has been added to the body of theoretical knowledge. Rigor is a question of the force of the argument i.e. how persuasive is this argument. The force of the argument is a function of an appropriate research question/s, useful data, and analysis thereof and a logical/rational and clear argument. • Some researchers argue that “rigor is the strength of inference made possible by the given research study”. Staw, B. M. (1985) Reports on the road to relevance and rigor: Some unexplored issues in publishing organizational research. In L. L. Cummings & P. J. Frost, eds. Publishing in the Organizational Sciences (pp 96-107). Homewood Illinois, Richard D. Irwin, Inc.

6. Definitions of Rationality ….1 • A principle aim of this chapter has been to build the foundation upon which a clear understanding of the concept of “rationality” could be erected. Clarity does not necessarily imply simplicity, however. Roughly speaking, rationality is concerned with the selection of preferred behaviour alternatives in terms of some system of values whereby the consequences of behaviour can be evaluated. Does this mean that the process of adaption must be conscious, or are unconscious processes included as well? It has been shown that many of the steps in mathematical invention – than which there can presumably be nothing more rational – are subconscious; and this is certainly true of the simpler processes of equation-solving.

7. Definitions of Rationality ….2 • Moreover, if consciousness is not stipulated as an element of rationality, are only deliberate processes of adaption admitted, or non-deliberate ones as well? The typist trains herself to strike a particular key in response to the stimulus of a particular letter. Once learned, the act is unconscious, but deliberate. On the other hand, any person instinctively withdraws a finger that has been burned. This is “rational” in the sense that it serves a useful purpose, but is certainly neither a conscious nor deliberate adaption.

8. Definitions of Rationality ….3 • Shall we moreover, call a behaviour “rational” when it is in error, but only because the information on which it is based is faulty? When a subjective test is applied, it is rational for an individual to take medicine for a disease if he believes the medicine will cure the disease. When an objective test is applied, the behaviour is rational only if the medicine is in fact efficacious. • Simon, H. A, (1997) Administrative Behaviours: A Study of Decision-Making Processes in Administrative Organizations. New York: The Free Press.

9. Clarity of argument http://www.online-utility.org/english/readability_test_and_improve.jsp

10. Getting Academic Papers Published Fog Index 25.2 (p63)

11. Getting Academic Papers Published

12. Two major approaches to rigor • Rigor may be seen as a function of process • This is the tick the box approach where we are checking that all the necessary tasks have been appropriately addressed • Rigor may be seen as a product of evaluation • To be able to apply rigor as an evaluation you first have to performed a rigor checklist type approach • Then you need to have a standard to which to compare the research under review

13. Rigorous websites • http://www.wjh.harvard.edu/nsfqual/Ryan%20Paper.pdf --- What Are Standards Of Rigor For Qualitative Research? Gery W. Ryan RAND Corporation • http://www.nova.edu/ssss/QR/QR8-4/golafshani.pdf--- Understanding Reliability and Validity in Qualitative Research , NahidGolafshani, University of Toronto, Toronto, Ontario, Canada • http://www.nova.edu/ssss/QR/QR7-4/pare.html---Enhancing the Rigor of Qualitative Research: Application of a Case Methodology to Build Theories of IT Implementation, Guy Paré+ • http://faculty.babson.edu/krollag/org_site/craft_articles/staw_rel_rigor.html --- Staw, B. M. (1985) Reports on the road to relevance and rigor: Some unexplored issues in publishing organizational research. In L. L. Cummings & P. J. Frost, eds. Publishing in the Organizational Sciences (pp 96-107). Homewood Illinois, Richard D. Irwin

14. Some more useful rigorous websites • http://www.youtube.com/watch?v=SW9N34LH7KQ a different use of the word rigour. • http://www.youtube.com/watch?v=RZ4a8pQIwfE a general remarks • http://www.youtube.com/watch?v=Nnp4qZX9H1U in mathematics

15. Some questions about the context of rigour • Has an adequate case been made for the conceptual framework of the research? • Is there a convincing case for the importance and relevance of the research question? • Does the data required as specified by the researcher resonate with the research qustion • Has a well established methodology/method been chosen or has a methodology/method been created by the researcher?

16. Characteristics of a rigor • Careful and precise uses of definitions • Making any assumptions used visible • Avoiding any obfuscation • Step by step logical path • Exposition demonstrated by examples • Heightened sense of being critical • Has constant comparison been a characteristic of the study

17. Rigor as a process----1 • What are the processes of academic research? • Producing a suitable research question • Demonstrating the position of the research question within the extant literature • Establishing the connection to theory or arguing for grounded theory • Identifying an appropriate research methodology and discuss its advantages over other possibilities • Collecting/gathering/developing data/evidence

18. Detectives are Researchers

19. Rigor as a process----2 • Managing the data and the software required (if any) • Analysing the data and producing the results • Postulating our finding and conclusions • Discussing the research through reflection including using the audit train and arguing for its relevance and rigor • Developing management and/or business guidelines

20. Producing a suitable research question • We are not concerned here whether the research question is interesting, topical or useful. • We are concerned to see if the research question is clear, unambiguous and answerable and that it is associated with some theory. • The same thinking will also apply to sub-questions.

21. The purpose of literature in research • A knowledge of the literature is of paramount importance to academic research for 2 key reasons. • The literature positions the research in terms of what is known in the field of study about the topic and way of researching in that field. • The literature is used to call on authority to support the arguments made by the researcher. Be careful of the authority you invoke as some of the material published is wrong.

22. Demonstrating the position of the research question within the extant literature • All academic research is underpinned by the extant literature. • The key rigor issues here are:- • Peer reviewed journals need to dominate the reference list. • A significant number of the papers need to be drawn from highly rated journals (ABS) http://www.the-abs.org.uk/?id=257 • The majority of the papers should be contemporary • Do not rely too heavily on an author or one journal • Other sources such as books and less formal publications may be use sparingly

23. Identifying an appropriate research methodology and discuss its advantages over other possibilities • Research methodology needs to be clearly articulated and rooted in the researcher philosophical view point • The chosen research methodology needs to be justified on its own and • It has to be justified relatively to other methodological options.

24. Collecting/gathering/developing data/evidence • For the purposes of your research data needs to be defined • A philosophical stance on the nature of data collection is important • At least two philosophical understandings of data need to be considered:- • Data of a product of mining – this has nothing to do with data mining. • Data as a product of travelling

25. Managing the data and the software required if any • There are issues related to the storing and management of the data and keeping hard and soft copies • Data will normally be coded and transferred to a computer • Version control is imperative as is backup copies • Software selection is important and the need for skills required in using the software is important

26. Analysing the data and producing the results • Rigor relies on a demonstration of being able to use appropriate data analysis to explore the research question/s • What was the nature of the data used? • How much data was used? • How were data suppliers chosen? • From whom was it collected and how many? • What was the stance of the data suppliers? • Have the assumptions underpinning the analysis been complied with? • Are the results significant and in what way?

27. Postulating our finding and conclusions • The findings of the data analysis need to be clearly articulated • The findings need to be interpreted • The researcher needs to point out what the findings suggest and what they do not suggest • This requires skill at rhetoric

28. Developing management and/or business guidelines • Research findings are not the same as the results of research • One of the problems which sometimes occur in the presenting of business guidelines is the apparent attractiveness of motherhoods. Motherhoods have to be avoided. They are superficial generalities which have no place in academic research.

29. Experimental and Quasi-experimental • The so-called gold standard of rigor in experimental research is to use a random sample and to have a control group. • A quasi-experiment does not fully comply with the rigor required for an experiment. The difference is that in a quasi-experiment the participants/informants will not have been randomly chosen.

30. Rigor as a product of evaluation • How do we do this? • Any evaluation means comparing what is being evaluated to a standard---- Here we are looking for a gap between what we have and an ideal of what we want i.e. standard----We need the “application of precise and exacting standards” • We need to avoid shallow analysis • What issues in academic research need to be evaluated?

31. Each chapter has a list of issues to be examined Each issue needs to be checked for completeness and for depth and where possible the conclusions need to be supported by authority Issues in academic research to be evaluated

32. Part 2

33. Analytical rigor • The need to avoid shallow analysis • How do we tell what is rigorous • Rigor is following the rule -- too simple to be useful • How do you assess the quality of an analysis? Rigor starts with getting the research question right. The research question leads to the hypothoses or propositions.

34. Some issues • The breath and depth of exploration of alternative analysis. • How much critiquing has been done. • 8 different kinds of characteristics of a Meta judgement of the rigor of an analysis • The facts are nothing more than ………reading the news • Reconsidered from diverse points of view • Build a chain of reasoning • How has the reasoning and logic been checked • Quickly generating a hypothesis is an indication of low rigor • Avoid closing the issue too soon. Pre-mature closure • Are you going to have the best hypothesis on the topic. • Find on topic material ---broadening check

35. A multi-attribute measure of sufficiency • Rigor is a product of the culture of the school • A revised definition of rigor, reframing it as an emergent multi-attribute measure of sufficiency rather than as a measure of process deviation.

36. What is meant by shallow analysis? • Shallow analysis is not adequate to deliver a fuller meaning of the work which has been performed. • Shallow analysis is incomplete. • Shallow analysis may miss the point of the whole research or findings.

37. What is meant by shallow analysis?..1 • A poor set of initial papers. The most appropriate papers were not read and considered. • An inadequate critique of the literature? • There is an unsatisfactory research question/s and this hypothesis • The importance of the research was not adequately argued • The methodology was not adequately argued • There are clear rules about this issue • More subjective • The test here is to seek support from supervisor, colleagues and outside influencers • The force of the argument is the main issue here • There are clear rules about this issue

38. Hypothesis -- the world is getting warmer – Global warming?

39. Support • Make a list of all those who support your research besides yourself and your supervisor. • Have you encountered any paradoxes or contradictions in your research so far? • Have you any lurking doubts about the way your research has gone so far? • Lurking doubts are a central issue in academic research. If you have no lurking doubts you do not understand the problems/challenges you are facing Remember the aphorism “if you are not confused and fearful you do not understand the issues you are facing”.

40. What is meant by shallow analysis?...2 • An unsatisfactory sample was used – no cherry picking • Are the analytical procedures been conducted properly conducted? Any biases? • Have the results of the analysis being understood correctly? What alternative candidate explanations have been considered? • Is there a convincing discussion? Has there been any supporting information. • Have the results been convincingly converted to management guidelines or policy options? • There are clear rules about this issue • There are clear rules about this issue • This is a more creative issue • This is a function of rhetoric and support from supervisor, colleagues etc • This is a creative issue and there is no cookbook answer

41. What is meant by shallow analysis?....3 • Has an adequate suggestion been made concerning future research? • Has adequate attention been given to the principle of parsimony? • Is the written language at an acceptable standard? • Have all the universities requirements been complied with regard presentation? • This is a more creative issue • There are clear rules about this issue • There are clear rules about this issue • There are clear rules about this issue

42. Exploring challenges • The researcher needs to have thought about a list of potential criticisms which examiners/reviews may use

43. Introduction • Is the topic of the research clear ? • Is the objective of the research clear ? • Are all the terms adequately defined? • Has a possible theoretical context been highlighted? • Has the importance of the research been argued? • Has the course of the research been described?

44. Literature • Have all the important authors in the field been included? • Is the literature being explored on a concept or timeline basis or a combination of both and why has the approach been chosen? • Is there an adequately critical approach to the literature? • What gaps are there and how does the research question relate to the gaps?

45. The methodology was not adequately discussed and argued • What type of investigations is being proposed and why? • What is the unit of analysis? • What time horizon is being suggested? • What data/evidence is required? • What type of instrument is needed? • How will the data be analysed? • How will bias be minimised? • Alternatives?

46. The research • What did the research protocol look like? • How were the ethical issued addressed? • What was actually done? • How were informants chosen? • How may? • Where? • How was the data/evidence collected? • What data management issued arose?

47. Findings and conclusions • Was the analysis conducted with appropriate knowledge and care? • What do the findings mean? • What are the limitations of the analysis? • How were the conclusions drawn from the research?

48. Audit trail and reflections • How did the research process develop? • What were the major miles stones in the thinking? • How has the researcher developed over the period of the research? • Looking over the whole experience how does the research feel about what has been achieved?

49. Management guidelines and future research • So what? • What’s new? • Who cares? • How can the research finding be put to use? • What are the other interesting issues encountered during the research process?

50. Zen and the Archer • Many years’ ago in Japan, there was a warrior – one of those itinerant Samurai known as ronin. He had an ambition to find fame and fortune as the finest archer in the land. In pursuit of his dream, he travelled the length and breadth of the country looking for a master-bowman to help him improve his technique. Few of those he encountered had much to teach him but he continued searching and eventually, just as he was about to give up, he chanced upon some dilapidated farm buildings far from other habitation. All over the buildings, in the most unlikely and inaccessible places, were hundreds of hand-painted targets, each with an arrow at the exact centre of the bulls-eye.