Is Information Systems Research Relevant?

1. Is Information Systems Research Relevant? Detmar Straub Georgia State University Editor-in-Chief, MIS Quarterly Graphics available at: detmar straub.com IS Colloquium University of Georgia November 2010

2. Agenda The Underlying, Real Issue Framing the Conceptual Investigation Framing the Empirical Investigation for RQ#1 Improving Topic Usefulness Framing the Conceptual Investigation for RQ#2 What Kind of Evidence Do We Need (Not Need) for RQ #2 Prior Literature Addressing the Real Issue….I.e., none Prior Literature Addressing the Fictitious Issue…some, but thin & overwhelmingly conceptual Need for More Research on Relevance What Would Good Research Here Look Like?

3. Main Points • IS academics are both: Educators & Scholars. • When most people address the question of relevancy, they only see us in our role as scholars. • This creates untold confusion and harm.

4. Main Points • Many people believe there is a natural tension between academic scholarship and practice. • Why should this be? • This presentation argues that if we understand the highly defensible goals of each, the tension could fade into the background or even disappear. • “Rigor versus relevance” is the wrong framing; practitioners generally cannot understand nor can they directly use academics’ “methodological rigor.” • They may learn IT research findings through other avenues, but NOT thru scholarly journals (RQ #2). • Sadly, there has been NO literature germain to the underlying, real issue (RQ #2) of whether IS research is relevant.

5. 1. The Underlying, Real Issue • Business, management, and information schools are applied schools, meaning that they have a mission to contribute knowledge to the work practices of certain disciplines. • Medical schools, also applied schools, have a mission to improve the practices of doctors, dentists, and nurses, for example.

6. 1. The Underlying, Real Issue We are applied scientists, then, not engaged in “basic” research and teaching. E.g., Do kanagaroos experience “love”?

7. 1. The Underlying, Real Issue • What does “relevance” mean, then, in this context? • Is our teaching relevant? • Presumably this would mean that we are teaching our students skills and critical thinking that will allow them to be better practitioners. • Is our research relevant? • Presumably this would mean that: (1) our research findings are applicable to practice and (2) that these findings are being communicated to practitioners in one or another appropriate format (Straub & Ang, 2008).

8. 2. Framing the Conceptual Investigation • If we are in agreement with these essential principles, then there are two underlying questions that frame the issue of whether IS research is relevant: RQ #1. Are the findings useful for practitioners? RQ #2. Are they being communicated to practice? • Please note that (1) academic, (2) academic-practitioner, and (3) practitioner journals have not entered the discussion thus far, but this is where research shows its true colors. • Why do we separate journals into these three types?

9. 3. Framing the Empirical Investigation for RQ#1 • Why do we have different types of journals? • The types refer to targeted readership. • Academic journals (1) target scholars/professors. • Academic-practitioner journals (2) have a hybrid readership of academic-savvy practitioners and practitioner-savvy academics. • Practitioner journals (3) target the practical issues confronting practitioners…in a sense, this is where people discuss their best practices. • Evidence is proffered in the December 2009 MISQ article by Baskerville & Myers

10. 3. Framing the Empirical Investigation for RQ#1 Academic Journals American Economic Review Decision Science European Journal of Information Systems European Management Journal IEEE Transactions on Engineering Mgmt Industrial Marketing Management Information & Management Information Society Information Systems Research Journal of AIS Journal of Business Logistics Journal of Economic Behavioral Organization Journal of General Management Journal of Global Information Management Journal of Management Journal of Management Information Systems Journal of Strategic Information Systems Long Range Planning Management Decision MIS Quarterly Management Science Managerial Auditing Journal Organization Science

11. 3. Framing the Empirical Investigation for RQ#1 Academic-Practitioner Journals Academy of Management Executive Business Quarterly California Management Review College & Research Libraries Communications of the ACM Harvard Business Review Human Resource Planning Journal of Euromarketing Journal of Systems Management MISQ Executive Practitioner Journals ABA Banking Journal Bank Management Best's Review ( Life/Health) CFO: The Magazine for Senior Financial Executives CIO Government Executive Industry Week Internal Auditor InternetWeek Management Accounting - London McKinsey Quarterly Purchasing & Supply Management Supply Management TMA Journal

12. Office automation* * Source: Baskerville, R.L., and Myers, M.D. 2009. "Fashion Waves in Information Systems Research and Practice," MIS Quarterly (33:4, December), 647-662. Used with permission.

13. CASE (Computer-aided software engineering)

14. Business process reengineering

15. Electronic commerce

16. 3. Framing the Empirical Investigation for RQ#1 • Analyzing 900 academic and academic-practitioner articles in the IS literature, Alavi and Carlson (1992) concluded that IS scholarship was relevant because over a third of the 900 articles they catalogued were "devoted to describing and illustrating how…MIS concepts and models could be applied" (p. 56). • Evidence is also proffered in the December 2008 MISQ editorial by Straub and Ang (2008) • Overlaps in triangle are considerable.

17. LEGEND Knowledge sources acad = academic profile point ap = academic-practitioner profile point practice = practitioner profile point Antecedents Triggers a1 = strategic realignment a2 = poor profitability a3 = poor performance a4 = resource deficiency a5 = regulatory influence a6 = supplier presence Inhibitors ag1= increase in org. risks ag2= strong org resistance ag3= poor funct. performance ag4= loss of funct. resources ag5=high transaction costs ag6=lack of suitable suppliers FIGURE 1 Correspondence Analysis Mappings LEGEND (continued) Process p1= planning for outsource p2 = manage vendor obligations p3=manage client obligations p4 = manage employee obligations Consequences Positive cp1 = increased cost advantage cp2 = increased profitability cp3 =enhanced performance cp4 = increased resources Negative cn1 = increased org. risks cn2=decreased funct. performance cn3 = reduced resources cn4=increased transaction costs Definitions d1 = execution d2 = control d3 = location d4 = ownership ag6 ag6 CLUSTER 3 CLUSTER 3 cp4 cp4 ag2 ag2 CLUSTER 2 CLUSTER 2 cn4 cn4 p3 p3 a6 a6 ap ap cp3 cp3 CLUSTER 4 CLUSTER 4 ag4 ag4 p1 p1 ag1 ag1 d4 d4 ag3 ag3 cn2 cn2 a1 a1 p4 p4 ag5 ag5 a4 a4 acad acad a2 a2 cn1 cn1 d3 d3 a3 a3 Axis 2 p2 p2 d2 d2 cp2 cp2 practice practice cn3 cn3 d1 d1 a5 a5 CLUSTER 1 CLUSTER 1 CLUSTER 5 CLUSTER 5 cp1 cp1

18. 3. Framing the Empirical Investigation for RQ#1 *Lau, Kevin, Soon Ang, & Detmar Straub, “Knowledge Structures of Academic and Practitioner Discourse: A Content Analysis of the Outsourcing Literature,” working paper, 2000. • In the outsourcing literature, practitioners focus on consequences.* • Scholars tend to focus on: • Nomologies (or antecedents leading to consequences), • Processes , and • Definitions.

19. 3. Framing the Empirical Investigation for RQ#1 • Szajna, B. (1994). "How Much is Information Systems Research Addressing Key Practitioner Concerns?" DATABASE, May, 49-59. Two out of three of her tests asking whether topics published in academic journals aligned well with those of the “CIO Key Issues” surveys (Dickson and Nechis, 1984; Brancheau and Wetherbe, 1987; Niederman, Brancheau, and Wetherbe, 1991) were SIGNIFICANT. • Abstract: The results of a series of three surveys, published by MIS Quarterly, have designated issues of primary concern to IS practitioners….Findings suggest that IS researchers have been focusing on important practical issues but that these issues were critical to IS practitioners in the early 1980's. While some of these issues such as end-user computing, data management and software development may be of such abiding interest that its appropriate to continue research in these areas, other topics have yet to receive the attention they might deserve. The findings also specify practitioner issues that have experienced an increase / decrease in research interest and the publication outlets that emphasize primary practitioner issues.

20. 2. Framing the Conceptual Investigation: RQ #1 • A forthcoming article by Taylor et al. (MISQ, 2010) offers strong evidence that the IS community has always focused its attention on IS strategic issues, a topic which appears prominently in every single CIO survey since 1984 (Dickson and Nechis, 1984; Brancheau and Wetherbe, 1987; Niederman, Brancheau, and Wetherbe, 1991; Luftman, 2004; Luftman and Ben-Zvi, 2005; Luftman and Kempaia, 2008; Luftman et al., 2009; Luftman and Ben-Zvi, 2010). • Analyzing the co-citation patterns of over 30,000 citations, Taylor et al. (2010) found that IS strategy was the dominant focus of a group of IS scholars who morphed over time, but who always attacked this same topic.

21. 3. Framing the Empirical Investigation for RQ#1 • RQ #1. Are the findings useful for practitioners? Or Topic Usefulness (Not Alignment) • RQ #2. Are they being communicated to practice? Or Knowledge Transfer

22. 3. Framing the Empirical Investigation for RQ#1 *Szajna, B. (1994). "How Much Is Information Systems Research Addressing Key Practitioner Concerns?," DATABASE, May, 49-59. • In a 1994 study of key executive IS issues versus what the IS journals were publishing, Szazna* concluded that there was always not a good alignment between practitioner issues and academic issues. • But she may have been attacking a fictitious issue. • I would argue that usefulness is the issue, not alignment. • Why would academe need to be perfectly aligned with businesses? Indeed, the evidence in Straub and Ang (2008) is that they are not.

23. LEGEND Knowledge sources acad = academic profile point ap = academic-practitioner profile point practice = practitioner profile point Antecedents Triggers a1 = strategic realignment a2 = poor profitability a3 = poor performance a4 = resource deficiency a5 = regulatory influence a6 = supplier presence Inhibitors ag1= increase in org. risks ag2= strong org resistance ag3= poor funct. performance ag4= loss of funct. resources ag5=high transaction costs ag6=lack of suitable suppliers FIGURE 1 Correspondence Analysis Mappings LEGEND (continued) Process p1= planning for outsource p2 = manage vendor obligations p3=manage client obligations p4 = manage employee obligations Consequences Positive cp1 = increased cost advantage cp2 = increased profitability cp3 =enhanced performance cp4 = increased resources Negative cn1 = increased org. risks cn2=decreased funct. performance cn3 = reduced resources cn4=increased transaction costs Definitions d1 = execution d2 = control d3 = location d4 = ownership ag6 ag6 CLUSTER 3 CLUSTER 3 cp4 cp4 ag2 ag2 CLUSTER 2 CLUSTER 2 cn4 cn4 p3 p3 a6 a6 ap ap cp3 cp3 CLUSTER 4 CLUSTER 4 ag4 ag4 p1 p1 ag1 ag1 d4 d4 ag3 ag3 cn2 cn2 a1 a1 p4 p4 ag5 ag5 a4 a4 acad acad a2 a2 cn1 cn1 d3 d3 a3 a3 Axis 2 p2 p2 d2 d2 cp2 cp2 practice practice cn3 cn3 d1 d1 a5 a5 CLUSTER 1 CLUSTER 1 CLUSTER 5 CLUSTER 5 cp1 cp1

24. 2. Framing the Conceptual Investigation: RQ #1 *Lau, Kevin, Soon Ang, & Detmar Straub, “Knowledge Structures of Academic and Practitioner Discourse: A Content Analysis of the Outsourcing Literature,” working paper, 2000. • In the outsourcing literature, practitioners focus on consequences.* • Scholars tend to focus on: • Nomologies (or antecedents leading to consequences), • Processes , and • Definitions.

25. 4. Improving Topic Usefulness *“Trans-Situated Learning: Supporting a Network of Practice with an Information Infrastructure,” Information Systems Research, Vol. 20, No. 4, December 2009, 547-564. • Emmanuelle Vaast & Geoff Walsham* argued that sharing knowledge is “situated” and tacit knowledge can only be rarely and with great difficulty communicated through communities of practice (CoPs), networks of practice (NoPs), and information infrastructures (IIs) . • Scholars are distinct from practitioners in terms of CoPs, NoPs, & IIs (Nielsen, KIWISR-5, 2010).

26. 4. Improving Topic Usefulness *“Toward Improving the Relevance of Information Systems Research to Practice: The Role of Applicability Checks,” MIS Quarterly, March, 2008, 1–22. • Michael Rosemann and Iris Vessey* proposed that we design and conduct IS research as usual…. • But at a research project’s beginning or end (or both), add an applicability check. • That is, you first explain the research project and its model to a practitioner focus group. • The group then provides thoughts, suggestions, and changes that you can report in your research and that could influence future related research.

27. 4. Improving Topic Usefulness *“Marshaling the Professional Experience of Doctoral Students:  A Contribution to the Practical Relevance Debate,” MIS Quarterly, December, 2008, 675-686. • Heinz Klein and Franz Rowe* proposed that we require doctoral students to have significant experience in practice before admitting them to doctoral programs. • Alternatively, they could gain such experience while in their programs. • In any case, the goal would be to help them produce more relevant research.

28. 4. Improving Topic Usefulness Antonacopoulou, Elena P. (2010). “Making the Business School More 'Critical': Reflexive Critique Based on Phronesis as a Foundation for Impact,” British Journal of Management, March, Vol. 21, pp. S6ff. Abstract: This paper explores how the business school can become more critical by advancing the notion of reflexive critique. Drawing on diverse literatures propounding a critical perspective, this paper integrates the various interpretations of 'what it is to be critical' and proposes phronesis as a foundation for responding to and extending the relevance and rigour debate by articulating what it means for business schools to have a critical impact on management practice. A phronetic analysis of management education provides an innovative lens for understanding the power of critique in engaging academics and business practitioners in the co-creation of knowledge. This is illustrated by distilling the main insights from the experience of introducing an innovative course entitled 'Critical Thinking' offered to MBA students over a five-year period. The paper discusses the importance of critique in the business curriculum and explains the rationale for introducing the course and its objectives, as well as the learning and teaching techniques employed. The analysis considers how reflexive critique can be a platform for integrating a critical analysis of management informed by management research and academic thinking in relation to business practitioners' practical experiences of managing. The paper concludes with a review of the main lessons learned and the implications for future initiatives intended to foster engagement of theory and practice and the collaboration of academics and business practitioners [bolding, coloring, & italics added for emphasis].

29. 4. Improving Topic Usefulness Kieser, Alfred and Lars Leiner (2009). “Why the Rigour-Relevance Gap in Management Research Is Unbridgeable,” The Journal of Management Studies, May, Vol. 46, No. 3; pg. 516ff. Abstract: In this paper we discuss, on the basis of system theory, the rigour-relevance gap in management research and the proposal to overcome it. From a system theory perspective, social systems are self-referential or autopoietic, which means that communication elements of one system, such as science, cannot be authentically integrated into communication of other systems, such as the system of a business organization. Social systems can only irritate - provoke - each other, i.e. alter conditions in such a way that other systems are forced to respond. Because of the differences between management science and practice it is impossible to assess relevance of research output within the system of science. On the basis of our analysis we show that neither action research nor Mode 2 research nor recent approaches to collaborative research can succeed [bolding, coloring, & italics added for emphasis] in producing research that is rigorous as well as relevant. Researchers and practitioners cannot collaboratively produce research; they can only irritate each other. However, sometimes irritations or provocations turn out inspiring.

30. 4. Improving Topic Usefulness BUT…. Hodgkinson, Gerard P. and Denise M Rousseau (2009). “Bridging the Rigour-Relevance Gap in Management Research: It's Already Happening!,”The Journal of Management Studies, May 2009. Vol. 46, No. 3; pp. 534ff. Abstract: Kieser and Leiner (2009) maintain that the rigour-relevance gap in management research is fundamentally unbridgeable because researchers and the researched inhabit separate social systems. They argue that it is impossible to assess the relevance of research outputs within the system of science and that neither action research nor related approaches to collaborative research can succeed in producing research that is rigorous as well as relevant. In reply, we show how their analysis is inconsistent with available evidence. Drawing on a diversity of management research domains, we provide counter-illustrations of work where researchers, in a number of cases in collaboration with practitioners, have generated knowledge that is both socially useful and academically rigorous [bolding, coloring, & italics added for emphasis].

31. 4. Improving Topic Usefulness Mentzer, John T. (2008). “Rigor versus Relevance: Why Would We Choose Only One?”,Journal of Supply Chain Management, April, Vol. 44, No. 2; pp. 72ff. Abstract: The gap between scholarly research and practice in business has existed for decades. Academics argue that the rigor is so important to what they do that relevance is not important. This is no less true in the business disciplines, including supply chain management. The central mission of business scholars and educators is to conduct research that contributes knowledge to a scientific discipline and to apply that knowledge to the practice of management. To do this, the authors need to design their research so that it provides understanding of the practical problems facing a profession, and they need to develop good theory so that their research advances the knowledge that is relevant to both the discipline and practice. For research done well, the authors do not have to choose between rigor and relevance. Researchers, editors and reviewers should pay attention to rigor. Researchers and practitioners should dedicate themselves to regularly interacting to understand the timely and timeless issues faced in their discipline.

32. 4. Improving Topic Usefulness

33. 4. Improving Topic Usefulness

34. 4. Improving Topic Usefulness • Could we do it better? • Probably, Yes. • Could we already be relevant from the POV of RQ #1? • Likely. We need more studies, naturally.

35. 5. Framing the Conceptual Investigation: RQ #2 • We have logical and prima facie evidence that IS research is possibly relevant by RQ #1. That is, RQ #1. Are the findings useful for practitioners? • What about RQ #2? (Knowledge transfer) • The basic assumption of most researchers who have tried to study this is that the communication of research findings must take place through print. • Mostly journals of some type.

36. 5. Framing the Conceptual Investigation: RQ #2 Or, on the road… • What kind of PRINT do practitioners read?

37. 5. Framing the Conceptual Investigation: RQ #2 • The Fictitious Research Question: Do practitioners read academic journals? • The Real RQ: Do practitioners learn about IT research findings (knowledge transfer through any kind of medium)?

38. 5. Framing the Conceptual Investigation: RQ #2 * Kim, Shin, & Grover (2010). “Investigating Two Contradictory Views of Formative Measurement in IS Research, MIS Quarterly, Vol. 34 No. 2 (June), 345-365. • The Fictitious Research Question: Do practitioners read academic journals? • ANSWER: No • Why not?: (1) They have different interests; (2) They do not have the educational background that would allow them to effectively interpret the articles. • Example?*

39. 5. Framing the Conceptual Investigation: RQ #2 * * Kim, Shin, & Grover (2010). “Investigating Two Contradictory Views of Formative Measurement in IS Research, MIS Quarterly, Vol. 34 No. 2 (June), 345-365.

40. 5. Framing the Conceptual Investigation: RQ #2 • In Straub & Ang (2008), we argued that alternative (indirect) forms of knowledge transfer might be: • Academic-practitioner journals (e.g., MISQ Executive), • Textbooks that reflect the best theoretical and practical thinking in the business disciplines, • Higher education courses and degree programs, • Continuing education programs, • Short courses or seminars, • Public speaking engagements by academics, • Newspaper articles, • Brochures that describe in lay terms the ongoing research of research centers,

41. 5. Framing the Conceptual Investigation: RQ #2 • To this we can add: • Teaching students the principles of IT consultancy • Corporate training by academics • Certificate programs • Collaborative research between academics and practitioners • Sponsored conferences based on research findings • Faculty internships • Findings presented to university advisory groups • White papers and policy briefings • Executive doctoral programs • Academic-practitioner journals (e.g., MISQ Executive, Academy of Management Executive) • Scholarly journals (a very limited proportion of the overall knowledge transfer

42. 6. Moreover…What Kind of Evidence Do We Need? • “Need”…to prove that our research was either relevant or irrelevant…. • Analyses of the literature plus…empirical studies using: • The journal archive • Fresh data collection from relevant subjects • Experiments??

43. 6. What Kind of Evidence Do We (NOT) Need? • Assumptions (Herculean) • Unsupported opinions and assertions • Anecdotes • Dogma • Innuendo • Impressions • Rumors • Popular beliefs • Urban myths • Snide asides • Logic that misses the point

44. 7. Prior Literature Addressing the Real Issue

45. 7. Prior Literature Addressing the Real Issue • Have there been many studies to date that have really researched either “rigor” or “relevance”???? • Not according to Palmer et al. (2009)* • They say there is little empirical research on: 1) historical trends in theoretical and methodological rigor, 2) historical trends in relevance, and 3) the relationship between the two. • This is partly because the measurement of theoretical and methodological rigor, as well as of relevance, is fraught with conceptual problems. *Palmer, Donald, Brian Dick, & Nathaniel Freiburger (2009). “Rigor and Relevance in Organization Studies,” Journal of Management Inquiry. December, Vol. 18, No. 4; pg. 265ff.

46. 8. Prior Literature Addressing the Fictitious Issue (Herculean Assumption) Dess, Gregor G. and Livia Markoczy (2008). “Rather than Searching for the Silver Bullet, Use Rubber Bullets,” Journal of Supply Chain Management, April, Vol. 44, No. 2; pp. 57ff. Abstract: Organizational scholars have long recognized the divide or "gap" between academic rigor and practical relevance. This recognition goes back to at least to 1949, when Merton asked social scientists to more carefully consider the usefulness of their work. Many others have since addressed this issue. The research process is often referred to as a continual process of rediscovery. Unfortunately, the research-practice "gap" was discovered a long time ago and it appears to be continually discovered with a little bit of light -- but perhaps a lot more heat. The authors have offered a few ideas on the source of the problem and some recommendations or "rubber bullets" to help bridge the "gap." However, they recommend caution in efforts directed toward dosing the "gap." After all, the authors hope that "the cure is not worse than the disease."

47. 8. Prior Literature Addressing the Fictitious Issue Desouza, Kevin C., Omar A. El Sawy, Robert D. Galliers, Claudia Loebbecke, & Richard T. Watson (2006). “Beyond Rigor and Relevance Towards Responsibility and Reverberation: Information Systems Research That Really Matters,” Communications of the Association for Information Systems, Vol. 17, pp. 1ff. Abstract: This paper is an outcome of a panel discussion that was held during the 2005 International Conference on Information Systems (ICIS). The panel, titled, "IS Research that Really Matters: Beyond the IS Rigor - Relevance Debate," was organized to foster a discussion on conducting IS research that is significant, in terms of impact, and responsible to the needs of stakeholders. To this end, panel members were asked to push the debate on IS research beyond the traditional arguments of rigor and relevance in research towards IS research that reverberates and is responsible. In doing so, the panel members shared their views on the definition of significant research, who are the stakeholders of IS research and what are their needs, and how do we move the field ahead.

48. 8. Prior Literature Addressing the Fictitious Issue (By Implication) Desouza, Kevin C., Omar A. El Sawy, Robert D. Galliers, Claudia Loebbecke, & Richard T. Watson (2006). “Beyond Rigor and Relevance Towards Responsibility and Reverberation: Information Systems Research That Really Matters,” Communications of the Association for Information Systems, Vol. 17, pp. 1ff. “Significance - I guess everybody would agree – mainly depends on the stakeholders. The always repeated and old story refers to journals, rigor and relevance, and promotion committees. But the title of this discussion is on IS research that really matters. So, to whom do journals and promotion committees matter? Mainly to the authors, i.e., to us!” (p. 347)

49. 8. Prior Literature Addressing the Fictitious Issue (By Direct Statement) Desouza, Kevin C., Omar A. El Sawy, Robert D. Galliers, Claudia Loebbecke, & Richard T. Watson (2006). “Beyond Rigor and Relevance Towards Responsibility and Reverberation: Information Systems Research That Really Matters,” Communications of the Association for Information Systems, Vol. 17, pp. 1ff. “The outcome is that research does not appreciate the complexity of environments, both at the industry and societal levels. Hence, research products, especially those that are written up in our mainstream journals, seldom get any attention from external stakeholders.” (p. 350)

50. 8. Prior Literature Addressing the Fictitious Issue Clinebell, Sharon K. and John M. Clinebell (2008). “The Tension in Business Education Between Academic Rigor and Real-World Relevance: The Role of Executive Professors,”Academy of Management Learning & Education, March, Vol. 7, No. 1; pp. 99ff. Abstract: Business schools are walking a tightrope between the academic side of business and the practitioner side. In reaction to early criticism of being trade schools, business schools have emphasized the academic nature of business education. The pendulum has now moved in the direction toward greater inclusion of real-world experience. As a result, many business schools are hiring business practitioners to teach courses. We examine the current state of business education and the tension that exists in many business schools today concentrating on the use of executive professors in the business school. Recommendations are provided regarding the use of executive professors.