Writing and reading the results: the reporting of research rigour tactics in information behaviour research as evident in the published proceedings of the biennial ISIC conferences, 1996 – 2014

Introduction

In the introduction to their paper about the information needs of physicians at a hospital in Valencia, Spain given at one of the earliest ISIC conferences (Sheffield, 1998), Abad-Garcia, Gonzalez-Teruel and Sanjuan-Nebot (1999) point out that in addition to sharing results of studies in order to improve information sharing, research reporting can also lead to ‘the recognition of problems in methodology which are revealed when studies which have been reported are analyse’ (p. 209). Problems in methodology point to limitations and problems with the findings of a study. The authors are talking about the concept of research rigour and its importance in all stages of the research process, including communicating about research studies.

According to Merriam-Webster, the concept of rigour refers to ‘the quality or state of being very exact, careful or strict’ (Rigor, n.d.). Similarly, Pickard (2013) in her research methods text for information professionals and scholars, provides a definition for “Rigour” in her glossary of research terms: ‘[the] degree to which research methods are scrupulously and meticulously carried out in order to recognize important influences occurring in an experiment’ (p. 326). In his editorial statement “Rigor – the Essence of Scientific Work” published in an issue of the Electronic Journal of Biotechnology, Allende (2004) explores this concept, writing:

Rigor is many things. It is dissatisfaction with uncertainty, with inaccurate answers, with unprecise (sic) measurements, with the spread between the plus and the minus. Rigor is also being methodical commitment (sic) to experimental procedure, to the need of controlling all parameters that can affect the results of our tests . . . Rigor is in the essence of scientific work, in each one of the stages of the research work. Rigor implies a structured and controlled way of planning, developing, analyzing and evaluating our research and a special care in adapting the presentation of the results to the demands of the audience we communicate the results of our investigations. (n.p.)

One could argue that the emergence of information behaviour as a distinct, cohesive and important area of study was marked by the first ISIC conference in Tampere, Finland in 1996. As such it is still a relatively young area of study. We agree with Abad-Garcia, Gonzalez-Teruel and Sanjuan-Nebot (1999) that rigour is central to the conduct of information behaviour research. We found ourselves wondering if this viewpoint was shared by others and, more importantly, if it was evident in the published work of other information behaviour scholars.

Literature review

In a very short recent article in Nature, Pagan and Torgler (2015) remind readers that ‘4Rs’ are required to assess papers for research rigour: reproduction, replication (provision of enough information to allow a study to be reproduced), robustness (strong design), and revelation (communicating enough information to allow for accountability and transparency) (p. 34). In their library and information science research methods text, Williamson and Johanson (2013) note

reviewers and editors pay particular attention to the elaborate detailing of research methods and techniques with a tacit assumption that the adoption of ‘rigorous’ and ‘reliable’ research methods is a guarantor of the validity of research outcomes.(p.115).

Researchers, whether qualitative or quantitative, agree that rigour in research methods is intended to guarantee that study results are valid, reliable, and trustworthy, and that published work can be replicated and evaluated.

In our review of the literature we were initially surprised to find substantially more material about rigour in qualitative, as compared to quantitative, research. However, it is likely this trend reflects the expansion of qualitative methodologies from disciplines such as anthropology and sociology to others including health sciences, education, and library and information science, a transition marked by the publication of Lincoln and Guba’s often cited landmark work Naturalistic Inquiry in 1985.

General overviews of rigour in quantitative research are most commonly found in research methods texts. For example, Babbie in his 5^th edition of The Basics of Social Research identifies appropriate and strong research design, appropriate statistical analysis, appropriate sampling techniques, the ability to replicate the work, and disclosing the limitations of a study as criteria which contribute to research rigour. Connaway and Powell (2010) present a similar list, organizing it under the broader categories of universality, replication, control, and measurement. Journal articles and reports dealing with rigour in quantitative research often focus on good practices within particular disciplines such as the Society for Neuroscience’s (n.d.) “Research practices for scientific rigor: a resource for discussion, training and practice,” Claydon’s (2014) “Rigour in quantitative research,” which focuses on the particular requirements for Nursing, and the Federation of American Societies for Experimental Biology’s (2016) “Enhancing Research Reproducibility”.

For qualitative work, Lincoln and Guba (1985) provide the fundamental description of  trustworthiness, based on earlier work by Guba (1981), who focuses on credibility, dependability, confirmability, transferability, and authenticity, as markers of rigour in qualitative work. According to Krefting (1991), there are multiple tactics to achieve trustworthiness in qualitative work. To achieve credibility, a researcher should attend to prolonged field experience, time sampling, reflexivity, triangulation, member checking, peer examination, interview technique, researcher credibility, structural coherence, and referential adequacy. To achieve transferability, scholars may use nominated samples, compare their samples to demographics data, time sample, and use dense description. To achieve dependability, researchers can apply a dependability audit, dense description of research methods, stepwise replication, triangulation, peer examination, and code-recode procedures. Finally, to achieve confirmability, researchers can use a confirmability audit, triangulation, and reflexivity. Houghton, Casey, Shaw and Murphy (2013) specifically point to techniques such as prolonged engagement, persistent observation, triangulation, peer debriefing, member checking, audit trails, reflexivity, and thick descriptions as methods to achieve trustworthiness. Shenton (2004), writing in the library and information science context, provides a long and useful list of specific actions and tests for qualitative researchers to apply in a quest for trustworthiness. Many others have written about techniques to ensure trustworthiness, such as Cope (2014), and Long and Johnson (2000). Freese (2007), Kingsley and Chapman (2013), and Funder et al. (2014), emphasize transparency of data and data collection methods. Davies and Dodd (2002) point to ethical practices as being central to rigour. Rigour in qualitative work, they note, demands consideration for 'attentiveness, empathy, carefulness, sensitivity, respect, honesty, reflection, conscientiousness, engagement, awareness, openness, and context.' Many of these ethical stances point to reflexivity, which is the focus of significant writing about qualitative work (Finlay, 2006). Hall and Callery (2001), for example, focus on reflexivity and relationality with interview participants when doing grounded theory work, and Jootun and McGhee (2009) offer very specific techniques for promoting reflexivity. Porter (2007) suggests that qualitative researchers follow seven principles, and suggests questions related to each. The first is transparency (is the process of knowledge generation open to outside scrutiny?). The second is accuracy (are the claims made based on relevant and appropriate information?). The third is purposivity (are the methods used fit for purpose?). The fourth is utility (are the knowledge claims appropriate to the needs of the practitioner?). The fifth is propriety (has the research been conducted ethically and legally?). The sixth is accessibility (is the research presented in a style that is accessible to the practitioner?). And finally, the seventh is specificity (does the knowledge generated reach source- specific standards?).

While there is a significant body of literature which describes research rigour and how to achieve it, we were unable to identify, both generally and specifically for information behaviour research, any studies which systematically examined the reporting of research rigour tactics in published reports of empirical research.

Research questions

To address the gap identified in the literature we explored the following research questions:

1. Do information behaviour researchers report methodological tactics used in their research projects so that other researchers have enough information to replicate the studies? What do they report? What do they not report? And if they report rigour tactics, where do they report them in their papers?
2. Do information behaviour researchers share enough information so that readers can assess rigour and the validity/trustworthiness of findings? If so, how is this done?

Method

To address our research questions, we conducted a content analysis (Krippendorff, 2013) of the 193 full papers which were reports of empirical research studies found in the published proceedings of the biennial ISIC conferences from 1996 to 2014. While human information behaviour researchers communicate their work through a variety of publications and conferences, ISIC was chosen as being one of the major international venues for dissemination of human information behaviour research. While this may be regarded as a limitation to the study, it resulted in a sample which made data collection and analysis feasible and which, we argue, constitutes a reasonable representation of the information behaviour research.

The ISIC proceedings include a variety of types of papers, such as theory papers, method papers and reports of empirical research studies. Some years of the proceedings include both full and short papers. We did not analyse the short papers as the length restriction is likely to have limited the discussion of research rigour. Our first step was to look carefully at the full papers to identify the reports of empirical research, the corpus examined in this study.

Our next step was to compile a list of the tactics used by researchers to ensure rigour in their design, implementation and reporting of research studies. These were derived from the following prominent research methods texts, chosen with an emphasis on works commonly used for both teaching and research within library and information science:

• Babbie, E. (2010). The practice of social research (12^th).
• Leedy, P.D., and Ormrod, J.E. (2013). Practical research: planning and design (10^th).
• Lincoln, Y.S., and Guba, E.G. (1985). Naturalistic inquiry.
• Miles, M.B., Huberman, A.M., and Saldana, J. (2014). Qualitative data analysis: a methods sourcebook (3^rd).
• Pickard, A. (2013). Research methods in information (2^nd).
• Connaway, L.S., and Powell, R.R. Basic research methods for librarians (5^th).
• Wildemuth, B.M. (2009). Applications of social research methods to questions in information and library science.
• Williamson, K., and Johanson, G. (Eds.). (2013). Research methods: information systems and contexts.

To name individual research rigour tactics we adopted the language used by the authors of the texts consulted. Individual research rigour tactics were also identified as being primarily used in quantitative studies (designated by *), primarily used in qualitative studies (**) or used in mixed methods (both) approaches to research. During the coding process we did not make an assessment of whether or not a research rigour tactic was used appropriately but did look for evidence of implementation of the tactic before coding for the presence of that tactic. The following research rigour tactics, grouped by stages in the research process, were identified and provided a start list for our coding of the papers:

Design

• Triangulation (methods, participants, researchers, data collection sites)
• Pilot tests

Sampling

• Theoretical/purposive sampling **
• Sampling to saturation **
• Sampling strategy (type of quantitative sampling) *
• Sampling frame
• Sample size
• Response rates
• Characteristics of participants

Data collection

• Prolonged engagement in the field **
• Keeping an audit trail **
• Keeping a reflexive journal **
• Replication

Data analysis

• Member checking **
• Peer debriefing **
• Inter-coder reliability
• Intra-coder reliability
• Negative case analysis
• Appropriate statistical analysis *

Research reporting

• Thick description **
• Transparency
• Inclusion of limitations to the study
• Inclusion of “non/negative” findings
• Inclusion of data collection instruments (e.g., surveys, interview schedules)

Other

Papers were also coded for disciplinary affiliation and country of the first author, the types of research methods used in the study, theories used, whether or not research rigour tactics were explicitly reported, whether or not enough information was provided to allow readers to assess the rigour of the study and/or replicate the study, where research rigour tactics were reported in the article (title, abstract, introduction/literature review, method, results, discussion/conclusion) and the presence of citations to support the choice of research rigour tactics. During the coding process we kept reflective notes to document theoretical insights and kept track of particularly cogent examples related to the authors’ use (and non use) of research rigour tactics.

In order to provide a preliminary assessment of the validity of the coding instrument, all authors coded a small sample of three randomly chosen papers as a basic test for inter-coder reliability. The unit of analysis for this was the individual decision point; in other words, every time a decision was made (e.g., were research rigour tactics explicitly reported or not reported?) comprised an instance of the unit of analysis. Our coding scheme included 54 decision points. Frequencies and percentages were calculated for the number of agreements. While it is agreed there are problems associated with this particular measure, research indicates percentage agreement is the most commonly (69% of studies) used measure (Lombard, Snyder-Duch, and Bracken, 2010). For our purpose of getting a rough sense of the validity of our instrument, it worked reasonably well. The rate of agreement was 90.0%, suggesting that the coding scheme was likely valid and reliable.

The data were analysed quantitatively with frequencies and percentages calculated for all categories. Although our sample size was relatively large (n=193), the data did not meet the criterion of having five or more data points in each cell to be able to calculate tests of Chi Square to look for relationships between variables. We also looked for themes and examples apparent in the data as we read through the articles.

Results

Characteristics of the papers analysed

While the affiliations and geographic locations of authors were not reported consistently across all the ISIC proceedings (in particular this was done differently for the first two conferences in 1996 and 1998), we did code for these variables to give some sense of the background of the authors of the reports of empirical research analysed for this study. The majority of the 165 first authors with a reported affiliation to a particular discipline or organization were associated with library and information science programs (n=140; 84.8%), 21 (12.7%) with other academic programs, two (1.2%) with research centres, one (0.6%) with professional practice and one (0.6%) with another context. Of the 176 first authors for whom a geographic location was reported, 57 (32.4%) were from North America, 45 (25.6%) from Scandinavia, 36 (20.5%) from another European country, 15 (8.5%) from Australia and New Zealand, 13 (7.4%) from Asia, six (3.4%) from the Middle East and two (1.1%) from each of Africa and South America.

 Researchers used a variety of methods in their studies. Quantitative methods were used in 32 (16.6%) studies, qualitative methods in 105 (54.4%) studies and a combination of quantitative and qualitative methods in 52 (26.9%). Four (2.1%) of the 193 reports of empirical research did not provide enough information to classify them as quantitative, qualitative or both in research method approach. The most frequently used method was interviewing (n=118; 61.1%), followed by surveys (n=62; 32.1%), observation (n=50; 25.9%), diaries (n=15; 7.8%), think-aloud protocols (n=11; 5.7%), focus group interviews (n=9; 4.7%), transaction log analysis (n=9; 4.7%), social network analysis (n=7; 3.6%), experiments (n=6; 3.1%), application of standard tests (n=6; 3.1%), and bibliometric analysis (n=3; 1.6%). Other methods (including approaches such as the critical incident technique, case study, conversational analysis, eye tracking and secondary data analysis) were used a total of 32 (16.6%) times. Eighty-two (42.5%) of the papers listed one or more theoretical approaches which were used to design the studies and/or to interpret the results.

The reporting of research rigour tactics

Authors reported research rigour tactics in a variety of places in their papers. Not surprisingly this occurred most frequently in the methods section (n=140 papers; 72.5%), but also was included in results (n=17; 8.8%), discussion/conclusion (n=39; 20.2%), the introduction and literature search (n=17; 8.8%) and even in the title of two (1.0%) papers. A surprising 43 (22.3%) of papers included one or more research rigour tactics in the very brief space allocated to abstracts; in almost all cases this was a statement of sample size or methodological triangulation.

Across the 193 papers, research rigour tactics were reported 698 times, for an average of 3.6 and a median of 3 times per paper. The number of tactics reported ranged from none (n=26 papers; 13.5%) to a high of 20 (n=1; 0.5%), with only 8 (4.1%) papers describing 10 or more. Seventy-nine papers (40.9%) included one to three rigour tactics, 58 (30.1%) four to six, and 22 (11.4%) seven to nine. Frequencies for individual tactics are summarized in Table 1. In Table1 * is used to identify primarily quantitative research rigour tactics, ** primarily qualitative tactics and tactics which are used by both qualitative and quantitative approaches are not marked. The four papers which could not be classified as quantitative, qualitative or mixed methods (both) are not represented in a separate column but are included in the total number of papers and in calculations which show frequencies and percentages in relation to the total number of papers.

Sampling tactics (n=320; 45.8% of the total number of tactics) were the most frequently reported, with sample size included in 100 (51.8%) of the papers and the characteristics of respondents in 77 (39.9%). Research reporting tactics constituted the next largest group with 158 incidents (22.6% of the total) of research rigour reporting tactics. Noting the limitations of the study (n=65; 33.7% of papers) and using thick description (n=53; 27.5% of papers) were the two most prominent tactics evident in this group. What is interesting about this is that these rigour tactics are foundational to quantitative research. It may be that their higher incidence is simply associated with greater familiarity of the authors with the discourses of quantitative research writing. Data analysis tactics (n=83; 13.3%), design tactics (n=81; 11.6%), and data collection tactics (n=44; 6.3%) accounted for most of the remainder of research rigour tactics reported. Other tactics listed, while quite important for research rigour, were addressed, surprisingly, in no more than one or two papers each and included things like building rapport with study participants, practicing emergent design and analysis, thoroughly training project staff, and using standard tests and instruments which had worked well in other studies.

When considered individually, quantitative papers reported more research tactics (an average of 4.4 per paper) than qualitative papers (3.5 tactics per paper) and mixed methods papers (3.6 per paper). As noted in the literature search, this may be because quantitative research methods have a longer history of use within LIS with better defined practices which are more generally understood. However, these differences are not very large.  Papers in each research approach reported an overall percentage of tactics close to the their percentage of papers overall: quantitative papers comprised 16.6% of the sample and reported 20.3% of tactics;  qualitative papers (54.4%), 48.7% of tactics; papers employing both qualitative and quantitative approaches (26.9%), 27.1% of tactics.

Examples of research rigour tactics

We encountered many particularly fine examples of research rigour writing. The following, chosen from the eight studies including more than ten rigour tactics, are representative of what we found:

Credibility and truth value were enhanced by the extended period the researcher spent in the field. Persistent observation over an extended period avoids any tendency to come to conclusions prematurely. Over a year was spent in the field conducting participant observations and interviews, and analyzing the data took another year’s time. (Hersberger, 2001, p. 127)

[Prolonged engaged; thick description]

Analytical rigor was enhanced through robust sample size and triangulation of interview approaches (narrative and elicitation) and data sources (women and health professionals). The following measures were taken to strengthen credibility: identification and exploration of negative incidents; discussion and debriefing with peers; and careful documentation at all research stages. Rich description of findings and theoretical implications facilitates transferability.(Genuis, 2015)

[Sample size; triangulation of methods and participants; negative case analysis; peer debriefing; keeping an audit trail; thick description]

Data collection used in-depth, semi-structured interviews. An interview guide (Foster, 2005 and included in Appendix A), provided an agenda for open-ended questioning. . . . By far the most important method for credibility (Lincoln and Guba, 1985) was the use of member checking; that is, the process of using participants to review the researcher's recording and interpretation of their contribution. In the study reported here member checking was utilised in four ways: (1) at the pilot stage to develop initial questions, (2) as feedback and checking on examples and themes arising throughout interviews, (3) in post-interview discussions, (4) a sample of participants reviewed full transcripts of their own interview, reviewed the general findings, and were introduced to the model, and given opportunity to discuss its relevance as a reflection of their information behaviour. The study makes no claim for generalisability, as befits naturalistic inquiry, but ensures transferability and further development of the research themes by rich description and reporting of the research process (Lincoln and Guba, 1985; Sanjek, 1990). Dependability and confirmability were addressed through research notes, which recorded coding decisions, developing themes and interpretations, and emergent theory. (Foster, 2005)

[Including data collection instruments; member checking; noting limitations of the study; thick description; keeping a reflexive journal; citations are included to support and describe rigour tactics]

There were also many particularly poor examples, including the following. The author(s) has/ have not been listed; our intent was to explore research rigour reporting practices across information behaviour researchers as a group and not to criticize the work of individuals.

This action research project occurred in two phases. The first phase involved an appreciative inquiry process resulting in an organizational realignment of personnel and the introduction of shared leadership. The second phase involved the co-design of organizational information and communication systems and subsequent implementation of initiatives.

The number of groups and participants in this study was never identified and data collection and data analysis were only described in very general terms. Other aspects related to study design, data collection and data analysis were also only presented very generally, providing no basis for assessing rigour.

Even when a paper seemed to do a relatively good job, one or more key pieces of information related to rigour was often missing. For example, Genuis’s (2015) study does a good job of describing the method used (“Semi-structured interviews”), the sample size (“n=28”), characteristics of the participants (“women who had been or were engaged in information gathering and/or decision-making related to menopause management”) and uses thick description, quoting from a methods text (“The goal of the sampling was to gather rich qualitative data “so that others outside the sample [but in the target population] might have a chance to connect to the experiences of those in it (Seidel, 1998, p. 47-48)” to support this choice. However Genuis does not provide a copy of her interview guide; nor does she identify the sampling frames from which her participants were drawn.

Other results

Showing a serious commitment to reporting research rigour tactics used in their work, authors of 34 (n=17.6%) papers provided citations to support their use of particular research rigour tactics.

As indicated at the beginning of this paper, one of the most important reasons for writing about rigour in research is to allow the reader to assess the quality of the research project and its results. We asked two more holistic questions of each paper to assess how effectively information behaviour scholars do this. We found that 44 papers (22.8%) provided enough information so that a study could be replicated (i.e., described exactly what was done during data collection and analysis so that another researcher could repeat the study to explore whether or not the results were reliable); unfortunately 149 (77.2%) did not. While more authors (n=76; 39.4%) provided enough information to assess the quality of a study, many more (n=117; 60.6%) did not.

Finally, although we could not statistically test for a relationship, research rigour reporting practices did not appear to vary over time.

Conclusions

Overall, the findings of this study were disappointing. Both quantitative and qualitative empirical work is not being reported in ISIC papers in ways that clearly demonstrate research rigour, nor assure replicability. On the whole these authors are not poorly trained, nor is their work of poor quality; however, their reporting of it is unsatisfactory. It could be that authors are not viewing the ISIC conference as a venue where methods need to be particularly detailed. Perhaps these oversights would be rectified if the conference papers were subject to a second review prior to publication. However, it is surprising that the initial review for the conference fails to ensure more consistently careful reporting of methodological rigour. In order for information behaviour scholarship to receive the respect and have the impact it deserves researchers in the field must practice the basic literacies, the ABCs, of research reporting, writing their papers so that readers can fully make sense of the work. 

About the authors

Lynne McKechnie is Professor in the Faculty of Information and Media Studies, The University of Western Ontario, London, Ontario, Canada. She received her PhD from the University of Western Ontario. Her research and teaching interests include the intersection: of children, reading and public libraries; information behaviour; and, research methods. She may be contacted at mckechnie@uwo.ca.
Heidi Julien is Professor and Chair in the Department of Library and Information Studies at the State University of New York at Buffalo. She received her PhD from the University of Western Ontario. Her research and teaching interests include information behaviour, digital literacy, and research methods. She may be contacted at heidijul@buffalo.edu.
Roger Chabot is a PhD candidate in Library and Information Science in the Faculty of Information and Media Studies at The University of Western Ontario, London, Canada. His research interests include religious information behaviour and the application of Buddhist philosophy and psychology to topics in LIS. He may be contacted at rchabot2@uwo.ca.
Nicole Dalmer is a doctoral candidate in the Faculty of Information and Media Studies at The University of Western Ontario in London, Ontario, Canada. She received her MLIS from the University of Alberta. Her research and teaching interests include intersections of family care work and information work within aging in place policy, and the development of public library services for aging populations. She may be contacted at ndalmer@uwo.ca.
Cass Mabbott is a doctoral student at the Graduate School of Library and Information Science, University of Illinois, Urbana-Champaign, USA. Her research and teaching interests include social justice and youth in public libraries, information behavior of young children, and the history of children’s literature. She currently is a graduate assistant working with The Comic Book Readership Archive, with Dr. Carol Tilley. She may be contacted at mmabbot2@illinois.edu.