Vol. 9 No. 2, January 2004
This paper presents both theoretical aspects and practical examples from the on-going and large-scale JUBILEE (JISC User Behaviour in Information seeking: Longitudinal Evaluation of Electronic Information Services) project, now in its fifth year. Particular emphasis will be placed on the importance of using robust theory and methods as the basis for reputable user studies, especially those undertaken by practitioners. Theory underlying the development of the JUBILEE project and Toolkit is outlined, followed by a demonstration of its practical use and impact during the project's on-going programme of monitoring and evaluating. Themes underlying the paper concern the importance to practitioners of the need to understand and adopt a well founded methodology and sound methods with which to conduct projects, the additional benefits they can derive by so doing, and the pitfalls and dangers of not so doing. JUBILEE is used throughout as the example project to demonstrate the points made.
In a paper written in 1980 and re-published in 2000, Wilson discusses the methods used in information research, which he sees as a particularly diverse field of study, more of an applied than a basic science (Wilson, 1980). The outputs of information science research are applied in a given managerial or organizational setting. Wilson stresses the importance of method. He sees that social survey methods are often misused to collect a great deal of data without the researcher having a theoretical (or even descriptive) framework into which to fit the data. He suggests that, in effect, collecting data becomes a substitute for thinking about the problem. This is a particular risk for practitioner-based research.
Studies are commonly seen as being either quantitative or qualitative. Quantitative data are seen as being objective, quantifiable, hard, generalisable, based on numbers, whereas qualitative data are seen as being subjective, socially constructed, soft and non generalisable, based on words. User studies are, by definition, about people, behaviour and contexts. They need both quantitative and qualitative approaches to be combined to produce the both the holistic view and the robust data needed to triangulate and thereby validate data collected. Indeed, Salomon, as described by Miles and Huberman (1994) sees the quantitative versus qualitative argument as ill-founded, seeing instead an "analytic" approach to understanding a few controlled variables, and a "systemic" approach to understanding the interaction of variables in a complex environment. Both approaches can coexist if required by the objectives of the particular study.
User studies cover a wide range of topics and foci, for example:
Given that user studies are context driven, and are typically multidimensional covering at least some of the topics and foci described above, they will generally require multidimensional methods with which to study their associated research questions.
Models describe and explain some aspect or aspects of a situation or behaviour, and can be used as the basis for prediction. Models typically focus on more limited problems than do theories, and sometimes may precede the development of formal theory (Case, 2002). User studies may well produce a model as part of their data interpretation process, which may form the basis for a project output such as a 'toolkit', or set of guidelines for practitioners to apply to their own institutions. JUBILEE is an example of a project which has developed a model (of electronic information service development in institutions), which underpins a toolkit for use by practitioners, and which uses a mix of methods in its data collection process.
In this paper particular emphasis is placed on the importance of using robust theory and methods as the basis for reputable user studies, especially those undertaken by practitioners. Theory underlying the development of the JUBILEE project and the Toolkit is outlined, followed by a demonstration of its practical use and impact during the project's on-going programme of monitoring and evaluation.
Themes underlying the paper concern the importance to practitioners of the need to understand and adopt a well founded methodology and sound methods with which to conduct projects, the additional benefits they can derive by so doing, and the pitfalls and dangers of not so doing. JUBILEE is used throughout as the example project.
The Joint Information Systems Committee (JISC) of the Higher and Further Education Funding Councils in the United Kingdom funds a number on national information services for the use of research and learning communities in the post-16 education sector, which has recently been expanded from higher and further education to include Adult and Community Learning, and will also develop further to include school sixth forms. Information services provided by the JISC include online bibliographic databases, bulletin boards, software and data archives, socio-economic, scientific and digital map databases and electronic mail discussion lists. An aim of the JISC is to promote best practice and the use of agreed standards. The JISC strategy for 2001-5 outlines the vision of a single world-wide information environment that will support learners, teachers, researchers and administrators: it is designed to help the sector to manage change (JISC, 2000).
In 1999, the JISC established its Framework for Monitoring and Evaluating User Behaviour in Information Seeking and Use of Information Technology and Information Services in UK Higher Education to focus specifically on the development of a longitudinal profile of the use of electronic information services (EIS) and the development of an understanding of the triggers and barriers that affect such use. One of its great strengths is that the mixed qualitative and quantitative methodology generates a rich picture and a set of deliverables that have the potential to address the perspectives of, and be of value to, a variety of different audiences (Rowley, 2002).
JUBILEE (JISC User Behaviour in Information seeking: Longitudinal Evaluation of Electronic Information Services) is one of two large-scale projects funded under the JISC's MEUB Framework. It began in August 1999, and has since been extended to also include the further education sector, and is about to expand again to include Adult and Community Learning. JUBILEE is undertaking an holistic, longitudinal monitoring and evaluation, which seeks to predict, monitor and characterise information-seeking behaviour in relation to EIS, and is providing illuminative and contextualised pictures built up over time, of users and non-users in different disciplines.
Key questions for the Framework projects to answer are:
In summary, JUBILEE is discipline-based, collecting longitudinal data on the basis of annual cycles of fieldwork in institutions, followed by analysis and development of JUBILEE outputs.
JUBILEE has twin aims:
Methodology used in JUBILEE combines the approaches of social science, both qualitative and quantitative, with those more commonly used in an organizational setting, such as action research. The methodology must deliver a multidimensional insight into user behaviour in relation to electronic information services. JUBILEE is seeking to embed its findings in practice, and needs buy-in from practitioners in all stages of the research in order to realise this objective. A particular challenge for JUBILEE has been to use and then develop methods to specifically address the project aim of collecting and manipulating longitudinal data, with the outcome of providing a tool to promote good practice in institutions.
Methods for the JUBILEE project therefore use a mixed qualitative and quantitative approach including a literature review; questionnaires in paper and e-mail versions; face-to-face and telephone interviews; focus groups; and developmental workshops with practitioners.
|Methodology, methods||Outputs, outcomes|
—qualitative and quantitative, social survey
—applied research, action research
|Overall framework for study: |
—longitudinal, discipline-based, monitoring and evaluating user behaviour
|Monitoring user behaviour:
—survey: sampling (HE/FE, disciplines etc.),
—questionnaire, interview, focus group
|Understanding user behaviour:
—how people use EIS
—longitudinal dataModel of EIS behaviour to underpin Toolkit for use in institutions
—Toolkit design: analysis framework for improvement (themes, development stages)
|Evaluation toolkit/action plan for managers, to improve institutional practice|
|Impact and outcome assessment:
—data collection: re-visits, questionnaires
—action research projects- feedback
|Basis for staff development activities for LIS staff|
—participation in project giving changes in academic practice
—feedback leading to re-design/extension of subsequent cycles of project
Survey-based fieldwork activity has been undertaken in annual cycles of activity in Higher and Further Education institutions, and, over the four cycles of the project to date, has resulted in the following data being collected.
|Higher Education||Further Education||Action Research||Total|
|Staff interviews and focus groups||217||191||32||440|
|Student e-mail questionnaires and focus groups||34||33||89||156|
In-depth case studies are discipline-based, with a range of disciplines targeted for data collection in each cycle in a mix of large and small, and old and new institutions covering all geographical areas in the United Kingdom (England, Wales, Scotland and Northern Ireland). Academics, students and LIS staff are surveyed by questionnaire and interview in each institution. Snapshot case studies are shorter, conducted in institutions where deeper investigation is not feasible, with less data being collected. The breadth of fieldwork data is shown in the following table.
|Higher Education||Further Education|
|Cycle 1||Six in-depth case studies (health, business, English)||[No studies in Cycle 1]|
|Cycle 2||Six in-depth case studies (history, sociology, computing)||Six month pilot study:
Four in-depth case studies in the North-east of England (history, sociology, computing)
|Cycle 3||Six in-depth case studies (art & design, geography,
Three re-visit case studies (health, business)
|Five in-depth case studies in the North-east (art & design, geography, law, business)
Ten snapshot case studies in the North-east
Three action research case studies in the North-east
||Six in-depth case studies (politics, biology, film studies and media)
Three re-visit case studies (sociology/ history)
One action research case study
|Four in-depth case studies (politics, biology, film studies and media)
Five snapshot case studies
Three action research case studies
One off in-depth study: JUBILEE International
Small-scale action research projects within JUBILEE are providing close-to-practice data, which can be used to inform on the impact of electronic information services on institutional agenda items such as the impact of its use on student learning. Action research can obtain otherwise unavailable information, and is also a valuable dissemination mechanism to diffuse knowledge, both within institutions and more widely. It aids understanding, provides lived in experience, and is of demonstrable practical value to participating institutions. It increases practitioner acceptance of the research process, thereby reinforcing research outcomes in institutions. It should be of mutual benefit to researchers, and researched (Coulson et al., 2003).
Data collected through survey fieldwork are synthesised and interpreted to provide a longitudinal monitoring of changes in information behaviour in relation to electronic information services, within and between institutions and disciplines. This monitoring is aiding understanding and informing the development of policy by the JISC.
The synthesis and interpretation of the fieldwork baseline data collected in cycle 1 of JUBILEE permitted the modelling of critical success factors in relation to the development of EIS in institutions. This has formed the basis for the themes, which underlie the JUBILEE Toolkit, which is the second main aim of the project, under its evaluation remit.
Evaluation is the assessment of all data collected during the monitoring process, both quantitative and qualitative, with a view to determining the 'success', 'usefulness' or 'value' of the innovation (Wilson, 1980). For JUBILEE, the evaluation approach needed to be user-centred and holistic, encompassing the human and organizational aspects of a broadly defined system, that is, electronic information services..
An outcome of the project, as stated in the original project tender, is to produce:
...a benchmarking tool, in the form of an Action Plan for the use of HE managers, based on the characterisation of user-based success criteria in relation to EIS, as seen from the users' points of view. HE managers will be enabled to see how well positioned they are to exploit JISC resources and to support their decision making with respect to EIS.
In designing the JUBILEE evaluation, with the ultimate aim of producing a practical toolkit, various questions had to be answered, such as: what to include, and for whom? How to do it, and why? There are different approaches to evaluation, which suit different situations. The literature of evaluation suggests the importance of including qualitative as well as quantitative data as the basis for evaluation (Lancaster (1997), McClure (1999), Clarke (1999). Saracevic (2000) identifies the ethnographic, sociological, systems, economic and political approaches, which will be appropriate for differing audiences, with differing goals for their evaluation. The systems approach is the one most usually found in the context of electronic library projects, and indeed lies at the heart of the Tavistock model of evaluation used in the UK eLib projects (Kelleher, 1996). This is where aspects of information system performance lie at the heart of the evaluation, with data on the impact of the system on users, and data that synthesise impact more widely, also explicitly collected.
The key building blocks underpinning the evaluation are:
The interrelationship of these building blocks forms the basis for modelling the development of electronic information services in institutions, and forms the basis for the layout for the JUBILEE Toolkit as demonstrated in the following section.
The JUBILEE Toolkit is being constructed with the aim of improving the uptake and use of EIS in higher and further education institutions in the UK. Improvement resides in the fact that the Toolkit is designed for self application by individuals in relation to their own practice, and in relation to institutions. Individuals must identify their own strengths and weaknesses, and work towards solutions using the knowledge base and process model contained in the toolkit. The model of EIS development in institutions, which underpins the JUBILEE Toolkit characterises both the ingredients of information behaviour in relation to EIS, and a staged improvement process. The resulting framework is populated with fieldwork data, giving a knowledge base of practice based examples. It offers a template and indicative exemplars of good practice to guide practitioners in the their self-development and the development of their institutions.
Toolkits are decision making frameworks based on expert models (Oliver and Conole, 1999). The model underlying the JUBILEE Toolkit characterises both the ingredients of information behaviour in relation to EIS, and a staged improvement process:
The format of toolkits means that they can be used in a standard, linear fashion, or can be 'dipped into' by users whose level of expertise is stronger in some areas of the design process than others (Conole, 2000).
The JUBILEE Toolkit is constructed for user self-diagnosis and use. It permits flexible use giving several entry points to the data, by discipline, theme, level and stage of development, and user task (which is to be added in present cycle).
Critical success factors are those factors of critical significance, where effort should be concentrated, if the system is to be effective (Wilson and Huotari, 2001). Grounded theory can be used as the basis for deriving critical success factors, supported by documentary analysis and other methods used to determine the organizational context. They are useful in identifying organizational objectives, and in relating the information needs of various groups to those objectives. This is the approach used in JUBILEE. Themes were identified, which emerged from the year 1, cycle 1, fieldwork data. The focus was on optimising conditions which would result in the seamless and effective use of EIS. In year 2, cycle 2, of the project, the Toolkit was tested and refined, and was extended to the FE sector. No research of this type has previously been carried out in the FE sector, and it has been valuable in establishing a baseline of such work.
The rationale for the development of the Toolkit was described in the cycle 1 report as follows (Banwell et al., 2000):
At its most general level, the information provider's goal is to create a situation where users, academics and students, will use information seamlessly, irrespective of type or source, with confidence and trusting in its quality, in support of their learning and leisure. An individual's success criteria for information searching will be linked to achieving that goal personally. Evidence from cycle 1 fieldwork in JUBILEE suggests that the prerequisites for achieving this goal are for users to perceive they are in a situation where:
—Access to information, including EIS, is easy for all users
—The resource base (technical and human resources) itself is good to excellent
—Users have the technical and evaluative skills to use information, including from EIS
—EIS are embedded in course design and delivery, and in the research process
—EIS are embedded in student learning
—Quality assurance processes are in place for internet-based information
—Seamlessness is achieved [in all aspects of service delivery to the user]
These user-based critical success factors became the basis for Toolkit themes, which provide the framework for driving change, around which a development path is being constructed as the project progresses. Barriers will be encountered by users, creating dilemmas for individuals and institutional managers. Enablers will provide solutions and will permit movement along the development path. Institutional and discipline contexts and constraints will determine the exact nature of the development path appropriate in each individual and institutional context.
In the cycle 2 report it was noted that the development of the JUBILEE toolkit was building on a recent study (Banwell et al., 1999) undertaken in the School of Information Studies at Northumbria University for the UK Office of Library Networking (UKOLN). In the 1999 project, Managing Organizational Change in the Hybrid Library, a development matrix for hybrid libraries in higher education was produced. In this project five development stages were identified (Baseline, Change, Congruence, Embedding, Full Integration) and were applied to themes to produce development paths. The themes had been identified through consultation with experts and through fieldwork and were:
Benchmarks were produced to indicate the point at which the Library and Information Service could move to the next level of development. These findings provided context for the JUBILEE project, and a similar analysis, undertaken with specific reference to EIS, helped form the basis for the JUBILEE Toolkit. Benchmarks to indicate the point at which EIS information behaviour in the organization moves from one level to the next would be increasingly characterised in subsequent cycles of JUBILEE. Generic characteristics were described for each development stage, derived from fieldwork evidence, and were applied to JUBILEE cycle 2 data. These stages are presented below in table 5.
By building up exemplars of good practice in different disciplines at different sites, evidence is being presented of baseline, intermediate and advanced development in the use of EIS. Each theme is being developed in detail to provide a development path showing the interaction between barriers to development, dilemmas for resolution and enablers, which could be employed at the local level to overcome the barrier.
The JUBILEE project is testing and developing this methodology in the context of EIS. Based on cycle 2 evidence collected in both higher and further education institutions, the project team was satisfied that the methodology from the UKOLN project is transferable to JUBILEE, in both types of institution.
|1. Baseline. Starting point, status quo.||No systematic practice, sporadic surveys, historical picture, not a
Understanding of broader issues.
Introduction of new systems to adapt to/accommodate change.
Some early decisions made.
|2. Change. Point at which there is recognition of a need to change.||User needs, objectives, service targets established. |
On-going methods of satisfying needs etc. devised e.g. working parties, informal strategy document, service evaluation (surveys, staff involvement).
Resource use monitored.
Relationships being strengthened.
|3. Congruence. Stage at which the vision is starting to be implemented.||Provide and monitor effectiveness of skills training.
Institution wide policies and standards established.
Collect and optimise statistical and qualitative data.
|4. Embedding. Appropriate partnerships are developing from congruence and are accepted as part of the culture.||User education and monitoring, restructured framework for needs assessment.
Partnership between students, researchers and staff.
Widespread implementation of strategies and policies at all levels.
Ensure meeting of targets.
|5. Full integration. All the diverse elements are assimilated signifying maturity and the ability to fully exploit available potential.||Individual behaviour recognised and satisfied.
Generic frameworks adopted and adaptable.
Student and user-centred strategies, involvement in the curriculum.
Appropriate multidimensional systems in place.
Continuous performance measurement.
From Cycle 3 of JUBILEE on, the Toolkit has been developed in practice. In addition to the main, on-going fieldwork activity of the project as part of the longitudinal study of EIS, further longitudinal data are being collected by undertaking small-scale re-visits of case study sites from earlier cycles to determine whether evidence exists of any changes in EIS and information seeking over time. Interviewees are asked to reflect on the evolution of EIS provision and use by charting and analysing their past and current positions, using the Toolkit outline (see table 6) and a summary of their institution's case study key findings. Used together, these allow greater detail to emerge, highlighting examples of significant areas that sustain or hinder EIS development at the institution.
|Theme||Stage||Potential Characteristics||Your Position in:|
|2000/ 2001||2002/ 2003|
|Access||1||Inadequate user to PC ratio, with limited scope for remote access to EIS.|
|2||User to PC ratio being improved and users' access needs beginning to be considered in detail.|
|3||IT infrastructure adequate to meet needs of most users, and strategies being developed to improve EIS access both on and off campus.|
|4||Access to EIS facilitated via multiple avenues, enabling users to use EIS both on and off campus. Extensive opening hours of IT areas.|
|5||Highly developed provision and access policies meeting EIS needs of users, whether on campus or remote to it. Continual process of review.|
|Library/Academic Liaison||1||Limited, uncoordinated communication between library staff and academic staff|
|2||Relationships being strengthened.|
|3||Some reciprocal presence on library and academic boards and committees.|
|4||Effective partnership between students, researchers academic and library staff.|
|5||Ongoing partnership and collaborative working to maintain and improve EIS awareness/use.|
|Training||1||Any EIS skills training fragmented, no formal statement of responsibility, little user needs assessment.|
|2||EIS training available, with increased 'joined up' thinking regarding how provided and who provides it.|
|3||Provide and monitor effectiveness of skills training.|
|4||User education and monitoring, restructured framework for needs assessment.|
|5||Consensual LIS/academic institution-wide training programme, tailored to best fit user needs. Any such training subject to frequent review.|
|Integration of EIS||1||Little awareness of EIS available and no strategy for embedding EIS in learning/teaching.|
|2||Awareness of appropriate EIS developing, and some localised use of EIS in individual courses.|
|3||Maintained current awareness, and informal strategies being developed to embed EIS.|
|4||Wide awareness of EIS, and embedding into courses and curricula by groups of academics.|
|5||University wide student/user-centred embedding strategies, curriculum involvement and continuous performance assessment.|
|User Behaviour||1||Reliance on printed sources of information, with little use or awareness of relevant EIS and limited IT literacy.|
|2||Printed resource use prevalent, but with greater use and awareness of EIS among some groups of users.|
|3||Use of both print and EIS material, increased IT literacy, but limited awareness of EIS other than Internet search engines.|
|4||Increasingly integrated use of different media, and a critical awareness of EIS married with the acquisition of information seeking skills.|
|5||Individual behaviour recognised and satisfied, (with seamless use of the most appropriate resource, be it print or EIS)|
Another objective of the re-visits is to gain qualitative feedback and evaluation from academics and library staff as to the usefulness and validity of the Toolkit. There was a positive response from research respondents regarding the use of the Toolkit to measure changes in EIS attitudes, provision and use:
Involvement in the project is a good way of reflecting on these things for a little while, what we are doing, so it is good for us to be involved too. (Library staff member)
Fieldwork sites are providing detailed evidence of their involvement with JUBILEE resulting in positive changes in their institutions in areas such as:
The JUBILEE Toolkit is currently being populated with more data, from more disciplines and more institutions. By the autumn of 2003 it will contain data from around 25 higher and 40 further education institutions, with twelve broad disciplines included. It will have been validated both through re-visits to JUBILEE fieldwork institutions from earlier cycles, and through dissemination throughout both higher and further education. The original aim for the Toolkit was to be a tool for improvement. It is becoming clear that it is also a tool for the measurement of impact in institutions: the impact of EIS on student information behaviour, and the impact of the JUBILEE project itself through the promotion of EIS in institutions, with the use of the Toolkit as a staff development tool (Banwell et al., 2003).
The Toolkit is proving itself to be a discriminating method of monitoring change in institutions and in individuals' information behaviour in relation to electronic information services, and, in turn, use of the Toolkit is itself helping to improve practice in institutions and by individuals.
Themes underlying the paper and presentation are summarised as:
As JUBILEE matures as a project, now in its fifth year, with increasing amounts of longitudinal evidence available, the Monitoring and Evaluating User Behaviour Framework will increasingly be usable by the JISC to gauge the impact of electronic information services in the post-16 Education sector in the United Kingdom.
|Find other papers on this subject.
© the author, 2004.
Last updated: 1 December, 2003