Mapping the development of user constructs of relevance assessment as
informed by topicality.
Department of Information Studies
University of Technology
This paper reports on research examining the dynamism of a Šuser' view
of topicality exhibited in the information retrieval experience of
individual searchers. Experienced researchers presenting complex search
requests to an electronic information retrieval system are observed. The
information retrieval activities of an individual searcher are situated
within the complex and constructive information seeking process. Viewed
holistically within this process, that individual's relevance assessments
are social constructions which are fluid, interactive and dynamic. To
explore relevance and topicality within the broader context of information
retrieval, the searcher's full experience must be recorded and evaluated.
This paper discusses preliminary findings from pilot testing attempting to
'capture' this broader context. Within a qualitative framework, I will
explore evolving views of 'topic' throughout the individual information
Relevance is a fundamental concept in information science which, despite
more than forty years of debate, is still not fully understood. Notions
like topicality, pertinence, or situational relevance have appeared in the
literature over time, but support for one view has often meant excluding
all others. For example, relevance is currently recognised as a
multidimensional and dynamic concept extending beyond the traditional
definition so closely associated with topic-matching and an identification
of relevance as the relation between a document and a search question
(Schamber et al, 1990). With this user-centred shift, topicality has been
dispensed with even though criteria derived from users in real search
situations show that subject content and related qualities remain
important. I believe that the notion of topical relevance is more dynamic
than thus far considered in much of the literature. Green and Bean(1995), for instance, contend every imaginable
My thesis will focus on the interactive construction of meaning
attributed to the 'topic' of an individual search. A 'symbolic
interactionist' framework will allow me to explore the context and the
process behind an action -- in this instance, relevance assessment. In this
way I hope to get closer to the human factors associated with information
retrieval and the search process. Symbolic interactionism recognises that
people are constantly involved in processes of interpretation and
definition as they move from one situation to another. Individuals shape
the meaning and action associated with their relevance judgements through
unique experiences and interpretations (Blumer, 1969; Denzin, 1989).
Meanings of 'relevance' change throughout an information search as a result
of their encounters with people, things and ideas (Cool, 1993). Using the
symbolic interactionist framework, I can view information retrieval as
social communication. Relevance will be defined as a "social
object" and relevance
Relevance, information retrieval and the search process
Information retrieval is viewed as a communication process in which
judgements of relevance are made. The judgements upon which my research
intends to focus are those made by the individual searcher performing
information searches in an electronic environment. Individual views of
'topic' are shaped through the information seeking process, making the
search a learning process (Blumer, 1969; Kuhlthau, 1991). The notion of
'topic matching' may be a useful method of identifying documents held
within a retrieval system. However, seeking out the human meaning of
'topical relevance' requires a better understanding of the learning process
associated with individual relevance assessments. The articulation of what
is 'on the topic' for a particular search and the selection of 'relevant'
information are influenced by interactions during the information retrieval
process as well as the broader information search process. This thesis
aims to develop a thorough exploration of those user constructs in the
Previous research of user-derived relevance criteria suggests that all
the criteria derived from the user perspective are inter-related (Barry,
1994; Barry & Schamber, 1998; Park, 1992; Schamber, 1991). These
earlier inquiries asked individual searchers to make judgements of criteria
directly associated with a document. Recognising the interrelationships of
relevance criteria, however, means relevance must be explored by charting
the whole information retrieval experience. From a searcher's perspective,
'topicality' is more about the articulation of a particular topic and the
information answering the specific inquiry. Thus, my inquiry examines those
interrelationships by focussing on user constructs of 'topic' without
reference to a particular document.
My research applies an ethnographic approach to study information
retrieval and record the 'experience' using human-computer-interaction
facilities. Given the unique character of each person and their individual
search, any attempt to understand how an individual derives topicality
priorities must explore all phenomena influencing that person's relevance
assessments during their information search. A naturalistic inquiry is
required to study the complex interplay of all elements in a real¨not
contrived¨context. Recognising that human activity is not context free, a
naturalistic inquiry seeks out all factors involved in interaction (Guba
& Lincoln, 1985; Erlandson et al, 1993; Mellon, 1990). Understanding
the meaning of 'topical' for a user will require a thorough understanding
of the context within which these individual decisions are made. In my
research, developing an understanding of changing topicality priorities
involves three elements:
- detailed explanations about each individual's approach to information
evaluation in a real (personal) situation (to elicit relevance assessment
criteria from users);
- direct observation throughout each
subject's search process; and
- observation of personalised
information seeking sequences within each individual's framework of
evaluation and use.
This research attempts to study relevance by absorbing and integrating
theories from associated disciplines and bringing to the foreground the
notions of human behaviour which have been shown to be so significant in
the user-centred view of information retrieval (Barry, 1994; Park, 1994;
Schamber, 1994; Sugar, 1995). Notions from computer science, ethnography
and sociology, for instance, enrich the exploration of human factors in
electronic information environments (Monk & Gilbert, 1995). Such an
approach suggests a bridge between research on the information search
process and user-centred relevance criteria. By applying other frameworks,
a new means for evaluating and critiquing relevance criteria can be
Work plan and progress to date
My candidature began in early 1997 and I am due to complete the degree
by December 1999. I recently completed pilot tests of my data collection
method. My work plan uses six-monthly reports as progress milestones, to
ensure that I can complete my research on time. The stages of research and
target dates for completion which I have hitherto met are: literature
review and research question (April 1997 - January 1998); establishment of
research framework (April - November 1997); pilot testing of method
(December 1997 - April 1998); data collection (Phase One, November 1998;
Phase Two, early 1999); preparation of transcripts, summarising data,
coding (November 1998 - March 1999); interpretation and analysis (November
1998 -October 1999); final revisions (November - December 1999).
I recently completed method testing which involved observing the search
behaviour of two individuals. This experience clarified the data collection
strategy, which is the focus of my work for the next few months. Asking
questions and engaging in conversation with participants about the
individual's information retrieval practices, decisions, thoughts and
reactions during searching proved enlightening and in fact essential to
understanding the context of the searches.
The first stage of data collection will involve detailed observation of
three participants. An emerging theory about user constructs of topicality
will be tested in the second phase of data collection with three or more
additional participants later this year. I am presently arranging access to
a group of experienced researchers who regularly conduct searches in
electronic environments. Their search process will be recorded from first
interactions with an electronic database through to decisions to take
actual items or documents away for further review or study. The interaction
will be recorded using video and audio facilities in a
human-computer-interaction laboratory. Screen activity will be recorded
electronically. Discussions with participants during their searching will
be used to develop the case studies about individual views of 'topicality.'
Questions will revolve around each person's criteria for selecting and
rejecting items judged 'relevant' by the retrieval system in use.
Detailed case studies in the first phase of the research will describe
not only the criteria used to select items but also assumptions (as
expressed by the research participant) that influence individual decisions
to select or reject an item. All changes in the search query and evaluation
criteria will be recorded and explored with each participant. Observation
of a search will finish when the subject decides to take a book out of the
library, photocopy an article, or download a document identified as
'relevant' to the search. The language of each individual searcher's
interaction with an information retrieval system will be used to describe
the process. By encouraging participants to 'think aloud' during their
searches, I hope to develop a better understanding of the individual
context of relevance judgements.
Applying naturalistic inquiry methods
A significant stage in the development of my research design has
involved establishing the 'natural' context of the phenomenon under
investigation and determining how and where data can be collected. In their
discussion of relevance, Sperber & Wilson (1986) note that
communication involves inferring meaning from the context in which it
occurs. Human-computer-interaction research and usability studies suggest
that observations of the system-searcher interaction in a computer facility
can capture a 'natural' context (Blomberg, 1995). While the individual
interactions with information on the screen and not the system interface
are the focus of my inquiry, attempting to 'capture' the full information
retrieval experience can benefit greatly by following the example of
usability research to conduct research in a computer search facility. A
workable replication of the 'natural' context of the individual search
experience can be created if:
- searcher-system environment is similar to the individual's usual
- search problem is a 'real' question for the
individual, not imposed by researcher;
- the task(a computer search)
is not performed in isolation from the process of information
While conceding that this environment cannot be considered the 'natural'
environment of the individual searcher, use of a computer search facility
appears to preserve the main elements of the searcher-system interaction.
It is particularly important that the individual is still able to pursue a
'real' search problem ű posing search questions and using databases of her
own choosing. Moreover, much of the physical context of the individual's
search can also be preserved in such a facility. Computer equipment can be
arranged to mimic speed and interface, for instance. Observing the
information search in this environment facilitates recording of search
requests and all possible modifications from the first request through to
the selection of 'suitable' information for searcher. Recording
interactions using the video and audio facilities of such a facility also
simplifies analysis of screen activity as well as discussions with
participants during their searching. This approach thus offers a greater
The current focus of my research has been devising a method for
observing how people interpret 'topic' during information retrieval.
Instead of exploring evaluation criteria directly associated with a
document, this inquiry aims to:
- examine the whole process of shaping the topic;
- describe the communication between system and searcher regarding that
- relate user constructs to a system description of
In a pilot test, conducted in June 1998, the data collection method
that seemed to address these aims was evaluated for suitability.
Attempting to 'capture' the full retrieval experience of the participant
involves discussions with a participant during the individual search. The
pilot was divided into four sections:
- interview just prior to the search
- search session
- post-test interview
- subsequent review of retrieved items.
The interview with the participant was used to learn what was 'known'
about the subject of the search and to gain insight into his experience,
expectations, and attitudes to electronic searching. The search was
conducted in a computer facility ű recording all activity on screen and all
discussions with the participant. The searcher was encouraged to discuss
thoughts, reactions and actions during the search process. In addition to
pursuing the individual's search process, the post-test interview explored
the searcher's reactions to the test. The review of retrieved items
involved asking about the items related to the desired topic. The first
three stages were conducted consecutively over a period of four hours. The
final stage was conducted at a later date.
Recording of this search session was linked to a usability software
package known as DRUM- Diagnostic Recorder for Usability Measurement. The
tool helps organise and analyse activities during the search session, by
facilitating the logging of significant 'events' during the test.
Preliminary evaluation of pilot test results suggests a valuable link
between naturalistic inquiry of information retrieval and established
usability testing methods.
Preliminary findings also indicated there are indeed significant
benefits to observing searches of individuals in this environment, namely a
computer search facility. The pilot participant, an experienced researcher
who is a frequent searcher of electronic databases, appeared comfortable
with conducting his search in the facility. The databases and interface
were no different from that of his office workstation. He was eager to
begin his search and appeared unfazed by the video and audio recording
equipment. In the interview and discussion after the search session, he
made no comment about feelings of discomfort or unease.
During the computer search session, the participant voiced his thoughts
and described his actions at stages of the search. In addition, he was
asked to explain individual actions throughout the session. Despite this
potential for disrupting the 'natural flow' it was also evident that the
dialogue between researcher and participant was necessary for identifying
the individual's thoughts, reactions and actions throughout the search.
During the search session, there were many occasions when the searcher's
evaluations and selection decisions were markedly changed. To better
understand this dynamism, it became imperative to ask the searcher to
explain actions and choices. This provided a valuable discussion about the
participant's search behaviour and selection decisions.
Capturing relevance assessment in the broader information retrieval
In my research, relevance assessment is related to
contextual factors of information seeking in electronic environments. My
exploration of the interaction of an information seeker with an electronic
information retrieval system extends beyond the initial interaction to
include the broader context of information retrieval ű one which situates
the activities observed within a search process involving much more than
electronic information retrieval of information. My preliminary findings
suggest that studies of user-centred relevance which address evaluation
criteria only after a particular information retrieval system has been
selected are not adequately addressing the concept. Many decisions about
'relevance' are made even before the search begins. To explore relevance
and topicality within the broader context of information retrieval, the
searcher's full experience must be recorded and evaluated. Recognising the
significance of context also suggests the importance of situating
individual information re
- Barry, C. L. (1994) "User-defined relevance criteria: an exploratory study." Journal of the American Society for
Information Science 45, 149-159.
- Barry, C. L. & Schamber, L. (1998) "Users' criteria for relevance evaluation: a cross-situational comparison." Information Processing and Management 34, 219-236.
- Blomberg, J. L. "Ethnography: aligning field studies of work and system design." In A. F. Monk & G. Nigel Gilbert (eds) Perspectives on HCI: Diverse Approaches. London: Academic Press. pp.175-197
- Blumer, H. (1969) Symbolic Interactionism: Perspectives and Method. Englewood Cliffs, NJ: Prentice-Hall.
- Bogdan, R.C. & Biklen, S. K. (1982) Qualitative Research for Education: An introduction to theory and methods. Boston: Allyn & Bacon.
- Cool, C. (1993) "Information Retrieval as Symbolic Interaction: Examples from Humanities Scholars." 56th ASIS Annual Meeting: 274-277.
- Denzin, N. K. (1989) The Research Act. Englewood Cliffs, NJ: Prentice-Hall.
- Eisenberg, M. & Schamber, L. (1988) "Relevance: the search for a definition." In ASIS '88: Information and Technology: Planning for the Second Fifty Years. Proceedings of the 51st ASIS Annual Meeting held in Atlanta, Georgia on October 23-27, 1988. Volume 25: 164-168.
- Erlandson, D.A., Harris, E.L., Skipper, B.L. & Allen, S.D. (1993) Doing naturalistic inquiry. A guide to
methods. Newbury Park, CA: Sage.
- Froehlich, T.J. (1994) "Relevance reconsidered - towards an agenda for the 21st Century: Introduction to Special Topic Issue on relevance research." Journal of the American Society for Information Science 45(3): 124-133.
- Gluck, M. (1995) "Understanding performance in information systems: blending relevance and competence." Journal of the American Society for Information Science 46(6): 446-460.
- Green, R. & Bean, C.A. (1995) "Topical relevance relationships. II. An exploratory study and preliminary typology." Journal of the American Society for Information Science 46(9): 654-662.
- Kuhlthau, C. C. (1991) Inside the search process: Information seeking from the user's perspective. Journal of American Society of Information Science 42(5): 361-371.
- Lincoln, Y. S. & Guba, E.G. (1985) Naturalistic inquiry. London: Sage.
- Mellon, C. A. (1990) Naturalistic Inquiry for Library Science: Methods and applications for research, evaluation, and teaching. Contributions in Librarianship and Information Science, Number 64. New York: Greenwood Press.
- Mizzaro, S. (1997) "Relevance: the whole history." Journal of the American Society for Information Science, 48, 810-832.
- Monk, A. F. & Gilbert, G. N. (eds) (1995) Perspectives on HCI: Diverse Approaches. London: Academic Press.
- Park, T.K. (1994) "Toward a theory of user-based relevance: a call for a new paradigm of inquiry." Journal of the
American Society for Information Science, 45, 135-141.
- Park, T.K. (1992) The Nature of Relevance in Information Retrieval: An Empirical Study. Doctoral dissertation, School of Library and Information Science, Indiana University, Bloomington, IN.
- Saracevic, T. (1996) "Relevance reconsidered '96." In P. Ingwersen, N.O. Pors (Eds) Information Science: Integration in Perspective ű Proceedings of CoLIS2. Copenhagen: Royal School of Librarianship.
- Schamber, L. (1994) "Relevance and Information Behavior." Annual Review of Information Science and Technology 29: 3-48.
- Schamber, L.(1991) Users' Criteria for Evaluation in Multimedia Information Seeking and Use Situations. Doctoral dissertation, Graduate School of Syracuse University, Syracuse, NY.
- Schamber, L., Eisenberg, M.B., Nilan, M.S. (1990) "A re-examination of relevance: toward a dynamic, situational definition." Information Processing & Management 26: 755-776.
- Sperber, D. & Wilson, D. (1986) Relevance: Communication and Cognition. Oxford: Basil Blackwell.
- Sugar, W. (1995) "User-centred perspective of information retrieval research and analysis methods." Annual Review of Information Science and Technology 30: 77-109.
Information Research, Volume 4 No. 2 October 1998
Mapping the development of user constructs of relevance assessment as informed by topicality, by Theresa Anderson
Location: http://InformationR.net/ir/4-2/isic/anderson.html © the author, 1998.
Last updated: 25th September 1998