Learning analytic devices – co-forming, re-forming, in-forming
Theresa Dirndorfer Anderson and Simon Knight
Introduction. This work-in-progress paper explores the intersection of theorising in human-data-interaction, information studies and learning analytics as part of a discussion about the role informative artefacts play as agents of learning.
Method. The artefacts crafted by learners through collaborative work in two different classroom context are considered both as representations of and representations about learning.
Analysis. Framing analytic devices crafted through collaborative work in these classroom examples as boundary objects draws attention to their value as carriers and constructers of ideas within and beyond the classroom.
Results. The fluid, transient nature of the activities contributed to their value as informative artefacts in individual and collective sensemaking. Through the constant refreshment and reinvention of the material forms that students exchange with one another (and ultimately with their instructors) information is produced.
Conclusion. By playfully allowing for multiple means of interaction, the artefactual agents in the two examples create a range of multimodal action possibilities as material and informative artefacts. The paper invites further conversation about these possibilities and the valuable "social life"(Brown & Duguid, 1996) of analytic devices that shape the ways that learning is understood and enacted as objects of assessment.
This work-in-progress paper explores the intersection of theorising in human-data-interaction, information studies and learning analytics as part of a discussion about the role informative artefacts play as agents of learning. In building on previous work about the sociomateriality of information practices, creative ecologies of learning and learning analytics, we offer a conceptual explanation for classroom activities where we sense that the artefacts being produced are players in the learning. Using this explication of the role played by these informative artefacts in the individual and collective sensemaking of our students illustrates how the act of representing has value for learning. Making visible the qualities of representation and inscription in analytic artefacts contributes to the understanding of how the knowledge embodied in these artefacts comes to be represented and communicated in any given context (or within any given community). In the rest of this paper we introduce learning analytics, highlighting the synergies between concerns for ‘algorithmic accountability’, and work in the information science domain. We articulate our work with respect to two short classroom-based examples, before concluding with points for further discussion.
Learning analytics as an information agent
The last five years, from the first learning analytics and knowledge conference in 2011, have seen a growing interest in the use of data from learning contexts, to support the learning occurring in and through those contexts. The field of ‘learning analytics’ has developed as one response to this challenge, making use of the increasing amount of data from learning interactions in (often online) software platforms to gain insight into learning. The field is thus emerging as a discipline occupying a ‘middle space’ (Suthers and Verbert, 2013) between learning, computational, and information sciences. This bridge between learning analytics and information studies offers opportunity for a transdisciplinary conversation about the dynamic, hyper-coordinated spaces within which people create, curate and use information.
Thus, in learning analytics, transdisciplinary work is engaged to bring to bear the tools and theorised approaches of each domain on the boundary object of the learning target. For example, a growing interest in learning analytics is in its potential for the analysis of processes of learning (providing student feedback on dialogue, for instance) and the social and discursive constituents thereof (Buckingham Shum and Ferguson, 2012). In such analysis, each domain brings to bear its own theoretical and empirical perspectives on the problem, for example, how learning dialogue is to be conceptualised, structured, and investigated. Elements of each analysis will be domain-specific – providing, for example, particular methods or constructs emergent from one domain or another. However across domains a shared understanding of the core constructs and methods must be built, to develop a ‘transdisciplinary’ model of an analytic device that is grounded in a shared perspective. Thus, it has been argued (Knight and Buckingham Shum, Forthcoming), that these devices in their operationalisation of constructs into technologies and contexts commit to particular assumptions about learning. Indeed, it has been suggested (Knight, Buckingham Shum, and Littleton, 2014) that these commitments can be thought of in terms of a triad of epistemology, pedagogy, and assessment (which learning analytics are a form of), such that devices implicitly or explicitly commit to particular stances on what it is to know, how one learns (or comes to know), and how that knowledge is assessed.
Learning analytic devices as inscribed informative artefacts
In this paper we thus argue that learning analytic devices become information artefacts inscribed with particular commitments that both shape, and become shaped by, the learning contexts in which they are deployed. That is, analytic devices both shape the ways that learning is understood and enacted as objects of assessment, and are interpreted, reinterpreted, and acted with as a dynamic part of that very context. This broad discussion has noted that code ‘acts’ in education (Williamson, 2015), such that:
as algorithms are increasingly being designed to anticipate users and make predictions about their future behaviours, users are now reshaping their practices to suit the algorithms they depend on. This constructs ‘calculated publics,’ the algorithmic presentation of a public that shapes its sense of itself. (Williamson, 2015, p. 30)
Thus, the ways in which analytic devices become active agents in learning – both inscribed with policy and practice commitments, and enacted or enactive informative artefacts – has led to calls for greater ‘algorithmic accountability’ (Diakopoulos, 2014), to ensure that the pedagogic aims of analytic devices are transparent across a range of stakeholders.
One means through which algorithmic accountability can be approached is to make the invisible more visible, by acknowledging the representational work of coding and classifying embedded in an analytic device, for instance. This articulation work, which Suchman and Trigg (1993) also refer to as craftwork, draws attention to representational devices as central actors in the structuring of practice. Building on this theoretic tradition, Anderson (2007) argues there are two conditions of human experience essential to acknowledge when studying information practices in context:
- The situated, embodied character of human experience; and
- Active examination and analysis of the socio-material (and increasingly socio-technical) context.
The legibility of analytic devices
Conveying learning analytic information across stakeholder audiences with their respective skills and needs (from individual students up to institutional leaders) is a challenge, requiring consideration of collaborative sensemaking (Knight, Buckingham Shum, and Littleton, 2013). As approaches such as learning analytics become increasingly available, the need to explore human interactions with this data/information grows, with fields such as ‘Human Data Interaction’ (building on work in human computer interaction – HCI) emerging to explore how to "support end-users in the day-to-day management of their personal digital data..." aligning with our own view of data as of an "inherently social and relational character" (Crabtree and Mortier, 2015, p. 1). In such approaches, interactions with analytic devices would be seen as a "distinctively socio-technical problematic, driven as much by a range of social concerns with the emerging personal data ‘ecosystem’ as it is by technological concerns, to develop digital technologies that support future practices of personal data interaction within it" (Crabtree and Mortier, 2015, p. 3). Of particular interest to our concerns is the notion of ‘data legibility’ (Crabtree and Mortier, 2015, p. 18), a concern to understand the ways in which personal data is made legible through visualisation and processing, to support people in understanding and investigating their own data.
Analytic devices, as objects that both shape and are shaped by learning contexts require complex analyses to make them legible to learners and educators. To do so, analysis of the theory and operationalisation behind any given learning-target (for example, a particular form of dialogue), alongside the methods for collation and feedback, should be given. Moreover, agents should understand how their data-feedback is both an ends of the analytic, and a fundamental shaping component in the analytic device. Thus, from Suchman’s theorising (2007; Suchman and Trigg, 1993), we recognise that a ‘situated action’ unfolds in the doing, in interaction with the circumstances that make that situation. The close relationship between product and process in document work is thus increasingly recognized by researchers in the information and learning sciences. Lund’s broad defining of a document refers to:
… any results of human efforts to tell, instruct, demonstrate, teach or produce a play, in short to document, by using some means in some ways, is very focused on activities around making documents, in other words on practices (Lundh, 2010, p. 744).
In information studies, researchers like A. Lundh (2010; A. H. Lundh and Dolatkhah, 2016), Trace (2007) and Sköld (2013) note that this activity must be understood as constituting both physical and mental activities at both individual and collective levels of practice. Building on this notion, we have begun to consider how analytic devices are brought into meaningful interaction in learning contexts, as both a shaping part of those contexts and a shaped output from them through minded acts of documentation. Thus, in the earlier example, the analytic devices both shape the feedback on dialogue and are shaped by the dialogue. We thus argue that analytic devices become inscribed in their unfolding cultural context.
Informative artefacts as representation of and representation for learning
With this paper we wish to open discussion about the collaborative meaning making that lends value to these informative artefacts as agents of learning. To do so we draw on insights gleaned from two examples from our classrooms, briefly outlined here.
From tabletop to desktop: studying technology
As part of an undergraduate subject focussed on social studies of technology, students worked in teams to explore one of six emerging technologies under study that semester. In individual and collective activities staged over six weeks, students discussed their discoveries in relation to their team’s assigned technology. Classroom exercises the first weeks were deliberately analogue. Using sticky notes, markers and paper, students performed ideation work via analogue techniques like sketching and doodling to mediate discussion on each week’s theme.
The decision to encourage analogue ideation for a time was deliberate for two reasons:
- as a way to emphasise the co-evolving, bi-directional influence of technologies on human practices; and
- as a technique to slow down thinking and make the articulation work of individual and collective sensemaking more visible.
The fluidity of each teams’ (and each individual’s) representation of the topics under discussion informed the in-class and between-class activities. As the ideas took shape in relation to the students’ assessment tasks, this work became increasingly digitised within an online interactive whiteboard web application. This collective work (in the form of a series of analogue and digital posters and sketches) was shared with all members of the class, serving as source material for the artefacts each student produced and submitted for assessment.
From big data to dear data
As part of a masters level subject focussed on data science and innovation we are planning an exercise in the quantified self, to orient our students’ considerations of ‘big data’ to the personal, representational, and qualified in a manner similar to that discussed in Anderson and Martinez-Moldonado (2016). The project mirrors the ‘analogue drawing project: dear data’ (www.dear-data.com), in which two visual designers send hand-drawn personal-data postcards to each other (see, Lupi and Posavec, Forthcoming; and some preliminary thoughts in education at Knight, 2015). Thus, students will be asked over a period of weeks to collate data on a theme, by whatever means they wish, and visually represent this data for sharing. To date the exercise has been protoyped informally with small groups of students ahead of formal, full-scale implementation in the subject. These early experiments with the activity suggest that by encouraging students to articulate the data collection and representation through hand-crafted artefacts, we can draw attention to:
- The space of possibilities in representation – highlighting the variety of ways in which the same thematic data might be collated, segmented, and visualised.
- Representational interactions – by engaging with each other’s representations, not only is the range of potential spaces highlighted, but the necessity of human sensemaking, explication or qualification, on a personal level.
- The performativity of information representation – that representations are created for a purpose, that they are situated in that purposeful context, but that they also act on it to frame discussions and actions (in this case, both through raising awareness of the data one is collating about oneself, and through the sharing of these personal-data artefacts).
As with the first classroom example, students can transition this deliberately analogue exercise into digital visualisations as their work progresses.
Communal conversations centring on the production and interpretation of deliberately analogue artefacts are fundamental to the learning design of each of these exercises. The significance of the artefacts formed and reformed in the activities described above is illustrative of the valuable social roles that Brown and Duguid (1996) attribute to documents and their assertion that, "…we need to see the way that documents have served not simply to write, but also to underwrite social interactions; not simply to communicate, but also to coordinate social practices." It is through the deliberate and visible act of producing and representing that the students learn about the subjects under study. We speak of the social life of these artefacts because their formation is in partnership with the learners’ and yet separate from their control within each of these classroom contexts. The significance of activity and the movement of ideas, as represented in the material forms students craft, are critical to each learning community’s conversations, giving these documents a social as well as an informational role. Through the constant refreshment and reinvention of the material forms that the students exchange with one another (and ultimately with their instructors) information is produced. The artefacts produced in each classroom are instrumental in getting students to make their thinking more visible to themselves and to others (peers and teachers), confirming Brown and Duguid’s contention that looking at the content alone cannot explain their value.
Framing the informative artefacts crafted through collaborative work in these classroom examples as boundary objects allows us to appreciate their socio-materiality and value as carriers and constructers of ideas within and beyond the classroom. As has been discussed elsewhere (see, for example, Anderson, 2007; Huvila, 2012) applying the concept of the boundary object (Star and Griesemer, 1989) to the study of information practices helps to make the sociotechnical contexts of the making, using, sharing and curating of information visible. Critical is the appreciation of the role boundary objects perform as translational devices. Because the making and nurturing of boundary objects initiates and advances shared understanding, they are important agents of learning in the classroom. Here we illustrate two contexts where those boundary objects tool very material, analogue forms. Through conversation about and circulation of the analogue representations of ideas in both of these examples, students form their views about the topics under discussion in each respective classroom/subject.
The emergent, evolving boundary objects -- shaped in the slowly unfolding, collaborative activities within the two examples presented here -- help the individual learner to make their own kind of sense of the activity in question in concert with a growing communal understanding. In these examples we can recognise and value both information and learning perspectives. Information studies helps us value the collective forming and reforming of these material artefacts as information practices and the minded acts of documentation. Learning analytics helps us explore these artefacts as agents of learning, encouraging us to identify ways that the individual and collective practices of information curation, creation and use they enable shape the ways that learning is understood and enacted as objects of assessment.
Conclusion: the sociomateriality of learning
The recursive link between data and information discussed by Crabtree and Mortier (2015) compels us to pursue further discussion of analytics (and the objects containing them) as information. Both of our examples embed an unfolding co-creation in the learning activities where data is both an object and a player. Consequently, the analytic device reshapes the context in which the interaction takes place and reshapes the artefactual agents. Therefore, the ‘same’ data may be presented at various levels of a stakeholder diagram (from individual students through to teachers), at various times, while representing varied information through a range of possible manifestations. Through ‘playing’ with the artefacts being developed in each classroom example presented here, students come to engage practically with the space of possibilities and meanings in representation; commensurately, each artefact itself may be seen as a player in learning context.
We are deliberate in our choice of the term ‘player’ when describing the evolving artefacts crafted through learner interaction in both of the examples as players of learning. In earlier work presenting classrooms as creative ecologies of learning, Anderson (2013) discusses the value of nurturing the creative capacities of students through classroom strategies that make a deliberate effort to build in reflective practice (pause) and opportunities to tinker with ideas and take risks (play) to become better at engaging with information in unexpected and exponentially changing ways. It is a very activity-oriented position that connects to discussions of the haptic value of drawing and doing in the classroom (see, for example, A. H. Lundh and Dolatkhah, 2016; Madsen, 2013). We see meanings unfold over engagement with these artefacts, or as Knorr-Cetina refers to them "epistemic objects" with their "unfolding, dispersed and signifying (meaningproducing) character" (2001, p191-3). Playfully allowing for multimodal interaction, the artefactual agents in our two examples create a range of possibilities: as material objects (representing a transition from analogue-activities to the encoded/inscribed-digital) and as informative objects (via their documentary status).
The desire to support playful engagement with the ideas at the heart of both exercises informed the fluid, transient nature of their design and, we believe, contributed to the learning value of the informative artefacts produced via such individual and collective sensemaking. Further discussion and exploration of additional learning contexts and analytic players shaping learning contexts will contribute a fuller understanding of the ways that creative unfolding shapes learning with, and from, these playful and informative agents of learning.
About the authors
Theresa Dirndorfer Anderson is an Associate Professor in the Connected Intelligence Centre at the University of Technology Sydney, where she is Course Director of the Master of Data Science and Innovation. She received her Bachelor's degree (honours) in Diplomatic History and Soviet/Russian Studies from the University of Pennsylvania (US), a Master’s Degree in International Relations and Security Studies from the University of Lancaster (UK), a Master’s Degree in Information Studies and PhD from the University of Technology Sydney (Australia). She can be contacted at: firstname.lastname@example.org
Simon Knight is a Postdoctoral Research Fellow in the Connected Intelligence Centre at the University of Technology Sydney, where he also teaches into the Master of Data Science and Innovation. He received his B.Sc (Intl Hons) in Philosophy and Psychology from Leeds University (UK), a Master’s Degree in Philosophy of Education from University of London (UK), MPhil in Educational Research from University of Cambridge and Doctor of Philosophy, Knowledge Media Institute (KMI), Open University (UK). He can be contacted at: Simon.Knight@uts.edu.au
- Anderson, T. D. (2007). Settings, arenas and boundary objects: socio-material framings of information practices. Information Research, 12(4). Retrieved from http://www.informationr.net/ir/12-4/colis/colis10.html (Archived by WebCite® at http://www.webcitation.org/6miJelebf).
- Anderson, T. D. (2013). The 4Ps of innovation culture: conceptions of creatively engaging with information. Information Research, 18(3) paper C28. Retrieved from http://InformationR.net/ir/18-3/colis/paperC28.html (Archived by WebCite® at http://www.webcitation.org/6miK6Gced).
- Anderson, T. D., & Martinez-Moldonado, R. (2016). Building a "qualified self" around lifecycles of experience and thinking. In For Richer, for Poorer, in Sickness or in Health...The Long-Term Management of Personal Information, CHI 2016 Workshop on Personal Information Management (PIM 2016). San Jose, CA. Retrieved from http://pimworkshop.org/2016/papers/PIM_2016_paper_11.pdf (Archived by WebCite® at http://www.webcitation.org/6miKJM7pd).
- Brown, J. S., & Duguid, P. (1996). The social life of documents; introduction by Esther Dyson. First Monday, 1(1). Retrieved from http://firstmonday.org/ojs/index.php/fm/article/view/466 (Archived by WebCite® at http://www.webcitation.org/6miKLr7sN).
- Buckingham Shum, S., & Ferguson, R. (2012). Social learning analytics. Educational Technology & Society, 15(3), 3–26.
- Crabtree, A., & Mortier, R. (2015). Human data interaction: historical lessons from social studies and CSCW. In ECSCW 2015: Proceedings of the 14th European Conference on Computer Supported Cooperative Work, 19-23 September 2015, Oslo, Norway (pp. 3–21). Springer.
- Diakopoulos, N. (2014). Algorithmic accountability. Digital Journalism, 3(3), 398–415.
- Huvila, I. (2012). Authorship and documentary boundary objects. In 45th Hawaii International Conference on System Science (HICSS), Maui, HI (pp. 1636–1645). New York, NY: IEEE.
- Knight, S. (2015, July 26). Dear learner. Retrieved from http://sjgknight.com/finding-knowledge/2015/07/dear-learner/ (Archived by WebCite® at http://www.webcitation.org/6miKxkbKF).
- Knight, S., & Buckingham Shum, S. (Forthcoming). Theory and learning analytics.
- Knight, S., Buckingham Shum, S., & Littleton, K. (2013). Collaborative sensemaking in learning analytics. In CSCW and Education Workshop. San Antonio, Texas, USA. Retrieved from http://oro.open.ac.uk/36582/ (Archived by WebCite® at http://www.webcitation.org/6miL3MjKb).
- Knight, S., Buckingham Shum, S., & Littleton, K. (2014). Epistemology, assessment, pedagogy: where learning meets analytics in the middle space. Journal of Learning Analytics, 1(2). Retrieved from http://epress.lib.uts.edu.au/journals/index.php/JLA/article/view/3538 (Archived by WebCite® at http://www.webcitation.org/6miLAqbh1).
- Knorr-Cetina, K. (2001). Objectual practice. In T.R. Schatzki, K. Knorr-Cetina, E. von Savigny (Eds.), The practice turn in contemporary theory (pp 184-197). London: Routledge.
- Lund, N. W. (2010). Document, text and medium: concepts, theories and disciplines, Journal of Documentation, 66(5), 734 - 749.
- Lundh, A. (2010). Studying information needs as question-negotiations in an educational context: a methodological comment. Information Research, 15(4), colis722. Retrieved from http://informationr.net/ir/15-4/colis722.html (Archived by WebCite® at http://www.webcitation.org/6miLK6bEw).
- Lundh, A. H., & Dolatkhah, M. (2016). Reading as dialogical document work: possibilities for library and information science. Journal of Documentation, 72(1), 127–139.
- Lupi, G., & Posavec, S. (2016). Dear data: the story of a friendship in fifty-two postcards. Harmondsworth, UK: Penguin.
- Madsen, J. (2013). Collaboration and learning with drawing as a tool. Teaching and Teacher Education, 34, 154–161.
- Norman, D. A. (1988). The psychology of everyday things. Basic books.
- Sköld, O. (2013). Tracing traces: a document-centred approach to the preservation of virtual world communities. Information Research, 18(3), C09. Retrieved from http://www.informationr.net/ir/18-3/colis/paperC09.html#.WE9hyWR941I (Archived by WebCite® at http://www.webcitation.org/6miLXUzpG).
- Star, S. L., & Griesemer, J. R. (1989). Institutional ecology,translations’ and boundary objects: Amateurs and professionals in Berkeley’s Museum of Vertebrate Zoology, 1907-39. Social Studies of Science, 19(3), 387–420.
- Suchman, L. A. (2007). Human-machine reconfigurations: plans and situated actions. Cambridge, UK: Cambridge University Press.
- Suchman, L. A., & Trigg, R. H. (1993). Artificial intelligence as craftwork. In S. Chaiklin & J. Lave (Eds.), Understanding practice: perspectives on activity and context. Cambridge, UK: Cambridge University Press.
- Suthers, D. D., & Verbert, K. (2013). Learning analytics as a "middle space." In Proceedings of the Third International Conference on Learning Analytics and Knowledge (pp. 1–4). New York, NY: ACM.
- Trace, C. B. (2007). Information creation and the notion of membership. Journal of Documentation, 63(1), 142–164.
- Williamson, B. (2015). Coding/learning: software and digital data in education. In B. Williamson (Ed.), Organizing algorithms in digital education (pp. 27–33). Stirling, UK: University of Stirling.
How to cite this paper
|Find other papers on this subject|
© the authors, 2017.
Last updated: 21 February, 2017