Engagement Analytics: discussion logs as virtual tapas bars for your unit

Does your unit feel like a ghost town?

If you are like me, you’ve probably wondered a fair bit how to get your students to hang around a little longer in your part of the academic countryside.

You know — that bit beyond the social media, pokemon, part-time work, and night life city limits: the little town you’ve set up with colourful bunting hanging from every power poll, a welcome banner over the road into the city centre, and dozens of thoughtfully constructed venues along the main street, enticing a would be learner to engage with your topics.

Sure, your students might take the bus to your little town once or twice a week - a lecture here, a tutorial there, but inevitably, once the show is over, the mic is turned off, they seem to flick open their smart phones and check their transport app for the first bus outta there. (Don’t they care that we know?)

In educational terms, the issue at stake is, of course, ‘engagement’: how engaged are my students with my topics? How can I increase their level of engagement — real engagement (not the, ‘I’m going to delete random parts of the lecture materials so that you have to turn up’ engagement)?.

In this piece I want to describe briefly my approach and then spend most of the time on analysis: 'engagement analytics’, we might call it.

Building an always-on tapas bar

About 7 years ago I set on the idea of using online discussion forums as a sort of always-on tapas bar in my unit’s little town. The idea being, if I could get the students to hang around at the bar, they’d not only be directly spending more time in the town — with others from the unit — they’d also be more likely to explore the other venues I’d created next door, or across the street, thereby indirectly increasing engagement across all of my unit’s attractions.

My discussion logs would be the natural ‘hang-out’ for my students. And like all good bars, time would perhaps melt away, the conversations and thought-lines continuing on into the wee hours, students eventually stepping back out on to the street nudging each other at what a great time they’d had, eager to get back to the bar next.

To do this, I get my students to 'sign up’ to the 'discussion log’ task in week 1. I then allocate them via an algorithm into 5-6 student groups, and, by Tuesday of week 2, open their group's discussion log area,  setting them the task of first summarising and then discussing about 2-3 relevant texts per week. Faculty (myself and a dedicated discussion log tutor) have access to all groups, but for any student, their group is all that they see. This approach avoids the anonymity of the crowd. My tapas bar has private tables, encouraging trust, openness and good, old fashioned, philosophical pub talk.

Marking is achieved via three ‘checkpoints’, after weeks 4, 9 and 13 (stuvac), each worth 5% of their grade, 15% in total. Marks are awarded individually on the basis of effort, engagement with the task, and the quality of their posts.

Measuring the bar: Engagement Analytics

First up, we’ll look at the posting activity over the semester. In the plot below we show the total posts, total words contributed, and average words per post, per week of the semester. Posting only begins in week 2, and ’S’ stands for the stuvac week (week 13).

First up, the impact of the three checkpoints are obvious to see, especially as the semester wears on. Checkpoints 2 and 3 see a ramp-up in activity. Checkpoint 2 is especially pronounced, due, I suspect, to the students having two written assessments, due in weeks 6 and 9 respectively, worth 35% in total. Their effort is presumably drawn to these other assessments in weeks 5-8 and then they hit the discussion logs hard during week 9 to ‘catch up’. There is no restriction on when they should be posting on which article, even though I do list them by week. I prefer the bar to be full of all kinds of topics inter-weaving and overlapping — nobody likes the bore who commands you to talk about their chosen topic and nothing else.

Next, it’s interesting to see the ramping up in the average words per post over the first 4 weeks, before settling into a remarkably stable 300 or so word pattern for the rest of the semester (excepting week 10 where I’m sure they are writing quick posts to coordinate the final weeks of readings while reeling from their assignment work). One could conclude that the 300 word average is pointing to a regular ‘paragraph’ of thought.

However, if we look at the distribution of post sizes, we can see that it is actually very broad, and non-symmetric.

What’s going on here is that I intentionally look to cultivate leadership in each group by suggesting that one student take responsibility for each text, opening up a new discussion thread for the text with a summary of the text mixed in with a personal reflection on it. Ideally, the first post includes some challenging views or reflections that will get the group going. In terms of post length, the first post is typically much longer — sometimes as much as 1,500 words as can be seen in the figure above (students are often apologising, “Sorry this got so long!!”).

Now remember, the entire assessment is worth 15%. So how much effort are students expending? You can see in the next figure that a remarkable 856,052 words were written by my 136 active students in 2016. That’s a median contribution of around 6,000 words over the semester, with some students contributing over 15,000 words. Of course, the text here is less formal than essay prose, but the point remains that the level of contribution here is astonishing.

So far, so engaged. But have I really got them coming to my town all through the week? Or are they treating the discussion logs like any other assignment, cramming the night before, or only playing my ball game right after the lecture?

Here’s the distribution of posting activity over the week, otherwise known as ‘punch-card’ analysis.

When I first saw these stats, so many iterations ago, I knew that I’d shifted the game entirely. I had my students coming to my unit’s town every day! Sure, they do a little less on the weekend, but through Monday to Friday they are collectively checking in to my unit every single day. To do this, they are logging in to Moodle, pulling up the forum, starting a new thread, or checking in to an existing one. The mechanics of this mean that they are necessarily walking by dozens of other links, videos and downloads that I’ve got on Moodle. A little window shopping, anyone?

But is this activity concentrated during each day?

The short answer is again, ‘no’. Aside from the hours between 3 and 6am, the students are getting online and joining in. In the above plot I’ve highlighted the region which falls within the ‘university timetable’. It’s obvious that students don’t see the discussion logs as a ‘work hours’ thing. They are just as likely to be online at 10pm on a Thursday night as they are at 10am on a Monday morning (my lectures in this semester were Mon 8-9am and Wed 8-10am (yes, I know!)).

OK, so we’ve established that the tapas bar is pretty successful at both increasing, and smearing out, attendance at my imaginary unit’s town. Students are contributing phenomenal effort, are highly engaged, and seem to be having addictively interesting discussions with each other at their little tables.

But can we learn more?

Well, yes. I’m a complexity scientist, and so, I’m pretty interested in networks. Let me show you...

Thread analysis: how others respond to your pub talk is revealing

Suppose that you were a silent observer at one of these imaginary tapas tables. Suppose that you were interested to know which of the guests was really driving the conversation? Which one was the ‘last word’ kind of guy? Which guest, when making a contribution, seemed to electrify the other guests causing them to go off in several eddies of conversation?

At the table, getting at this kind of information would require a very large note-pad, and an astonishingly fast note-taking style.

Thankfully, because my students are visiting a virtual tapas bar, ‘conversing’ through textual posts online, the cookies of their conversations are being gathered all the time, automatically. All I needed to do was find a way to crack open the cookie jar. .. A good coding challenge for a raining morning at the office.

Let me show you a couple of thread diagrams to get you thinking.

In the visualisations, we represent each student by a number. So in Thread A above, we’ve got 5 unique students in the conversation, with student 1 kicking off the thread (circled, at top), and replying within the thread a further five times (row 3). We use an arrow between two students to indicate the direction of response. For example, the first arrow from student 1 (at top) to student 2 (row 2, at left) indicates that student 2 responded to student 1’s opening post.

In all, thread A has 22 posts, and you can see from the visualisation that actually there were at least around four main lines of conversation that developed after the first post. In fact, we can think of the width of a visualisation like this as a measure of how multi-faceted the discussion has been. In this case, we’d say that at most, row three is 10 wide: 10 concurrent  lines of discussion occurred. Moreover, we can look at the height of the visualisation to think about conversation depth. Here, we’ve got a maximum of 7 rows of exploration, the 1-2-1-2-3-2-4 conversation on the left hand side. Depth in this sense doesn’t mean that other branches of the thread were more or less ‘deep’ (in a Bloom’s taxonomy of learning kind of ‘deep’), just that the conversation in other branches of the thread didn’t carry on for as long.

Now, let’s contrast Thread A with Thread B below. (Note: student numbers do not refer to the same actual students.)

Using the same approach as before, we thread B has maximal width of just 4 (row 5), but has 3 conversation branches which have made it to a depth of 6. On average, this thread has a depth of 5.75, whereas Thread A has an average depth of just 3.7.

It’s easy to see that notions of width and depth capture salient features of the threads, at least in this visualisation. But can we use this sort of analysis to actually get at the quality of the conversation?

My answer is yes. Or at least, thread features do not arise randomly, they carry latent information.

Think about it for a moment, student 5 appears just once in Thread A, but their single contribution sparked four replies! Contrast this with student 4 in Thread A, they contributed twice (once in response to student 5, and once to student 2), but on both occasions no-one carried the conversation onwards. Whilst we’d be wise not to draw strong conclusions from a single thread, if, on average, this pattern continued across a semester for students 4 and 5 in this group, we’d be highly likely to infer that student 5 is making far more thought-provoking, impactful contributions than student 4.

Or consider another angle, in Thread B, this student 5 made the first post, and has managed to spawn three major parallel discussions, each of considerable depth. Now, again, we’d be careful not to quickly ascribe the strong depth response to their post from a single thread, but if, on average across the semester, when this student 5 posts to the group, conversations of high depth predominate, we’d be likely to conclude that student 5 sets up intriguing, powerful, thought-lines that their group members enjoy chewing over.

The underlying premise here is that the way that other students respond to a post carries useful information. This is no different to recommender systems used on Amazon, or the powerful page-rank system of Google search: how a network of agents/actors (web-pages, products) interacts with each other carries rich information about the kind of agents/actors we have in question.

Unit engagement leads to private dining for 5?

We started by wondering about engagement. We ended in a small-town tapas bar, listening in (and analysing) private table conversations.

As you can probably see, online discussion logs are not only a remarkable tool for enhancing engagement, but also provide near endless ideas for analysis. In this post, I’ve tried only to whet the appetite, and cover some of the basics of what I’ve tried.

If you’d like to find out more, please get in contact.