TWin of Online Social Networks Deliverable D5.2 Definition of Metrics

This report proposes a set of metrics to determine the deliberative quality of discussions on social media in general and TWON-platform from the TWin of Online Social Networks-project in particular. Chapter 2 lists the key indicators of: Exposure to political content; Engagement with political content; Contributing political content; Diversity of exposure; Quality of exposure. Chapter 3 explains how and why this set of indicators differs from the typical list of deliberative indicators and proposes to view deliberation from a summative rather than an additive perspective. The traditional additive perspective simply argues that to achieve societal deliberation, people need to deliberate. The best way to strengthen deliberative democracy would thus be to organize good deliberation, which is then ``added'' (-> additive) to the societal debate. Often, organizing such deliberation is challenging at a large scale, especially on social media platforms. In contrast, the summative approach does not require any one venue of good deliberation to achieve deliberative democracy at a societal scale but rather aims to realise it only at the collective, i.e. summative level. In this view, social media do not need to aim at perfect deliberation within one platform; rather, the goal is to contribute to deliberation at a societal scale via the platform. We propose that social media can contribute especially by offering an avenue for users (citizens, journalists and politicians alike) to be exposed to political debate, but also to engage and participate in that debate. In addition, social media can connect otherwise unconnected users and expose them to ideas they might otherwise have missed. Ideally, these ideas are substantiated with arguments and evidence. Chapter 4 evaluates a large set of automatic classifiers to determine the degree to which social media comments meet several deliberative criteria, specifically whether comments are rational, interactive, diverse and civil. Results show how more modern techniques like fine-tuned transformers and generative large language models have improved our ability to reproduce manual codings automatically, but also that results vary considerably between models.Chapter 5 integrates the aims of Chapter 3 with the results of Chapter 4 and translates them to the case of TWON to arrive at the metrics proposed in Chapter 2. It adds tests of the performance of different classifiers to determine whether a comment is political or not. Chapter 6 takes a look into the future, beyond what is currently feasible for TWON, to explore whether new techniques can help determine the deliberative quality of online social media debates to the more fine-grained level of specific claims and shows some promising first results.