I really enjoyed by the post u/Norishoe wrote last week analyzing the data from the live chatters. I work as a data engineer, so I was inspired to put together a similar dataset and look at it in more ways.

I put together a dataset of every single chat message from the past 50 live streams to visualize in a few different ways.

Here's a link to the dashboard

You'll find a few different dashboards there, which look like little tabs at the top of the view. I really recommend making the dashboard Fullscreen (button on bottom right) when viewing. Also, just a heads up, the dashboards don't render very well on mobile, there's just too much to fit on screen.

• Data Set Overview
  This dashboard shows some general stats about the data, to understand the context
• Top Chatters
  This is a sort of leaderboard of the people who've been in the chat. One view is by total number of chats, and the other is by chats per minutes. Also play around with the filters to see how it changes things, One that's really useful is "Message is all Emotes", it lets you filter out messages that are just emote spam.
• Top Emotes
  This view shows the top emotes that were used during the 50 live streams.
• Chatter Representation
  This view shows the percentage of chat messages written by the top chatters compared to the rest of the chatters. You can change the number of chatters considered "top" by changing the value in the top right box of the dashboard named "Top N Chatters". The default is 100.
• Show Breakdown
  I found it interesting to look at some stats broken down by show name (H3TV, After Dark, etc.). There's 3 of these views, the first normalizes metrics on a per video basis, the second normalizes in a per minutes (of video), and the last normalizes on a per video view basis.
• Messages Over Time
  This dashboard shows when, during the stream, people are chatting. The bars show the average number of chats sent at a given percentage of video completion time. This was the best way I could think to normalize it, since videos aren't all the same length. If you only select one video from the filter, however, you'll see the actual number of chats at that point in the video.

There's more data elements from this that I didn't end up exploring, but if there's interest I can put together some more views. The data is also downloadable from Tableau Public for anyone to use, so feel free to explore it for yourself or ping me on discord (captainshell) if you want to build something together.

Some Interesting Observations

• As u/Norishoe pointed out in their post, the top chatters (those who write the most in chat) make up only a small percentage of the total number of chatters, but a significant portion of the total messages sent in the chat. Across all videos in this dataset, the top 100 (0.3%) chatters, wrote 18.61% of the messages in the chat!

• If you look at the show breakdown (by any metric), you'll notice that there's far less engagement in the chat during Leftovers, but the chatters that do participate during that show tend to write longer messages on average. I attribute that to the Hasanabi heads (I'm one of them, which also explains the length of this post).

• If you look at the top emotes dashboard, you'll see that the tomato emote was the most used emote by far. While it is a pretty popular emote in general, it's mostly due to the fact that it was used nearly 12,000 times during the live show when Jimmy Lee did his bit. If you remove that show from the chart, the most used emote is the :red_heart:, followed closely by the :_h3h3Byedog: emote.

If you've read this far, thanks for reading, I hope you enjoy looking through the data!

Below this line you can read through all the tech details (if you're a nerd like me and are interested).

---

Data Aggregation

I'll briefly discuss how I collected all this data. I used the python package scrapetube for collecting past videos, I pulled data from 50 videos for this data set, and used the python package chat_downloader to pull the individual chat records from each stream. From there I wrote a script to flatten the records and wrote it to an Excel file.

Why only 50 videos?

Initially I wanted to pull together a lot more data for this, and I did pull more than 50 videos, but I ran into some technical limitations. First, I don't have a personal Tableau license, meaning I had to use Tableau Public, and the data sources I can connect to with that are pretty limited. I decided to go with an Excel file because it has the best relationship support among local file types. Unfortunately this imposes another limitation, there's a 1,048,576 row limit per sheet, so that's the maximum number of chat messages I could collect.

Ideally the data would be stored in a real relational database (I was going to use sqlite until I realized Tableau Public can't connect to it...), but I figured 50 videos was good enough. There's probably a workaround by splitting messages into multiple sheets and unioning them in Tableau, but I was too lazy to try that.

Data Elements

The collected data includes 3 tables (stored in individual Excel sheets)

• Videos - A table of records for each video in the data set
• Chat Messages - A table of records for each individual chat message
• Emotes - A table of records for each individual emote used in each individual message

Emotes in messages

I realized that a lot of messages were only Emotes, so I used Regex patterns to identify most cases where a message was just emote spam. I didn't filter those messages out, but I did add a flag to the data which can be filtered in the dashboards if desired. It's not perfect, it doesn't catch things like :) or unicode emoji, but it get's all the channel emotes and standard ones, so I felt it was good enough. For those curious, this is the regex pattern: (:[a-zA-Z1-9_-]+:\s*)+

Using the Data

You can download the Tableau Workbook from the dashboard should you want to do your own analysis. Tag me if you find something cool!