Source
r/Twitch•
koskeller

AI that watches your stream and highlights interesting moments

Discussion

UPD Dec 2:

Thank for your feedback and all your upvotes. We have hit our goal and are going to start development immediately. I will keep this thread updated with latest news.

Original post:

Hey everyone! We are working on AI that can create stream highlights in real time. The idea is very simple in that we are going to use computer vision algorithms to watch stream, detect interesting moments and cut them into twitch clips automatically (we can deliver clip links to stream chat, so your followers can share them instantly). Afterwards it can create a video with all highlights and upload to Youtube, etc.

I wonder if it would be valuable for you? If you think so, please vote or share your thoughts in the comments. If this post hits 100 points, we will make every effort to make this happen asap.

P.S. It may sound a bit futuristic, but we did a small proof of concept and it looks promising. For those who are interested in the technical details, we are using OpenCV and ffmpeg to work with video stream and machine learning to recognise patterns.

Posted Dec 1, 2016, 1:13 PM UTC · edited

14952
PiiSmithhttp://www.twitch.tv/piismith•

How can it find the interesting parts? This is a hard problem to do as a human. I would really by surprised if any machine can do it.

14
PMmeyournaveltwitch.tv/pairagamers•

I presume it would detect when there is an increased amount of comments in chat, or something to that effect. Still a great concept!

16
RetroGaijintwitch.tv/retrogaijin•

Hmmm. I do a golden kappa check once or twice a day that gets chat moving 10x the speed of normal. It would probably highlight that moment for me. Maybe if you filtered out the word "Kappa" from the list of chatlines that the bot uses, then it would sound promising!

3
MoosecoopGaming•

topkek

1
koskellerOP•

We can detect game events, such as when a player kills someone and analyse chat log in the same time. If there was a kill and everyone is laughing in the chat, it may be signal that the moment is interesting.

8
PiiSmithhttp://www.twitch.tv/piismith•

OK, so you are basing this on chat activity. It might work for a channel with many or at least enough users to make this a good measure. Also it could be a problem that the chat activity was not triggered by the stream, but rather by chat itself. Like troll posting something, which caused a lot of answers, though there was no highlight in the stream itself.

5
JPili•

He also said it analyses game activity too. I'm not sure how it does that but I just wanted to let you know that it looks at more than just chat.

3
koskellerOP•

Chat is only one of the pattern sources in this case. The other three are the following: game UI, sound and overall picture (colour histogram, object distance, movement, etc.).

1
bondjaybondtwitch.tv/TheKillsburyPwnBoy•

I'm curious to see how this works with host raids and donation trains.

3
fruitbatsstwitch.tv/fruitbats•

We can detect game events, such as when a player kills someone

Do you need to manually program the AI for what constitutes an interesting moment on a game-by-game basis? "A player kills someone", after all, does not look the same in every game.

1
koskellerOP•

Yes, some patterns may be trained only manually, but some may be generalised for all games.

2
Tritemaretwitch.tv/Tritemare•

Very valuable, please contact me via tritemare at gmail when you are ready to test it. I have hundreds of highlights and have a somewhat slick system to help moderators make them that speeds the process up, but it is more like a script and less like an AI program that auto detects.

5
nikitheligerNikitheliger.com - Twitch Marketing Enthusiast & Growth Hacker•

This would be really interesting to see! :) One addition that many streamers totally forget when posting something is hashtags. If the AI would detect the game and use that as a hashtag + of source mention that specific streamer. So he/she knows about clips posted it would certainly benefit quite a lot of folks :)

5
koskellerOP•

Did you mean Youtube hashtags? That's a brilliant idea! We can fill hashtags with meta information, like game names, maps, game modes, heroes or champions, etc.

2
nikitheligerNikitheliger.com - Twitch Marketing Enthusiast & Growth Hacker•

I actually meant for Twitter hashtags. They are reeeally powerful for marketing purposes. So when the actual clip would be shared for the chat that people can share for example Twitter: Something #Game by @streamer + link. It would greatly help out on the marketing aspect. Youtube information sounds really nice also!

3
koskellerOP•

Got it! Very good idea.

1
warrboytwitch.tv/walters954•

Are you guys beta testing this service? I would be interested in trying it out.

3
koskellerOP•

We are going to test it soon. I can invite you if you wish.

2
NotKarateKidd•

This is also very valuable to me as editing would be a lot easier for YouTube posts

1
warrboytwitch.tv/walters954•

Sure I would like an invite.

1
koskellerOP•

I see the problem that after streaming session you have a 6 hour (for example) video-file that has only a name. There's no navigation or meta information that can help navigate through the video and watch a particular moment. Is this the problem for you? How many of your followers are interested in past content like streams or highlights?

3
Haughingtontwitch.tv/haughington•

Honestly I think I'd be happier with something that just lets me press a hotkey to create a timestamp so that I can easily find a moment later. I'm not tellin' you to scrap the AI idea, just saying that this little way to do it manually would be a great addition.

2
koskellerOP•

What about meta information? For example, which champion did you play, on which map, who were your opponents or what was the final score?

1
Trololoo•

This has the potential to be one of the greatest streaming tools ever! I wish you the best of luck with your production! If there is anything I can do to help with testing let me know!

3
0x1027twitch.tv/DeckerzMc•

What would be nice is something like a program that can be a hotkey to make a clip of say the last minute. So if something happens, i push a key say alt + \ and it creates a clip and saves the URLs to a text file.

That way as a streamer if i want to save something i dont have to tab out of the game

2
Sigma476twitch.tv/sigma476•

what about something like shadow play? IIRC shadow play will keep a buffer running you can save. press a hotkey and save the past 3 minutes and the next two to use as a clip

1
0x1027twitch.tv/DeckerzMc•

Yeah but that involves editing and uploading. whereas this is instant

1
mirr0rballtwitch.tv/eskea•

I'm curious too! How will you filter all? :o

2
0xkeentwitch.tv/0xkeen•

Huh. Where are you using OpenCV? For the chat logs, I assume you're using NLP (and can probably short-circuit on busy streams when a short list of terms appear rapidly in a duration), but are you targeting regions of the screen (e.g. the killfeed on overwatch or cs:go)?

Let me know if you open source it -- I'd be curious to take a peek.

2
koskellerOP•

Yes, we are using OpenCV to analyse colour histogram (to detect teamfights in games like dota and lol) and heroes pictures. Tesseract OCR to analyse UI text elements.

4
0xkeentwitch.tv/0xkeen•

Ah cool. Tesseract is great -- I use it in one of my applications and found that passing the resultant corpus through a spellcheck (with an augmented dictionary for online and context-specific terms (e.g. "lol", "kappa")) was breathtakingly accurate.

Cool idea, thanks for sharing.

4
Bitlife•

When you say it watches your stream, do you mean it's a program running locally on your machine and never actually enters chat or your channel? Or that it actually counts as a viewer in your channel? If it's the latter, I'd be more worried about the violation of ToS as that would count as a viewbot. Even if it is just one viewer.

2
Bitlife•

I'd really like to hear if this ai counts as a viewer in the channel. Like I said, if it does, that's a violation of ToS and could get the streamer using it banned if they're caught.

1
koskellerOP•

Thanks for this information. Yes, it is going to be like a viewer on a channel. I'm going to talk with Twitch team about ToS. Hopefully our case may be an exception.

1
Weebjager•

the future is now what the fuck

2
Hobi_Wan_Kenobi•

I'd be interested, for sure. But as an entertainment streamer, my question is: would it know when I've done something funny or made a joke? Or is it more of a "got 3 kills in quick succession" kinda thing?

2
koskellerOP•

That's an interesting point. We can definitely recognize speech easily, and in some cases machines can understand what are you talking about. But, jokes are one of the most complicated things for machines. We can try to use other patterns to recognize that moment, like a chat reaction. We will think about it. Thanks for this insight.

3
Hobi_Wan_Kenobi•

No problem. Glad to help.

1
xPennyz_•

This would be amazing.

1
IsChodan•

I would like to try this. If you guys need some testers, let me know!

1
PalidMisttwitch.tv/PalidMist•

Sounds like an impressive endevour. I look forward to seeing more information and/or a demo/beta.

1
itsgamerdoctwitch.tv/itsgamerdoc•

something that would be amazing

1
sippycupgamestwitch.tv/sippycupgames•

Interesting concept. Would like to invite the Twitch version of SkyNet to watch/clip me.

Seriously, would be happy to test it out. My concerns echo those of others, I'm sure. What if something cool happens to a dead chat room? What if you are fairly eccentric in stream anyways, how would it differentiate visually something awesome vs something that is ones norm?

On the note of dead chat, if something cool happens and clip is dumped in chat, you (the streamer) lose that clip from bot timers or the like, could the AI also PM the link to the streamer? You wouldn't otherwise see it in "Your Channel Clips" as clips need 10+ views before seen by channel owner, so theres an idea for avoiding clip loss.

1
koskellerOP•

Chat is only one of the pattern sources in this case. The other three are the following: game UI, sound and overall picture (colour histogram, object distance, movement, etc.). We are still thinking about epic moments without game events and boring moments with a lot of game events. I hope we can find a solution and train machine properly ))

1
koskellerOP•

Yes, we will keep them in our base for cases like this, so you can navigate easily.

1
sippycupgamestwitch.tv/sippycupgames•

Fair enough. I think this is a really cool and ambitious idea. Would love to see it come to fruition.

1
edwinksl•

That seems like a ton of computational resources that you have to dedicate to training and using the models.

1
nipplepotamustwitch.tv/dogeofwarr•

It would be nice to have an AI I can link to a command or something where myself or a mod can use it, much like Xbox recordings, that'll put the last "x" amount of time into a clip. This amount then can be changed to however much time I want

That would be nice

1
GuardianAngelzTTV•

I'd be very interested in trying this out too - I've a small, but growing channel, highlights and stuff happens - but chat activity isn't always astounding. if you want a diverse testing group - if you wanna focus on making it work, I understand :)

1