Pointsman scores TTR using camera+AI
Posted Apr 12, 2019, 8:09 PM UTC
31279
Every game requiring point tallying at the end should have this type of app. That's the future I dream of. But on Android instead of iOS.
31
Everyone's favorite part of every game. Tallying up points to determine who won. Yay addition!
3
My game group doesn't even track TTR points during the game, and just has me count everyone from scratch at the end of the game.
This is even for games I'm not playing. It's just that I'm faster at counting all 5 players points by myself, than all 5 players counting themselves at the same time.
I don't even like TTR
7
That’s how we do the point-counting in TTR. Unfortunately, it is not possible with the United Kingdom expansion. Learned that the hard way...
1
We have an entire ritual for counting the points at the end of the game. (We don't track during)
We sort our tickets from lowest to highest (finished or not) and go round the table one ticket per player. This adds to the suspense since the score jumps are increasingly bigger. We leap frog over each other until someone runs out of cards or has very high-scoring tickets left.
After that, the scores might be close (sometimes they're not). We count all the train routes starting from the lowest player. They'll jump the highest player and feel like they're ahead.
In the end it results in the feeling of the scores "being close."
1
My favorite part:
- tally up all the points
- find out you lost by 1 point
- sit there recounting the points while everyone starts cheering at your demise
- find out you miscounted and missed 10 points from your score
- look up at the winner and say "sorry bud, I have some bad news bears for you..."
2
Race for the Galaxy does, and I'm pretty sure it's on android. Not quite accurate enough for me, but cool all the same
1
Why not both? 🤷🏻♀️
1
I was excited until I realised it was iOS only. Seriously though, great app!
10
Oh man this looks amazing! Please do make an android version
6
Ever since I used the app that does this for Race for the Galaxy, which has existed for at least 5 years, I've wondered why this isn't more prevalent. I'm assuming it's because it's very hard to do.
5
Didnt that app get pulled?
1
I'm not sure, I know it doesn't work for me anymore though. I know it depended on another app to work, that may have been its downfall.
2
I love that you made it pretty and added some sounds to it. Looks very professional and clean, thanks for the hard work!
5
aww, thx!
3
Really love this. Great work on the image recognition, the tracking looks really sharp.
What tech stack did you use to build this? My first guess would Swift if it’s a native iOS app, but would love to hear about the deciding factors on language.
4
Swift, with a little C thrown in for some performance critical areas; my current preference is Swift + PlanetSwift + Anchorage + Laba. Beyond that it uses ARKit2, SceneKit, and CoreML. All of the models are custom and trained by myself; Python + Keras + TensorFlow.
All-in-all the project effort was about 80% ML and 20% app.
8
This is pretty cool.
Does it only work for the base game train colors? I have the translucent promo purple trains that I like to use.
2
I do not have the purple trains, so it is not trained specifically for those. It possible it might pick those up consistently as another color, if you give it a try let me know how it goes!
3
So how does this work exactly? I would guess the map is modelled as a graph and AI uses separate classifier for each edge, while route cards map to two vertices. But maybe I'm overcomplicating things :)
2
You're on the right track. Here's the 10,000 ft view:
-
Use ARKit/SceneKit to capture a consistent image of the game board (I take a running mean of 200 frames to smooth out any jitters / auto focus blurriness)
-
Use CIImage's CIPerspectiveCorrection filter to transform the image to a known size for processing
-
Using ARKit means at the time we have the game board image, which know which game board it is. I have a typical GameBoard class to house all of the shared rules, and subclasses for each supported game (GameboardUSA, GameboardEurope, etc...)
-
I assign each track on the board a trackID, each city a cityID, then I extract crops around each individual track and city. These are what is fed to the machine learning, which is housed by CoreML in the app. The Gameboard class keeps track of the confidence levels the ML infers for each track and city.
-
I run some basic error correction in Gameboard once the initial inference is completed, things like "if 3 trains of a 4 train connection are all red and one is black, make them all red" (although specific implementation is much more generic than that). This helps bolster the ML predictions and ensures what is scored is constrained by the ruleset.
-
As you guessed, cities and track connections are stored in a basic graph format in Gameboard; this combined with the error corrected ML predictions is then used to score everything appropriately.
-
After the trains are all scored, the player can then scan tickets. These are basically the same as scanning the board game but much simpler. ARKit to find the ticket, same running mean to get an image, a much, much simpler ML model to say which specific ticket it is. Gameboard stores cards to players, uses them in the scoring.
5
I’d have guessed that the route cards just map directly to the points they’re worth, and users only scan complete ones.
As for the trains: the developer has stated that it’s trained for specific maps, so I’d guess it just looks at where routes are expected be, and determines the colour. I’m not sure if it would have predetermined lengths but that seems likely given the repeat use of maps.
Source: just guessing Edit: was wrong! See below for explanation.
1
users only scan complete ones.
Users can scan all of their tickets, the app will determine whether they are completed or not and score them appropriately since not completed tickets results in negative scores.
It even scores Europe properly with its concept of stations. In Europe a station can only be used for a single connection during scoring, so the app will examine all possible combinations of routes based on tickets and everything to find the highest scoring combination.
3
Wow! That is very cool. Thanks for explaining it, I had forgotten about the fact you lose points when you miss routes too. Can’t wait to play and try it out!
1
This is brilliant, awesome idea and execution!
2
Nice work!
2
Just tried it out, had a few issues with trains being placed between double tracks but that was sorted out quickly. Made scoring 100 times easier and quicker!
Good job man
2
Is that a standard thing, putting the trains between a double track (assumably for games here you have too few players to use the double tracks)? If it is, I could look to add support for that in the future.
Thx! Do me a solid and leave a review on the App Store :)
1
I do it just to remind us that in 2-3 player games you can’t use the second track. Easy enough to move the trains onto a track when it’s scoring time though.
1
That’s really cool. Will download and rate it later :)
1
iOS 12 only.... shame
1†
It relies on ARKit2, which is iOS 12 only. I could have wrote it without ARKit2, but that would have been a lot more work.
11
This is so cool! Look forward to trying it out if/when it comes to android
1
This looks amazing! I can't wait to try it out on my next game. I have a question though...
Can you make it upload to BGG for tracking games played?
1
neat, I didn't know they had an API! I certainly could.
In the meantime, after you score a board you can swipe to the bottom of the score ticket and press the email button. This should open up an email with the game board and score ticket images in them, which you can share however you like.
2
There's a great app called BGG Stats that lets you log game plays and syncs it with your BGG account.
Also, do you think you could teach your app to score Carcassonne fields at the end of a game?
2
We are living in the future.
1
Wow!! Can you please adapt this to Between two castles of madking ludwig?
1
Between two castles of madking ludwig
Oh, that game looks cool! I don't have it but i watched the video. Certainly seems possible that I could.
2
Very cool family game with one flaw, counting points takes like 10 minutes. I think you ll find a great response! If you need photos, lists etc i can share.
2
It's done! Check out the Mad King's Steward on the app store.
3
Omg Sir, you are a genius!!! Did you share the app with the publisher? There is also fb about the game.
1
"AI". That term doesn't mean anything anymore. Sounds more like machine learning and computer vision
1
Looks cool, I thought about building something like this for 18xx games (1846) to calculate the route profits for the companies by filming the gameboard but I don't know enough about image recognition to know if it is feasible.
1
http://chimerasw.com/pointsman/
Full disclosure: I'm the developer. The app is iOS only, 100% free, and works for USA/1910, Europe/1912 and the Great Lakes.
Happy to answer any questions!
Noooooooo!!!
Any plans for an Android version?
Yeah, if the iOS version does well I can make an android version.
Awesome job! What sort of challenges did you face implementing this?
By far the largest challenge are stations on Europe boards. They are tall enough that from most angles they can cover up trains/tracks behind them. This can lead to confusing the AI during training, as the data sample for the track might be labeled “red train” but be completely covered by a green station.
Very interesting - are you able to have a person required to pan around the board a bit while you capture multiple images? Or is that too labour intensive training wise?
That was one of my original ideas actually, starting at like 15 degs to the plane and moving the camera up to 90 degs. The result though was that it just made capturing the game board difficult, without a whole lot of benefit.
The stations challenge is almost completely solved now. The standout issues and resolutions were:
As it stands now there are only two, single track routes that have a larger likelihood of being mis-classified due to a station, being single tracks there is not additional information to error correct with. Those are from Paris to Dieppe and Bruxelles to Amsterdam.
This looks awesome! How did you train the system, photos of completed games?
Yup, lots and lots of them. I’ve got a system down now, so for a brand new board I start with setting up a bunch of boards to ensure each tack has one of every train represented. Then I set up more boards through mock play for the validation set.
Will it work for the 10th anniversary edition?
It is not specifically trained for that edition. If the game board and/or tickets are too visually different than the "regular" edition then it will not work. I can train it to work with any of the games/editions, but I'd need access to the specific one for a number of weeks to perform the data collection & training.
I’m guessing it would probably struggle. It’s very similar but all the train sets are different and unique. Thanks though :)
Probably, but the app's free so if you want to give it a try and let me know if it just works I'd greatly appreciate it.
Can I donate money to you to get you to port this to Android?
Quite possibly. Give me a week or so to see how things go, but a Kickstarter to fund android support is not out of the question.
This is glorious. What was it that drove you to make this? Convenience? Challenge? Add an item to your resume? Burning passion for TTR?
All of the above. I started tinkering with machine learning about two years ago. During that time I did a number of little experiments like making an iPhone play a real pinball machine. However they were just experiments, not anything “production worthy” if you know what I mean. This is my first attempt at that. My family and I love playing games, and TTR is one that we love. Like others have said, keeping score while playing is just tedious. Now we play without scoring to the end, and use the app to save the 5-10 minutes of counts and recounts to play the next game.
That's awesome! I'm really impressed.
Why would it care what the map looks like? You can rely on the players to only have made valid placements, so you're only looking for blobs of color. I suppose it helps determine city locations, but most of the time a-b-c is not a straight line or someone else is also at that city, so you've got other ways to determine connection points.
I use ARkit the locate and identify the game board, allowing me to extract the game board image for processing by the ML. If the game board is visually different enough then arkit will not recognize it.
Does it work with non English versions ?
I do not own any non-english versions; why don't you give it a try and let me know how it goes?
I would gladly but I have Android
I’ve thought about doing this myself, but didn’t find the time to do this. I’m very happy you did this, so I “massively” supported it :-D
you are awesome friend :)
Good work. You beat me to my homebrewed version. Can score hive and scythe using a webcam on the board and a trained vision model. This looks more polished.
I was imagining this type of app when scoring Between Two Cities. Should be easy to recognize colored borders and algorithmically calculate the scores.
Looks great. Any chance of a smartphone version ?