Source
r/Jeopardy•
nihlaak

Trivia Trainer (my attempt at Roger Craig's Jeopardy knowledge tracker)

Hey everyone, I wanted to share a training site I made for myself and get feedback on how I can make it more useful.

www.trivia-trainer.com

DEMO GIF. This gif shows the basic functionality, including what the stats page looks life after you've answered some questions.

 

  The initial motivation for this is Roger Craig's knowledge tracking site. My original goal was to link all of the questions and answers to Wikipedia articles so that after I miss a question I could make a reading list of Wikipedia articles. I'd like to eventually add these Wikipedia inks to the Review page.

Some other things I want to add:

  1. More intelligent question selection. Within each category it's basically random right now. I'd like to change it to use spaced repetition and not re-show questions you've gotten correct.
  2. Suggestions for what category to study. If a category occurs a lot and has a high average dollar value but I don't know many of the answers then I have a lot to gain and should spend time there.
  3. Improve the top-level categorization. I automatically clustered all Jeopardy clues and came up with these 53, but I think they can be refined and made more accurate.
  4. More filters for the quiz. E.g. specify the dollar amount.
  5. Track performance over time. E.g. see that my performance on "Shakespeare" has increased over the past month.

Please let me know if you have any ideas about how to make the site more useful!

[Edit: formatting the list correctly]

Posted Oct 13, 2017, 11:55 PM UTC · edited

4550
echothree33•

I just tried this quickly without logging in but it looks pretty amazing and very useful!

6
nihlaakOP•

Cool, thanks! Let me know if you have any suggestions!

3
brickhanson•

Really impressive work!

I have privately been working on my own version of this for a while, and yours is so much better than mine! Because I have been in the weeds with this for a while, I have so many questions. To start:

  • I am very impressed with the quality of your categorization. Did you use a third-party API or do the clustering on your own? I used MeaningCloud to do do text classification into IPTC classifications, and then wrote a simple routine to sort those into one of 16 categories (the same 16 that Roger Craig used). However, I feel like clustering is the way to go. It’s what Roger did. I just have no idea how that works.
  • Do you have a way (or have you considered a way) to track progress over time?
  • I would love an interactive bubble chart, like Roger Craig shows in his talk. He uses Google Visualization Motion Charts, but I’m pretty sure those are deprecated (I couldn’t even get them to load). I used Google Charts, which works well, but doesn’t allow a slider to show progress over time. Vizabi (https://github.com/vizabi/vizabi) has all the features I would like, but the current alpha version is not ready for prime time.
6
nihlaakOP•

Thanks! What have you built? Have you primarily been working on the categorization part? I'd love to see what you have!

I am very impressed with the quality of your categorization. Did you use a third-party API or do the clustering on your own?

I did it using the Python scikit learn library. I created one document for each Jeopardy category on a given day and its clues (so the category text and the text of all the answers and questions for the 6 questions). I did the clustering in two steps. For the first one, I just tried clustering all of those documents. I think I used something like 100 clusters, and then looked through the clusters. It worked pretty well, but there were plenty of useless categories. After cleaning up the categories, I made another pass where I took each unclustered Jeopardy Category and tried assigning it to one of my already existing categories.

PM me if you want to see the code.

Do you have a way (or have you considered a way) to track progress over time?

Not yet, but I'd definitely like this at some point! I store enough data to make this easy, so the work is basically just getting the charting library set up. I'd love to use something Highcharts, but it's expensive (and I'm not making any money from this so I can't really justify paying so much).

I probably have time over the next few months to implement either stats over time w/ some graphs OR some kind of Jeopardy study guide using Wikipedia. Which do you think would be more useful?

3
brickhanson•

Thanks for responding! Later this week, I will try to PM you with what I have. This is my first coding project ever, just for my own educational purposes, so I have been learning as I’ve been going. At this point, I’m just figuring out how to put my server on a real server rather than just localhost.

Mine is designed to mimic playing the game as much as possible. A TTS voice reads clues out at the same average pace Alex does (I measured 🤓). After the voice finishes reading the clue, a light turns on indicating that you can now buzz in, then it measures your reaction time. If you buzz in too soon, it penalizes you with a delay, just like on the show. To practice, I bought a signaling device, which connects via USB, that is very similar to the jeopardy buzzer. After that point, it’s similar to yours/Roger Craig’s. You can reveal the answer and then mark whether you got it correct or not. Whenever you finish a session, it shows you the bubble graph with your category strengths/weaknesses for the session and some statistics, like average reaction time. It also estimates your Coryat Score and what you game score would have gotten before Final Jeopardy in a game against players with average reaction times.

I definitely think stats over time would be more useful, but tell me more about your study guide idea. Like Roger Craig discusses, one of the more useful ways studying comes through spaced repetition. And there is no need to reinvent the wheel there, because there are such amazing apps like Anki already in existence.

A few people in this thread (and other forums) ask for spaced repetition for Jeopardy clues, but that seems like a waste of time for me, because the chances are slim that the random clues you get will be the most useful ones to know. My approach to studying is that I have been slowly converting/updating the Chuck Forrest book “Secrets of the Jeopardy Champions” into an Anki deck. It focuses on the most crucial pieces of information to know for Jeopardy.

I know more can be done with text mining the clue archive for common themes and responses, I just have no clue how to get started with that stuff.

3
nihlaakOP•

That sounds like a great project! My goal was to get a super simple UI working so that I could start tracking my stats, but I really like your approach of making it more realistic. How'd you build your version, is it a website? Is it up anywhere?

For the Wikipedia study guide, the goal was to study up on areas (e.g. "Geography") by reading about all the relevant Wikipedia articles. I wanted to feel less like I'm memorizing facts and more like I'm just reading.

2
doitneatly•

I have been slowly converting/updating the Chuck Forrest book “Secrets of the Jeopardy Champions” into an Anki deck. It focuses on the most crucial pieces of information to know for Jeopardy.

Cool!

I have a pretty good (I think) Anki deck that I put together based on some manual j-archive searching plus u/mystcmage 's excellent "pavlov" data discussed and linked in this jboard thread

I'll let you know once I get my Anki deck cleaned up and shared on ankiweb.

Are you planning/willing to also share yours?

1
mystcmage•

I'm also impressed with the categorization. I would love to see the code.

It's one of the major improvements I need to make to my training app: https://play.google.com/store/apps/details?id=com.mystcmage.android.flashcard

3
nihlaakOP•

Sure, PM your email address and I'll send you a gist.

2
doitneatly•

add me to the list of impressed parties!

I just signed up and poked around briefly, I’ll PM you any specific feedback I might have.

6
nihlaakOP•

Great, thanks!

2
skeltor57•

I'm glad you're looking at implement spaced repetition, that's going to completely elevate the site to a whole new level. Might there be a way to filter out questions that predate a certain year? Watching older jeopardy episodes(pre 2000), there's a lot of question I know wouldn't be asked on modern jeopardy.

5
nihlaakOP•

Cool! How do you think I should integrate spaced repetition? Do you think it'd work within the current UI (e.g. I just make sure to re-show a clue N days later) or would you want a way to review previously answered questions? I could try to make that easier by emailing clues that you've missed, but I really wouldn't want to spam people.

I loaded all questions from j-archive, but currently it's set to not show questions from before 2000 (for exactly the reason you said). Do you think that's good enough?

5
skeltor57•

Yeah, it's a tricky conundrum how exactly to best implement spaced repetition. I think your idea of re-showing the clue after N days is a good one. I'd love to hear other's redditor's better ideas though.

Secondly, I think you could allow the user to create an amalgam of selected categories to pull from if they want to focus on a hand full of different categories, not just one or all.

As far as not showing clues before a certain year is concerned. I think if you added more flexibility, maybe in the form of a slider allowing you to select how far you wanted to go back or check-boxes for each year, that would be perfect.

Also, if somebody really likes a clue, maybe give them the option to "finish the category". I've encountered numerous fun clues that have made me hungry for the remaining ones in the category with no easy way to access them.

Lastly,I'm not sure if you're aware of this, but clues involving videos or pictures are not displaying ATM.

Lot's of potential, can't wait for you to flesh it out more.

5
nihlaakOP•

For "finish the category", do you mean the specific jcategory? Right now it's possible to specify a particular category (e.g. Part of Speech. So I think you're proposing putting a link on the quiz page to this specific category? Right now those links are only surfaced on the stats page.

For the year slider, did you run into questions you found annoying/out of date? Do you remember any examples?

Thanks for flagging the video problem. I had started disabling all clues with the text "clue crew" or "seen here" or "heard here", but I hadn't had a chance to push it yet. Good to know someone else found it annoying too!

Lot's of potential, can't wait for you to flesh it out more.

Great, thanks! I built this for myself, but it'd be fun if others kept using it.Realistically how often do you think you would use this? Never again? Once per week? I wouldn't take offense-- just curious how interested people are in something like this.

3
EatMoarToads•

Realistically how often do you think you would use this?

For me, daily, but that is because I am currently in the pool. Not sure otherwise.

4
Runandwin•

I'll definitely be using this site, very nicely done.

5
nihlaakOP•

Thanks! Anything in particular you found useful or that could be improved?

4
Runandwin•

The stat tracker is really helpful. I can't think of anything to be improved for now but I'll message you if I find anything.

4
nihlaakOP•

Cool! How do you find it useful?

2
EatMoarToads•

This looks fantastic! Would absolutely love performance tracking. Would be interesting to compare by dollar amount and also whether questions were DDs or FJ.

My only issue so far was that one of the clues I had was a "Kelly shows this diagram on the monitor". That might be next to impossible to code in what you're doing, but sure would be helpful if you could add a link to click on that brought up the media.

4
nihlaakOP•

Thanks!

Can you say a little more about the "compare" part? Are you thinking something like "your correct rate for $200 is 70%"? How would you want it shown in the UI?

I got all the questions from j-archive, and it looks like all the media links are broken. For example, one of the Clue Crew clues links to this video, which is a broken link. I'm not sure how to get in touch with someone at j-archive to see where the content is, but for now I'll just disable these questions.

2
EatMoarToads•

Can you say a little more about the "compare" part? Are you thinking something like "your correct rate for $200 is 70%"? How would you want it shown in the UI?

You know what, maybe skip that and don't worry too much about the presentation.... tbh I am not exactly sure what I want to see right now and what would be useful. Maybe just let the user download a .csv of their history and let them figure out what they want to do with it. As your user base grows, you'll have better feedback as to what type of reporting people want to see in the app.

One thing I think is missing in the answers is something to click when you know the answer but didn't come up with it in J! speed. Maybe that's the same as "Don't know" for all intents and purposes, but I think it's a good distinction if your goal is to get an honest feel for how you'd do on the show and how to improve.

3
nihlaakOP•

Yeah, definitely. I've been using "Don't Know" for both "I have no idea" and "I couldn't come up with the answer in time". I try to force myself to answer in a few seconds, and just click "Don't Know" if I can't come up with it.

What do you think a good name would be for the button when you're not fast enough?

2
OhmyLord247•

What about 'Guess' and 'Don't Know' Before the answer is revealed.

3
jduck23•

You might just be my new favorite person. I love it!

3
nihlaakOP•

Thanks! I mostly built this for myself, but did spend a good amount of time putting it online so that's nice to hear! Just curious: did you answer many questions or check out your stats? If so, what felt useful?

I feel like people who saw this on reddit so far have been positive (and I've seen that quite a few people registered). Do you have any advice on how to show more people?

5
jduck23•

I haven’t gone through it a ton yet, but my first impression is good. I like the amount of categories you have. It’s very well organized.

3
TheChosenJuan99•

This is a dumb and kinda irrelevant thing, but a night mode would be super nice.

3
nihlaakOP•

Haha, no ideas are dumb or irrelevant! When would you find this useful? Do you use mobile or desktop?

3
TheChosenJuan99•

I've found myself using the site at night on mobile a fair bit, and it's a bit jarring with the white backdrop taking up a decent portion of the screen. Otherwise I absolutely love the website though.

3
nihlaakOP•

Cool, I'll add it to the TODO list. (Although I think I'll get to non-UI sooner.)

3
TheChosenJuan99•

You da best.

2
augustt198•

Keyboard shortcuts for the buttons would be great

2
nihlaakOP•

Definitely, I want that too! I'll add it to my TODO list.

1
OhmyLord247•

Is there a way for you to tell me if I was 'Correct' or 'Incorrect' in my guesses? I feel like multiple choice would ruin it but I still feel a little unsatisfied by clicking 'correct' 'incorrect' or 'i dont know.' There are times in Jeopardy where I would risk and guess more and times I wouldn't. Maybe if I click 'I don't know' before the answer is revealed?

Sorry if this confusing. I think I just would be more satisfied with the trainer if there was more accountability.

2
nihlaakOP•

I responded to your comment above about making "I don't know" it's own button-- I like it!

2
OhmyLord247•

Also, great job. I love it :]

1
Tpxdmd•

This is stunning and fantastic. I registered and played several rounds.

I've been trying to build something similar for a while and I don't have the technical skills and am just frustrated.

I did end up sitting and manually categorizing questions in a CSV, and I've limited myself to 18 top-level categories and then a number of subcategories to make the tracking easier. And then found I can't track because I can't get to this!

This is brilliant. I wish I had technical skills to offer real help.

2
nihlaakOP•

Thanks! Were you building something just for fun, or are you trying to get on Jeopardy at some point?

Let me know if you want any advice on the technical side.

1
Tpxdmd•

For the show, should I ever get called after an audition...

I wouldn't mind any advice on pulling the questions from the J-archive. I managed to do it once, and then after some time went by I tried to get the ones since the last pull and I could not figure out how to get it to work again.

2
nihlaakOP•

I just used these scripts I found on Github.

1
squiderror•

Signed up and played for a few minutes and love it! I like the simple design on mobile.

Edit: ignore what I said previously in my comment; I'm not common-sense smart at 2am.

2
nihlaakOP•

Great, thanks!

1
squiderror•

hopefully images get integrated soon!

2
nihlaakOP•

I'm planning on just removing those questions for now, but I definitely would look to handle these well eventually.

1
questions123abc•

are you only pulling questions that fit the categories you made? It seems like I keep getting the same questions over and over.

I'm also not going broad questions, for example I haven't had any questions about geography yet, which seems strange.

2
nihlaakOP•

Yeah, it's only questions from the categories I made. The idea was that these are the knowledge areas that have a lot of questions, so it's better to focus on those than ANYTHING that can show up in Jeopardy.

But if you're getting the same questions that's no good. Can you link me the URL you're using? Is it just www.trivia-trainer.com/quiz, or are you selecting a category (e.g. http://www.trivia-trainer.com/quiz?category=geography)?

1
questions123abc•

Ok got it. It seems strange though that I haven't had things like State Capitals, Presidents, Potent Potables etc. since that shows up a lot in Jeopardy.

I love the tool you built! And thank you for sharing it. I think if you could add more questions that are outside of the categories you made it could be even more useful.

1