oopsleonOP•

This is a project I've been working on for a few months, and I recently got a job offer to do similar work so all progress on the project is about to halt. Figured this would be the best time if ever to submit it here, so I'd love to hear any feedback/suggestions.

In addition to being a project exploring conversation models in TensorFlow, it is also a good reference for anyone looking to get a simple app deployed on a website using their TensorFlow models. I experimented with both Heroku and Google AppEngine but stuck with GAE.

For anyone looking to play with sequence to sequence models and/or write your own, this project would be a good reference (imo).

To those that may ask: yes, I have seen Google's seq2seq repository, and I came across it about 2 months into working on this. They did some similar work, but the difference (of course) being their focus on NMT instead of conversation. I also feel that the more specific goals of my project allow the code to be a bit more transparent than Google's seq2seq (less layers of abstraction).

Although I would not consider this project finished by any means, it has matured significantly since I started and I think it has useful enough content to share here. I would've waited longer to post but, as I said, it is unlikely I'll be doing any more serious work on it in the near future.

21
8solutions•

Thanks for posting!

3
snylekkie•

Looks good

2
hastor•

how does it compare to opennmt?

1
oopsleonOP•

I'm not too familiar with OpenNMT, but one large difference is that they use Torch instead of TensorFlow. They are also a lot more polished, e.g. their site says "OpenNMT is used as provided in production by major translation providers." My project certainly makes no claims of being production-ready.

Although my project allows someone without too much TensorFlow knowledge to get a model up and running, the people that would benefit most from it are perhaps people moving past "beginner" TensorFlow usage and looking for more intermediate well-documented examples. In other words, perusing the contents would probably be most useful for someone looking to implement/expand on its functionality in this project or their own.

3
juharris•

Great! I looked through your README but I didn't see anything stand out about if you just use the previous message like most chat bots or if you use more context. Could you describe the little more about that?

3
oopsleonOP•

Right it is indeed doing just that, but improvements like beam search were top of the "next-things-to-do" list. If by context you are referring to information regarding the encoder sentence, then yes the project supports Luong and Bahdanau attention mechanisms which can be specified via an attention_mechanism: LuongAttention under your model_params (config file or via command line args). I'll definitely update the README with this and other info I'm getting as feedback. Thanks!

If this didn't answer your question, let me know and I'll do my best to clarify more later. Any ambiguities/unclear parts of the project will be clarified and updated within the project!

2
oopsleonOP•

Okay, some of these conversations are pretty hilarious. Had to post some of back-and-forths I've seen. Rest assured I have no idea who y'all are, all I have are the user-bot conversation pairs and a label for the user if you were kind enough to give yourself a nickname. Thanks everyone!


UserBotty
are you smartRetarded!

UserBotty
Do you know what 1+1 is?Two nights.
almost very clever.Always enough.
UserBotty
Do you ever feel lonely?I said it would.
You personallyWhat?
Do you feel lonelyYeah!
Why?Deceased.

UserBotty
Who is president of USA?"banana solo"?
Who is banana solo?<link>

UserBotty
you fuck bitchBitch.
bitchWhat?
you are a fucking bitchEnough.

UserBotty
My name is Inigo Montoya.Excuse me.
You've killed my father. Prepare to die.The only cut.
Can you possibly fathom the current state of affairs? I thought not.What do you want?

Great stuff! Remember that the web models were only trained on their specific dataset constrained to inputs and output sentences where each were less than or equal to 10 words in length. The responses get considerably more colorful when you increase the length, but training takes much longer as well. Also, as dumb as the web models are, it is pretty impressive considering how simple they are! Gotta give lil botty some credit, botty is basically just two RNNs connected to each other (oversimplification of seq2seq).

Also, to the user who said "ubuntu is shit," I couldn't agree more haha. Definitely the noisiest dataset and I could've done a better job cleaning it. You can actually see what I did for ubuntu in the relevant jupyter notebook, and you can get an idea of the (low) quality sentences in that dataset. Cornell is basically a perfect dataset, but is dramatically smaller than the rest.

3