Did anyone manage to reproduce MemN2N results on bAbI dialog tasks?

Hi all,

I'm playing with a MemN2N on the bAbI dialog Task 1. And a very surprising thing I see is that it learns dialogues pretty well, but on the most interesting part, actual API calls, it's pretty terrible (0.7% per-utterance accuracy). Apart from API calls, there's little to learn in Task 1 (e.g., there are only 7 distinct interlocutory responses but 159 distinct api_calls in the trainset), so it performs nowhere near what's reported in the paper (the implementation I'm using is not original. In fact, there's no original MemN2N dialog task setup available).

Now, if trained to predict exclusively API calls based on the full dialogue contexts, its per-utterance accuracy is at 96% on training and 0% on testing which makes me think it does nothing but memorize its training data.

My MemN2N input is <story, question, answer> triplets where story is all turns prior to the current question; story sentences as well as questions are of the form <turn index, agent name, word_1, ..., word_n>. Training hyperparameters are as follows:

20 embedding size 3 hops 12 memory size 100 training epochs

I tried playing with embedding size, hops number, memory size - nothing of it made difference. I then tried to compare it to a simple LSTM classifier (not seq2seq just to be closer to the original setup) - it performed horribly on API calls.

If you worked with a MemN2N on bAbI dialogue tasks, did you notice similar behavior? Maybe you have an idea of how to fix it?