[D] What would be the best way to convince an academic audience that your reinforcement learning algorithm is "good" or "competitive" with other algorithms?
Posted Oct 27, 2017, 8:51 PM UTC
515
Uh........show better results than other algorithms?
If you can't do that, you're job is much, much harder: you need to convince people why they should give up accuracy or performance for whatever you're proposing.
2
The problem is: on what task? One openAI gym? one atari game? Shouldn't there be a collection of representative tasks that demonstrate the overall quality of the algorithm?
3
I would say if you want to cause a stir, show competitive results on a selection of classic Atari games, and then try to find one or two that existing methods struggle with and show SOTA performance there.
In particular, I know reward sparsity is a problem for many single player RPG-style games.
6
Thanks especially for the last suggestion. Our team is searching for well-known problems that are particularly hard for SOTA. If you have any other examples like this it would be really appreciated!
1
Without knowing the details of your method, it's hard to know what to focus on. But Montezuma's revenge, of the Atari problems, is famously punishing for RL. It requires you to explore while avoiding difficult enemies under conditions of sparse rewards, which is obviously very difficult. Last I'd heard, there'd been some progress made using intrinsic exploration rewards and hierarchical methods, but the SOTA still wasn't great relative to a human baseline.
Another thing that would draw interest would be doing a selection of the classic ATARI problems for which superhuman results exist, and replicate that using substantially fewer training examples, if you believe your method is more sample-efficient than DQN or its successors.
3
In all seriousness, there are popular benchmarks out there (you've named two good ones). Ideally you try your algorithm in all the major arenas to show that it's robust.
3†
Your two comments were a bit condescending, and actually didn't give me any additional information.
Are you saying that there is no established method for demonstrating how good an RL algorithm is aside from just try everything?
1†
There's a pretty generic ML evaluation recipe that works for any area:
Go find a paper similar to yours that is highly regarded. Hopefully you've done a literature search and know of a couple.
Do an evaluation that is the union of all of the experiments in those papers, and show your method is better than the proposed methods on those benchmarks.
Add one example that captures the intuition of why your method does better, and show your method is much better on this example.
9
Apologies for sounding condescending, I was aiming more for cynicism.
The issue is that there are a lot of open problems in RL, and making progress on single hard tasks (take Montezuma's revenge as an example of planning in a sparse environment) doesn't necessarily mean you'll do well in other challenging arenas.
So if you have a particular angle you're swinging for, then the best thing to do is find whatever specific benchmarks are being used in similar papers (like the other poster said) and beat them.
If you're arguing for a more general "RL improvement", then I return to my first answer.
2
The whole point of the Atari benchmark is to run the same agent with the same hyperparameters across the entire suite. People fiddle with individual games sometimes when it makes sense, mostly the really hard ones like Montezuma's Revenge, but single game results are usually not informative.
2
In my opinion, RL (at least for now) should be aimed at specific applications (or classes of problems: certain markovian assumptions, smoothness, whatever). We're still at the point where the goal should be solving a particular task in a fast and impressive fashion. So the question becomes: what problem (or class of problems) is your RL algorithm trying to solve?
[deleted]