[D] How do people come up with all these crazy deep learning architectures?

For the past few days, I've been reading TensorFlow source codes for some of the latest DL architectures (e.g. Tacotron, Wavenet) and the more I understand and visualize the architecture, the less sense it makes intuitively.

For vanilla RNNs/LSTMs and ConvNets, it's quite easy to grasp why they would work well on time-series/image data. Very simple and elegant. But for these SOTA neural networks, I can't imagine why putting all these pieces (BN, highway networks, residuals, etc) together in this seemingly random way would even work.

Is there some kind of procedure people follow to compose these Frankenstein networks? Or just keep adding more layers and random stuff and hope the loss converges?