Bernard of Chartres is attributed with the concept of learning through building on the discoveries of others; however, it was Isaac Newton who said "If I have seen further, it is by standing on the shoulders of giants." To be clear, that's exactly what I'm suggesting here.
If I were going to design a network architecture for a new deep learning problem, the first thing I would do is try to find a similar problem that has been solved before in a satisfactory way. While it might be that no one has solved the problem you're tasked with, something similar likely exists.
There is a very good chance that several possible solutions exist. If that's the case, and if time permits, the average results ...
