An evaluation benchmark is time boxed by design; you run the benchmark and report the figures. Although, an optimization benchmark is more blurry. Concretely, you can spend a whole year optimizing a simple web service just because of the layers it uses and all the small tuning options you can test, from the network configuration, through the JVM memory, to the caching solutions.
Thus, it is crucial before starting to benchmark an application and optimizing it to define how long you will spend benchmarking your application. It can also be linked to a renting period and may require an estimation phase to work with the operation and development teams. But if you don't do it, the risk is that you will spend ...
