Part I
Optimization
In statistics we need to optimize many functions, including likelihood functions and generalizations thereof, Bayesian posterior distributions, entropy, and fitness landscapes. These all describe the information content in some observed data. Maximizing these functions can drive inference.
How to maximize a function depends on several criteria including the nature of the function and what is practical. You could arbitrarily choose values to input to your function to eventually find a very good choice, or you can do a more guided search. Optimization procedures help guide search efforts, some employing more mathematical theory and others using more heuristic principles. Options include methods that rely on derivatives, derivative-free approaches, and heuristic strategies. In the next three chapters, we describe some of the statistical contexts within which optimization problems arise and a variety of methods for solving them.
In Chapter 2 we consider fundamental methods for optimizing smooth nonlinear equations. Such methods are applicable to continuous-valued functions, as when finding the maximum likelihood estimate of a continuous function. In Chapter 3 we consider a variety of strategies for combinatorial optimization. These algorithms address problems where the functions are discrete and usually formidably complex, such as finding the optimal set of predictors from a large set of potential explanatory variables in multiple regression analysis. The methods ...
