Nerds on Wall Street: Math, Machines, and Wired Markets

3.5. A Scientific Approach: Mathematics, Behavior, and Discovery

The obvious shortcomings of these simple strategies motivated several generations of mathematically based algorithms, using increasing levels of mathematical and econometric sophistication to include models of market impact, risk, order books, and the actions of other traders. The idea of an efficient frontier of trade path strategies and the use of optimization established a conceptual foundation, analogous to the efficient frontier in portfolio theory.

Markets have become even more fragmented and complex, and with decimalization (trading in pennies instead of eighths or sixteenths), less information is conveyed by the best bid and offer (BBO).^[] The ability to rapidly cancel and replace limit orders has a similar information-reducing effect on the limit order book. New markets, electronic communication networks (ECNs), and others create a need to exploit new order types and to access dark liquidity. This has given rise to behavior-based algorithms, probing for liquidity, driven by procedural logic and stimulus-response principles and as well as mathematical models. These are often called "gaming algos," and many traders and liquidity providers are actively deploying anti-gaming measures.

Algorithms at the edge probe, learn, and adapt. They need to make effective use of analytic tools, and know how to recognize their limitations. Many use small probe orders, examining the fill and replenishment response in multiple ...