O'Reilly logo

PowerShell Deep Dives by Edited by Jeffery Hicks, Richard Siddaway, Oisín Grehan, and Aleksandar Nikoli

Stay ahead with the world's most comprehensive technology and business learning platform.

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, tutorials, and more.

Start Free Trial

No credit card required

Chapter 15. Scalable scripting for large data sets: pipeline and database techniques

Matthew Reynolds

An online retailer needed to learn which of their Domain Name System (DNS) records were getting the most queries after an advertising campaign. They wrote a script to get this information from their DNS server logs, but as the logs grew, the script slowed to a crawl. Worse, when they tried to run the script on multiple servers remotely the script failed with an OutOfMemoryException.

A web search for the terms PowerShell and OutOfMemoryException returns many thousands of hits. People are clearly struggling to manage large data sets.

The typical problem is the use of a fragile pattern that works in the lab but doesn’t scale with real-world ...

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, interactive tutorials, and more.

Start Free Trial

No credit card required