SQL Server 2012 Data Integration Recipes: Solutions for Integration Services and Other ETL Tools

CHAPTER 13

Organising And Optimizing Data Loads

Once you have succeeded in getting a set of data into SQL Server the next challenge is how to load it automatically, faster, more smoothly and more easily. Over and above this, you want your processes to run as reliably as is feasible with as little manual intervention as possible. This can involve many techniques, such as:

• Handling multiple identical source files for serial loading.
• Loading multiple files from one or more directories.
• Ordering file loads (to load files by size, for instance).
• Setting up parallel file loads—and dispatching files to multiple processes, with load balancing if possible. ...