Microsoft® SQL Server® 2008 Integration Services: Problem-Design-Solution

10

Implementing Scale-Out ETL Process

Now that you’ve worked through several aspects of building an SSIS solution, let’s turn now to some advanced concepts. In this chapter, you learn how to scale out SSIS, where SSIS packages are working in tandem on multiple machines. This effective way to handle large volumes of data has the added benefit of fault tolerance if one of the servers goes down.

Chapter 11 transitions to review advanced scripting in those situations when you just can’t accomplish what you need to with out-of-the-box components and you need to compose code to help accomplish your goals (both control flow and data flow). The final chapter, Chapter 12, is about troubleshooting performance issues and optimizing packages for efficient use of the data flow, and other advanced topics (such as how to handle partitioned tables). Overall, these last chapters give you the direction you need to deal with both complexity and high volumes of data.

One of the biggest benefits of SSIS is its capability to process and transform data in memory, effectively reducing disk I/O, and taking advantage of better memory availability and 64-bit architecture as well. With BI solutions now becoming more commonplace than just a few years ago, companies are realizing the need to process larger and larger amounts of data, and to try to do so as quickly and efficiently as possible. As a result, ETL architects and SSIS developers must start thinking about ways to scale out ETL processes so that they ...