Professional Visual Studio 2017

28Data Warehouses and Lakes

WHAT’S IN THIS CHAPTER?

• Understanding the ideas behind Apache Hadoop and HDInsight
• Executing Hive queries and Pig job against a Hadoop cluster
• Examining the performance of Hadoop jobs

WROX.COM CODE DOWNLOADS FOR THIS CHAPTER

The wrox.com code downloads for this chapter can be found at www.wrox.com by searching for this book’s ISBN number (978-1-119-40458-3). The code and any related support files are located in their own folder for this chapter.

In an industry where buzzwords abound, “big data” is currently sitting at the pinnacle of hype. One would think it was a cure for cancer, the eradication of poverty, and the latest cat video all rolled up into one. Even if you’re skeptical about the hype, there is bound to be a nugget of potential usefulness in the technology. With Visual Studio 2017 and some recent additions to Windows Azure, now is the time to not only look at what big data really is, but also to consider the tools that are available to help with it. In this chapter, we discuss some of the basic ideas of Apache Hadoop, along with an examination of the tools that Visual Studio 2017 brings.

WHAT IS APACHE HADOOP?

You’re definitely not the first person to ask that question, and you won’t be the last. If you’re developing in the Microsoft space, it’s quite likely that you’re also asking about HDInsight. Fortunately, those two questions are related. HDInsight is the cloud-based implementation of Hadoop found in Azure. So a ...