DIVING DEEPER INTO DATA MANAGEMENT
In the previous chapters, we looked at the hardware and software on which a pipeline is built. In this chapter, we will look at how pipelines are designed. In particular, we will explore four key topics:
• Directory structures
• File-naming conventions
• Version control
• Metadata and databases
Some of these concepts were introduced in Chapter 5 ...