Developing Web Applications with Haskell and Yesod

Appendix B. Conduit

Conduits are a solution to the streaming data problem. Oftentimes, laziness allows us to process large amounts of data without pulling all values into memory. However, doing so in the presence of I/O requires us to use lazy I/O. The main downside to lazy I/O is non-determinism: we have no guarantees of when our resource finalizers will be run. For small application, this may be acceptable, but for a high-load server, we could quickly run out of scarce resources, such as file handles.

Conduits allow us to process large streams of data while still retaining deterministic resource handling. They provide a unified interface for data streams, whether they come from files, sockets, or memory. And when combined with ResourceT, we can safely allocate resources, knowing that they will always be reclaimed—even in the presence of exceptions.

This appendix covers version 0.2 of the conduit package.

Conduits in Five Minutes

While a good understanding of the lower-level mechanics of conduits is advisable, you can get very far without it. Let’s start off with some high-level examples. Don’t worry if some of the details seem a bit magical right now. We’ll cover everything in the course of this appendix. Let’s start with the terminology, and then some sample code.

Source

A producer of data. The data could be in a file, coming from a socket, or in memory as a list. To access this data, we pull from the source.

Sink

A consumer of data. Basic examples would be a sum function (adding up ...