The Usual Suspects
Every programmer will eventually face a bug that defies their attempts at fixing it. The bug might seem impossible to reproduce, or it might show up only occasionally on a remote customer's machine. Some bugs are so difficult and time consuming to investigate that living with them can be cheaper than trying to fix them. On the other end of the scale, some bugs are so costly that no option exists but to fix them. Struggling with hard bugs is such an important programming experience that I am skeptical of any programmer who claims to be an expert, but who cannot tell at least one or two bug-fixing horror stories.
Here is one of my horror stories: I was asked to help a team who had been struggling for about a week with an elusive bug. This bug had a particularly high profile because it involved the loss of data in a system that managed the personal information of children and families. The team could not reproduce the problem despite daily reports of data loss.
Compounding the difficulty, the support team was given access to the problematic customer database for only a few hours daily and only after obtaining written authorization from the client each day. There was also uncertainty about which version of the database was being used by the customer. All the stored procedures, functions, and views in the database (hundreds of them) were encrypted and the database had no proper version information so the support team was forced to guess or experiment to ...