Beautiful Testing

The Memory Leak That Wasn’t

It was almost two months after the “OK, but what’s a performance test?” episode before we were ready to start ramping up load with a reasonably complete system usage model on the eVersity project. The single-user and 10-user tests on this particular build had achieved better than required performance, so I prepared and ran a 100-user test. Since it was the first run of multiple usage scenarios at the same time, I made a point to observe the test and check the few server statistics that I had access to while the test was running.

The test ran for about an hour, and everything seemed fine until I looked at the scatter chart, which showed that all the pages that accessed the application server started slowing down about 10 minutes into the test and kept getting slower until the test ended. I surfed the site manually and it was fine. I checked the logfiles from the load generation tool to verify that I wasn’t seeing the effect of some kind of scripting error. Confident that it wasn’t something on my end, I ran the test again, only this time I used the site manually while the test was running. After about 15 minutes, I noticed those pages getting slow. I picked up the phone and called Sam, the architect for the project, to tell him that he had a memory leak on the application server.

Sam asked if I was running the test right then. I told him I was. I heard him clicking on his keyboard. He asked if the test was still running. I told him it was. He said, “Nope, no memory leak. It’s your tool,” and hung up.

I was furious. For the next two days I ran and re-ran the test. I scoured logfiles. I created tables and graphs. I brought them to project meetings. I entered defect reports. I sent Sandra the URL to the performance test environment and asked her to use the application while I ran tests. Everyone seemed to agree that it was acting like a memory leak. By the end of day two, even Sam agreed that it looked like a memory leak, but followed that up by saying, “…but it isn’t.”

Late on the third day after I’d first reported the issue, Sam called me and asked me to try the test again and hung up. I launched the test. About 20 minutes later, Sam called back to ask how the test looked. It looked great. I asked how he fixed it. He simply said, “Installed the permanent license key. The temp had a limit of three concurrent connections.”

Sam didn’t talk to me for the next couple of weeks. Since he wasn’t very talkative in the first place, I wasn’t certain, but I thought I’d offended him. Then a surprising thing happened. Sam called me and asked me to point “that test from the other week” at the development environment and to bring the results upstairs when it was done.

When I got upstairs with the results, Sam said to me, “Impressive work the other week. It took me over 20 hours to track down the license key thing. The tests we ran looked like a memory leak, too…except that the memory counters were showing tons of free memory. Anyway, from now on, why don’t we look at weird results together?”

From that time on, Sam demanded that management assigned me to all of his projects. He’d frequently ask me to design and run tests that I didn’t completely understand, but that would result in massive performance improvements within a day or two. I’d often call him and say, “I’ve got some odd-looking results here, would you like to have a look?” Sometimes he’d tell me why I was getting weird results, sometimes he’d want to take a look, and other times he’d ask me to run the test again in an hour, but he never again dismissed my results as a tool problem. And I never again announced the cause of a performance issue before confirming my suspicions with Sam.

Of course, the beauty here is that Sam came to see me as a valuable resource to help him architect better-performing applications with less trial and error on his part. In retrospect, I only wish Sam had been more talkative.