11.12.15 Estimating Application Performance by Running an Application

A common problem when developing applications for the IBM i is trying to estimate how they will actually execute on a "busy" or "loaded" production system.

If you test a function and "guess" that it seems to run okay, then you run the following risks:



     You may test this program and find that the panel appears in 1 second on average. However, you may find that if you sign onto the computer at the start of the day and then attempt to access the program initially it takes 4 seconds.

     If you then repeatedly access the program from the menu, you find the access time gets less and less. The faster you access the program and return to the menu and (re)access the program the lower the access time seems to get. In fact you might find that the access time drops down to an average of 1/4 a second or less.

     You might also notice that if you stop accessing the program for 5 minutes, and have a coffee break, your next initial access climbs back to 3 seconds or so.

     How can this be, when you know that the program is doing exactly the same amount of work every time it is accessed (assuming that it is not an *HEAVYUSAGE program) ?

     The reason is simple. When you access and re-access the program very quickly, the program and the objects that it accesses tend to remain in the computer's memory. However, if it is the very first access they are not in memory. Likewise, if you take a long break between interactions, it is likely that the objects are "rolled out" of memory back onto disk. If you wish to re-access them, they must be "rolled in" to memory again, causing you to wait while this is done.

     This phenomena is not new, and applies to many "virtual memory" operating systems such as the IBM i, but you should be aware of its existence when you are testing your applications.

     The faster you work, the quicker the system will respond. The slower you work, the slower the system will respond. This effect is very pronounced in systems that have very long "think times" such as complex database inquiries, and least pronounced in system that have very short "think times" such as high speed data entry systems.

     You should allow for this in any testing. If your system has a long "think time", you should emulate this in testing, or you may be using the operating system in a way that does not truly reflect the way that users of your system will.

Hopefully all of these points combined are enough to convince you that attempting to estimate how an application will perform in a production system is not a simple task and it should not be just doing by some simple testing and then concluding that "it seems to run okay".