Fast NSF

A Background on Domino Testing Tools

I originally joined Lotus as a product manager to help with Notes server performance (this was in the Notes V3 days, the Domino moniker didn't exist yet). At the time their were no public testing tools for Domino. Administrators were hungry for perfomance and sizing information and clamored for any data to right-size their servers. Hardware vendors wanted a way to stress test Domino on their servers so they could provide reliable capacity planning information, typically required to close sales engagements. Lotus was very tight lipped about making any specific tuning recommendations simply because it hadn't done much (actually it hadn't done any) performance testing for public consumption. Many hardware vendors were happy to send us servers with the intent that Lotus would endorse their hardware as the "top hardware platform for Domino" (which of course wasn't going to happen).

I remember talking with screaming customers, vendors, and a depressed support staff who took the brunt of customer performance complaints (and still do). At the time, there was a single tool used for load generation, and internal Iris testing tool called TESTNSF. There were many debates as to whether we should just give customers and vendors TESTNSF and let them do their own performance testing. Thoughts of supporting TESTNSF sent violent shivers up the spine of most Iris Engineers (at this time Iris was still a very small company with less than 80 employees who were always very busy and didn't have time to field questions on an internal tool that was never meant to be used outside the company). TESTNSF was a self documented tool. In other words, there was little to no documentation, and to use it, you simply had to read the code, or talk to someone who used it before you. Throwing it over the wall to the public was not an option, yet simply telling the customers and vendors "no" wasn't working anymore.

As time went on, a happy /less painful medium was reached, which we've all come to know as the NotesBench consortium and the NotesBench testing tool. NotesBench was simply TESTNSF in lock down mode- you could only execute several predefined scripts and not execute custom scripts (as you could with TESTNSF). This was a great move because it pushed the responsibility of data generation to the hardware vendors. They were now responsible for producing their own performance reports for internal and public consumption. Since NotesBench could only execute predefined scripts (workloads), this allowed for some basic apples to apples comparisons among server lines, and between OS platforms. We (Lotus/Iris) simply needed to support a handful of vendors using NotesBench. What was great about the consortium was that the performance reports were audited. You couldn't simply throw a report over the wall for public consumption (well, you could, but you would be slapped on the wrist if you did). This model worked very well (from Lotus/Iris perspective) as it some what got the performance monkey off our backs and we could get back to improving the core product.

Fast forwarding beyond the IBM acquisition of Lotus, the need for the admin, consultant, or developer to perform in-house testing still existed- and there cries were getting louder. This group insisted on doing their own characterization in house. We realized that we had to give the client base something. At this point I fled Lotus for Iris and joined the relatively new Iris Performance Team. In a variety of discussions with development, support, and product management, the consensus was that whatever the tool, it had to be free and easy to use. I volunteered to create this tool with very grandiose ideas running through my mind. I wanted wizards, server profilers, and the like. Fortunately my boss brought me back to earth with more practical and realistic goals.

Server.Load is essentially a GUI based version of TESTNSF with limits (you can only simulate 2048 users per computer in Server.Load 6.5.x and up), and only a subset of the TESTNSF commands were exposed to Server.Load). The biggest limitation is that it only runs only on Windows (although it can drive a load against any Domino server on any supported OS). It's strongest benefit is that it allows the end user to create their own scripts, run canned scripts, and capture testing results. Who could ask for anything more from a free tool? The name Server.Load followed the same naming scheme as Domino.Doc and other "dot" notation Lotus products. Surprisingly, it still retains the name. I remember giving a Lotusphere Session in 1999 about it, and there was a decent amount of excitement and interest, with tons of questions. It was generally accepted as a decent testing tool within IBM as well.

Page 1

1   2   3   4   5   6   7   8   9   10   11   12   13   14   15   16   17   18   19   20   21   22   23   24   25   26   27   Next->  

1. David Russell07/10/2007 10:55:26 PM
Homepage: http://www.russellsphere.com

I think this is a great resource, a quick blog to reference admin tips. Thank you for taking the time to put this together

2. Janez Kogovsek07/17/2007 05:17:48 AM
Homepage: http://www.src.si

I'm a Notes developer and I have to create a metology to perform stress tests against (existent) extensive LN/Domino applications for both Notes client and Web client.
As far as I know Server.Load is the only tool which I can use to (partly) do this job.
Unfortunatelly Server.Load is mainly focused on Mail & Scheduling.
I struggle to find adequate commands to simulate real applications.
Since every Notes apllications use dozens of @DbLookup/GetDocumentByKey ... functions/methods I must find corresponding Server.Load command to perform the same action. In fact there are methods FindByKey and FindByName but there is no way to "randomize" parameter (key). If we use the same key over and over again we won't get proper response time since Domino caches search results.
Since you're the author of Server.Load I feel it's a little bit inappropriate but I still dare to ask you if you know of any other tool mainly focused on performing tests against real LN/Domino applications. If the answer is 'no', can you help me with some detailed informations about existent Server.Load commands? I also have a couple of questions about undocumented commands, which are uses in build-in scripts.
Best regards, Janez

3. Greg08/08/2007 03:41:07 AM

What a great article and thank you for investing the time to write it !!!

I am in a similar position as Janez where I have a production Domain that I have just added a Web server to that I would like to stress test prior to allowing real user access.

Being a Production Domino Domain, I would rather not register 3000 to 5000 test users to perform load testing.

Are there any tricks we can use (ie Anon access) to simulate heavy browser use ? Are there custome scripts publically available that would assist with this ?

Kind regards,
Greg.

4. Adalton10/16/2007 02:17:21 PM

The following link { Link } is broken on page 25 of the article. This is the one that suposed to have the mail files populated with documents. Is there some other place we can download it from?