SystemDataRecorder

Challenge

SystemDataRecorder is perhaps the most complete and simple package for professional performance monitoring, performance analysis, system sizing, and capacity planning in challenging business environments. It provides consistent data recording across systems and devices, permits access to raw data and is designed for time series analyses. It may not seem that way for non-software experts, but it is indeed a very powerful and long-awaited solution to a long-standing problem.

Even though computers have been around for more than 25 years, there is still no consistent performance monitoring between different operating systems, as each environment deploys its own type of monitoring and data collection tools. Consistent data recording across different operating systems is highly difficult to do without purchasing separate software for each environment or installing third-party solutions. Even the format used for recording data varies from system to system, making data collection and analysis a logistic nightmare.

Looking closely at the individual operating systems, it is easy to see that the methods for obtaining performance data are actually similar, but different interfaces are used with varying terminology according to the individual vendors and the specific type of implementation. The idea behind SystemDataRecorder is to have several standard data collecting agents to obtain metrics from the system interfaces to exporting this information in a unified way. For the exported data, a plain text file can be used, foregoing any special file formats, to make this file usable for any available analytic tool. In this way, a simple data recording module can be used for system troubleshooting, performance analysis, system crash analysis etc., and it is easy to enable across a large number of hosts in a data center, and the operating system used no longer plays a role.

SystemDataRecorder was already available and proving to be the light at the end of the tunnel for Linux and Solaris systems, when Redwerk was contacted about adapting it also to Windows 2003 and 2008 server environments. The goal was for these servers to work with similar metrics as SystemDataRecorder for Linux or Solaris. The solution had to use as few system resources as possible, be small in size and robust enough in order not to create system overheads. The recorders had to be implemented as Windows services, i.e. native applications for 32-bit and 64-bit systems. All data had to be stored in a plain text file.

An additional challenge was to port the ability to monitor the output of the raw data files to Windows. For every change in a file, a HTTP/HTTPS POST had to be sent to a backend system in order to be analyzed later. The software had to be distributed as a software package suitable for automatic installation on Windows without the need for user interaction.

Solution

Our research showed that a service written in C/C ++ receiving dozens of parameters and writing data to file uses about 2 MB RAM and only minor CPU resources. The same service in C# uses a minimum of 6 MB, and while it is easier to develop it also requires a minimum of .NET 2.0. After analyzing the advantages and disadvantages, the code was developed in C. This way we succeeded in minimizing abstraction and were able to use simple routines to access Windows kernel metrics without the overheads which the use of an object-orientated language would have entailed. All the data for the service was obtained from the WMI API.

For each individual recorder, the key parameter to be used is the number of seconds between each measurement. The default value was 60 seconds. We also came up with a solution for launching with a certain parameter on Windows. Recorders launch and write data in their own, individual files. After this initial step, the files are rotated and compressed. In Linux or Solaris this is implemented using cron, but for Windows we had to find our own solution.

The development process was very sound and based on logical steps of research, testing and deployment. The testing included some very interesting issues. We had to make sure that the recorders would run without any memory leaks or other deviations for at least 48 hours. We even left them to run over the weekend, to be able to monitor them properly, running suitable acceptance tests.

For the process and the solution to be solid and thorough, also emergency situations like power outages or unplanned system reboots have to be taken into account. To deal with such scenarios, services must re-start automatically during system boot and continue to record and store data in the output files.