SystemDataRecorder is perhaps the most complete and simple package for professional performance monitoring, performance analysis, system sizing, and capacity planning in challenging business environments. It provides consistent data recording across systems and devices, permits access to raw data and is designed for time series analyses. It may not seem that way for non-software experts, but it is indeed a very powerful and long-awaited solution to a long-standing problem.
Even though computers have been around for more than 25 years, there is still no consistent performance monitoring between different operating systems, as each environment deploys its own type of monitoring and data collection tools. Consistent data recording across different operating systems is highly difficult to do without purchasing separate software for each environment or installing third-party solutions. Even the format used for recording data varies from system to system, making data collection and analysis a logistic nightmare.
Looking closely at the individual operating systems, it is easy to see that the methods for obtaining performance data are actually similar, but different interfaces are used with varying terminology according to the individual vendors and the specific type of implementation. The idea behind SystemDataRecorder is to have several standard data collecting agents to obtain metrics from the system interfaces to exporting this information in a unified way. For the exported data, a plain text file can be used, foregoing any special file formats, to make this file usable for any available analytic tool. In this way, a simple data recording module can be used for system troubleshooting, performance analysis, system crash analysis etc., and it is easy to enable across a large number of hosts in a data center, and the operating system used no longer plays a role.
SystemDataRecorder was already available and proving to be the light at the end of the tunnel for Linux and Solaris systems, when Redwerk was contacted about adapting it also to Windows 2003 and 2008 server environments. The goal was for these servers to work with similar metrics as SystemDataRecorder for Linux or Solaris. The solution had to use as few system resources as possible, be small in size and robust enough in order not to create system overheads. The recorders had to be implemented as Windows services, i.e. native applications for 32-bit and 64-bit systems. All data had to be stored in a plain text file.
An additional challenge was to port the ability to monitor the output of the raw data files to Windows. For every change in a file, a HTTP/HTTPS POST had to be sent to a backend system in order to be analyzed later. The software had to be distributed as a software package suitable for automatic installation on Windows without the need for user interaction.
Our research showed that a service written in C/C ++ receiving dozens of parameters and writing data to file uses about 2 MB RAM and only minor CPU resources. The same service in C# uses a minimum of 6 MB, and while it is easier to develop it also requires a minimum of .NET 2.0. After analyzing the advantages and disadvantages, the code was developed in C. This way we succeeded in minimizing abstraction and were able to use simple routines to access Windows kernel metrics without the overheads which the use of an object-orientated language would have entailed. All the data for the service was obtained from the WMI API.
For each individual recorder, the key parameter to be used is the number of seconds between each measurement. The default value was 60 seconds. We also came up with a solution for launching with a certain parameter on Windows. Recorders launch and write data in their own, individual files. After this initial step, the files are rotated and compressed. In Linux or Solaris this is implemented using cron, but for Windows we had to find our own solution.
The development process was very sound and based on logical steps of research, testing and deployment. The testing included some very interesting issues. We had to make sure that the recorders would run without any memory leaks or other deviations for at least 48 hours. We even left them to run over the weekend, to be able to monitor them properly, running suitable acceptance tests.
For the process and the solution to be solid and thorough, also emergency situations like power outages or unplanned system reboots have to be taken into account. To deal with such scenarios, services must re-start automatically during system boot and continue to record and store data in the output files.
The best products are those which make complex problems look simple. This is especially true in the world of C software development. SystemDataRecorder is one of those solutions. Redwerk’s expert software engineers had to face a few hurdles but overall had few problems in developing the required services for Windows environments, with periodical reviews being scheduled (3-4 reviews of the main code and binary product before the final release). The solution was tested in a very thorough fashion to see how it would behave under different types of Windows operating systems.
Out client ran their own acceptance test at the end of the development phase, checking the product in terms of CPU usage, memory consumption and robustness. We certainly had nothing against these kind of “exams”! It was highly interesting for our team to work on such an unusual product with a very specific usage scenario, and we were delighted that our solution was up to the high standards our client expected from us. We take great pride in having contributed to such a great tool!
We selected Redwerk for their dedication and professionalism designing and developing SystemDataRecorder performance monitoring system for Windows platforms. Long hours testing and ensuring the software works without problems makes Redwerk a solid and mature partner for software development.
Need a dedicated development partner?Contact Us
Other Case Studies
Helped WorldNow solve lost captions problem while live streaming, enhancing TV experience for multi-million US audience
Built Android app for self-employed cleaners across Germany and Austria, soon acquired by Helpling.de
Helped #1 E-Government vendor in Netherlands and Belgium develop several modules for council decision-making platform