SystemDataRecorder

Light-weight native Windows system service

to log CPU, disk, hardware and network usage was developed by Redwerk

SystemDataRecorder software development partnersSDR Dynamics OyEspoo, Finland

SystemDataRecorder is a tool for performance analysis, system sizing, and capacity planning in challenging business environments.

All Customers
Product Development

Being a one-stop shop for software development, Redwerk has implemented all the System Data Recorder Windows services from the ground up. We went through every phase here: requirements analysis, architecture, development, testing, installation package creation and delivery.

Learn more
Startups & Innovation

We have developed products for multiple tech start-ups from the ground up, who trusted us to build their entire technology.

Learn more

Challenge

SystemDataRecorder is perhaps the most complete and simple package for professional performance monitoring, performance analysis, system sizing, and capacity planning in challenging business environments. It provides consistent data recording across systems and devices, permits access to raw data and is designed for time series analyses. It may not seem that way for non-software experts, but it is indeed a very powerful and long-awaited solution to a long-standing problem.

Even though computers have been around for more than 25 years, there is still no consistent performance monitoring between different operating systems, as each environment deploys its own type of monitoring and data collection tools. Consistent data recording across different operating systems is highly difficult to do without purchasing separate software for each environment or installing third-party solutions. Even the format used for recording data varies from system to system, making data collection and analysis a logistic nightmare.

Looking closely at the individual operating systems, it is easy to see that the methods for obtaining performance data are actually similar, but different interfaces are used with varying terminology according to the individual vendors and the specific type of implementation. The idea behind SystemDataRecorder is to have several standard data collecting agents to obtain metrics from the system interfaces to exporting this information in a unified way. For the exported data, a plain text file can be used, foregoing any special file formats, to make this file usable for any available analytic tool. In this way, a simple data recording module can be used for system troubleshooting, performance analysis, system crash analysis etc., and it is easy to enable across a large number of hosts in a data center, and the operating system used no longer plays a role.

SystemDataRecorder was already available and proving to be the light at the end of the tunnel for Linux and Solaris systems, when Redwerk was contacted about adapting it also to Windows 2003 and 2008 server environments. The goal was for these servers to work with similar metrics as SystemDataRecorder for Linux or Solaris. The solution had to use as few system resources as possible, be small in size and robust enough in order not to create system overheads. The recorders had to be implemented as Windows services, i.e. native applications for 32-bit and 64-bit systems. All data had to be stored in a plain text file.

An additional challenge was to port the ability to monitor the output of the raw data files to Windows. For every change in a file, a HTTP/HTTPS POST had to be sent to a backend system in order to be analyzed later. The software had to be distributed as a software package suitable for automatic installation on Windows without the need for user interaction.

Solution

Our research showed that a service written in C/C ++ receiving dozens of parameters and writing data to file uses about 2 MB RAM and only minor CPU resources. The same service in C# uses a minimum of 6 MB, and while it is easier to develop it also requires a minimum of .NET 2.0. After analyzing the advantages and disadvantages, the code was developed in C. This way we succeeded in minimizing abstraction and were able to use simple routines to access Windows kernel metrics without the overheads which the use of an object-orientated language would have entailed. All the data for the service was obtained from the WMI API.

For each individual recorder, the key parameter to be used is the number of seconds between each measurement. The default value was 60 seconds. We also came up with a solution for launching with a certain parameter on Windows. Recorders launch and write data in their own, individual files. After this initial step, the files are rotated and compressed. In Linux or Solaris this is implemented using cron, but for Windows we had to find our own solution.

The development process was very sound and based on logical steps of research, testing and deployment. The testing included some very interesting issues. We had to make sure that the recorders would run without any memory leaks or other deviations for at least 48 hours. We even left them to run over the weekend, to be able to monitor them properly, running suitable acceptance tests.

For the process and the solution to be solid and thorough, also emergency situations like power outages or unplanned system reboots have to be taken into account. To deal with such scenarios, services must re-start automatically during system boot and continue to record and store data in the output files.

Need a team to develop a system service?

Request Quote

2

developers worked on the project

1

QA engineer

600

man-hours spent

46,650

lines of code

Results

The best products are those which make complex problems look simple. This is especially true in the world of software. SystemDataRecorder is one of those solutions. Redwerk’s expert software engineers had to face a few hurdles but overall had few problems in developing the required services for Windows environments, with periodical reviews being scheduled (3-4 reviews of the main code and binary product before the final release). The solution was tested in a very thorough fashion to see how it would behave under different types of Windows operating systems.

Out client ran their own acceptance test at the end of the development phase, checking the product in terms of CPU usage, memory consumption and robustness. We certainly had nothing against these kind of “exams”! It was highly interesting for our team to work on such an unusual product with a very specific usage scenario, and we were delighted that our solution was up to the high standards our client expected from us. We take great pride in having contributed to such a great tool!

Visit kronometrix.com

Professional perfomance monitoring

At SystemDataRecorder Oy we focus on performance monitoring and data analysis for computer systems and support many types of Linux and UNIX based systems, like RedHat, SuSE, Debian, FreeBSD, Solaris. But we have not supported Microsoft Windows platforms to date. Since many enterprises rely on Windows-based platforms to support their web services, as well as their application or database tiers, we certainly needed to fill this gap.

However, this project looked both technically challenging and time consuming. To help us accomplish our goal we approached our partner RedWerk to help with the design, implementation and testing of a complete set of data recorders for Windows 32 bit and 64 bit platforms.

With RedWerk's help, we wrote and tested five key data recorders in C/C++ that had minimal overhead while recording all the important Windows operating-system metrics. Testing was a very important part of this project. Our data recorders are required to run for weeks without introducing any additional load on the systems being measured and they need to run very reliably without human intervention. The testing confirmed that these goals were achieved.

Consequently, we released the first version of our data recorders as part of a free open source distribution that is now available to interested parties in the performance monitoring community.

Thank you RedWerk for your contribution and dedication during this project.

Stefan Parvu SDR Dynamics partner

Stefan Parvu, Lead System Engineer SDR Dynamics Oy

Need a dedicated development partner?

Contact Us
SystemDataRecorder installation
Performance analysis
System sizing software
Capacity planning software
Device data recording
System data recorder for cpu

Impressed? Hire us

Contact Now
Request Quote
×