PlusPlus

Challenge

PlusPlus is a chatbot for Slack made for entertainment. It allows to plus, minus and score things your friends say and do on Slack by addressing to objects and users with ‘@’. It has a global leaderboard and every action affects the general results.

The challenge for Redwerk team was to completely rewrite the existing bot, originally developed in Python, using Ruby as the core technology. We were also tasked to increase stability, make it possible to scale and optimize the overall performance of this high load project.

Solution

We’ve started with the most important part — code refactoring and fixing. The old version of the bot was written in Python and we needed to port it to Ruby, keeping in mind all the additional changes it might bring, fixing existing problems and implementing new functionalities. When the infrastructure was planned, we based our assumptions about the load on the average statistics of old Python bot usage: approx. 500-700 thousand user actions per minute. We extended this level up to ~1 million user actions/minute in order to have some reserve capacity and provided a set of stress tests on staging.

And here’s where another challenge appeared. It turned out that Slack limits testing to 1 message per second and doesn’t have the functionality to test high load projects. The actual live load of PlusPlus showed over 20-50 million actions per minute, and the existing infrastructure couldn’t support it. To cover the new demand, a major part of application logic had to be reworked.

Users and objects logic optimization

Redwerk’s dev team needed to carefully define and optimize any gaps and even the smallest leaks of memory and CPU usage. The actual project load sent millions of actions containing both @usernames and @things in a single line. Objects required more advanced logic than users, because they needed to be parsed, then checked if existed in database, created a new entry if didn’t, then updated score, leaderboard and the global trending. With millions of such actions, most of them happening within a single message, it overloaded the DB very quickly, as well as the memory and CPU.

Another high load task was to optimize score updates in the leaderboards, as every action had to influence the position in the global leaderboards of users and objects. The incredibly high amount of items that had to be parsed, checked, and updated, created additional challenges for development.

In order to optimise all these actions, our team developed functionality that uses multiple CPU cores on each server, along with a specific Bot Manager, which routes each bot to a specific node (Server + CPU core) — a kind of task which Amazon load balancer couldn’t handle. After that, we rebuilt request logic to decrease the number of requests.

We also switched from Ubuntu to CentOS. Ubuntu didn’t allow us to create additional connections to a database. CentOS in its turn provided more stable environment while working with MongoDB and supported the excessive action load. It also allowed us to re-configure AWS server and update OS core settings on nodes to improve the file descriptor and threads limits.