Gigmit

Challenge

The nature of our client’s business requires continuous data harvesting to ensure the end users have an ever-increasing choice of relevant and up-to-date options. In simple terms, gigmit ensures artists can easily find desirable music festivals, fairs, shows, or special event venues and contact the organizers.

To achieve this goal, gigmit collects publicly available information from a plethora of music & entertainment websites into a single database. Before our partnership, gigmit gathered the necessary data with the help of Scrapy, an open-source web crawler for extracting data with APIs and general-purpose web scraping.

However, they faced certain limitations with this tool, so they turned to Redwerk to help them perform data scraping and parsing of a select number of web resources. Naturally, we were expected to do this in a way that doesn’t harm the operation of the mentioned resources and without getting blocked. Also, we had to work under time constraints.

Solution

The first step was to evaluate the existing web scraping strategy and select a new tech stack that would pose fewer data extraction and display restrictions.

We chose Django REST to build data models and handle interactions with the database. This framework allowed us to organize and manage the collected data efficiently.

We created asynchronous task queues with Celery and RabbitMQ to avoid overloading the servers with API calls. This way, we ensured our requests are processed in an orderly and controlled manner.

Contact information was only sometimes readily available, so we leveraged the power of Hunter.io, an AI-driven email finder.

The biggest challenge was to find the balance between parsing speed and the limits of hosting servers. Still, we ethically collected all the needed data without service disruptions or complaints from the web admins.