Linktiger

Challenge

Broken links create a big problem for websites, e-commerce, etc. They can frustrate and drive away visitors, while stopping online buyers in their tracks. Broken links interrupt user interaction and diminish overall user experience regardless of whether the link is a local or remote resource. Finding and correcting broken links are critical to online sales, as well as building or maintaining the site`s (and company`s) credibility.

Redwerk`s software developers were challenged to build a sophisticated system with an extensive backend, operating on fairly light hardware. The ability to run multiple crawlers was necessary in order to handle checking thousands of websites every day.

Solution

After in-depth evaluations of existing commercial and open source spiders, Redwerk`s web crawling software developers team decided that creation of a web crawler from scratch was in order. During the development phase and continuing through years of maintenance, our software engineers constructed a highly mature crawler capable of handling almost every single peculiarity of the specific web server, and occasionally, even browser behavior.

Our development team designed an advanced architecture that is quickly and easily scalable by simply plugging in additional servers as Linktiger`s load grows. Always striving for the highest performance level, Redwerk`s software developers went through a series of optimizations that resulted in the system processing terabytes of downloaded documents daily. It`s even capable of extracting links from Javascript code.

All of this processing power functions seamlessly behind its user-friendly user interface. Additional features include:

• Automated e-mail notifications alert subscribers about broken links almost instantly
• Daily report results are visually represented on a dashboard with pie charts and graphs, which can be exported, printed emailed and shared with colleagues
• The report`s user interface allows filtering by error and file type, searching by document name and exporting link lists in CSV format
• A “Show me” function leads users through the site`s pages containing broken links via Next/Previous buttons. The broken links are highlighted in orange, making them easy to locate and quickly fix
• Users manage their accounts, customizing reports, adding, editing and removing websites through the dashboard without any changes necessary in the source code
• No software to download, order upfront or maintain. No hardware required. Just automated site scanning without any work on the user`s part.

In addition to crawling HTML code, Linktiger also hunts down broken links in CSS stylesheets, JavaScript code, Word and Excel documents and in PDF, RTF and TXT files.

Interesting Fact

In 2011, Redwerk provided a technology to facilitate a mutually beneficial partnership between two of our customers: Linktiger and Hosting.com. We developed a Linktiger widget for HDC`s customer portal so now Hosting.com customers can easily subscribe to Linktiger and keep their websites free of broken links. A partnership between the two companies made sense and was a win/win for both and tens of thousands of their subscribers.