In the last few years, we’ve discovered that many of our customers have been putting up with artificial clients hitting their websites. Scraping competitors’ websites for business intelligence purposes is quickly becoming a common practice. Recently, an article by Slashdot shared that people are even using bots and scripts to score restaurant reservations. Likewise, Variable Pricing software and methods are being used by many e-commerce websites in order to reduce their sales costs and obtain greater control over the buying process.
Although the legality of the practice of gathering data from competitors has been questioned, it continues to occur every day. Many companies we’ve talked to have shared that more than 60% of the traffic being served from their website is attributed to scrapers and bots. Some of the scrapers are visitors they wish to allow, like price comparison websites, while others are competitors attempting to get a pricing edge on them. As a result of this trend, many companies have expressed service interruptions, page slowdowns up to 30 seconds per page, web server crashing and loss of revenue.
An article in the New York Times noted that during the holidays, most major retailers put strategies in place to undercut their competitors. Meanwhile, shoppers are becoming more aware of this game and are utilizing purchase-decision websites such as www.decide.com. Browser extensions can also be installed to help consumers scrape the web for price comparisons.
Outside of just direct competitors gaining business intelligence, we’ve found “business intelligence” companies selling pricing data. These companies will do the service of scraping websites for you. Some of them utilize cloud-hosting solutions that allow them to change their IP addresses daily, making traditional IP blocking methods more difficult. These are some of the reasons why companies serve more than 60% of their bandwidth to competitors.
Some interesting questions to ask are: What would happen if you detected the bots, knew it was your competitors or people you didn’t want knowing your prices, and you purposely LOWERED prices below cost to the bot? With the variable pricing software and intelligence index methods, could your misinformation campaign cause your competitors to make bad decisions? What if you lowered prices below cost, then raised prices sky high to the bots, while keeping your prices low? Could you change the landscape in your favor? How much does it cost for you to host bandwidth and servers for your competitors?
What technologies are you using to monitor, stop and change this behavior that inevitably hits your website every day? Some of our customers have integrated both InFlight and DefensePro to help eliminate bots and allow legitimate users on their site. They have found value in being able to stop competitors from automated anti-competitive behavior.
What are you doing to mitigate these intrusions?