Bots and Scripts for Abuse


In the last few years, we’ve discovered that many of our customers have been putting up with artificial clients hitting their websites. Scraping competitors’ websites for business intelligence purposes is quickly becoming a common practice. Recently, an article by Slashdot shared that people are even using bots and scripts to score restaurant reservations. Likewise, Variable Pricing software and methods are being used by many e-commerce websites in order to reduce their sales costs and obtain greater control over the buying process.

Although the legality of the practice of gathering data from competitors has been questioned, it continues to occur every day. Many companies we’ve talked to have shared that more than 60% of the traffic being served from their website is attributed to scrapers and bots. Some of the scrapers are visitors they wish to allow, like price comparison websites, while others are competitors attempting to get a pricing edge on them. As a result of this trend, many companies have expressed service interruptions, page slowdowns up to 30 seconds per page, web server crashing and loss of revenue.

An article in the New York Times noted that during the holidays, most major retailers put strategies in place to undercut their competitors. Meanwhile, shoppers are becoming more aware of this game and are utilizing purchase-decision websites such as www.decide.com. Browser extensions can also be installed to help consumers scrape the web for price comparisons.

Outside of just direct competitors gaining business intelligence, we’ve found “business intelligence” companies selling pricing data. These companies will do the service of scraping websites for you. Some of them utilize cloud-hosting solutions that allow them to change their IP addresses daily, making traditional IP blocking methods more difficult. These are some of the reasons why companies serve more than 60% of their bandwidth to competitors.

Some interesting questions to ask are: What would happen if you detected the bots, knew it was your competitors or people you didn’t want knowing your prices, and you purposely LOWERED prices below cost to the bot? With the variable pricing software and intelligence index methods, could your misinformation campaign cause your competitors to make bad decisions? What if you lowered prices below cost, then raised prices sky high to the bots, while keeping your prices low? Could you change the landscape in your favor? How much does it cost for you to host bandwidth and servers for your competitors?

What technologies are you using to monitor, stop and change this behavior that inevitably hits your website every day? Some of our customers have integrated both InFlight and DefensePro to help eliminate bots and allow legitimate users on their site. They have found value in being able to stop competitors from automated anti-competitive behavior.
What are you doing to mitigate these intrusions?

David Hobbs

As Director of Security Solutions, David Hobbs is responsible for developing, managing, and increasing the company’s security practice in APAC. Before joining Radware, David was at one of the leading Breach Investigation Firms in the US. David has worked in the Security and Engineering arena for over 20 years and during this time has helped various government agencies and world governments in various cyber security issues across all sectors.

Contact Radware Sales

Our experts will answer your questions, assess your needs, and help you understand which products are best for your business.

Already a Customer?

We’re ready to help, whether you need support, additional services, or answers to your questions about our products and solutions.

Locations
Get Answers Now from KnowledgeBase
Get Free Online Product Training
Engage with Radware Technical Support
Join the Radware Customer Program

CyberPedia

An Online Encyclopedia Of Cyberattack and Cybersecurity Terms

CyberPedia
What is WAF?
What is DDoS?
Bot Detection
ARP Spoofing

Get Social

Connect with experts and join the conversation about Radware technologies.

Blog
Security Research Center