40m QIL Cryo_Lab CTN SUS_Lab TCS_Lab OMC_Lab CRIME_Lab FEA ENG_Labs OptContFac Mariner WBEEShop
  40m Log  Not logged in ELOG logo
 List  |  Find  |  Login  |  Help 
Entry  Wed Feb 5 18:36:56 2014, rana, Frogs, elog, MicroSoft BingBot is attacking us 
    Reply  Thu Jul 31 18:54:03 2014, Chris, Frogs, elog, MicroSoft BingBot is attacking us 
       Reply  Thu Jul 31 21:21:49 2014, Koji, Frogs, elog, MicroSoft BingBot is attacking us 
Message ID: 10309     Entry time: Thu Jul 31 18:54:03 2014     In reply to: 9603     Reply to this: 10311
Author: Chris 
Type: Frogs 
Category: elog 
Subject: MicroSoft BingBot is attacking us 

Quote:

 The ELOG was frozen, with this in the .log file:   

GET /40m/?id=1279&select=1&rsort=Type HTTP/1.1

Cache-Control: no-cache

Connection: Keep-Alive

Pragma: no-cache

Accept: */*

Accept-Encoding: gzip, deflate

From: bingbot(at)microsoft.com

Host: nodus.ligo.caltech.edu

User-Agent: Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)

  (hopefully there's a way to hide from the Bing Bot like we did from the Google bot)

 

Yesterday elog was excruciatingly slow, and bingbot was the culprit. It was slurping down elog entries and attachments so fast that it brought nodus to its knees. So I created a robots.txt file disallowing all bots, and placed it in the elog's scripts directory (which gets served at the top level). Today the log feels a little snappier -- there's now much less bot traffic to compete with when using it.

We might be able to let selected bots back in with a crawl rate limit, if anyone misses searching the elog on bing.

ELOG V3.1.3-