Forum not reachable today 4/22/2017
- Thread starter zeynel
- Start date
Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.
@zeynel - I am guessing earlier today?
We now have 10x the bandwidth to the hosting environment. As a result, both the botnets and Google Bot can now hit the forums much faster and that blew up the nginx access log to the point that it filled the disk.
Fixed this morning. Longer time fix is re-platforming the forums, which I need to do. Also setting less verbose logging to the access logs.
We now have 10x the bandwidth to the hosting environment. As a result, both the botnets and Google Bot can now hit the forums much faster and that blew up the nginx access log to the point that it filled the disk.
Fixed this morning. Longer time fix is re-platforming the forums, which I need to do. Also setting less verbose logging to the access logs.
Yea well, Googlebot alone just started generating 150x as much traffic per day. (not kidding)
That remote syslog will happen as part of the forums re-platform.
Based on what I have been seeing, I almost think it is hitting the "new posts" page constantly. Very strange.
I mean, from a hardware/ network side it is not an issue for us to service. But imagine if you had a small VPS/ AWS instance.
It is about 1mbps traffic during those peaks.
In that case you'd set the max crawl per-day in GWT. Still seems like an insanely lot of crawling for a site this size... I wonder if there's a boat load of dupe content / missing canonicals due to the forum software? IE: Forum posts/threads may be perfect but maybe guides, users, etc, type pages are not setup properly and spawning tons of duplicate content.
Back when I was running a LISTSERV instance, I saw the same behavior from Googlebot. Of course, back then there were actually people you could talk to at Google, and they fixed it. I miss the old Google...
