Sorry for the slow responses from git.friendi.ca this evening.

There was a massive and globally distributed scraper storm targeting expensive, randomly generated filter combinations on the Friendica Addons pull-request overview.

Even though #Anubis was active, far too many requests still reached #Forgejo.

Some statistics from the incident:

• The storm started at approximately 21:28 CEST.
• Anubis issued 144,192 challenges to 115,724 different source IP addresses.
• At the peak, Anubis issued 8,965 challenges per minute.
• 70,402 connections were abandoned while being proxied.
• Only 33 requests matched an explicit Anubis deny rule.
• 193,401 requests reached the Friendica Addons pull-request overview.
• The peak was 5,580 requests per minute, or about 93 requests per second, on this single page.
• 44,478 requests completed with HTTP 200.
• 148,923 requests ended with HTTP 500, often after triggering expensive database and Git operations.
• Forgejo briefly consumed about four CPU cores and 3.6 GiB of memory.
• The server load exceeded 80.

The traffic started declining significantly at around 22:11 CEST. Git operations such as clone, fetch and push were not the target, but the excessive load slowed down the entire service.

We are investigating additional global request and concurrency limits that protect the web interface without disrupting normal Git clients.

Sorry for the disruption.

!Friendica Support