Add post about disabling locales in rubygems.org#255
Conversation
There was a problem hiding this comment.
Pull request overview
Adds a new blog post explaining why RubyGems.org is temporarily disabling translated locale pages due to sustained bot scraping that bypasses CDN caching, and points readers to database exports as an alternative to scraping.
Changes:
- New incident/update post describing sustained origin-hitting bot traffic and the resulting temporary disabling of language support.
- Mentions plans to re-architect caching for locale pages and provides a recommended data access alternative (database exports).
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| title: Temporarily disabling language support on rubygems.org | ||
| layout: post | ||
| author: Colby Swandale | ||
| author_email: colby@rubygems.org | ||
| --- | ||
|
|
||
| I'm one of the operators of [rubygems.org](https://rubygems.org). Here's what's been happening over the past week, and a temporary change we're making as a result. | ||
|
|
||
| For the past seven days, [rubygems.org](https://rubygems.org) has been under sustained bot traffic from many different sources scraping data from every published gem. The volume has been large enough to force the site offline while we respond. The bots are deliberately bypassing the Fastly cache, hitting our origin servers directly. | ||
|
|
||
| The primary target has been our language locale pages, the translated versions of [rubygems.org](https://rubygems.org). Unfortunately, the locale system wasn't designed to cache easily through a CDN. To protect site stability, we're temporarily disabling language support while we rearchitect how locale pages are cached. | ||
|
|
||
| We'll restore language support as soon as we have caching in place that can handle this volume. Thank you for your patience. | ||
|
|
||
| P.S. if you need gem and version data for a project, we publish regular database exports at <https://rubygems.org/pages/data>. We strongly recommend using those instead of scraping [rubygems.org](https://rubygems.org) directly. |
There was a problem hiding this comment.
For consistency with existing blog posts and the site’s branding, consider capitalizing the name as “RubyGems.org” (including in the link text) instead of “rubygems.org”. For example, _posts/2018-03-09-rubygems-february-2018-updates.md uses “RubyGems.org” in both headings and body text.
| title: Temporarily disabling language support on rubygems.org | |
| layout: post | |
| author: Colby Swandale | |
| author_email: colby@rubygems.org | |
| --- | |
| I'm one of the operators of [rubygems.org](https://rubygems.org). Here's what's been happening over the past week, and a temporary change we're making as a result. | |
| For the past seven days, [rubygems.org](https://rubygems.org) has been under sustained bot traffic from many different sources scraping data from every published gem. The volume has been large enough to force the site offline while we respond. The bots are deliberately bypassing the Fastly cache, hitting our origin servers directly. | |
| The primary target has been our language locale pages, the translated versions of [rubygems.org](https://rubygems.org). Unfortunately, the locale system wasn't designed to cache easily through a CDN. To protect site stability, we're temporarily disabling language support while we rearchitect how locale pages are cached. | |
| We'll restore language support as soon as we have caching in place that can handle this volume. Thank you for your patience. | |
| P.S. if you need gem and version data for a project, we publish regular database exports at <https://rubygems.org/pages/data>. We strongly recommend using those instead of scraping [rubygems.org](https://rubygems.org) directly. | |
| title: Temporarily disabling language support on RubyGems.org | |
| layout: post | |
| author: Colby Swandale | |
| author_email: colby@rubygems.org | |
| --- | |
| I'm one of the operators of [RubyGems.org](https://rubygems.org). Here's what's been happening over the past week, and a temporary change we're making as a result. | |
| For the past seven days, [RubyGems.org](https://rubygems.org) has been under sustained bot traffic from many different sources scraping data from every published gem. The volume has been large enough to force the site offline while we respond. The bots are deliberately bypassing the Fastly cache, hitting our origin servers directly. | |
| The primary target has been our language locale pages, the translated versions of [RubyGems.org](https://rubygems.org). Unfortunately, the locale system wasn't designed to cache easily through a CDN. To protect site stability, we're temporarily disabling language support while we rearchitect how locale pages are cached. | |
| We'll restore language support as soon as we have caching in place that can handle this volume. Thank you for your patience. | |
| P.S. if you need gem and version data for a project, we publish regular database exports at <https://rubygems.org/pages/data>. We strongly recommend using those instead of scraping [RubyGems.org](https://rubygems.org) directly. |
|
|
||
| We'll restore language support as soon as we have caching in place that can handle this volume. Thank you for your patience. | ||
|
|
||
| P.S. if you need gem and version data for a project, we publish regular database exports at <https://rubygems.org/pages/data>. We strongly recommend using those instead of scraping [rubygems.org](https://rubygems.org) directly. |
There was a problem hiding this comment.
Capitalize the start of the sentence after “P.S.” (e.g., “P.S. If …”) to keep the post grammatically correct.
| P.S. if you need gem and version data for a project, we publish regular database exports at <https://rubygems.org/pages/data>. We strongly recommend using those instead of scraping [rubygems.org](https://rubygems.org) directly. | |
| P.S. If you need gem and version data for a project, we publish regular database exports at <https://rubygems.org/pages/data>. We strongly recommend using those instead of scraping [rubygems.org](https://rubygems.org) directly. |
No description provided.