Work Text:
I have been a Sysadmin for 33 years. For the last 18 years I have been working at charity that has a sizeable estate of hardware. While my day job is not directly relevant to this talk I find it useful to be able to install bare metal machines at scale, Not to be afraid of large systems. And be used to specifying hardware.
I am a Amateur SRE
I have been a volunteer for organisation for transformative works for nearly 14 years and for the last 11 years I have been the technical lead of the systems group, and a member of AD&T, the group that have commit access to the repo and designs new features.
The Archive was created because fan fiction kept on being deleted. It is part of a gift culture.
We are inclusive of content - Queer content was often removed from other sites
Because of this we are blocked in Russia and China.
We want to resist censorship, whatever the source.
This means we reduce the number of levers others can have on us.
For example not having advertising means we can’t be forced to drop content due to losing advertising revenue
We collect minimal data, if we don’t have it we can’t be forced to disclose it.
And as a point of principle we own our servers to make it harder to de-platform us
AO3 source code
AO3 terms of service
The organisation for transformative works Is a 501(c)(3)
The org has an Elected board and collects donations
The main product although not the only project of the organisation for transformative works is the Archive of our own, known as AO3.
These are my own views, I do not speak on behalf of the organization for transformative works.
Google implemented prefetching for us.
This is from the introduction of a document that was shared.
The last line says “Because of its popularity, improving page load speed for them can move global average user experience.”
I have seen cloudflare report, that the Archive had over one hundred million page views in a single day. The least I have seen is about 87 million.
We have had about 18 hours of downtime this year. It costs us nothing to be down.
For us putting the site in to maintenance mode means that some deploys and changes are considerably easier.
Our only significant variable cost is how much we spend on new hardware.
In 2023 we bought some storage and virtualisation systems, we spent 270 thousand dollars in total, or about $740 dollars a day.
2024 Budget Update
In 2022 we bought three expensive database servers at about 67 thousand dollars a machine
OTW Finance: 2023 Budget Update
We spent about 44k on new hardware in 2021.
OTW Finance: 2022 Budget Update
Having no employees means that all work on the Archive is done in people’s spare leisure time. But it does mean that we don’t have to worry about paying for people's salaries.
Given we have no employees how does anything get done at all ?
We have a large number of volunteers, however some people are in multiple groups. The two groups I am a member of :
- AD&T, has 11 people
- Systems has 8 people
And both groups share two people between them, not everyone turns up to the meetings every week, not everyone does work every week
Our choices and therefore environments are shaped by our constraints. The choices you make have to be right for you and your constraints.
- We don’t have to make a profit, most people do.
- All of the workers are volunteers, this means that things go slowly
- Our founders were worried about being deplatformed.
- So we have no advertising, only donations.
- The less we know about our users the better. We can’t tell if a user is a donator or not.
- Worrying about being deplatformed is the main reason why we own the servers
- We have no incentive to drive people back to the site, it costs us money to provide a service
- An example of this is the conscious choice not to have an algorithmic recommendation system.
- We care about preserving fic, if we are offline it's not a complete disaster
- The choices made 14 years ago still steer our decisions today.
- being able to vote in the election for the board of directors
- small bits of merchandising.
- We have no employees, we want keep things as simple as possible.
- Our support costs with MariaDB is based on the number of instances not the size of them.
I joined the org in January 2011
Not long after our MySQL server was going really slowly, it was being crushed under the load of requests, mostly sorts. So we migrated the MySQL tmp on to tmpfs and things went much better for a bit.
Nine months later the database server was yet again being crushed under the load.
This time after a lot of discussion we disabled a core, feature tag searching and filtering as we didn’t have the hardware to do this using MySQL and we didn’t have the enough money to throw hardware at the problem.
We also moved the servers behind a firewall, I was relieved, even though I missed a dot in a zone file…
It took another six months before the tag searching and filtering was re-implemented using Elasticsearch pre 1.0, We didn’t have any spare machines for elastics servers so we colocated elastic on the application servers.
This was actually in many ways a harder time to look after the Archive than today as we were beginning to get a fair amount of traffic ( many 10’s of pages a second ) but we hadn’t learned how to handle the load and we didn’t have a lot of money to spend on new hardware.
In 2012 we moved from pets to cattle by moving to cf-engine instead of making changes by hand.
Migrating to Galera has meant that I got a lot more sleep. Having three MySQL servers rather than one is a big improvement.
When we migrated text field from utf8mb3 to utf8mb4 we learned to love pt-online-schema-change.
COVID and lock down meant there was a 20% increase in traffic, going from 280 million page-views a week to 340 million 2 weeks later ,We had to disabled features to reduce database load until they were re-implemented.
Last year we were hit with a DDOS attack that we could not ignore, Our data center reported that they saw traffic exceeding 1.2 terabits per second for short periods of time.
We needed help from a DDOS protection service . We created a free account with Cloudflare
And at the same time we made a request for a business account with our finance committee.
A fan who was working at Cloudflare and knew I was a volunteer reached out on discord and guided me through requesting support from project Galileo, which is Cloudflare’s assistance program for vulnerable organisations.
We were back on line 5 and half hours after we created the free account, running under project Galileo.
Later attacks had apparently peaked at 65 million requests per second
The Archive of our own
The AO3 July/August DDoS Attacks: Behind the Scenes
Hardware is not our only expense, our hosting fees are dependent on how many racks we have ( currently 3 ) and how much bandwidth we use ( about 600MB/s ) .
We also pay for spam protection and an issue tracking system.
There are two donation drives a year.
The majority of our income comes from these.
We rely on our volunteers and passionate small dollar donors
We have about 15,000 members of the org each of which will have donated a minimum of ten dollars, the only privilege is :
Companies provide free services for, not for profits. Sentry offered us a free service when our previous APM provider withdrew the product we were using. When the Internet links to the Australasia area were congested and people were having trouble connecting to us, Cloudflare added Argo smart routing to our free account, which has a list price of, I think of about, $17,000 a month for our bandwidth
The systems committee are the org’s SRE team
We use FAI to install the operating systems on bare metal machines
And Ansible to configure the operating systems and applications stack.
For a change I have the acknowledgements in the middle.
AD&T are the core contributors who steer the project, we design features and choose the contents of each release.
We have access to the rails console and use this to provide support to other committees.
I do want to highlight, that even though we have good code coverage at 93%, All issues are still manually verified by two different people ( not including the coder ).
Problems do still get past us in to production, It's hard for us to test at scale.
We are a Legacy app, Hindsight is a wonderful thing, but I still think rails is a good choice today.
One of the reasons we chose rails was that it follows convention over configuration. If you follow the conventions then lots of work is already done for you.
The choice of rails was steered by one of the early ideas: We wanted lots of people to be able to contribute and for them to be able to do so even if they were beginners.
We use rails in the normal way that relies on a relational database, so migrating to a NoSQL database for the core data is not an option.
I have found that MySQL is hard to scale compared to elastic and simple app servers.
Shout out to sarken who really keeps everything together.
We are not a normal open source project
Only some people inside the org can make an issue.
We only accept pulls requests for which an issue has been created.
Other groups have brought up instances of the archive, however it is not easy
The Archive software is not designed for others to use easily, there are lots of moving components and lots of assumptions about how the software stack is deployed.
Implementing OTW's Code To Build SquidgeWorld
The Archive was created to protect fanworks
We work on the principle of a gift culture where effort is gifted to others without an expectation of future reward
This is why using the Archive for commercial gain is against the terms service.
Why I Support The OTW, by Speranza, aged mumble-mumble
An Archive Of One's Own (post by astolat)
These elasticsearch servers were bought with the best single processor we could buy and plenty of RAM
Note the cpu usage which is on the high side.
Later we will talk about what happens when we run out of resources.
The redis server was bought to have good single threaded performance, as redis uses only a single core per instance (at this time)
We have 5 database instances so we more have database capacity via sharding ( on a single machine ), and we use each instance for a different purpose eg autocomplete, async jobs, feature flags etc.
None of the data in redis is critical, it can be regenerated, we can survive the loss of the service , losing at most an hours reading history, however the website would be down until we moved the service to a new server.
Note the consumer disk, this was a mistake
When the disk was failing it’s smart tests we announced downtime, giving 24 hours notice.
We prepared an old application machine as the new redis server and then took the site down in our change window.
We were lucky.
We got 5 years of uninterrupted use of redis without having to worrying about the system
And yes I know all the reasons why this is awful. But it did work for us.
When a service has been running uninterrupted for more than 5 years you find that lots of things will have changed in that time.
For example we found that one of the instances now took more than 90 seconds to load, meaning that systemd needed fixing before it would start.
While we currently don’t use sentinel, we had started preparing for it and having sentinel half configured caused the instances to go read only.
If you are relying on your own hardware then you need to have plenty of excess capacity.
Note the order date February 2022 and the production date January 2023, 11 months from ordering the machines to them going in to production.
We ordered these while COVID was affecting delivery times, after quite a few months we had the three servers delivered and physically installed. And then we discovered we had one server had been delivered with a bad processor and that machine had to be returned. As the machines were to be a cluster and two machines by themselves were not sufficient, we had to wait until the final machine was returned before we could finish the work. Although the other machines were used to test the migration.
At this time we were on Percona’s version of MySQL and that worked fairly well. However we have been buying support from MariaDB and not using their version of MySQL seemed to be a mistake.
Migrating from Percona to MariaDB was really painful, with MariaDB not reading a modern MySQL servers database files, we had to have an older version of MariaDB as a secondary to the primary cluster of Percona and then use secondaries to upgrade the database format to a modern MariaDB system.
We buy big machines and this is unusual, Most people would shard their databases and this is supported in rails 7 although it would add complexity to the code base ( and that complexity would be seen by other instances of the Archive )
So why don’t we shard: ?
In general we say that you should protect the databases, as they are hard to scale. Saying this I have seen transactions being open for more than 20 minutes from our asynchronous workers, and I have not found the time to chase this down.
We found that MariaDB has different edge cases than the Percona variant.
Galera Cluster is a really good multi master version of MySQL and in combination with MaxScale, Galera allows us to scale MySQL server horizontally ( the maximum number of servers supported is 15 ), if you are using galleria, then pt-online-schema-change is essential, it makes it appear as if a migration happens instantaneously with the magic of triggers and a background data migrator that swaps tables over at the end of the process.
Why did we choose Galera rather than a more traditional multiple tiered replication system ?
Galera with MaxScale appears to developers much more like a single database system, there is no additional application code to handle multiple servers. As an SRE I spend very little time doing production replication work.
MariaDB works really well until it doesn’t,On the left hand we see our normal workload with a full backup running on one of the servers.
However on the right we see an example of a cascade failure, sometimes the only way to recover is load shedding.
If we find ourselves with only a single database server running and the others having to do full state transfers rather than an incremental we take the site down to minimise the possibility of data loss, this downtime is acceptable for us, we generally bring the site back when we have two of the three MySQL servers running.
Having really big servers is unusual, systems this big are not tested until people like us get the release.
I am practically certain I am going to move to 5 MySQL servers in future so we have more resilience and so I can spend less time worrying about MySQL.
Metastable Failures in Distributed Systems
We have about 33,000 queries per second.
I believe we have about 40 queries per page and the only way we can get away with having so many queries per page is by serving them really quickly, each page spends about ten milliseconds in MySQL queries.
If we end up with a large number of queries running in parallel which do not die down naturally we can end up with a cascade failure. Although this is really quite rare.
We use a MySQL proxy, MaxScale which has an embedded SQL engine which we use to detect writes and steer those to a nominal primary to reduce conflict, and to steer reads to the fastest available server.
MaxScale also has query throttling and maybe I should look at for asynchronous workers ?
However I am a amateur ,there is no free time, it’s not on fire at this instant so it may end up on an infinite backlog, or maybe one of the other system people will take on as something interesting and it will be investigated.
One of the costs of having only volunteers is that no one can be ordered to do a task, work is only done because someone thinks its worth spending some of their leisure time doing it.
Database servers require lots of ram and fast disks. An advantage to buying your own hardware is that you can specify the components, at least with super-micro rather than Dell.
Given we have so much ram we tried to use normal NVMe disks ( a mirror of 6 striped Intel P5510 ).
However the disk array could not keep up and we needed to get a stripe 5 of Optane, these disks were hard to locate, we had to use multiple vendors to have sufficient disks.
I believe that latency is really important but it can be hard to find out for most hardware.
Intel Optane was a wonderful product, I am really disappointed that it has been discontinued.
So much so that when we started thinking about the next generation of database servers I decided to buy the Optane disks while they could still be bought before we decided on the rest of the hardware.
I hope CXL will allow low latency storage and that people produce it.
Conventional wisdom says that you should set the innodb_buffer_pool to about 80% of your free ram, with 4TB of ram this would be about 3.2TB however after talking to MariaDB we agreed that this was the cause of most of the pain we were seeing.
By setting the innodb_buffer_pool to a relatively small size ( 128Gigabytes ) we have reduced those problems.In one way we are nearer the standard configuration ( a more standard amount of memory for the innodb buffer pool ) in another further away ( we are relying on the vfs cache ).
If you end up having to use a large vfs cache you also need to make sure that you do not let dirty writes accumulate. If you do then you will see IO stalls as the dirty pages are flushed and your database will freeze until the flush completes.
We also moved some autocomplete functions from MariaDB where they took about 90 seconds to Elasticsearch where they take 25ms.
It has been a long time since we bought a server just to be an application server.
In general I buy 3 MySQL database servers every 2 years and the old database server get recycled into applications servers.
At this instant we have about 1000 cores for our users load and about another 600 cores for everything else.
Memcached is really simple isn’t it ?
As I said we recycle old database servers to be application servers which means they have a lot of memory left.
So the obvious thing to use the spare memory is a cache.
Each page has on average 38 lookups, taking in total about 5ms per page.
The graphs here show an issue we found
Earlier in the day a deploy had completed, leaving a node in a strange state.
That node was running out of TCP sockets and while we didn’t notice: health checks were hiding any issues with the app server.
The socket exhaustion was making Memcached perform poorly. Once we fixed the issue Memcached did run smoothly again.
However this demonstrates that co locating services while being cost effective does mean that systems are more complex and harder to maintain.
You don’t need much hardware to deliver 2 million emails a day however you do need to get the systems configured correctly to get the emails delivered:
Spf ( sender policy framework ), dmarc ( email authentication ) and keeping a good ip reputation are all important
I did use SES for this AWS comparison as I thought that bring your own ip was out of scope for the comparison.
I just want an opportunity to complain about American power, if I pay for 30 amps of power I think it is reasonable to be able to draw up to 30 amps of power, not be told that only 24 Amps is usable because otherwise the cables might melt.
This is a summary of the hardware we have and a rough guess at AWS prices using 3 year reserved instances paid in advance. I am aware that discounts do exist but these are list prices.
This is from our annual reports which includes our expenses over the last three years.
And an estimate of the costs of running the site on AWS using 3 years reserved instances paid in advance
This is just lift and shift which is not the right solution to a cloud migration
We all have a complexity budget, and we choose to spend it in a different way.
We choose to keep things as simple as possible with no dynamic control plane to go wrong, I say this but we have maxscale and galleria
We choose not to build Kubernetes clusters or to use openstack, we don’t get enough advantage to pay back the complexity that those systems have.
Cloud services do have their advantages it is a method of swapping capex for opex. It has the concept of pay as you go which works really well if your infrastructure costs scales linearly with profit. Cloud Services allows people to outsourcing complexity for a fee.
Given we don’t use Cloud services for most things what do we do to reduce our complexity ?
Moving IP address around is unpleasant and the systems that do it well are fairly complex.
So how do we do load balancing in a simple scalable way ?
We have an instance of haproxy on each machine that needs to access another service, be that our front end machines or our application servers or even our database servers.
We have no single point of failure for our distributed load balancers.
As well as haproxy we have a MySQL application specific proxy MaxScale which works well.
We could also have an elastic coordination nodes on each application server however this is not recommended by elastic as each coordination node needs to be part of the elasticsearch cluster.
Being part of the cluster would mean that they would have to receive cluster state changes which has scaling implications.
I am going to talk about how we use haproxy both to shape our traffic and as a simple circuit breaker to protect Elasticsearch in particular.
The user’s browser connects too Cloudflare and Cloudflare connects to our pfsense firewalls,
haproxy on the firewalls connects to nginx on our front ends which inspects the url and method and tags the request
Nginx then sends the request to the local haproxy on a specific port on the frontend server.
Haproxy on the frontend servers then shapes the requests dependent on the port and forwards the request to unicorn which is an old web server for rails on the application server,
the rails app inspects the tag from nginx and uses that with a gem, rack attack to apply any rate limits.
When the rails app needs to talk to a database it will connect with haproxy on the application server to contact elastic or MaxScale to contact MySQL both proxies running locally on the application server.
Here we see an url path, which places a high load on the elasticsearch servers, but is relatively low use.
If there was lots of traffic then it would exhaust the elasticsearch servers capacity.
When it is heavily accessed we fail quickly allowing users to continue to access the site. This is working as a circuit breaker.
This is a snippet from our nginx configuration.
We access a variable called upstream_result the next slide will show how that is set.
We set a header so we can read it in the application later, this is used for rate limiting.
And we go to a upstream dependent on the variable.
Lastly note that we log nothing
if we don’t log browsing data we can’t release it if we were subpoenaed for it.
It's possible to read on the Archive without an account completely anonymously. If you do have an account then you would need to turn off your reading history.
The Archive does record some ip address that are used to change the state of the system, for example posting a work or making a comment.
Technically Cloudflare does record a sample of browsing data for 30 days so we may be able to see some of the information.
Here we set the upstream_result variable, which is dependent on the method,and for a GET we return the cache variable.
The cache variable is dependent on the request_uri in this case the expensive uri returns unicorn elastic bookmark
We can see each of the tags has a different upstream defined.
Different names can use the same back end port meaning that the that traffic is collected together and shaped as one.
However the traffic is treated differently at the rate limiter.
This is the relevant snippet of the haproxy configuration.
While we are using this to protect Elasticsearch the same method could be used to protect any api server.
The time out controls how long a request can sit in a back-end queue. This is effectively the circuit breaker component.
A back-end can have multiple back-end servers but we are going to consider an example with only a single back-end server.
A back-end server will accept Minconn requests before requests are queued, and will accept at most Maxconn requests at any point
Fullconn is the amount of requests sent to a back-end before a back-end server will be sent Maxconn amount of requests
There is a straight line between Minconn and Maxconn which is different per back-end server.
This is with a single back end server to simplify the arithmetic.
This table shows for each back-end queue length , how many requests will be processed by the back-end server simultaneously.
Minconn is set to 2 so the first two requests go straight through to the back-end server with no queuing
Fullconn is 8 so at 8 requests in the back-end queue, Maxconn requests (that is 4 requests ) are being processed simultaneously by the back-end server.
When there are 5 requests in the back-end queue then 3 requests are sent to the back-end server.
We are assuming that each request takes a single time slice to run.
At time 6 we can see we have one request queued, as we received three requests and we only process two requests.
At time 10 we get a sudden request spike of 27 requests
We let 4 requests in and continue to let 4 requests in until the queue is drained, in this case none of the requests stay in the queue for more than 8 seconds so they are all processed by the application and the time out or circuit breaker has not come in to effect.
We can see that the red line which is the requests sent to the back-end has been smoothed and therefore the Elasticsearch servers have been protected, the queuing acts as a shock absorber for the database.
What happens when you run out of capacity in one of the areas ?
Here you can see that we were hitting the limits of cpu on our Elasticsearch servers.
On the left hand side you can see the elastic cpu, with peaks of 100% visible on short term graphs, that is Bottom left
Servers take a long time to deliver, It might take 3 weeks to get a quotation, 3 weeks to get it financially approved, 10 weeks for delivery, 2 weeks for the machine to be installed in the DC and then maybe a week to install the machine and get the application running on it. That would be a total of 4 months which is probably optimistic.
This is one area cloud really does have an advantage.
Our application servers are mostly ex MySQL servers and really quite capable.
To solve the lack of capacity in Elasticsearch we can redeploy an application server to a new role.
The link at the bottom, is actually a postmortem about the side effects that reducing the application capacity and how that produced a much larger impact than one would expect.
Postmortem of August/September 2024 Archive Issues
We had increased the number of servers we have in our Elasticsearch cluster, but we were we still getting the circuit breaker kicking in ?
I had forgotten to increase the number of requests we could process.
Once I did that the load on the elastic cluster went up and the the errors went down.
Lastly rate limiting, we use a gem “rack attack” which implements rate limiting.
We have this little snippet of code in the Archive’s initialiser that defines the rate limits.
In this way the deployed code is unchanged but we can control rate limits by adjusting the application configuration.
We can adjust the URI’s that go to each rate limit bucket by changing the map statement in the nginx configuration.
Therefore we can adjust the rate limits dynamically and the url patterns they use. All without a code deploy it is just configuration.
I do want to reiterate that this local rate limiting solves a different problem from the rate limiting we have in Cloudflare.
The rate limiting we have in Cloudflare protects us before their DDOS protection kicks in.
Large scale attacks need to be stopped before being routed to us.
The traffic shaping we have locally protects our database servers.
The local rate limits are much finer grained.
Counting in MySQL is hard, we can see that counting 13 million things takes a long time, having our front page take over 4 seconds to generate is less than optimal.
However there was resistance to caching these numbers, the user population really liked seeing these numbers go up.
By using approximate numbers you get users used to the idea that numbers can be the same when they make a change so that they accept the cached number.
Currently we only do this rounding on the home page.
By having multiple pools of applications servers for different tasks a problem in one pool will not affect the other pools.
This is assuming that you don’t bring you common resources, your database servers down.
Our previous APM charged us per server and we wanted to concentrate the APM on the machines that users directly interacted with. However it has other advantages.
Here we can see that our application servers which handle spiders, bots and AI scrapers had a higher load, this did not affect our real user traffic.
We don’t make money.
There are times when being in planned downtime to make a change means that making the change becomes much simpler.
Applications often have outliers that should not be counted, for example the orphan account is a special account that allows people to disown a work without deleting it.
We optionally allow users to leave a pseudonym on the work.
There is a page which displays the a list of pseudonyms for an account, the orphan account is an outlier.
Previously we blocked this page in nginx.
We did some work to make the page load more quickly and deployed it.
Our database load went up by a factor of 15 and the page for the orphan accounts pseuds still took more than 30 seconds to load.
We decided to restore the block that we have in nginx.
We needed to replace the software we using to generate pdf’s, it was not getting security fixes.
Calibre uses significantly more cpu to do the same task, however we choose to stay with it because it is only affecting application server load and not the database
And because we care about security, and when a similar issue was discovered in calibre there was a security fix.
Choose scale out over scale up ( obviously )
Elasticsearch works really nicely for this.
Galeria allows us to scale out MariaDB but the support costs being per instance pushes us towards scale up rather than out.
Scaling MariaDB up means we hit interesting scaling issues.
We pay for support where its not feasible for us to be an expert.
Buying hardware is both an advantage and a disadvantage.
It means we can specify our hardware environment completely.
It means we can’t change the amount of hardware we have dynamically.
You need to buy a lot more hardware than we strictly need. Otherwise you can not react to a sudden change in load or a service needing more capacity.
We do use cloud services our total bill is around $550 a month, which is fairly small ( mostly S3 which we used to host avatars ).
We are moving from an aws account created in 2012 to a set of accounts under control tower.
Storing MySQL backups in an S3 compliance mode bucket would make it hard for a rogue systems administrator to delete all the backups.
As the backups would be stored in a under a different administrative domain. This is not something that can do on our own hardware.
I want to thank everyone who has listened to this slide and given feedback.
Thank you