The Path to Liberty from the Cloud

As many know, we've embarked on a journey to liberate ourselves from (most) cloud services. As many service providers have found, the "cloud" is pretty inexpensive as you start but grows exponentially as the service grows, often to the point that keeping the service in the cloud doesn't make sense.

There's a sour spot between being small and being large where the RoI (Return on Investment) is truly not worth it. This is where we find Mastodon Canada today.

While we are the largest Canadian Mastodon instance on the Fediverse and the largest federated community in Canada, we've found ourselves in that expensive sour spot. We've been here for the better part of two years. Thanks to so many of our generous donors, we've been able to explore other options but they have made no appreciable difference. So, we've turned to purchasing our own hardware and maintaining it ourselves.

I'm pleased to say that we've finally had a very capable server arrive from Supermicro this past week after weeks of struggling with FedEx to get it across the US/Canada border. It is more than capable of serving Mastodon Canada and more*, consuming only 150W of power under peak load.

As I assessed this scenario, I considered colocation options. Specifically Wolfpaw, Rogers, and YEGTEL. While I can't share the cost of a 1U server at these providers, I can say that we wouldn't save any money for the benefits a data center provides that we can't tackle otherwise.

At its basics, a data center provides redundant power, internet connections, and conditioned air. We can tackle the first two rather easily. First and temporarily, an overspec'd UPS (1500VA utilizing 10%) will provide over an hour of backup power in the event of an outage. Second, using tunnelling via Cloudflare, we can utilize the existing backup internet connection I have with TekSavvy at my home to failover in the event TELUS Fibre has an outage. All of this hardware is backed by a separate 1500VA UPS.

Over the next year I have road-mapped installing a natural gas backup generator to maintain power. I expect we'll have that installed by July 2025. This is not a cost to our community but instead part of my resiliency plan for my home. Our utility provider is very reliable overall and has only had two 30-minute outages in five years of my living in this neighbourhood. The UPS will sustain the community if one of these rare outages occurs over the next 11 months.

In the interim, if an extended outage were to occur, we would have a gasoline generator that would be routed to the server closet to maintain the instance. I work from home, so an outage like this impacts not only the community but also my livelihood. Getting this generator up and running is the first thing I think about when a power outage occurs to make sure not only our community but also my furnace continues to run if the worst were to happen during the winter.

Finally, backup. We're going to retain our OVH Proxmox Backup Server to back up our containers and virtual machines weekly and stream our PostgreSQL database server in real time. We will auction off the other OVH server's lease which still has two years remaining. If you're in the market, let me know.

Our media, now climbing close to a whopping 2TB, will remain with DigitalOcean. I've tried reaching out to Fastly to join their "Fast Forward" program, but after first contact, I never heard back. Once the dust settles, I may try to reach out again. As it is, DigitalOcean offers its affordable Content Delivery Network (CDN), which significantly reduces the amount of bandwidth we consume and makes sure our members' media content gets out to other instances as quickly as possible without error. There's no scenario in which we move our media onto our new server where our members don't see a significant degradation in their post's reach. This also applies if we were to migrate it to the dedicated OVH server. A CDN is critical for what we're doing here, and DigitalOcean's offering is infinitely cheaper than us spinning up replication servers on each continent.

Ultimately, and most importantly, we're after reducing our operating costs. Times are tight, and everything costs more these days. I have a complete understanding of when folks choose to cancel their subscription donation to our community. The table needs food and that comes first. Second, we're after longevity and the ability to assure the community we'll be there for years to come as a source of news, current events, and community.

So, let's talk about how this migration process is going to take place...

Roadmap

(dates the week of. High-level estimates as of Aug 24, 2024, subject to change)
Aug 18: Server commissioning, performance testing, network integration
Aug 25: Tertiary services migrated (news.mstdn.ca, LibreTranslate, ElasticSearch, and elk.mstdn.ca)
Sept 1: Hold to analyze the performance of the tertiary services
Sept 8: Mastodon web services migrated, databases remain at DigitalOcean.
Sept 15: Redis migration; it is expected that timelines could be erased, but we are seeking to mitigate this.
Sept 22: PostgreSQL database migration; it is expected that we'll have a four to six hour hard downtime this week.
Sept 29: Hold to analyze the performance of primary services.
Oct 6: Vacation. Continued analysis of primary services.
Oct 13: Wind down and deletion of DigitalOcean virtual machines, load balancers, and databases. Planned upgrade to stable Mastodon v.4.3.0 if released.

Current Status

Aug 24: Server commissioned, news.mstdn.ca migrated. You're reading this on the new server. ElasticSearch and LibreTranslate migration initialized.
Sept 5: LibreTranslate migrated. First dry run of the PostgreSQL server.
Oct 13: Web services have been migrated, however the Streaming API is causing some troubles. Holding for one week to troubleshoot this and analyze our performance. Redis, PostgreSQL, ElasticSearch, and Elk are left to be migrated.

Please keep checking for updates. We endeavour to provide updates every Sunday.

Thank you to all who contribute financially to our community. Without your support, we couldn't keep our wonderfully vibrant and welcoming community alive.

Footnotes

* Pixelfed is very much on our horizon as long as the server allows. If you have ideas on what else we should host, email us.