Secondlife.Bio's Anniversary! (and Outage)
Hi Frens!
So Secondlife.Bio has hit our first year anniversary! 🙌
And then, we had our very first outage! 😢 So here's a breakdown (no pun intended..) of what went down (oh man..).
Some background information on our services - we run on scalable services to host the community-handles open source software (found here) - This typically requires a hosting service as well as a database provider. The software is designed to use postgresql out of the box, so we went with a serverless database provider. Since Secondlife.Bio was super new, we were on their free plan as it had enough resources for us to run everything. We calculated that we would not have to upgrade until we hit close to 100 users.
Great! All was working well. What we didn't realize was, Bluesky asks our system to look up who a handle belongs to like 5 times for each Bluesky lookup or refresh internally in their systems. Why 5 times? Well from what our investigations revealed, is probably they have their edge servers check and refresh their data themselves, instead of having one "registry". All of this was fine while we were less than 400 users.
But like a slowly creeping tide, the traffic and load on our free tier of services was exponentially growing until we hit about 480+ users, and we had an unusually busy weekend. That was enough to blow the database free tier capacity out of the water, so BOOM, our database froze up in the wee hours.
I woke up to an innocently looking message in Bluesky :
The secondlife.bio isn't showing up for your name
Dun dun dun.... Immediately, noticed that Bluesky was showing "Invalid Handle" under almost everyone who was online with the handle service! 😱 Quickly checked services and sure enough, the database provider had froze our service, pending upgrade payment. WHAT??
After reading their Terms of Service, came to realize that the free tiers would be frozen, should we hit the limit which was 100CPU Hours per month, until we upgrade our plan... So I upgraded on the post and services resumed!
Story over right? NO! It's never that simple 🤣 For all users who had been affected, Bluesky didn't fix it back for them! They remained at "Invalid Handle" until they did something about it.... whattt?!!
Immediately we sent out a post to help those who needed to fix their handle, but ALAS, trust was broken and rightly so. I had a chat with a few trusted contributors and advisors and we put a plan of action together!
So that's what happened in a nutshell, If there's interest in this blog and this format, we will share more details on what we did!
Secondlife.Bio Team