Recently in a code review session for a product under development, go to know the team was using Firebase for in-app notifications. Looked like a reasonable choice nevertheless reminded me of the infamous story how a startup almost went bankrupt using Firebase. I shared the story with the team & requested them to apply learnings if any from the same.
The story is really about how a startup got a bill of 72K USD overnight after launching their application. Following are few wisdom nuggets from the story. I jumped out of the bed, logged into Google Cloud Billing, and saw a bill for ~$5,000. Super stressed, and not sure what happened, I clicked around, trying to figure out what was happening. I also started thinking of what may have happened, and how we could “possibly” pay the $5K bill.
The problem was, every minute the bill kept going up. After 5 minutes, the bill read $15,000, in 20 mins, it said $25,000. I wasn’t sure where it would stop. Perhaps it won’t stop? After two hours, it settled at a little short of $72,000.
Techies should NOT skip the part 2 of the story where the author talks about the code in his application that led to this and how he fixed it. Here is a summary from part 2 for the busy bees.
While creating a Cloud Run service, we chose default values in the service. The max-instances is preset to 1000, and concurrency set to 80. In the beginning we didn’t know that these values are actually worst-case scenario for a test program.
Had we chosen max instances to be “2”, our costs would’ve been 500 times less. $72,000 bill would’ve been: $144. Had we chosen concurrency of “1” request, we probably wouldn’t have even noticed the bill.
Cloud overall is like a double-edged sword. When used properly, it can be of great use, but if used incorrectly, it can have consequences.
If you count the number of pages in GCP documentation, it’s probably more than pages in few novels. Understanding Pricing, Usage, is not only time consuming, but requires a deep understanding of how Cloud services work. No wonder there are full time jobs for just this purpose!
When the story was published around 2020, it was discussed quite a bit on internet. Here is an interesting thread from a Y Combinator forum.
Generally, quite a few folks who launched their startups successfully on cloud are leaving the cloud. A recent example is Basecamp & HEY quitting on cloud. It is estimated they will reduce their $3 million annual AWS bill by an estimated 60% . David Heine Meier Hansson, Co-Owner & CTO of Basecamp & HEY has written a nice article summarizing why he is leaving the cloud . According to the article, cloud makes sense only in following cases.
When your application is so simple and low traffic that you really do save on complexity by starting with fully managed services
When you have wild swings or towering peaks in usage. When the baseline is a sliver of your largest needs. Or when you have no idea whether you need ten servers or a hundred
Another example is from SEO crawler Prerender who reduced annual server costs by 80% — from $1M to $200k — by moving away from AWS. The whole journey has been documented in detail. Here are few wisdom nuggets for busy bees.
Moving data around is where you will start running into huge bottlenecks. The true hidden price for AWS is coming from the traffic cost, they sell a reasonably priced storage, and it’s even free to upload it. But when you get it out, you pay an enormous cost.
Small startups often don’t calculate the traffic cost, even though it can be 90% of their bill.
For example, if you are in the US West(Oregon) region, you have to shell out $0.080/GB whereas in the Asia Pacific (Seoul) region it bumps up to $0.135/GB.
In our case, it was easy around the $30k — $50k per month. By the end of phase two, we had reduced our total monthly server costs down by 41.2%.
A common thread, why folks want to quit cloud appears to be their outbound data transfer costs. These clouds trap you by allowing you to upload data for free (ingress) but if you want to transfer out your own data (egress) they will charge you a bomb.
Cloud is very unforgiving to developers. A small mistake in your code or not paying attention to billing plans and you are punished immediately. E.g., You do error handling in AWS lambda by retrying an operation, your bill will go up. Clouds in some way forces you to become better coder by writing code that is compute, memory & storage efficient. Unfortunately, the cost you pay to learn this could be very high.
So next time you are building your app & want to use a cloud service, remember this blog. Cloud is part of our life & cannot be completely ignored. But think hard , do your technical & commercial due diligence thoroughly and then may be take baby steps especially if you respect cockroaches more than unicorns.
