How we overengineered DevHuddle and paid the (literal) price.
Engineers engineer. That’s what we do! Sometimes (a lot of times) we are guilty of overengineering. When exploring the first version of how to deploy the architecture for DevHuddle, I (Rob) looked to cloud native technologies to alleviate us from having to manage infrastructure. Oh, and that whole bonus of being able to handle large volumes of traffic right out of the box. Win-Win.
As of this writing, we are hosting DevHuddle on Google Cloud Platform. Google Cloud Functions is one of their serverless offerings. On the surface, it was a great fit for our backend architecture. Where I went wrong, was over-functioning (wordsmithing!) our codebase in an attempt to compartmentalize functionality.
The idea was that each major area of the backend would be its own function. File upload, channels, messages, etc – each one of these would be its own. In the example above, a function exists to handle incoming webhooks from GitHub. This function is completely separate from the function that would handle channel management. If this is starting to sound a touch overengineered, you’re not wrong.
In addition to functional isolation, each function would have its own database. There were some exceptions to this rule. Functions triggered via an event from the message bus (PubSub) that were in the same functional area as their HTTP counterparts could share a database.
The Drawback
Serverless functions can scale to zero – yay! As a fledgling company, we need to save as much precious cash as possible. The downside of scaling to zero is that going back to one comes with a startup time penalty. For most of our functions, that simply wasn’t acceptable. The fix was easy – set the function to scale down to 1. Ship it!
The easy fix turned into an expensive fix…to the tune of a 300% increase in our next month’s invoice.
At this point, it was time to rethink compartmentalizing all of our functionality. Take our overengineered masterpiece and simplify it to something…well, simpler!
The Solution
Thankfully our solution was relatively simple – collapse all of the functions which handle HTTP events into a singular function. Note how I said simple, not easy! Getting all of the database code consolidated was an exercise in patience.
Looking forward, I think the most cost effective solution will be to spin up instances that sit behind Managed Instance Groups….but that means managing servers. Gross. 🤮Kicking the can down the road on that problem. For now, we need to go get some more users on the platform!
What serverless technologies have you successfully used? Stop by and share your experience with the rest of the DevHuddle crew: https://app.devhuddle.ai
