Scaling at Home: Part 1
I've been wanting to brush up on system design skills with some hands-on experience without the overhead of having to be concerned about production or cloud provider costs. This is one of the main reasons I built my own home server. It allows me to practice my skills and see the cause and effect with little to no consequences.
I'm actually going to be breaking this down into separate parts because I feel it is easier to reason with, document, and index all of the moving parts without everything turning into an even bigger wall of text.
Project Goals
The goal is to create and deploy an application that is composed of a few different microservices.
Ideally, the infrastructure would closely resemble a production environment that would be deployed on any of the major cloud providers, such as AWS or GCP.
At a minimum:
- The application and all of its supporting services should be easily distributable.
- The deployment should be done entirely through IaC.
- Creating and destroying the environment should be as simple as possible, and the results must be idempotent.
- The application and its underlying infrastructure should not interfere with the other services running on my server, remaining isolated, as much as the environment allows.
The Application
The application is meant to be a simple job/task execution pipeline.
The point of this exercise was not about the actual application itself, so I used an LLM (Claude Sonnet 4.6) to quickly implement based on the technical requirements I provided in each session.
While I was very specific about how the application architecture was designed, and even added test coverage, this project is not suitable for production, nor does it intend to follow all of the best practices.
Services
I went with Python and FastAPI to create the main application layer, responsible for managing DB models, managing real-time streams, as well as providing API endpoints for client applications to consume.
There are two PostgreSQL instances, a primary read/write and a read-only replica. If the replica goes down, all read queries are then routed to the primary as a fallback.
There are two Redis instances, one for handling async tasks, and another for Redis Streams to handle real-time communication.
There are two services that are written in Go, a general-purpose notification service and a task execution service.
The notification service handles job/task status updates, which are pushed onto the Redis Stream and made available for consumption by clients.
The task execution service is responsible for watching and pulling a task from the queue, spinning up an ephemeral container, and executing the supplied task payload in the isolated environment. The container is then torn down, and the results are persisted to the database.
What's next
As intended, the application is overly complex for what it does and has a lot of moving parts, which gives us a lot of different options for deployment.
Part 2 will cover the underlying infrastructure that the application will be running on, as well as explain what could be done to improve performance and availability as if this was a production-ready system.