About Scaling an Application

On the left you can see how a basic web app looks like. Let’s assume that as the vendor of AI-powered chat application is looking for its first client, their application runs on the bare minimum. There is one DB, one server running the back-end of the application and one server serving the frontend for the user devices. From here on, I will discuss the options for the company to scale up its software as its customer base is growing.

The topic of scaling an application is like a bottomless well. It’s hard to find a place where to stop. Therefore, I decided to stop after writing out 6 options.

Increase the number of instances of your backend and frontend

Increase your compute power or improve your ability to serve simultaneous network requests by simply adding more machines to do that same thing.

What is needed to make this work? Your services need to be stateless.

Pros: Increase in the throughput of the application proportional to the increase of resources you use. When you 2x the number of instances, you get 2x throughput for that service (overall performance benefit might be smaller because the performance bottleneck might shift elsewhere).

Cons: None. No reason not to use this option for stateless services.

Limitations: Eventually the database will become the bottleneck and then more complex solutions will be required.

Move your DB to a more powerful instance

Same as increasing the number of instances, but for DBs. The easy way out when you are not yet ready to take on the effort of splitting the DB.

What is needed to make this work? When you use AWS or something similar, this option should work out of the box with no or minimal downtime.

Pros: Solve your scaling issues with minimal overhead.

Cons: This is the easy way out. Using this option too liberally may allow you not to deal with the real issues such as missing indexes or overly complex queries.

Limitations: There is always the most powerful DB instance that your service provider offers.

Use autoscale to adjust your system to the actual load

Rather than manually adding instances or adding more compute/storage to your instance, monitor the state of your running instances and build a trigger to add more when you are nearing the limit.

What is needed to make this work?

1. You need to know what to measure, how to measure it, what are your thresholds
2. You need to implement a mechanism to automatically deploy another instance or add more power to your current instance.

Pros:

1. You use the resources that you actually need and thus pay for what you need to pay for.
2. You don’t have to pause your work to manually add an instance.
3. More resources can be added before the users feel things slowing down.

Cons: At an early stage, the effort to set this up, might be used better to do something else.

Limitations: There is always the most powerful DB instance that your service provider offers.

Deploy an instance of the application for each customer

Rather than having one application to serve the needs of all the customers, there will be a separate instance deployed for each of the accounts.

What is needed to make this work? The process of deploying and scaling the application must be automated and it must just work. You cannot have your engineers set up the application for each of your customers.

Pros:

1. Instead of having to scale up for how many users you have, you will only have to scale up for how many users there are on your biggest account.
2. Now there is no risk of mixing up the data of different accounts.
3. You will be able to gradually roll out the releases.
4. You will have an awesome deploy pipeline to run your integration tests and for your devs to set up their dev env.

Cons: At an early stage, the effort to set this up, might be used better to do something else.

Limitations: You will still need to deal with the scale issues which are caused by your largest accounts, but luckily those scale issues are much smaller than the ones that would be caused by the total number of users that you have.’

Work towards having a proper data structure

Say that you have a user_profile table, but this table does not include user status. So to get user status, you need to query the user_actions table to see what was the last status that the user assigned to himself. That’s not a proper data structure and it will make your queries slow.

What is needed to make this work? It seems to me that there are two strategies for achieving solid data structures.

1. Try to get it right from the start
2. Have a team that is capable of refactoring the DB as they learn more about the problem domain

My preference goes to option number two. For two reasons:

a) You are never going to get it right all the way, so you are still going to need to change the database structure at some point (or live with a suboptimal DB structure)

b) When you have the confidence that you can change it later, you can start off with the simplest thing that would work and leave the decisions about how to scale to be made in the future when you know more about the requirements of your system.

c) The longer you put off refactoring the DB, the more painful it is going to be.

Pros:

1. When your data structure is good, then writing application code on top of it becomes really easy.
2. When your team is used to proactively changing DB structure, they are going to be equipped to solve problems when the DB performance really hits the brick wall.

Cons: As a short term fix, writing code to work around bad data structure might seem like a more lucrative option.

Limitations: Eventually there will be a table that has so many rows and so many reads/writes that you still need to go beyond having a good data structure to keep it performant.

Use read-only replicas of your DB to ease the load on the master DB

Once the same DB table is accessed in 6–7 different situations, you will be able to see the pattern that there are quite few situations when a write is performed and more often data is accessed to read it. When your DB is becoming a performance bottleneck, you may ease the load on your master DB by creating a read-only replica(s) of it and directing the DB reads to the replica(s).

What is needed to make this work?

1. Implement one of these solutions https://www.postgresql.org/docs/11/different-replication-solutions.html
2. Figure out how to implement read from slave, write to master in your application code.

Pros:

1. Eases the load on your master
2. Provides you with a failover replica.

Cons:

1. Adds a layer of complexity
2. There will be a latency between the master and the replica, which might require to be handled in the application code

Limitations: As your load grows, you may run into a situation where your master DB cannot handle its load even when you only make write queries towards it.