All about scaling systems and infra

All though there are plenty of resources to learn about scaling systems from "100 to 100k users" but those are very different you mostly learn about topics which gets introduced at every timestamp . But when you start working with the actual scale you experience a totally new paradigm so here are few of my learnings from what I experienced till now.

What is scaling ??

For your app to be able to handle scale means that as users come on to your platform and start participating, your application services are able to handle the additional load.

A server must be involved in returning a response, serverless or not. Each request will take up a thread in the process, sometimes the only thread, and that thread consumes memory as it processes (load data sets into memory, sort, map, etc) and as more and more requests come in more Threads and therefore: CPU and memory is consumed. If you run out of Threads, CPU or memory then you will stop processing new requests until resources are available (intermittent traffic).

Scale can be accomplished in lots of ways (not all are listed probably):

How serverless and redis handle this

Serverless

Redis

Algorithm design becomes important when you want to scale things, you will get bound to understand core data structures like B, B+ trees and linked lists and such with all these understanding functional programming is also probably useful, as often these systems tend to be built under that paradigm.

You'll want to learn how to use messaging/eventing systems like Kafka. The basics would be to understand pub/sub, but also then the implementation details like managing partitions, routing inside a dockerized microservice environment or using something like Kubernetes, setting up both publishers and subscribers, and thinking about how you want to track things like latency here Uber is a pretty interesting case study you can read about, given how extensively they use Kafka across their microservice ecosystem

You'll want to understand various types of data storage besides RDBMS. Blob, file, table, etc. How to cost analyze them, and how to best use their features for performance a lot of average web/api devs know little about partitions, for instance. You will be needed to understand CQRS. When/why to use it, how to build a system based on it, how to setup and configure the different services to talk to one another. This really tends to live inside an eventing system.

You can find tutorials on almost all this stuff online, either for free on Youtube, or for a few thousand rupees a month on various platforms like Pluralsight or Udemy. Hopefully that gives you a head start. Of course, getting services to communicate is only one half of the scalability problem; the other is figuring out where and how you store the data.

Storage

Databases are reasonably easy to scale, you can either use a master/slave relationship to increase your ability to read data or add a master/master setup like mariadb's galera cluster to give you highly available writes.

Here I would avoid presenting Galera as a write-scaling mechanism. In a Galera cluster, writes are still coordinated across the cluster, so adding nodes doesn't give you linear write throughput.

I would rather say that 90% of scalability is picking the right storage solution. RDBMs can indeed handle thousands of queries per seconds. That does not really qualify as scale. No amount of RDBM will serve millions of requests per second unaided.

It is indeed true that most software systems will never reach a scale worth mentioning. Unfortunately, this has produced a cadre of developers who do not know how to address non-trivial scale, leading them into diminishing return query optimization.

soo how do you learn to address non-trivial scale?

One step is to read Martin Kleppmann's book. It gives you the fundamentals to distributed computing. Once you have done that, you can start trying to design systems that depend less on strong consistency, for example leveraging MVCC and other opportunistic strategies. Start thinking about "two-phase" APIs where you separate receiving the request but letting the caller poll for the answer. More generally, think about how to split work over multiple nodes that need as little coordination as possible.

And load test everything. When a developer (including myself) predicts a bottleneck in software, they are almost guaranteed to be wrong.

a really good skill to have is being able to tell a scale problem from a tuning problem. Like for an instance an app with 25k users and maybe 1000 concurrent doesn’t need scaling solutions, it needs tuning (along with a bit of common sense, like don’t deploy 4x servers in a cluster to run a single container instance with auto scaling turned off, that’s also defined as the smallest possible size I’m going to pull about $60k in annual cloud cost savings at work there).

Sharding ( partitioning work into independent batches, dispatching the requests to the appropriate nodes, and coordinating answers and errors ), load balancing ( smoothing resource spikes ), and caching ( read sharding ) aren't just for databases.

More on the scaling part

Handling large volumes is less about optimising your code (though it helps make running it cheaper) and more about making your application scalable across multiple servers. There are generally three parts to most applications that you need to scale, traffic ingestion (usually via a loadbalancer), your application and your datastore.

Depending on your infrastructure environment these can be more or less work, in cloud (Amazon AWS, Google Cloud) they have features that can take care of scaling the loadbalancer and datastores for you. This lets you only focus on your application.

The easiest way to scale is to make you application nodes stateless so that they do not hold any more info then what they need to serve that one request. Any longterm state should be in an external datastore such as a database. This allows you to add/remove nodes without having to worry about syncing state on each node. If you do need some temporary between request state on the node (such session data) you can either move that to a central cache or route clients back to the same node (though this does not work as well when you start to get really big).

If you manage your own servers or your provider does not offer autoscalling loadbalancers/databases you have to manage scalling these yourself.