Everything you need to know about the Google File System and Distributed Systems

I did not just read the GFS paper, but went through hours of questions-and-answers with Ai regarding it. Coz, many have read the research paper that came out in 2003, but not many really understand it.

this is my try in explaining the GFS, from perspective of someone learning distributed systems.

Architecture diagram to help you visualize

What is GFS?

It's a distributed file storage system, that was designed by Google back in 2003, to store extremely large sized files, and support reliable reads/and writes.

They did not create this to store small sized files but in large number, but to work with large sized files. And they optimized it for append only operations, when you write at the end of the file.

Also, don't forget this is 2003.

What all major components GFS has?

Google Client - It is what you will use to interact with GFS. Basically, it's like an SDK, you can call google's internal file methods egs WriteToFile(), ReadFile() etc. Fun fact: They decided not to make it fully POSIX compliant, so made their own.

Master Node - As the name says, it controls everything. It stores metadata about what the file is, where the file is stored, how many chunks it has, and where all the chunks are stored etc. And all of this information is stored in-memory.

Chunk Server - The huge sized files, are broken down into chunks of fixed size 64 mb, which are then stored in these chunk servers.

How does chunking work? Let's talk about Reliability

A file is broken into multiple chunks of size 64 mb. And GFS follows a replication factor of 3, meaning each chunk has 3 replicas. Each chunk is stored in 3 separate chunk servers, so in case 1 dies, you still have 2 copy of it to recover it from.

One of the 3 chunk servers is declared as primary chunk server for the chunk, and rest 2 are called secondary servers. This primary server is what is the most important and which decides the final state of the chunk/file.

Fun fact about the Master Node: Did this question come to your head, we are storing so much info-in memory? how is it possible?

Google did it's maths. And they found out to store a file of size 1 gb. you only need 1kb of metadatab. Which makes it to be able to store in-memory in RAM.

And master node, does a heartbeat check after every some time. It sends request to all servers of a chunk, to see which are healthy or not. If a server dies, it runs it's recovery mechanism.

How does Write-flow work?

Suppose you are writing into a file. You call GFS client to write it into the file.

The client, first sends the request to the Master node. Master node, returns the 3 ip addresses of the chunk servers where the client can find the chunk. It also mentions which server is primary and which are secondary.

So now the client knows, where the chunk exists.

Here's an interesting part, I will come back to it

It sends the data to write to all the 3 chunk servers. Now these servers instead of writing directly to their disk, they store the data in an in-memory buffer.

The primary chunk server then decides the order in which the write will happen. It sends this order to the rest 2 secondary servers so all of them write in the same order.

The client waits for ACK from all of the 3 servers, and then only notify the client that the write was successful.

Let's address the elephant in the house

CONSISTENCY

Let me ask you this. Now you know how GFS works, what do you think, it follows strong consistency? weak consistency? eventual consistency? or what?

This is the most confusing part.
Since the write is only successful after all the 3 servers write to their disk, and GFS ensures that all 3 servers follow the same order in which they write. At any time, all 3 chunk server, should have the same data, and in the correct order.

All of this made me think, GFS falls towards the strong consistency model.

But here's a catch.

While GFS guarantees, all 3 servers will have the same data. It does not guarantee while in case of concurrent writes, the data will be correct. It mentions in it's paper that the final output might be 'undefined'

Let me explain it.

Imagine A and B are writing at the same time at the same line in a file stored in GFS.
A writes XXX
B writes YYY

it is possible, that in the chunkservers final output becomes something like
XXX
YYY
XXX
YYY

instead of the correct one

XXX
YYY.

And if the data is not correct, you can't really use it for further processing.

Also, it is possible during concurrent write-read, that when the primary server is sending the order to the secondary ones, the network disrupts, and now we have different state among the 3 servers, and a read might happen then and based on what server it hits, it might show inconsistent data.

check out the og paper : GFS original research paper

That was it, I would encourage you all to brainstorm the points mentioned in this article with AI, since there is never one right answer in distributed systems.