has a vendor ever told you that open source Kafka is expensive?...

@kozlovski
Stanislav Kozlovski@kozlovski
30 views Dec 14, 2024 ~11 min read
Advertisement
1
has a vendor ever told you that open source Kafka is expensive?

here are 7 ways cost calculators lie to you

(featuring a real story)
Media image
2
the story begins in 2023 when WarpStream was first released with a poignant piece called "Kafka is Dead, long live Kafka"

It was an innovative Kafka-compatible system that avoided inter-zone networking and disks. 💡

It's main value proposition was that it was 10x cheaper.
Media image
3
They proved these claims through a cost calculator accesible on their website.

It sounded right!

• the cloud is super expensive
• Kafka racks up a ton of network costs
• Kafka without Tiered Storage uses a ton of disk & that isn’t cheap either.

So it must add up!
Media image
4
So much so that just 13 months after their incorporation, they got acquired by Confluent for $220 million.

That was perplexing... 🤔

Confluent had recently released a competitor product called Confluent Freigh which promised to be up to 90% cheaper than Kafka.
5
The price of the WarpStream product doubled (2.2x) soon after.

What's most interesting was that the percentage saved compared to Kafka didn't change.

Does that mean that Kafka's price 2.2x'd too?

Something smelled fishy.

So I started exploring.
Media image
6
Luckily, the calculator gave us a pretty clear breakdown of the costs and was client-side Javascript!

So we can compare in detail what changed.

And review the code 🤓
Media image
7
sure enough the Kafka price did double!

I asked myself:
• why does it deploy three times as many instances
• why did the inter-zone networking cost double? (1.8x)
• why did the amount of storage increase 1.6x, but the storage cost increase by 2.x?
Media image
8
btw - it's worth saying that the old version was expensive to begin with!

110 r4.4xlarge instances?

That instance has 16 vCPU and 122 GiB RAM.

And in this setup, it serves... 9.3 MiB/s of produce traffic
Media image
9
the calculator got worse.

now it deployed 346 instances.

other things got more expensive too!
Media image
10
the first gotcha? the underlying Kafka throughput changed.

the calculator has you configure PRE-compression throughput, because that's what WarpStream charges you on

the compression ratio changed from 5x to 4x, so those 5 GiB/s were now equivalent to 1.25 GiB/s for Kafka
Media image
11
This compares Kafka more disfavorably, since it increases the dimension Kafka is billed on by 25% but keeps the WarpStream dimension the same.

So we weren't comparing the right thing.

It is a super confusing input that can subtly change.

But let's call that my bad.
12
Without the compression gotcha (which everybody else I talked to fell for too), the cost of Kafka still increased.

By 64% 👀
Media image
13
ok, an accurate comparison finally

still doesn't make sense.

let's start dissecting 🔍
Media image
14
276 r4.4xlarge?

These are HUGE instances. They’re 16 vCPU and 122 GiB RAM each.

At 1 GiB/s cluster-wide, you’re looking at each instance serving a measly… 3.7 MiB/s of producer traffic.

Even the smallest r4.large type can do that (10x cheaper).
Media image
15
two hundred and seventy-six nodes?

I don’t think Kafka would work at all with that many brokers.

And if it did - it would be operationally impossible to manage.

But it’s true that without tiered storage you need a lot of instances (or use JBOD - multiple EBS disks per node)
16
why did the cross-zone networking cost change too? the throughput is the same.

1 GiB/s of produce at 3x RF and consumer fanout is:
• 682 MiB/s produce
• 2 GiB/s replication
• 2 GiB/s consume

total of ~4778 MiB/s of cross-zone, charged at $0.02 is $241,920 a month
Media image
17
I told you we have the code.

A bug was introduced that doubles the replication cost - literally a * 2

It already accounted for the replication cost by multiplying by RF - 1. Now it doubles it.
Media image
18
onto the disk changes.

if you notice carefully, there are two inconsistencies here:

• total GiB increased by 25%
• total price increased by 56% - twice as much as the GiB

since GiB is charged at a fixed cost, we would expect the total price to increase by no more than 25%?
Media image
19
the answer?

the pricing was simply increased. from $0.08 to $0.1.

Instead of using the cheapest gp3 SSD (which is overkill to begin with), it was changed to use the more expensive, older-gen gp2

and the comment reasons that it's for simplicity 💀
Media image
20
let's not get ourselves distracted. This was absurd to begin with.

Kafka was made to run on HDDs. Each broker in this deployment writes just 11.1 MiB/s to the disk.

and the cheapest AWS HDD (sc1) gives you 192 MiB/s at these sizes.

it's also 6.66x times cheaper 😈
21
this works because:

• sc1s give you just 192 IOPS
• Kafka has the OS batch larger IO writes from pagecache into disk due to the ample memory its deployed with and pagecache
• EBS HDDs have their own batching - they merge sequential IOs
22
if concerned, you can deploy a st1 for better performance.

but chances are if you're evaluating WarpStream, you don't care that much about latency.

(sc1 will usually give you better latency than S3)
23
in any case, the calculator overestimates storage costs by 6.66x.

the difference is $4.62M a year. that's 30% of the total (already wrongly-inflated) Kafka cost

and it's in fact LARGER than what my computed fair total cluster price is..
Media image
24
but there's more. the provisioned storage increased from 3.62M GiB to 4.53M GiB

the calculator says it targets 50% free space and 3.62M is correct, but the utilization was changed to 60% free space without updating the docs
Media image
25
Kafka does need free space, the only question is how much.

At these sizes, I think it's way overkill.

You have every Kafka broker use at most 6.4/16 TiB.

That leaves 9.6 TiB of unused (very-expensive) SSD space.
Media image
26
I think a fairer comparison would be a 37.5% free space, or perhaps even lower.

But it doesn't end there.

The impact isn't just the extra cost for storage. That's just the first-order effect.
Media image
27
The second-order effect is that you need to deploy a lot more instances (due to the 16 TiB limit).

And remember those instances are already 5-10x more expensive than they should be.

The ideal solution here is Tiered Storage or JBOD. if not - at least use cheaper instances...
Media image
28
ok enough criticism, it's easy to be a hater.

let me propose a solution:

• use 222 r4.xlarge - 5x cheaper instances (still too many brokers tbh)
• use sc1 with 50% free space (not 37.5% but still)
• compute networking without bugs

how much cheaper is it? around 3x
Media image
29
now, as I mentioned - this uses too many instances and free space already. I just wanted to keep the constraints the same to keep the comparison fair.

What if we try to optimize Kafka?

Use less disks and avoid consumer networking costs?

6x cheaper
Media image
30
turns out you can actually reduce Kafka cloud costs quite a bit!

KIP-392 removes consumer's cross-zone networking costs. 👍

KIP-405 stores data in the cheaper S3 and reduces the size of disks you need per broker. 👌
Media image
31
there is a path forward for further savings.

Kafka can introduce a change that has producers write to leaders in the same zone for unkeyed topics. (no ordering)

And maybe it can copy the write-to-S3 direct feature for special kinds of topics too?

That's a wish. 🕯️
Media image
32
The WarpStream team basically had a smear campaign against Kafka.

A lot of their blog was focused on bashing it, saying how it "can't be fixed".

Everything they said had a grain of directional truth, but like any good marketing - a large part isn’t true.
Media image
33
The way they did that is worth studying. They're talented writers!

The blog posts are very interesting, have a lot of good info, some half truths and then some extreme exaggerations that make no sense.

But they sound believable because the build-up to them was so good.
Media image
34
WarpScheme basically came in the industry, provided a good solution, bashed Kafka, and got acquired for a hefty sum by the Kafka company after 13 months.

But Kafka is an open source project.

It can’t bite back.

It's important that we as a community defend it. 🛡️
Media image
35
Why didn't anyone step up?

I should have done better.

So I'm making it up with this post.

I see it as my duty. My brand is based on Apache Kafka after all.
Media image
36
Here's the bite back 🐺

WarpStream is NOT cheaper than Apache Kafka. 🔥

That's right. Despite claiming 10x cheaper, WarpStream is more expensive in 2/3 clouds.

Only in AWS is it cheaper, but that's by only 32%. And you can get that down with cloud networking discounts.
Media image
37
Kafka is 4.3x cheaper than WarpStream in Azure

And 1.3x cheaper in GCP

So think again before you believe a marketing narrative 💡

source: gist.github.com/stanislavkozlo…
Media image
38
What's worse is that this public pricing was likely never their actual production cost. It was the MINIMUM.

WarpStream has pricing tiers, and the only tier that uses the public price has no SLA... 💀

So no... WarpStream is not cheaper than Kafka.
Media image
39
It's true - Kafka isn't perfect. There are products like Confluent, RedPanda and Pulsar that work hard to improve upon its obvious gaps.

Kafka is old too. Jay reportedly started writing it as early as 2008, and it was open sourced in 2011.

But old != outdated.
Media image
40
PostgreSQL was released in the 80s (1989) and it is still everybody’s go-to database in 2024! (despite the massive competition)

Kafka still has a very active community, perhaps its most diverse one ever so far, and is one of the most visited Apache projects.
Media image
41
Kafka is an open source.

It can't defend itself, which is a negative.

On the flip side?

Kafka also can’t hype itself up with marketing claims that aren’t true. That's probably a positive.
Media image
42
After I saw this, I started working on a solution.

I spent the last two months coding up an in-depth Apache Kafka deployment calculator, and today I’m releasing it.

See it here: 2minutestreaming.com/apache-kafka-c…
43
Admittedly, the calculator isn't open-source yet, so you'll have to trust my implementation.

It's also not feature complete. Today it just gives you the price.

It IS much more accurate than the WarpStream one, or any other calculator provided by a Kafka competitor.
Media image
44
But it's a starting point. I've also added a forum so we can have open discussions about assumptions and etc.

So I hope it helps have more transparent pricing.

That's the open-source ethos after all, isn't it?
Media image
45
To end this on a happier note, I want to add a bit of a disclaimer at the end.

I worked in Confluent for 6 years. Some people may see it as a bit of a backstab to speak out against a company they acquired.

I wanted to write out my thoughts in detail so there aren’t any questions about intention, malice, agenda, etc.

• I have only good things to say about Confluent. The company was always transparent, the founders are great people and the culture in there was great.
• I worked with talented, honest co-workers and friends there.
• I have equally published multiple posts praising Confluent's product and the company. And will continue to when the opportunity arises.
• I assume best intentions, namely that Confluent didn't know/notice, and that they will correct this.
• I see WarpStream and Confluent as separate, at least in the actions they’ve taken and the culture they've shown.
• I do think WarpStream has a very strong team and an amazing product, it’s a true testament of excellence what they managed to build.
• I think what WarpStream did in marketing was not right with relation to the open source Apache Kafka project.

I therefore believe a call out and a bit of parody in response is therefore fair game.

I don’t mean to build my brand off drama and controversy. I’ll be going back to my regularly-scheduled programming right after this post.

I realize the initial disclaimers on my newsletters were too vague regarding my intention, so I have updated them there too.

I don’t have a bone to pick. I realize other companies make similar non-truthful claims in their marketing too. And I think they should be called out too. 👮‍♂️

Especially if it goes too far.

I tried very hard to keep everything objective and backed by data. I have only used data I've publicly observed and expressed my opinion based on it.

I don't think I should censor myself simply and cherry-pick who I call out simply because I have some indirect history with them.

If there are any strict errors - please tell me, I will correct them and apologize.

I would prefer to say the truth. I would say it to a friend too.

If you don’t agree, then perhaps we can agree to disagree. 🤷‍♂️

You’re free to unfollow me and hold your own views. ✌️

It’s just some pixels of a post on the internet after all.

But I hold no negativity. Hopefully you don’t either. 😇

And if you side with the truth? Smash the repost button :)
Media image
46
The full post in a readable format: substack.com/home/post/p-15…
Actions
What You Can Do
  • Export as PDF or Markdown
  • Batch Export to Notion
  • Bookmark & Highlight
  • Screenshot Tweet
Create Free Account

Includes 7-day Premium trial

Advertisement