⚠️🚨 Very long read !! Data Center Latency!!🚨⚠️ Today the topic of...

Today the topic of where DCs needed to be physically located for AI inference came up. There are some claims that these DCs must be located near metropolitan areas, where the assumed usage would be coming from. The argument for this is network latency, and that the location of a DC that would be physically closer to users would mitigate latency, and ones that are located outside of major metropolitan areas would have latency that would be too high for AI inference use cases. Is this true? Or is this a narrative to justify a company’s own DC locations, while throwing a bit of shade on others?
If a DC’s physical location is so detrimental to it’s use for AI inference, then companies such as $APLD and Lancium/Crusoe are making grave mistakes, having committed billions of dollars to building AI DCs seemingly in the middle of nowhere, Ellendale ND and Abilene TX respectively. But are these DCs really in the middle of nowhere? Physically, yes. Logistically within cyberspace, absolutely not.
One of the best interviews I have watched on the topic of AI and DCs is of Jonathan Schildkraut from @CompassDCs. This video is so loaded with nuggets of info that I can write several posts related to it and not do it justice. In this interview he highlighted the point about plethora of new AI DCs being built in locations that are seemingly in the middle of nowhere. That are physically, however, in terms of their logistical location on the internet, he says some of these locations are prime real estate.
youtu.be/60PzGkHevPQ?t=…
To be on prime real estate on the internet is to be along the fiber optic backbones and trunks. In the past, it was thought to be a security risk to map out the internet infrastructure and deemed illegal. However, the Department of Homeland Security later created one and made it public, ironically. This map was published in 2015.
smithsonianmag.com/smart-news/fir…
The map at the top is the map that was released in 2015, and the one at the bottom is from service provider Lumen Technologies. I included the one from Lumen Technologies because it’s in better detail (but not the full backbone) and highlighted a specific trunk in yellow (from Amarillo to Dallas) for an example later in this post. The maps are practically identical aside from the areas that Lumen does not service. One can see from these maps why DCs were mushrooming up around places such as Des Moines IA and Cheyenne WY. They’re in the middle of nowhere in the physical world, but they are on the biggest East-West fiber optic backbone in the US, running along the 41st parallel. Prime real estate as Jonathan Schildkraut would like to call it, and tells the story of why this came to be, and in his opinion why future DCs will be built along these lines where large MWs of power are still available.
We now have an understanding of why DCs are being built in otherwise middle of nowhere locations. Some will say, great, they can be used for LLM training, but these locations would be useless for inference workloads, with latency being the cause. I would not necessarily agree to that argument. In a very recent fireside chat, Eric Schmidt, former CEO of $GOOG addressed this very topic. He did not believe that the latency of remote DCs made much difference in real time systems. More importantly, he makes the argument for training and inference to be in the same DCs for economic and logistical reasons, the biggest factors being cost of hardware and electrical costs.
youtube.com/watch?v=oC46Cz…
If Eric Schmidt doesn’t think there is much real-world difference due to latency, would there be any type of numbers to back that up? From my research the industry acceptable level of latency for AI inference is 100ms from end-to-end. The first reference is from a research paper by Nigade et al, where it cited this figure from another paper by Gujarati et al.
link.springer.com/article/10.100…
Another source that cited very similar numbers is a blog post from Pavel Lunin, VP of Network at @Scaleway, a French based cloud computing and web hosting provider.
scaleway.com/en/blog/unders…
The post from Pavel Lunin is quite informative, as he breaks down his calculations for latency, and gives real world examples of latency due to physical distance over a fiber optic network. The number he cites for latency on a fiber optic network is 1ms of round-trip time per 100 km, and also notes to multiply a factor of 1.5 for long transcontinental routes and multiply a factor of 2 for shorter terrestrial urban routes.
I purposely highlighted the latency from Paris to Amsterdam, as this was the closest example to the yellow trunk that I had previously highlighted on the fiber optic map. The section I highlighted is the trunk that runs from Amarillo, through Childress, to Dallas. I chose to use that trunk as an example and comparison not because any partiality to $IREN, but because they have been very transparent about the latency numbers of their DCs to major urban areas and hyperscaler DCs. Something I have not seen published by any other BTC miner (but would assume to be similar if connected by fiber optic internet). The latency numbers provided by IREN is as follows:
BC – less than 20ms
Texas – leas than 10ms
Childress is to Dallas is about 368km directly, and from the map of the fiber optic network, it is pretty much as straight line from point to point. But let’s just round that number up to 400km to be conservative. So going by Pavel Lunin’s formula (which I have seen elsewhere as well):
400 km, 4×2 = 8ms of RTT
So that number is below the 10ms provided by $IREN for Texas, which leads me to believe that IREN is being conservative to be on the safe side. Since I do not have a specific example of DC that is located in a metropolitan area with published latency numbers, I have to rely on asking Copilot on what the expected latency of DCs located within metropolitan areas connected to a fiber optic network are, and it gave me a wide range between 1 to 10ms. 1ms would be very low and 10ms would basically mean no difference between a location such as Childress. This is a wacky range, and for good reason, because we cannot specify a specific location, and have no idea what the actual network topology would be like for that area. But we’ll give the urban DC the benefit of the doubt and give it a very low 2ms of RTT. So even in a good case, an urban DC is only improving the latency by 6 to 8ms of RTT. In a bad case, that improvement in latency is even less or nil. For the good case in improvement in latency, is it economically justifiable to be building DCs on much more expensive land, when fiber optic connections solve much of the problem? Does it make economic sense to double spend of expensive GPUs? Eric Schmidt certainly does not think so. And with the way that $APLD and Lancium/Crusoe are betting with their AI DC build outs, they don’t think so. More importantly, those companies have put the money on the table.
I know that within the BTC mining sector, @SamirTabar of $BTBT and @fgthiel of $MARA have been particularly vocal about AI inference not working at locations not close to metropolitan centers. I would love to hear their thoughts and be able to provide some numbers for us to compare?



