Heyo! We’re Cody Haas and Ivan Vidal, and we’re engineers on the Riot Direct team. Our team is responsible for maintaining Riot’s global network - you may remember us from this awesome video and this series of articles.

Riot Direct exists because playing a game where someone has lag or high latency completely ruins the experience. Next to Wi-Fi, one of the biggest causes of lag and latency is the uncertainties of the internet. Riot Direct’s solution is to bring Riot’s network closer to our players by developing our own backbone network and collaborating with Internet Service Providers around the world. This has allowed us to limit these uncertainties and reduce lag and latency.

It’s been a while since you’ve heard from our team. So in this article, we’re going to tell you a bit about what we’ve done to reinforce consistent and stable connections, reduce latency, and improve the overall player experience for our entire multi-game portfolio.

From One to Many

When Riot Games was really just Riot Game, designing the network and everything inside it was less complicated.

With a suddenly expanding set of games, we transitioned from a network designed for:

• One game
• Game servers located in a specific place per region and per shard
• Primarily one protocol
• One latency threshold

To:

• Multiple games
• Multiple locations for game servers for each region
• Multiple different protocols
• Multiple latency thresholds

Moving to a multi-game environment while still supporting League of Legends meant we had to redesign the network while trying to avoid player pain.

Keep in mind - each game is not just a game server, but the platform, services, and people required for both the game and the infrastructure supporting it. All of that translates into more capacity and new features.

Doing all of this while dealing with daily operations was a major challenge.

What We’re Working With

Riot Direct has been around since 2014, so at this point, we’re pretty robust. This is a great place to start - now we needed to take this League-only network and its esports productions and adapt + extend it to properly support future games and potentially their esports.

Not all games are the same when it comes to networking. They have different requirements for protocols, latency, and server location.

Protocol: Fast-paced games like League of Legends and VALORANT rely on the speed of UDP at the expense of reliability, while a slower-paced game like Legends of Runeterra relies on the slower but more reliable TCP protocol.

Latency: Even though we all want that sub-15 millisecond ping for League of Legends, the game is still very playable at around 60ms latency on average. LoR can be played at even higher levels of latency. If you tried to play VALORANT with over 60ms latency, that would be pretty rough, so VALORANT optimized for a larger number of game servers.

Server Location: VALORANT game servers are located all around the globe. League of Legends game servers, however, are located in Chicago, Amsterdam, Tokyo, Seoul, São Paulo, Santiago, Istanbul, Miami, and Sydney.

VALORANT game server locations across the globe.

League game server locations across the globe.

IPv4 Address Exhaustion: IPv4 addresses are stored in a 32-bit unsigned integer, which means there are approximately 4,294,967,296 possible addresses. The number of available publicly routable addresses is even lower, because there are sets of rules that state which address blocks can be routed over the internet.

Denial of Service: It feels pretty awful when games are compromised due to a DDoS. While building out our networking solutions, we always have to keep these attacks in mind, because as we add more games, we increase our exposure.

All of these pieces combined meant that to support new games with individual server setups and network requirements, the original network designed for a centralized game server (like League and LoR) needed to also accommodate games with decentralized game servers (like VALORANT). All during an unexpected, unprecedented global pandemic.

COVID-19: The COVID-19 pandemic meant we had to accomplish all of this without the ability to work and collaborate in an office. We also had to manage the effect on hardware deliveries, which impacted our Points of Presence (PoPs) - and more server locations meant more PoPs.

Riot Direct

A Quick Overview

Riot Direct is basically a web of cables and routers specifically used for Riot’s packets, including packets for critical game-running processes.

To provide a little context, internet service providers (ISPs) default to hot-potato routing - this means they want to get customer traffic out of their network as soon as possible using the most cost-effective exit point. The way this impacts their customers (our players) varies widely based on the application they’re running and its overall bandwidth requirements.

The internet is a complicated place where we can’t just connect two points with a straight line. Each region is built based on individual geography, politics, and community policies, and whether via land or sea, we always need to adapt to their infrastructure.

By having Riot Direct present around the globe in key locations, we’re close to other ISPs, so we can connect directly using Private Network Interconnecting (PNIs) or via internet exchanges. This way, ISPs can fully offload traffic going to our games close to the source, so we avoid the internet as much as possible. Each one of these locations is called a point of presence (PoP).

Once traffic is inside Riot Direct, we use cold-potato routing - we use the best possible path back to players by keeping traffic inside our backbone for as long as necessary. To achieve this, we need to create specific policies that take into consideration all the carriers we peer with as well as the geographical constraints of each available path.

Here’s how it works. When someone decides to play one of Riot’s games, their client receives an address it needs to send packets to. The packets leave the player’s computer and travel onto their local network to an internet service provider’s (ISP) network. Ideally, the packets quickly leave the ISP’s network and enter Riot Direct’s network at one of our PoPs.

So basically, the PoP says “Hey, if you’ve got traffic trying to get to these addresses, hand it all to me, because I know the way there.” Traffic can then quickly and easily travel through one of Riot Direct’s dedicated fibers to the correct game server.

League of Legends Implementation

League of Legends was Riot’s first game, so our Riot Direct network was originally specifically tailored to run League games. Here’s a quick overview of how this works.

League’s servers are all located in the same place for each region - in North America, for example, this is Chicago - so the original Riot Direct network was designed to route traffic to this single location. The game servers all have public IPv4 addresses - this impacts scalability due to IPv4 address exhaustion and increased financial cost.

Another important piece is that League’s return traffic will try to use the best path back, and will succeed most of the time. This means traffic has several options. Sometimes it’ll travel Riot Direct’s fibers back to the original PoP, and sometimes it exits the game server’s location to ride the internet back. Network-wise, the best path is always calculated based on the point-of-view of the servers.

Return traffic is based on BGP best path selection. For us, the quality of the path is determined by how close we are. A PNI, for example, is always preferred over a transit (regular internet) because we peer directly with them. Not just logically, but also physically - there’s an actual dedicated fiber between us, while a transit is an indirect connection over another ISP (which is different from home ISPs).

VALORANT Implementation

Riot’s first person shooter, VALORANT, may look similar to League from a competitive point of view, but it has a drastically different set of networking needs. The VALORANT team’s early focus on high performance meant they deploy in multiple locations with many game servers to reduce lag, which can be absolutely game-changing in a first person shooter.

We worked closely with the VALORANT team to support these goals, which meant creating a totally new strategy that could work with any future game with a similar network setup. Put simply, our solution was to bring our network and servers closer to players. We accomplished this with a variety of strategies, including leveraging anycast networking, and solving for the IPv4 exhaustion that we were already dealing with for League.

Anycast Overview

When a player starts a VALORANT game, the platform determines their location and assigns a relevant game server IP. Unlike with League, this public IP does not originate from the game server, but from each PoP - these PoPs are distributed around a region. Each PoP within a region advertises the same IP address so players can reach the address from several locations, always ending up at the closest PoP - this networking strategy is called anycast.

Taking A Closer Look With Some Examples

Let’s say we have a VALORANT player located in Virginia, USA, and they receive an address of 192.207.0.1, entering Riot Direct’s network at our Virginia PoP. A week later, that player decides to take a trip to sunny Los Angeles, and logs on to VALORANT from their LA hotel. That SoCal match will still get the IP 192.207.0.1, but this time it’ll enter into Riot Direct’s network from our Los Angeles PoP.

By decoupling the subnet from the game server and moving the region selection to the platform, we can focus on optimal subnet use based on the internet topology in any given area.

For League of Legends, we had to use different subnets for the North America and Latin America North shards, which was a hard requirement on our end. But for VALORANT, we use the same anycast subnet for North and Central Americas.

To compare, the Latin America South and Brazil shards had two different subnets for League. We decided to also use two for VALORANT because if players were located in some regions in northern Argentina, the underlying internet infrastructure tended to route to Brazil instead of Santiago. So if we used the same anycast IP for both of these, it would be difficult from a routing perspective to determine what a player’s ISP would do. And since it’s the same IP, the platform has no control over where it’ll land. By having two anycast IPs we give the platform and players the flexibility to choose the best option.

Benefits of Anycast

While we have plenty of servers in strategic locations around the globe to ensure the lowest possible latency to all players, that doesn’t mean we have one server per PoP. Anycast origination is not tied to the game server itself, so internally we need to find the best solution for returning traffic between our edge and the game server.

Increasing Speed: Anycast networking means we can leverage the internet to choose the closest ingress point into our network. Player traffic is always pinned to its ingress point on the way back.

Avoiding IPv4 Exhaustion: With our implementation of anycast networking, we added Network Address Translation (NAT) to our game flow, which allows game servers to use private IP addresses instead of public ones. Private IPs don’t need to be unique like public ones do. This means we can add more game servers closer to players, reducing latency even more.

Optimizing Packet Return: NATing also guarantees a return from the server via Riot Direct’s network. So when packets return from the server to the player, they always go back to the PoP they entered at via our dedicated cables.

This resolves a major challenge we have with League routing - each routing decision between the edge and the game server is based on all available paths from the point of view of the server. With careful engineering, we can get it right most of the time, but due to the nature of the internet, sometimes it’s out of our hands.

Soaking Up DoS Traffic: When Denial of Service (DoS) or Distributed Denial of Service (DDoS) traffic is sent to the anycast address in any region, it’s distributed across each PoP that uses that specific location, soaking up the attack attempt.

Riot Direct’s edge soaking a 500Gb/s DDoS attack on VALORANT’s game servers.

Ultimately, lag and latency often depend on the highly variable public internet. By bringing more PoPs closer to players, we aim to reduce the uncertainty to ensure a consistent and excellent experience.

What About IPv6 Support?

So… where’s the IPv6 support? IPv6 would solve our IPv4 address exhaustion problem, so we’re currently working on adding IPv6 anycast addresses. We’ll still keep the IPv4 anycast addresses, but adding IPv6 will give players with public IPv6 addresses native support, saving them from latency by avoiding ISP carrier grade NATs. The IPv4 and IPv6 anycast addresses will work together to ensure as many players as possible have native end-to-end support for their Internet Protocol versions.

So Why Not Combine Them?

I know what you’re thinking. Wow, that VALORANT implementation seems pretty sweet - why not just do it for League?

While we’d love to just copy and paste our solutions over, it’s not quite that simple. League predates Riot Direct - its systems were already in place when Riot Direct was first formed. League is also an entirely different game, with its own client, backend, connections, and live player audience. We were able to collaborate with our friends over on VALORANT to build this new network routing much earlier in their development process.

Rest assured, this is only the beginning. The changes to Riot Direct described in this article are just the foundations for our multi-game future. Riot Direct exists to help our game teams create and support the best possible player experience, and each new game enables us to learn more and create better tools for all of our games.

Thanks for reading! If you have any questions, please post them in the section below.

Posted by Cody Haas and Ivan Vidal