Dragon Quest X for 3DS and the lag monster

What?

Dragon Quest X is a Square-Enix MMORPG that (as of this writing) is still a japanese-only release. Originally out on the Wii (installed to a USB stick!), then ported to WiiU and later PC. Since September 2014 it now also runs on a Nintendo 3DS.

The 3DS release is a bit weird: it's a streaming cloud service! See here for a quick overview. The game renders remotely on a cloud server, which encodes a video stream sent back to the 3DS.

Sadly the cloud service is hosted in japan. It's easy for most everyone living on a small island to have good latency for a cloud game, but across the pacific ocean the situation is a bit tougher.

The 3DS is a depressing region-locked device... So I recently acquired a "New 3DS" import and decided to give this a try.

Why?

Will it work? Can I now play this game from bed like a lazy sack of crap?

How well does it work?

You'll need a japanese VPN endpoint to cope with an IP lock on top of the devices' region lock, but thankfully paying for the service is trivial as nintendo will accept foreign CC's.

NOTE: The boxed version has 60 days of gametime included, please be careful when registering. If you already have an account from another platform you only have one chance to link it to the 3DS version! When the game first starts it'll ask if this is your first time playing DQX before. If you wish to link your old account, you must say no here. I am a moron and thought it was asking if this was the first time playing it on 3DS. It then eats your game time and you have to submit a support request to "unlink" the autocreated account from your 3DS.

From there, it's downhill. Once I logged into the game it was barely playable. Huge amounts of frame skipping, and navigating was impossible. My character became an enraged wallhugger at a bad daytime rave.

What was most surprising was how sensitive it was to my Wifi quality. If I stood a few feet from my access point, it would barely work. A few feet more behind a wall, it would grind to a near halt.

Fixing it.

See video demo - goes from unplayable to mostly playable.

I won't draw this out: I run OpenWRT on my router at home. What I did was configure a transparent proxy which would re-terminate the TCP connections between the 3DS and the servers in Tokyo.

Originally:

     130ms
3DS <-----> Tokyo

Now:

     0.5ms               130ms
3DS <-----> Home Router <-----> Tokyo

There is still a good amount of input lag, but the game is now generally playable. I've run it for an hour without hangups or stuttering or any issue at all. No more stuttering, and no annoying "your shit is slow!" message.

Why did that work?

I'm going to be a bit vague since I don't want to misrepresent any information. Time permitting, I'll update with some extra links.

The TL;DR summary is: A combination of the TCP receive window on the 3DS being set to 16k, and Wifi being crappy. Every 16 kilobytes of video sent to the 3DS, the remote server has to wait 130ms + Wifi jitter for our ACK packet before sending more data.

Quick breakdown of how the game works.

I like the streaming client (aside from its network failures), it does a few clever things:

The "game stream" is actually multiple data streams over different TCP connections. In the demo you may notice the video stuttering, but the audio does not. The game runs in at least three separate data streams:

* Game input stream
* Audio stream (smaller data, much less likely to hiccup)
* Video stream
* Text stream (might be part of input stream?)

The last one is from me observing that the in-game text appears overlayed on the video. It doesn't move quite in sync, doesn't have any compression artifacts, and has a consistent brightness. The 3DS resolution is so low that the kanji is still hard to read though...

The primary fault is that the video stream can't keep up, resulting in stuttering and wall-hugging.

The New 3DS is still 2.4ghz

If you live in an area close to other human beings, you'll understand how awful the 2.4ghz spectrum is. It's more able to go through walls, which means neighbors half a block away can interfere. It's so bad in my complex that at times of the day the 2.4ghz spectrum is nearly unusable. All channels are hosed. Not all devices support 5ghz yet, which isn't quite as prone to congestion.

While sitting on my network, pinging the 3DS shows lag bursts in upwards of 100-300ms, though usually much smaller. The further away from the AP I am, the more likely ping test considers some packets as dropped.

Having packets go weird on Wifi does not mean that it is congested. Often sending another packet right away will work. Eventually you'll get real congestion and have to slow down, but there is much more of a "background noise" with Wifi that is ignorable.

Transparent proxy, RTO, RTT, and window scaling.

I'm going to lose many readers here... What I'm doing is gaming TCP's Retransmission TimeOut, as well as optimizing the TCP receive window size for the 3DS.

The RTO is initially calculated during the three-way handshake of a new TCP connection:

SYN (from 3DS) <---> (from Tokyo) SYN/ACK <---> (from 3DS) ACK

When connecting the 3DS straight to Tokyo, TCP gets a Round Trip Time (RTT) of 130ms. This tells TCP a few things:

* It needs to wait a long time to detect possible dropped packets (RTO)
* It needs a large window to allow more packets to transfer at once without waiting for an ACK from the remote connection.

Now, mix that with my shitty Wifi signal: my network is causing either dropped, delayed, or possibly out of order packets to happen just between the 3DS and my router a few feet away.

However, TCP doesn't know this. If things go weird, it has to assume there is congestion between the two links and slows things down. Since the RTO is set very high it can also end up waiting a long time for a delayed or lost ACK packet.

SYN (from 3DS) <--> (from router) SYN/ACK <--> (from 3DS ACK)

then:

SYN (from router) <---> (from Tokyo) SYN/ACK <---> (from router) ACK

With the transparent proxy, the 3DS is establishing a connection with a proxy running on my router. The proxy understands where my 3DS intended to connect to, then makes a second TCP connection to the final destination.

Now the TCP connection the 3DS is dealing with has an RTT of 0.5ms instead of 130ms.

The RTO is shrunk by a huge amount. If Wifi causes a packet to be delayed too long, TCP will now immediately retransmit it, allowing the connection to recover more quickly.

The TCP window to Tokyo is now much higher. My router's a bit low on memory and has set it to a conservative 40k-ish, while the 3DS is still set to 16k. Given the very low RTT on the LAN, waiting for an ACK from the 3DS is very short, and it can keep up a lot better.

The videostream bitrate adjusts based on your connection, with a minimum of 750kbps. I typically see 1.2mbps to 1.5mbps. Lets do some math on the bandwidth-delay product !

3DS<->Tokyo: (1228800 * .130 / 8) -> 19968 bytes.

With a receive window of 16,000 bytes (it's actually closer to 15k), that means 1.2mbps stream will never work, as there's 3k too much data to fit in the buffer while waiting for an ACK. When running without the proxy, I always see the game cap itself at 750kbps, which is a bandwidth-delay product of 12,480 bytes instead. Just barely enough.

3DS<->Router: (1228800 * .0005 / 8) -> 77 bytes.

With only 77 bytes average outstanding, the tiny window on the 3DS is plenty. The problem then becomes the variance in Wifi: with RTT varying from 5-200ms the buffer will cap out sometimes. If your signal is truly terrible it might still be difficult to keep a smooth stream.

Finally:

Router<->Tokyo: (1228800 * .130 / 8) -> 19968 bytes again.

However this time the receive window on my router is 30-40k. This is more than enough, though I'm hoping to tune it to 65k or much higher.

Data Analysis

I did data captures of two short game sessions, comparing with the proxy enabled and disabled. Fascinating how different they were! First, pretty pictures:

Window scale without proxy:

Window scale with proxy:

It's not great in both cases, but it seems with the proxy the 3DS window scale jumps around a lot more. With a lower RTO it's more likely to have the window reset on it, or bounce around while idle.

Throughput without proxy:

Throughput with proxy:

This is the "throughput" of the remote server sending to us, and it is much more stable with the proxy enabled. It's still a bit weird looking due to the nature of the game and blips in the Wifi. Any upstream blips would slow this down as well.

Window scale from the router to Tokyo directly:

Throughput from Tokyo to the router directly:

This is from a packet capture from the router's re-established connection to Tokyo. The window size is much wider, which is a fantastic help, though still too low. Throughput is relatively stable as well. In tcpdump I can see the Tokyo server taking full advantage of the window we gave it.

The TCP dump had some further gems:

* Without proxy, there were no packet retransmissions, nor packet loss.
* With proxy, there were many, also accompanied by DUP ACK errors.
* The 3DS TCP stack doesn't appear to support TCP Window Scaling or timestamps

With the shorter RTO the router is more likely to give up on the 3DS and re-send a packet to de-jam things. However the wifi packets aren't lost, but simply queued somewhere (?). With the much higher RTO of the original connection packets eventually find their way. With the proxy, we de-jam quickly, but the original packet eventually shows up (and is then discarded).

Window scaling is a bit more damning. I streamed some video from the nintendo eShop, and found that it was using a full 64k window (and also had a non-Tokyo server to give me the data from). Window scaling (as per the link above) lets us make windows much larger than 64k. The connection between my router and Tokyo had a "send window" of 240k to the remote server, so I know it's supported on their end. I'm sure my router will allow a larger window once tuned.

... but 64k should be much better.

Cool beans.

Not a complete fix.

With a bad enough wifi signal, a high enough latency to the cloud servers, or not enough bandwidth (3-5mbps seems required to allow catching up when things stall), it will still run poorly. Midnight on a saturday at home, which is both peak 2.4ghz awfulness and peak play times in Tokyo, the game ran poorly. It was still better than with the proxy disabled during any other time, but is a reminder that it would be best if servers were nearby :)

What can Square-Enix do?

I hear that even in Japan the gameplay experience is touch and go. Initially there were some major launch issues - despite a relatively low number of units shipped (52,000?) Though likely a lot more with download purchases).

Even if TCP window scaling isn't supported, I was able to show the eShop using a 64k window, which would be much more appropriate for the video stream. Tuning this one trivial value would make the game run highly improved, even for people who live in Japan. They just need to push a new software update with one value changed.

What's my setup?

If there's interest I'll try to detail more of a HOWTO. This requires a router running a linux distribution (OpenWRT in my case). It needs to have a relatively large amount of flash space and RAM, or else you can't install the software or run the daemons.

I'm running Barrier Breaker's stable release on a router with 64MB RAM and 32MB flash space. Using a minimal haproxy configuration with iptables transparent proxy rules matching a statically assigned IP address on the 3DS.

The meat of the haproxy config is simply:

listen director
bind *:30000 transparent
mode tcp
server endpoint *

"endpoint" is an arbitrary server name. The first * is a magic value to listen on anything (though tying it down is a good idea). The second * means to treat the "server" of this proxy configuration as wherever the 3DS was originally trying to connect to.

The rest are iptables TPROXY rules (internet has a few tutorials), however adapting this to the firewall rules on your flavor of OpenWRT may be a bit difficult. Bug me if you want some examples.