I love the analysis of terminal latencies! And I'm in full agreement with the overall goal of less latency everywhere. But, of course, I feel like picking a few nits.
> And it turns out that when extra latency is A/B tested, people can and do notice latency in the range we’re discussing here.
Yes, this is true. But the methodology is important, and the test used doesn't really apply to typing in terminals. The test isn't a "type and see if you can tell it's slow" test, it's a hit the mark hand-eye coordination test, something you don't do when typing text. Latency when playing Guitar Hero is super duper important, way more important than most other games, which is why they have a latency calibrator right in the game. Latency when playing a Zelda game is a lot less important, but they still try very hard to reduce latency.
The same people who can distinguish between 2ms of difference in a drum beat also can't distinguish between an extra 30ms of response time when they click a button in a dialog box.
I'd like to see a stronger justification for why lower latency in a terminal is just as important as it is for hand-eye coordination tasks in games.
~2 msec (mouse)
8 msec (average time we wait for the input to be processed
by the game)
16.6 (game simulation)
16.6 (rendering code)
16.6 (GPU is rendering the previous frame, current frame
is cached)
16.6 (GPU rendering)
8 (average for missing the vsync)
16.6 (frame caching inside of the display)
16.6 (redrawing the frame)
5 (pixel switching)
I find this list pretty strange. It's generally right - there are a bunch of sources of latency. But having done optimization for game consoles for a decade, this explanation of game latency feels kinda weird.
Games that actually run at 60fps usually do not have greater than 100ms latency. They also don't miss vsync every other frame on average, that 8ms thrown in there looks bizarre to me. Render code and GPU rendering are normally the same thing. Both current and previous frame GPU rendering is listed, huh? Sim & render code run in parallel, not serially. The author even said that in his article, but lists them separately... ?
Consumer TVs come with like 50ms of latency by default. That's often half of it right there. Games are often triple-buffered too, that accounts for some of it. The ~2ms right at the top belongs in the 8ms wait, it disappears completely.
I just get the feeling the author of this list was trying hard to pad the numbers to make his point, it feels a like a very hand-wavy analysis masquerading as a proper accounting of latency.
It is gonna be a totally experimental feedback, but nonetheless it may help you relating to the importance of fast feedback latency ; the best stays to experience it yourself.
I noticed during my thesis on realtime systems that my brain had difficulties compensating for latency, even more for jitter (inconsistent latency). I do more typos when I have a high latency. I notice it when playing music in a high latency context, or when the ssh connection has a high ping. We did the experiment slowing down the click of the mouse by 100ms. Users hated the results.
I'm also more productive if I can notice my typos 100ms earlier, or confirm that everything I previously typed is good earlier. Even if it takes me 500ms to process the information displayed on screen, it stays on the critical path of the overall task speed performance.
I had a similar discussion with a colleague a few weeks ago. They were double counting latency in different pipeline stages when a delay in one stage causes high latency in the other stage.
My analysis would be something like this (assuming 60 fps):
1. Input latency - variable from hardware
2. Input event queuing - avg 8 ms (input event arrives in frame X but is handled in frame X+1)
3. Animation, game simulation, submitting GPU commands, other CPU work - 16.7 ms
4. GPU work - 16.7 ms
5. Display controller scanning out frame - 16.7 ms
6. Display panel latency - < 10 ms?
Steps 3 (specifically the submitting GPU commands part) and 4 can happen within one frame interval so you can be double buffered in that case. Otherwise you need triple buffering to maintain 60 fps. If you have a window compositor, then you need to account for when that latches and releases the buffer you submitted and if it performs copies or steals your buffer. This is the part that's especially murky. Games have it easy with fullscreen non compositing mode.
An interesting aspect of this is that for hand-eye coordination tasks, 60Hz provides a perceptual smoothness advantage over 30Hz. There's some benefit to having a higher frame rate even if it costs some latency. I don't know where the trade off point is, but it's not always better to reduce latency if it means a lower frame rate. It probably is always better to have both low latency and high frame rate, but that's hard.
As far as the latency analysis, I'd probably just lump display latency all in one and not try to break it down further. There's some number of ms it takes from the GPU output until pixels are on-screen. Consumer TVs are worse than computer LCDs because the TVs now are doing all kinds of filtering. OTOH, consumer TVs are also trending toward 240Hz refresh, so there is a counter-force causing latency to go down too.
Someone else here mentioned there are two kinds of triple buffering, which I didn't know, but FWIW I was talking about the 3-frames of latency kind. There's also a 2 frames of latency kind, and it sounds like that's the kind you're talking about?
Anyway, I tend to think about game latency as simply 1, 2 or 3 frames of latency, depending solely on how much buffering is going on. That explains almost all game latency, and the games are bad because they don't run at 60Hz, tons of games run at 15-30Hz with double or triple buffering, so the latency is automatically bad. No other sources of latency are needed to explain why. There is the 1/2 frame of latency between input and when the system polls the input and recognizes it, that's fair. So a typical double-buffered game has 2.5 frames of latency, and then add on the display device latency, and that's the sum total. It doesn't need to be made to look more complicated than that, IMO.
Typing text at speed is not really that different from playing an instrument at speed. When you pluck a string you're able to keep it in rhythm by using the resulting noise of your own playing as the cue to synchronize the next beat. When latency is high or varying, as sometimes happens with software synthesizers, even the simplest rhythm starts falling apart. Professional audio applications want very low latency, sub 20ms as a baseline, but preferably below 10ms.
In that light, it totally makes sense that typing latency matters. If you aim to type in a smooth rhythm and use the screen as feedback, any hiccup will slow you down.
Except that the big difference is the audio and the hand-eye coordination part.
You can more easily hear a 10-20ms delay than you can see one, it's a physical feature of our human hardware. And hand-eye coordination tasks are all about anticipating an event. Hitting or catching a baseball for example, we can see it coming, the pattern of it's trajectory is what allows us to compensate for the 100ms of delay in our nervous system & is brain and allow us to have 1ms accuracy.
Neither of those is true for typing. Don't get me wrong; I want lower latency in my terminals and editors. I just don't buy that it's particularly important until the latency hits a threshold of badness, which is probably around 100ms. People largely aren't complaining about terminal latency nearly as much as they complain about video game latency, even though both are widely used.
The reason guitar hero has latency adjustment controls and almost no other games do is because they're mixing audio with hand-eye coordination tasks. I can very easily tell the difference between 5ms of delay in Guitar Hero. But I have no idea what my terminal latencies are, and I generally don't care until it stalls more than probably 200ms. It makes a very subtle responsiveness difference when there's an extra 30ms latency while I type, but it doesn't make a large functional difference or compromise my ability to type in any easily felt or measurable way. With Guitar Hero, on the other hand, I drastically lose my ability to play the game when the latency is off by 20ms.
Anyway, I appreciate the response & discussion, but I still want to hear a stronger justification for typing latency being very important. There might be one, I just don't think I've heard it yet.
> And it turns out that when extra latency is A/B tested, people can and do notice latency in the range we’re discussing here.
Yes, this is true. But the methodology is important, and the test used doesn't really apply to typing in terminals. The test isn't a "type and see if you can tell it's slow" test, it's a hit the mark hand-eye coordination test, something you don't do when typing text. Latency when playing Guitar Hero is super duper important, way more important than most other games, which is why they have a latency calibrator right in the game. Latency when playing a Zelda game is a lot less important, but they still try very hard to reduce latency.
The same people who can distinguish between 2ms of difference in a drum beat also can't distinguish between an extra 30ms of response time when they click a button in a dialog box.
I'd like to see a stronger justification for why lower latency in a terminal is just as important as it is for hand-eye coordination tasks in games.
I find this list pretty strange. It's generally right - there are a bunch of sources of latency. But having done optimization for game consoles for a decade, this explanation of game latency feels kinda weird.Games that actually run at 60fps usually do not have greater than 100ms latency. They also don't miss vsync every other frame on average, that 8ms thrown in there looks bizarre to me. Render code and GPU rendering are normally the same thing. Both current and previous frame GPU rendering is listed, huh? Sim & render code run in parallel, not serially. The author even said that in his article, but lists them separately... ?
Consumer TVs come with like 50ms of latency by default. That's often half of it right there. Games are often triple-buffered too, that accounts for some of it. The ~2ms right at the top belongs in the 8ms wait, it disappears completely.
I just get the feeling the author of this list was trying hard to pad the numbers to make his point, it feels a like a very hand-wavy analysis masquerading as a proper accounting of latency.