The socket is fine for messages, but not for rendering. Here’s a simplified version of how 3D rendering apps usually work in different environments I have used.
Direct3D on Windows: app renders something and calls IDXGISwapChain.Present. The implementation communicates with the desktop compositor running in the dwm.exe process, dwm.exe renders the entire desktop composed of multiple windows, then communicates with physical GPU to wait for next vertical blank event.
Bare metal Linux with DRM: app renders something, calls drmModePageFlip(), then waits for next vertical blank event with poll() and drmHandleEvent() functions. Embedded Linux developers did an amazing job about DRM/KMS, in my experience the API is both reliable and efficient.
X Window systems have multiple different methods, glXSwapIntervalEXT, glXSwapIntervalMESA, glXSwapIntervalSGI, but in practice none of them is reliable, and on some systems none of them is even supported. Unfortunately, this makes rendering tear-free 3D content on X11-based desktops hard-to-impossible.
The originally posted article is about new 3D GPU-based backend in the GTK, which is unrelated to games.
Hardware accelerated 3D graphics is the best way to render pretty much everything on modern hardware, games or not games. High-resolution displays are omnipresent. Even cell phones often have FullHD or more pixels, like 2796×1290 or 2556×1179 in the iPhones.
>Hardware accelerated 3D graphics is the best way to render pretty much everything on modern hardware, games or not games.
A counterexample is video playback. The best way for video is to use the hardware decoder and hardware compositor. You don't want to render the video texture to a quad. Using the hardware compositor requires less computation and less power than using 3D graphics.
Microsoft strongly recommends that new code use MediaPlayer or the lower level IMFMediaEngine APIs to play video media in Windows instead of the EVR, when possible. Microsoft suggests that existing code that uses the legacy APIs be rewritten to use the new APIs if possible.
That lower level IMFMediaEngine API they recommend for Win10+ delivers uncompressed video frames in D3D11 textures.
Yes, but note no hardware composition is involved.
The composition is done by the 3D GPU like I said. Either in your app if you render that video yourself, or if you supply a swap chain’s back buffer to IMFMediaEngine.TransferVideoFrame method, dwm.exe will do it.
It's the compositor's job to composite in the most efficient way possible which in the best case would be using hardware compositing layers, if the hardware does not have enough layers than the compositor will use 3D graphics to combine layers together. Just because a GPU includes a 3D graphics pipeline that doesn't mean that everything has to go through it.
> Just because a GPU includes a 3D graphics pipeline that doesn't mean that everything has to go through it.
That’s how modern Windows does that in practice. Everything does go through the 3D graphics pipeline, and starting from Win8 it’s impossible to disable.
I’m not even sure modern PC hardware supports these hardware layers, except for one small extra layer for hardware mouse cursor. On my computer D3DCAPS_OVERLAY and DDCAPS_OVERLAY flags are unset. The driver reports to the OS the hardware doesn’t support any hardware overlay surfaces.
>I'm not even sure modern PC hardware supports these hardware layers
Yes, they do. I'm guessing you may be using an older nvidia gpu. In the case no overlay is available the video can become fullscreen and to take the only layer from dwm. For pcs the benefit is less about performance / power and more about lower latency.
Tested AMD Vega 7 inside Ryzen 5 5600U, and nVidia 1080Ti — dxcapsviewer.exe shows the same result, no overlays are supported. Only the D3DCURSORCAPS_COLOR hardware cursor is there.
> the video can become fullscreen and to take the only layer from dwm
Indeed, exclusive D3D full-screen mode allows to bypass dwm.exe compositor, even on modern Windows. But I don’t think it’s evidence of any hardware composition being used.
> it doesn't require using 3D graphics to put the texture on the display
In practice, 3D graphics is the best way to put pixels on the display. At this point, I believe other ways are remnants of the old GPUs which had 2D blitting hardware.
GDI is only used by legacy apps. It gonna stay there for quite a while for backward compatibility reasons, but modern Windows apps are using better GPU-centric APIs to render their GUI.
Specifically, WPF is based on DirectX 9, Direct2D is based on Direct3D 11, UWP and WinUI are based on Direct2D. On my computer, both Chromium and Firefox browsers are using Angle on top of D3D11, Firefox uses Direct2D for canvas only.
Also, even legacy apps who render their GUI with GDI are still using 3D GPUs to an extent. Some GDI operations like BitBlt are accelerated internally by the OS. And the OS composes windows on the desktop with D3D11, dwm.exe process does that.
Direct3D on Windows: app renders something and calls IDXGISwapChain.Present. The implementation communicates with the desktop compositor running in the dwm.exe process, dwm.exe renders the entire desktop composed of multiple windows, then communicates with physical GPU to wait for next vertical blank event.
Bare metal Linux with DRM: app renders something, calls drmModePageFlip(), then waits for next vertical blank event with poll() and drmHandleEvent() functions. Embedded Linux developers did an amazing job about DRM/KMS, in my experience the API is both reliable and efficient.
X Window systems have multiple different methods, glXSwapIntervalEXT, glXSwapIntervalMESA, glXSwapIntervalSGI, but in practice none of them is reliable, and on some systems none of them is even supported. Unfortunately, this makes rendering tear-free 3D content on X11-based desktops hard-to-impossible.