NuboScreenCaptureKit, VideoToolbox, and Metal: a frame’s journey
How capture, compression, and presentation work together, and why bounded queues matter.
Capture and encoding on Mac
Nubo uses ScreenCaptureKit to capture the desktop and VideoToolbox to encode video. These are separate stages: capturing an image is not the same as compressing or transmitting it.
Hardware acceleration reduces CPU work when available. Resolution, codec, and system load still affect processing time. Native APIs do not make processing free.
Decoding and presentation on iPad
Video travels over encrypted network channels. On iPad, VideoToolbox decodes frames and Metal presents them. SwiftUI manages the controls, rather than a sequence of desktop images.
The design aims to avoid unnecessary copies and conversions. That does not make the entire path literally zero-copy: compression, packetization, and encryption still require work.
Measure smoothness, not just FPS
Accumulating old frames increases the gap between your action and what you see. Bounded queues and quality adaptation help contain delay, but recovery must also respect codec dependencies.
For diagnosis, separate received FPS, presented frames, and network latency. A ProMotion iPad does not automatically receive remote video at 120 FPS. Network RTT is not a measurement of total visible latency.