Was the Light Really Red?

I was at a conference a while back and the presenter warned the audience about trusting the color of a traffic signal shown in a video. Yesterday, I did a bit of testing to see how worried we should be.

First, the potential mechanism worth investigating: the traffic signal occupies a small portion of the video frame, and the encoder is writing at a low bitrate, so it rations, spending its bits where the frame changes most. Then the light changes color, but a few pixels of change isn't worth the cost of recording it. As such, the light is green at the time of the crash, but the video still shows it as red. Let's do some experimenting…

We took two GoPros (1080p, 30 fps, wide) and recorded several light cycles. One camera stayed camped close to the light, as ground truth, close enough that no respectable encoder could miss the change. The other was set up 150, 300, and 450 feet away. At 450 feet, the light was only about 3 pixels by 3 pixels. Is that enough frame coverage to convince the encoder to write a change?

It turns out, yes. The video properly captured the changing light every time. But that’s at GoPro’s 45 Mb/s bitrate, where spatial and temporal compression are barely breaking a sweat. What happens when we compress the heck out of the video? What if we go from 45 Mb/s to 256 kb/s, less than one percent of the data? What if we take the GOP size from 15 to 600, twenty seconds before full refreshes? What if we jack up the quantization parameters [QP], forcing the encoder to round away detail? In other words, what if we create a worst-case scenario?

Left: original GoPro video. Right: severely degraded video.

The severely degraded version still captured the changing light… every time. I continued to degrade the video, upping the QP to 45, but at that point the light wasn’t discernable.

There’s a lot to this topic and diving all the way down the rabbit hole would not be To the Point, but here’s my current takeaway from this experiment: If the color of the light was discernable, the encoder kept it current. More testing would be needed to establish how broadly that finding applies, but it is consistent with how modern video encoders are designed to work: preserve visible change and discard what viewers cannot perceive.

However, macroblock analysis can add another layer of confidence. It lets you examine how the encoder handled the small region containing the traffic signal, frame by frame.

Macroblock analysis of degraded video.

That can help distinguish frames where the signal region contains newly coded information from frames where the video relied largely on prediction from surrounding frames. It does not make the answer automatic, but it gives the analyst another tool for evaluating how confidently the video reflects the signal’s displayed color.

Thanks for reading, keep exploring!

Lou Peck