Motion JPEG-Clear is the application of the JPEG-Clear algorithm to video files. It transforms a video file into a set of new video files, which together take up around the same amount of space as the original file, but allows lower-resolution video streams to be accessed without the need to generate any extra files.
Over an unreliable channel such as the Internet, a browser could subscribe to the higher-resolution “diffs” when bandwidth was high, and desubscribe from them when bandwidth dropped. This would allow the video to continue uninterrupted, with the resolution being dynamically and automatically adjusted by the browser. Contrast this to today’s common annoying occurrence of users having to wait as YouTube videos repeatedly pause.
For demonstration purposes, I have here made use of a simple 7.5-second video made by my wife and me at the Adelaide Zoo. The original file from the camera is tiger.avi, which is a 640 x 480-pixel file compressed by the camera at high quality with the mjpeg compressor, and is 14,097,396 bytes in length (14,685 kb/s).
For simplicity, and to allow a sensible comparison, I have used the ffmpeg utility to encode all video files in WebM format using default compression parameters. In practice, different codecs or compression parameters would be used; the Motion JPEG-Clear algorithm works equally well with any such choices.
The default WebM version of tiger.avi is 2,558,316 bytes long (2665 kb/s):
The Motion JPEG-Clear algorithm starts by repeatedly using the magic kernel to downsample each frame of the video by a factor of 2 until its larger dimension is between 32 and 63 pixels. In this case, it stops when it gets down to 40 x 30 pixels. These frames are then converted to WebM using ffmpeg, resulting in the “base” video stream tiger.mjc.a.webm, which is 132,194 bytes in length, i.e., 138 kb/s, or 5% of the bandwidth of the original tiger.webm. A browser employing Motion JPEG-Clear would retrieve this “base” stream as a bare minimum, and would use the magic kernel to upsample it back to its original dimensions:
The algorithm then upsamples each frame of tiger.mjc.a.webm by a factor of 2 using the magic kernel, and computes the “diff” between this approximation and the original frame at that resolution. These “diff” frames are then encoded into the video stream tiger.mjc.b.webm, which is 190 kb/s, or 7% of the bandwidth of the original video.
The “base” video stream tiger.mjc.a.webm together with the “diff” video stream tiger.mjc.b.webm allow a browser to reconstruct the “second approximation” of the original video, by upsampling using the magic kernel:
Although it is fuzzy, this is a useful approximation to the original video using only 12% of the bandwidth.
The algorithm continues encoding these “diff” streams until reaching the original resolution of the video.
The next diff is tiger.mjc.c.webm, which is 8% of the bandwidth of the original video. Together with tiger.mjc.a.webm and tiger.mjc.b.webm, it yields a useful approximation using only 20% of the bandwidth of the original:
The penultimate diff is tiger.mjc.d.webm, which is 25% of the bandwidth of the original video. Together with the previous streams (in total 45% of the original bandwidth), we now have:
The final diff is tiger.mjc.e.webm, which in this case (using default compression parameters) is 94% of the bandwidth of the original video. The full set of five diff streams yields a reconstruction that is essentially the original:
In practice, compression parameters can be chosen for each of the “diff” streams independently: the Motion JPEG-Clear algorithm ensures that a higher compression at a lower scale is automatically compensated for at each higher scale.
For example, if we (arbitrarily) tell ffmpeg to encode the five scales above with target bitrates of 50, 100, 200, 400, and 800 kb/s respectively, then we end up with the following files and successive approximations:
tiger-alt.mjc.a.webm (51 kb/s): yields:
tiger-alt.mjc.b.webm (100 kb/s): together with the above (total 151 kb/s) yields:
tiger-alt.mjc.c.webm (198 kb/s): together with the above (total 349 kb/s) yields:
Unfortunately, WebM support for ffmpeg is currently insufficiently developed for it to correctly encode low target bit-rates. When ffmpeg is no longer broken, I'll generate the last two files!
© 2006–2011 Dr. John P. Costella