Subsections


Related Work

Benchmarking Video Codecs

The performance of a video Codec is a function of the available video coding tools (the coding algorithm itself), the actual implementation of these algorithms, the characteristics of the input video sequences, and the architecture in which the Codec is implemented. Based on that, we believe that in order to make a comprehensive analysis of video applications, a good video Codec benchmark have to meet the following conditions: First, the benchmark should include complete applications (not only kernels) that implement all the features defined in the standard (or at least the most relevant ones) for a given application domain. Second, the benchmarks have to be optimized for high performance. The implementations of the video standars that are designed with verification purposes could produce misleading results in complexity and/or architecture evaluations. Optimizations can be platform independent (like fast algorithms for motion compensation) and platform dependent (like SIMD optimizations). Third, a complete set of inputs with different resolution, motion characteristics and spatial details have to be provided. Having only one sequence can lead to confusing results in performance evaluations. Fourth, a detailed list of the coding parameters have to be provided. Those parameters have to be tuned for the resolutions under study. The performance of the Codecs could change dramatically depending on the selected coding options. Not choosing the most appropriate options for a given application domain (ie HD video) could give misleading results. Fifth, the programs should be free (as in freedom) in order to be able to access the source code, analyze it, perform changes and optimizations, and be able to distribute them. The same apply for the input sequences. Sixth, the code has to be easy to port between different processor architectures, compilers and operating systems. Finally, the programs must be representative enough of real life multimedia applications, for example as part of multimedia players used in desktop operating systems. The desired characteristics for a video benchmark can be summarized as follows:

Multimedia Benchmarks


Table I: Description of Existing Multimedia Benchmarks

BENCHMARK

Release
date

License

Video Applications

Input sequences

Mediabench I

1997

Free 

MPEG-2 decoder (MSSG)

mei16v2.m2v: 352x240 pixels, 30 fps

MPEG-2 encoder (MSSG)

4 frames YUV sequence: 352x240 pixels

Mediabench+ 1999 Free

MPEG-2 decoder (MSSG)

n.a

MPEG-2 encoder (MSSG)
H.263 decoder (Telenor)
H.263 encoder (Telenor)

Mediabench II

2006

Free


MPEG-2 decoder (MSSG)

704x576, 10 frames, 25fps

MPEG-2 encoder (MSSG)

MPEG-4 decoder FMmpeg

MPEG-4 encoder FFmpeg

H.263 decoder (Telenor)

H.263 encoder (Telenor)

H.264 decoder JM 10.2

H.264 encoder JM 10.2

Berkeley Multimedia Workload

2000

Free 


MPEG-2 encoder (MSSG?)

720x576p, 1280x720p, 1920x1080p (16 frames)

MPEG-2 decoder (MSSG?)

EEMBC Digital Entertainment

2005

Commercial

MPEG-2 decoder (MSSG)

Graphic: 720x480p30 (50 frames)
Ralgrind: 320x240p25 (30 frames)
Sign: 352x240p25 (30 frames)
Zoom: 320x240p30 (30 frames)
Marsface: 192x192p25 (49 frames)

MPEG-2 encoder (MSSG)

MPEG-4 decoder (Xvid)

MPEG-4 encoder Xvid

BDTI Video Benchmarks

Commercial

H.264 like decoder

n.a.

H.264 like encoder


Table I provides a summary of the existing benchmarks for multimedia. Only the applications related to video processing are detailed.

Mediabench [9] is the most popular multimedia benchmark. For the video domain it includes a MPEG-2 coder and decoder based on the implementation of the MPEG Software Simulation Group (MSSG) with short input videos in low resolution (352x240 pixels). The MSSG Codec does not implement SIMD optimizations and, in general, it has low performance. An extension of the Mediabench called Mediabench+ [12] tried to solve the limitations of Mediabench by including MPEG-4 and H.263 video Codecs, but they selected the reference implementations (MoMusys and Telenor respectively) and they do not address high definition. Recently, a new version of the Mediabench (called MEdiabench II [13]) has been released in which more video Codec applications have been added: it includes Codecs for MPEG-2, MPEG-4, H.263 and H.264. The MPEG-2 Codec is the same MSSG implementation, the MPEG-4 is taken from the FFmpeg Codec library, the H.263 Codec is the Telenor implementation, and the H.264 is taken from the reference software (called JM). The main problem with this selection is the combination of reference implementations for some of the Codecs (MSSG for MPEG-2 and JM for H.264) with highly optimized version for others (FFmpeg for MPEG-4). On the other hand, although they have increased the resolution compared to the original Mediabench, they do not address HD applications and remains on Standard Resolution (SD). Additionally, Mediabench II provides only one short input sequence (10 frames) and the coding options are not tuned for HD applications.

The Berkeley Multimedia Workload [10] solved the problem of the low resolution of the input sequences by including inputs with higher resolutions, but they have selected only the MSSG implementation of the MPEG-2 Codec. The EEMBC Digital Entertainment [11] benchmark includes Codecs for MPEG-2 and MPEG-4 video standards using the MSSG and Xvid implementations respectively, they address low and standard resolutions and provide a different set of input sequences. Nevertheless, they do not have recent Codecs like H.264 and the coding options and input sequences are not publicly available. Finally, the BDTI Video Encoder and Decoder Benchmark [14] is a set of applications representative of modern video Codecs, but they are not complete video Codec applications. The Codecs seems to be similar to H.264/AVC but the details of the Codec, its sources, the coding parameters, and input sequences are not publicly available.

Thus, none of the available benchmarks for multimedia includes all the desired characteristics for a complete benchmark for emerging video Codec applications and for HD environments. HD-VideoBench try to solve all the before mentioned limitations by providing different video Codecs (including the most recent ones like H.264/AVC), including video applications optimized for high performance, and providing a complete set of input sequences and coding options tuned for HD applications.

Mauricio Alvarez 2007-04-17