H.264 Analyzer- Finalpresentation

H.264 Analyzer Utility Tool By Mukta Gore Shantanu Kurhekar Amruta Vaidya Gouri Patil

1

Project Statement n

Decoding the H.264 input bit stream step wise, restricting its purview to the Baseline profile.

n

Extracting NAL units from the bit stream, and displaying list of frames in the video sequence along with list of parameter sets encoded in the stream.

n

Decoding and displaying the syntax elements in the parameter sets.

n

Displaying user selected frame.

n

Extracting selected slice in selected frame, and decoding and displaying parameters in the slice header.

n

Decoding and displaying selected macroblock information. 2

Analyzers n n

n n

n

A software tool to help technical professionals parse bit streams conforming to international media standards Video standards specify only a syntactic description of compressed visual data and method for decoding it to reconstruct video sequence Since encoding methods are not specified, analyzers are needed to test for conformance to the standards Analyzers perform stepwise decoding of compressed bit streams and display results at every step to check for non-conforming parameters They also help in testing the working and efficiency of algorithms for motion estimation, rate control and scene change detection

3

A peek into our utility …

4

H.264 ENCODER

5

H.264 DECODER

6

Implementation of our Analyzer User Interface API Library

User Interface :

GUI in Visual Basic

API

:

Application Program Interface

Library

:

Dynamic Link Library in C 7

User Interface n n n

n

This acts as an interface between the user and the core functionality of the analyzer It enables user-controlled picture-by-picture analysis of .264 stream The user can analyze the bit stream at various levels in the hierarchy-sequence, frames, slices, macroblocks and macroblock partitions It displays the parameter sets, reconstructed sample values, errors in the bit stream, if any and the reconstructed picture

8

Option to Hide grid

Reconstructed frame

Parameter list Slice list Frame list

Macroblock list MB info choice

Detailed MB info based on choice selected

9

Library n

We have implemented the core decoding functions as a Dynamic Link Library because ¨ ¨ ¨ ¨ ¨

The analyzer uses decoding functions written in C which is roughly a 14000 line code DLLs link the object code to the executable at run time rather than compile time as done in static libraries So less memory is required to run the analyzer utility, and the executable file is compact Multiple applications can share the same DLL Multiple entry points are provided in DLLs to enable conditional decoding

10

Program Flow n n n

The functionality of the utility tool is event driven so depending on the action of the user, various tasks are performed. Processing is done at two levels: GUI level and DLL level Following events actuate entry into the DLL – ¨ ¨ ¨ ¨

n

Opening a file by clicking on the submenu Open in the menu File Clicking on a frame in the frame list Clicking on a slice in the slice list Closing a file by clicking on the submenu Close in the menu File

Certain events like clicking on SPS or PPS in the parameter list ¨ clicking on a macroblock in the MB list ¨ selecting MB info choice are handled in the GUI domain without calling the DLL again ¨

11

Event 1: Clicking on submenu Open of menu File GUI

DLL

Open File

PRIME

Display No. of NAL units,

Prime

Extract NALunit

parameter set list and frame list

Parse

12

Extract NAL unit

Parse

13

NAL unit structure

NAL unit Start code

NAL Header

Raw Byte Sequence Payload

0x000001

Video Coding Layer (VCL) data Bit 0 Forbidden bit Value:0

Bits 1-2 NAL ref idc Value:0-3

Bits 3-7 NAL type Value:0-31

14

Event 2: Clicking on an SPS from list

Event 3: Clicking on a PPS from list

Parameter set mechanism enables an encoder to signal important, infrequently changing sequence and picture parameters separately from the coded slices Sequence Parameter Set is applicable to a complete video sequence and contains profile type, the number of reference frames, decoded picture width and height, choice of progressive or interlaced coding etc Picture Parameter Set is applicable to one or more pictures within a sequence and contains selected SPS id, flag to select VLC or CABAC entropy coding, number of slice groups in use, number of reference pictures in list 0 used for prediction, initial quantization parameters etc

15

Event 4: Clicking on a frame from the list

16

Frame decoding n n

The following process will be iterated till all slices in the selected frame are decoded I or IDR frame: ¨ ¨

n

It contains only intra coded macroblocks and hence is decoded independently of previous pictures The starting point is first slice of the frame

P frame: ¨ ¨ ¨

It contains inter and intra coded macroblocks and hence its reference pictures need to be decoded The starting point is first slice of the most recent IDR To improve efficiency, if (recent IDR < last decoded frame < frame to be decoded) then starting point is first slice of last decoded frame else starting point is first slice of recent IDR 17

Slice decoding Slice Header n

Slice header ¨ ¨ ¨ ¨ ¨

n

Slice data

Slice type Picture Parameter Set referenced Frame number to which it belongs with respect to recent IDR Delta quantization parameter at slice level No. of active reference pictures in list 0

Slice data ¨

Macroblock layer

18

Macroblock layer n n

MB type (Inter/Intra/Skipped) Macroblock partitions for prediction ¨

Intra – 4x4, 16x16, PCM n n

¨

Inter – allows sub-block partitioning up to 4x4 n n n

n n

9 prediction modes for 4x4 4 prediction modes for 16x16 Separate motion vector for each partition or sub-partition Quarter pel motion vector accuracy Separate reference picture for each macroblock partition allowed

Delta quantization parameter at macroblock level Coded block pattern to indicate which of the blocks contain non-zero coefficients

19

Event 5: Clicking on a slice from the list

20

Event 6: Clicking on an MB from the list

The information extracted during frame decode is accessed in GUI domain for display, without calling DLL

21

Event 7: Clicking on submenu Close of menu File

22

Error handling n n

The utility checks if all parameters are within their specified range to test conformance with H.264 baseline profile Bit stream errors are handled at 4 levels ¨ ¨ ¨ ¨

n

General - Unsupported file format, No SPS, No PPS, No IDR SPS,PPS Slice header Slice data

On return from DLL, the error array is checked and corresponding errors are displayed with the relevant frame number

23

Challenges n n n n n

Datatype matching of structures in DLL and GUI domain DLL convention for parameter passing Accessing memory allocated in DLL from GUI Finding out reference frame for every macroblock partition Memory allocation and deallocation problems

24

Utility Features n n n n n n n

User-controlled picture by picture analysis Supports CIF (352 x 288), QCIF (176 x 144), SQCIF (128 x 96) video formats Stream level statistics collection Only Baseline profile supported Conformance checks (Display of non conformant parameters) Parameter set extraction facility (PPS, SPS) Frame Level Analysis ¨ ¨ ¨

n

Slice Level Analysis ¨ ¨

n

Frame type Number of slices in the frame Distribution of Macroblocks in the frame based on type Slice Header Number and address of Macroblocks in the slice

Macroblock Level Analysis ¨ ¨ ¨ ¨

Macroblock type, subtype and prediction mode Quantization parameters for luma and chroma MB grid overlay on picture with Hide Grid option Motion vector display - MB and sub-MB partitioning 25

Applications of analyzer n n n n

Getting insight into the coding quality of H.264 / MPEG-4 part 10 video-encoding software and hardware Enabling rectification of the functions responsible for encoding nonconformant parameters in the encoder Analysis of coded bit stream for no. of I and P frames, macroblock and sub-block partition for rate control Monitoring and validating large volumes of digital media down to inspecting and debugging any problems encountered at the bitstream level

26

Files tested File name & Compression

Brief Analysis

Akiyo.264 1 : 168

Low motion activity video of 150 frames (10 I and 140 P), 1 slice per frame, QCIF size

Silent.264 1 : 118

Medium motion activity video of 150 frames (1 I and 149 P), 1 slice per frame, QCIF size

Md_33.264 1 : 208

Low motion activity video of 60 frames (1 I and 59 P), 3 slices per frame, QCIF size

News.264 1 : 158

Low motion activity video of 97 frames (1 I and 96 P), 1 slice per frame, CIF size

Carphone.264 1 : 90

Medium motion activity video of 90 frames (1 I and 89 P), 1 slice per frame, QCIF size

Football.264 1 : 31

High motion activity video of 90 frames (1 I and 89 P), 1 slice per frame, CIF size

Stefan.264 1 : 27

High motion activity video of 90 frames, 1 slice per frame, CIF size

27

Non - conformant bit streams FILENAME

ERROR DISPLAYED

noStartCode.264

Unsupported file format

foreman.264

Profile not supported

md.264

Weighted prediction and weighted bi-prediction flag not zero

container_wipe.264

Illegal luma ac coefficients (Frame 3)

test.264

No IDR, no SPS, no PPS in bit stream

28

Future Scope n n n n n n n

Adding support for Main and Extended profile Displaying reference and current picture simultaneously for selected macroblock Visual display for motion vectors using arrows Slice group map type Displaying the macroblocks in the order they are decoded if arbitrary slice order is allowed Support for SIF file Support for simultaneous audio and video analysis

29

30