Dali Virtual Machine

1. Dali Virtual Machine Wei Tsang Ooi (With Brian, Sugata, Tibor, Steve, Haye, Matthew, Dmitriy)

2. Motivations • We know how to encode, transmit and store multimedia data today • People start looking into ways to process multimedia data

3. Video Editing • Concat, cross-fade, overlay, cut and paste

4. Gateway • Convert from one format to another • Change compression rate transcoder output video input video format B (format A)

5. DBMS “ Find all movies in the database that contain this image “

6. Set-top Boxes • Subtitling and close captioning • Mixing of incoming movies

7. Lecture Browser • Match video to slide • Switch between video streams

8. Current Solutions • Black box C code (mpeg_play) • Hard to reuse/adapt/break apart • Unpredictable performance • Standard library (PPM, IJG) • Least common denominator (RGB for all) • Video frames --> RGB --> gray

9. History • Rivl - high level scripting language for multimedia processing • Still suffers from “black box” problem

10. Experience • Real cost of processing • Inherently complex operations • Projective transform • Layered operations • Video decoding • Simple operations but lots of data • Copy

11. Dali Architecture User Expert User Dali Code DVM Code Dali Virtual Machine Dali Compiler MMX TriMedia C

12. Today’s Talk DVM Code Dali Virtual Machine

13. DVM Code : Design Goal • Small number of composable abstractions • Simple, predictable opcodes • High performance • Target for compiler • Easy to extend

14. Abstractions • Byte image • Bitstream • MPEG • JPEG • Bit masks • PCM

15. Abstractions • Byte image • Bitstream • MPEG • JPEG • Bit masks • PCM

16. Byte Image • Two dimensional array of byte • Can be either physical or virtual physical image virtual image

17. Byte Image Can Represent • Gray scale image • RGB image • YUV image • Alpha channel

18. Example Code : Fractal

19. image smaller newh neww smaller = byte_new (neww, newh); byte_shrink_2x2(image,smaller);

20. target dx image 2dy target= byte_clip(image,0,0,dx,2*dy);byte_set(target,0);target = byte_clip(image,3*dx,0,dx,2*dy);byte_set(target,0);

21. target dx image smaller newh neww target = byte_clip(image, dx, 0, neww, newh);byte_copy(smaller, target);

22. dx image 2dy smaller newh neww target = byte_clip(image, 2*dx, 2*dy, neww, newh);byte_copy(smaller, target);target = byte_clip(image, 0, 2*dy, neww, newh);byte_copy(smaller, target);

23. DVM Code Dx = 0.25 * byte_width(image);dy = 0.25 * byte_height(image);neww = 0.5 * byte_width(image);newh = 0.5 * byte_height(image);smaller = byte_new (neww,newh);for (i = 0; i < n; i++) { byte_shrink_2x2(image,smaller); Target = byte_clip(image,0,0,dx,2*dy); byte_set(target,0); target = byte_clip(image,3*dx,0,dx,2*dy); byte_set(target,0); target = byte_clip(image, dx, 0, neww, newh); byte_copy(smaller, target); target = byte_clip(image, 2*dx, 2*dy, neww, newh); byte_copy(smaller, target); target = byte_clip(image, 0, 2*dy, neww, newh); byte_copy(smaller, target);}

24. General Dali Strategies • Specific instruction • byte_shrink_2x2, byte_shrink_4x4, etc • Explicit memory allocation • byte_new • Reduce data • byte_clip • Composable abstraction • byte image

25. Performance run fractal with n = 4 on 800x600 gray scale image about 21 frames/sec for 320x240 video

26. Abstractions • Byte image • Bitstream • MPEG • JPEG • Bit masks • PCM

27. MPEG • Getting important and pervasive • Complex format • Complex code (mpeg_play) • Most decoders provides GetNextFrame() interface • Random access, direct transfer difficult

28. Abstraction for MPEG Video seq hdr gop hdr gop hdr seq end . . . gop gop

29. Abstraction for MPEG Video seq hdr gop hdr gop hdr seq end . . . gop gop pic hdr pic hdr . . . pic pic

30. DVM Code Examples • Skip to the n-th framefor (i = 0; i < n; i++) { mpeg_pic_hdr_find(bs); mpeg_pic_hdr_skip(bs); }

31. DVM Code Examples • Skip to the n-th framefor (i = 0; i < n; i++) { mpeg_pic_hdr_find(bs); mpeg_pic_hdr_skip(bs); }

32. DVM Code Examples • Skip to the n-th framefor (i = 0; i < n; i++) { mpeg_pic_hdr_find(bs); mpeg_pic_hdr_skip(bs); }

33. DVM Code Examples • Skip to the n-th framefor (i = 0; i < n; i++) { mpeg_pic_hdr_find(bs); mpeg_pic_hdr_skip(bs); }

34. Concat 2 Video Sequences inbs1 inbs2 hdr gop hdr gop outbs hdr gop hdr gop

35. DVM Code • Concat two sequences While not end_of(inbs1) { gop_hdr_dump(inbs1, outbs); gop_dump(inbs1, outbs); } While not end_of(inbs2) { gop_hdr_dump(inbs2, outbs); gop_dump(inbs2, outbs); }

36. Performance Analysis • Decode MPEG sequences to PPM • No output • Run on 3 different sequences

37. Performance

38. Performance

39. MPEG Strategies • Expose structure • Gop header, gop instead of just frames • Break up layer • Parse, decode, color space conversions

40. Show Time ! • Decode a MPEG sequence • For each frame scale it to “stamp” size • Create a contact sheet

41. Abstractions • Byte image • Bitstream • MPEG • JPEG • Bit masks • PCM

42. What Is Bitstream ? • Skip to the n-th framefor (i = 0; i < n; i++) { mpeg_pic_hdr_find(bs); mpeg_pic_hdr_skip(bs); }

43. Bitstream chunk of memory parser 1011001

44. Mode of Operations • File i/o static memory parser fread/fwrite

45. Mode of Operations • Network static memory parser socket

46. Mode of Operations • Memory map entire file parser

47. mpeg system stream inbs audio video audio video audio video

48. mpeg system stream inbs audio video audio video audio video filter videobs

49. mpeg system stream inbs audio video audio video audio video filter videobs parser

50. Bitstream Strategies • Predictable performance • Explicit I/O