Media Processing in the Network

1. Media Processing in the Network Wei Tsang Ooi

2. Research Area • How to build multimedia network applications ?

3. Outline • Dalí Multimedia Library • Problems and related work • Research proposal

4. Question • How to build multimedia network applications ?

5. Dalí

6. Current Solutions • black box library code • easy to write but too high level • need finer optimization, resource control • write from scratch • too time consuming

7. What is Dalí • A small set of simple, fast, reusable primitives and abstractions for image, audio and video processing

8. Example: Black Box Code • gd GIF programming library 1.3 gdImagePtr im;FILE *in;in = fopen("mygif.gif", "rb");im = gdImageCreateFromGif(in);fclose(in);gdImageRectangle(im,0,0,8,8,red);

9. f = fopen(argv[1], "r"); BitStreamFileRead(bs,f,0); BitParserWrap(bp, bs); GifSeqHdrParse(bp, sh); rmap = ImageMapNew(); gmap = ImageMapNew(); bmap = ImageMapNew(); n = GifSeqHdrGetCtSize(sh); GifCtParse(bp,n,rmap,gmap, bmap); ih = GifImgHdrNew(); ph = PnmHdrNew(); status = GifImgFind(bp); GifImgHdrParse(bp, ih); h = GifImgHdrGetHeight(ih); w = GifImgHdrGetWidth(ih); r = ByteNew(w, h); g = ByteNew(w, h); b = ByteNew(w, h); i = ByteNew(w, h); GifImgParse(bp,sh,ih,i); ImageMapApply(rmap,i,r); ImageMapApply(gmap,i,g); ImageMapApply(bmap,i,b); Example: Dalí code

10. Byte Image Audio Buffer Bitstream MPEG Headers JPEG Headers Lookup Table Bit Image DCT Image Vector Image Convolution Kernel Dalí : Abstraction

11. Byte Image Audio Buffer Bitstream MPEG Headers JPEG Headers Lookup Table Bit Image DCT Image Vector Image ConvolutionKernel Dalí : Abstraction

12. Byte Image • Two dimensional array of bytes • Can be either physical or virtual physical image virtual image

13. Composable Abstraction • Byte images can be used to represent: • Gray scale images • RGB images • YUV images • Alpha channels

14. Example Code : Fractal

15. image smaller newh neww smaller = byte_new (neww, newh); byte_shrink_2x2(image,smaller);

16. target dx image 2dy smaller newh neww target= byte_clip(image,0,0,dx,2*dy);byte_set(target,0);target = byte_clip(image,3*dx,0,dx,2*dy);byte_set(target,0);

17. target dx image 2dy smaller newh neww target = byte_clip(image,dx,0,neww,newh);byte_copy(smaller,target);

18. image smaller newh neww target = byte_clip(image,2*dx,2*dy,neww,newh);byte_copy(smaller,target);target = byte_clip(image,0,2*dy,neww,newh);byte_copy(smaller,target);

19. Dalí strategies • specialized primitives • byte_shrink_2x2 • memory sharing • byte_clip • explicit memory allocation • byte_new

20. gop pichdr pic pichdr pic ... Dalí : MPEG headers seqhdr gophdr gop gophdr gop ... seqend

21. Dalí : MPEG headers seqhdr gophdr gop gophdr gop gop ... seqend pichdr pic pichdr pic ...

22. Basic Header Primitives • find • parse • skip • dump • encode

23. Example • Skip to the n-th framefor (i = 0; i < n; i++) { mpeg_pic_hdr_find(bs); mpeg_pic_hdr_skip(bs); }

24. Dalí: Bitstream chunk of memory parser 1011001

25. Mode of Operations • File i/o static memory parser fread/fwrite

26. Mode of Operations • Network static memory parser socket

27. Mode of Operations • Memory map entire file parser

28. Strategies • expose structure • MPEG headers • explicit I/O • Bitstream abstraction

29. Novel Features of Dalí • explicit I/O • explicit memory management • predictable performance • programs are highly reconfigurable • push semantics to programmer • composable primitives, abstractions

30. Other Features of Dalí • fast • easily extensible • support wide ranges of formats (GIF, JPEG, MPEG-1, WAV, AVI, ..)

31. Dalí : Status • Currently in alpha release • C, Tcl and Java binding • Better performance than other tools (e.g. Rivl, mpeg_play)

32. Back to the question .. • How to build a multimedia network applications ?

33. capture process compress store source receiver de-compress display capture process compress Breaking it up .. on onemachine on twomachines

34. source receiver Optimization on onemachine 8 bit colors, 10 frame/s capture process compress store on twomachines de-compress display capture process compress

35. Optimization on onemachine 8 bit colors, 10 frame/s capture process compress store 8 bit colors, 10 frame/s on twomachines de-compress display capture process compress

36. Optimization • What if there are more than one receiver, and they have conflicting requirements ?

37. Problem and Current Solutions

38. Problem • Network heterogeneity 100Mbps S R1 36kbps R2 128kbps R3

39. Suggested Solution • Layered Multicast • streams are hierarchically encoded • each layer is multicast into a group • incrementally combine the layers to refine the quality of the streams

40. Example Layer 3 Layer 2 Layer 1

41. Layered Multicast 100Mbps S R1 layer 1,2,3,4,5 36kbps R2 layer 1 128kbps R3 layer 1,2

42. Layered Multicast • Receivers find out if they can join the next higher layer by experiments • Join the layer, if packet drops/low throughput detected, leave it

43. Problems • Needs layered encoding • More groups means more states to maintained • Uses a lot of multicast addresses

44. Alternative Approach • Move computation into the network • where to run ? • when to run ? • how to specify the computation ?

45. Related Work • QOS Filters (Yeadon 96) • where to run ? • when to run ? • how to specify the computation ?

46. Filters • transform one or more input streams into an output stream • Examples : • drop frames • decode frames • mix input audio • transcode to lower quality

47. filters filters flow spec Architecture R1 S R2 session mgr

48. filters filters Architecture ack R1 S R2 session mgr

49. filters filters Architecture data R1 S R2 session mgr

50. Filter Propagation