CS1315: Introduction to Media Computation

1. CS1315: Introduction to Media Computation Movies

2. Movies, animations, and video…oh my! • We’re going to refer generically to captured (recorded) motion as “movies.” • This includes motion entirely generated by graphical drawings, which are normally called animations. • This also includes motion generated by some kind of photographic process, normally called video.

3. Psychophysics of Movies:Persistence of Vision • What makes movies work is yet another limitation of our visual system: Persistence of vision • We do not see every change that happens in the world around us. • Instead, our eye retains an image (i.e., tells the brain “This is the latest! Yup, this is still the latest!”) for a brief period of time. • If this were not the case, you would be aware of every time that your eye blinks because the world would “go away” for a moment.

4. 16 frames and it’s motion • If you see 16 separate pictures in one second, and these pictures are logically sequenced, • That is, #2 could logically follow from the scene in #1. • 16 pictures of completely different things doesn’t work, • You will perceive the pictures as being in motion. • 16 frames per second (fps), 16 pictures in a second, is the lower bound for the sensation of motion.

5. Beyond 16 fps • Early silent pictures were 16 fps. • Motion picture standards shifted to 24 fps to make sound smoother. • Videocameras (digital video) captures 30 fps • How high can we go? • Air force experiments suggest that pilots can recognize a blurb of light in 1/200th of a second! • Video game players say that they can discern a difference between 30 fps and 60 fps. • Bottomlines: • Generate at least 16 fps and you provide a sense of motion. • If you want to process video, you’re going to have 30 fps to process (unless it’s been modified elsewhere for you.)

6. Processing movies • Our frames are going to be JPEG pictures. • One JPEG file per frame. • So, if we’re going to be processing movies, we’re going to generating or processing sequences of JPEG files.

7. Using MediaTools • To generate a series of frame pictures in a folder from an MPEG file. • To play a folder of frame pictures and to save it as a JMV file. • (JPEG Movie format.) • To play JMV or MPEG movies.

8. MPEG? QuickTime? AVI? JMV? • MPEG, QuickTime, and AVI are compressed movie formats. • They don’t record every frame. • Rather, they record some key frames, and then store data about what parts of the screen change on intervening frames. • MPEG is an international standard, from the same people who invented JPEG. • AVI is a Microsoft standard. • QuickTime is an Apple standard. • JMV is a file consisting of JPEG frames in an array. • All frames represented

9. Generating other kinds of movies • Other tools can generate other kinds of movie formats. • QuickTime Pro (http://www.apple.com/quicktime) can read a sequence of JPEG images and produce MPEG, AVI, or QuickTime movies. • ImageMagick (open source toolkit) can also read a sequence of JPEG images and produce MPEG movies.

10. Why do we compress movies? • Do the math: • One second of 640x480 pixels at 30 fps • 30 (frames) * 640 * 480 (pixels) = 9,216,000 pixels • With 3 bytes of color per pixel, that’s 27,648,000 bytes or 27 megabytes of information per second. • For a 90 minute feature movie (short), that’s 90 * 60 * 27,648,000 = 149,299,200,000 bytes (149 gigabytes) • A DVD stores 6.47 gigabytes of data. • So even on a DVD, the movie is compressed.

11. MPEG movie = MPEG frames plus MP3 soundtrack • An MPEG movie is actually a series of MPEG frames composed with an MP3 soundtrack. • It’s literally two files stuck together in one. • We’re not going to deal with sound movies for now. • The real challenge in doing movie processing is generating and manipulating frames.

12. Get the frames in order • Many tools (including os.listdir()) can process frames in order if the order is specified. • We specify the order by encoding the number of the frame into the name. • If you put in leading zeroes so that everything is the same length, the order is alphabetical as well as numerical.

13. Simple Motion def movingRectangle(directory): for frame in range(0,100): #99 frames canvas = makePicture(getMediaPath("640x480.jpg")) if frame < 50: #Less than 50, move down # Generate new positions each frame number addRectFilled(canvas,frame*10,frame*5, 50,50,red) if frame >= 50: #Greater than 50, move up addRectFilled(canvas,(50-(frame-50))*10,(50-(frame-50))*5, 50,50,red) # Now, write out the frame # Have to deal with single digit vs. double digit frame numbers differently framenum=str(frame) if frame < 10: writePictureTo(canvas,directory+"//frame0"+framenum+".jpg") if frame >= 10: writePictureTo(canvas,directory+"//frame"+framenum+".jpg")

14. A Few Frames frame00.jpg frame02.jpg frame50.jpg

15. if frame < 50: #Less than 50, move down # Generate new positions each frame number addRectFilled(canvas,frame*10,frame*5, 50,50,red) if frame >= 50: #Greater than 50, move up addRectFilled(canvas,(50-(frame-50))*10,(50-(frame-50))*5, 50,50,red) When frame = 1, addRectFilled(canvas,10,5,50,50,red) When frame = 2, addRectFilled(canvas,20,10,50,50,red) When frame = 49, addRectFilled(canvas,490,285,50,50,red) When frame = 50, 50-(50-50) = 50 addRectFilled(canvas,500,250,50,50,red) When frame = 51, 50-(51-50)=50-1=49 addRectFilled(canvas,490,285,50,50,red) When frame = 99, 50-(99-50)=50-49=1 addRectFilled(canvas,10,5,50,50,red) The trick here is all mathematics

16. Important cool thing: You can draw past the end of the picture! • addText, addRect, and the rest of the drawing tools will work even if you go beyond the edge of the drawing. • Drawings will clip what can’t be seen in them, so you don’t get an array out of bounds error. • This is a big deal, because it means that you don’t have to do complicated math to see when you’re past the end of the drawing. • But only for the drawing functions. • If you set pixels, you’re still on your own to stay in range.

17. Making a tickertape def tickertape(directory,string): for frame in range(0,100): #99 frames canvas = makePicture(getMediaPath("640x480.jpg")) #Start at right, and move left addText(canvas,600-(frame*10),100,string) # Now, write out the frame # Have to deal with single digit vs. double digit frame numbers differently framenum=str(frame) if frame < 10: writePictureTo(canvas,directory+"//frame0"+framenum+".jpg") if frame >= 10: writePictureTo(canvas,directory+"//frame"+framenum+".jpg")

18. tickertape(r"C:\Documents and Settings\Mark Guzdial\My Documents\mediasources\tickertape","Welcome to CS1315!")

19. Can we move more than one thing at once? Sure! def movingRectangle2(directory): for frame in range(0,100): #99 frames canvas = makePicture(getMediaPath("640x480.jpg")) if frame < 50: #Less than 50, move down # Generate new positions each frame number addRectFilled(canvas,frame*10,frame*5, 50,50,red) if frame >= 50: #Greater than 50, move up addRectFilled(canvas,(50-(frame-50))*10,(50-(frame-50))*5, 50,50,red) # Let's have one just moving around addRectFilled(canvas,100+ int(10 * sin(frame)),4*frame+int(10* cos(frame)),50,50,blue) # Now, write out the frame # Have to deal with single digit vs. double digit frame numbers differently framenum=str(frame) if frame < 10: writePictureTo(canvas,directory+"//frame0"+framenum+".jpg") if frame >= 10: writePictureTo(canvas,directory+"//frame"+framenum+".jpg")

20. addRectFilled(canvas,100+ int(10 * sin(frame)), 4*frame+int(10* cos(frame)),50,50,blue) • What’s going on here? • Remember that both sine and cosine vary between +1 and -1. • Int(10*sin(frame)) will vary between -10 and +10 • With cosine controlling y and sine controlling x, should create circular motion • frame=1 • x is 108, y is 9 • frame=2 • x is 109, y is 4

21. Frames from two motions at once

22. Moving something else:Remember this? def copyBarbsFaceSmaller(): # Set up the source and target pictures barbf=getMediaPath("barbara.jpg") barb = makePicture(barbf) canvasf = getMediaPath("7inX95in.jpg") canvas = makePicture(canvasf) # Now, do the actual copying sourceX = 45 for targetX in range(100,100+((200-45)/2)): sourceY = 25 for targetY in range(100,100+((200-25)/2)): color = getColor(getPixel(barb,sourceX,sourceY)) setColor(getPixel(canvas,targetX,targetY), color) sourceY = sourceY + 2 sourceX = sourceX + 2 show(barb) show(canvas) return canvas To move Barb’s face around, we have to do this for each frame, moving the target each time.

23. Note: This will take a while • On my fastest computer, it takes over a minute. • Think about it: Why? We’ll get to it next week. • You might to know where you are. • Print isn’t too useful here. • Print doesn’t print until the program is done. • There is a JES function called printNow() that takes a string and will print it immediately. • That way, we can see what frame number we’re at. • Most important thing to know: DID WE GET PAST FRAME ONE? (Very important to know as code gets slower!)

24. Moving Barb’s head def moveahead(directory): barbf=getMediaPath("barbara.jpg") barb = makePicture(barbf) for frame in range(0,100): #99 frames printNow("Frame number: "+str(frame)) canvas = makePicture(getMediaPath("640x480.jpg")) # Now, do the actual copying sourceX = 45 for targetX in range(frame*3,frame*3+((200-45)/2)): sourceY = 25 for targetY in range(frame*3,frame*3+((200-25)/2)): color = getColor(getPixel(barb,int(sourceX),int(sourceY))) setColor(getPixel(canvas,targetX,targetY), color) sourceY = sourceY + 2 sourceX = sourceX + 2 # Now, write out the frame # Have to deal with single digit vs. double digit frame numbers differently framenum=str(frame) if frame < 10: writePictureTo(canvas,directory+"//frame0"+framenum+".jpg") if frame >= 10: writePictureTo(canvas,directory+"//frame"+framenum+".jpg")

25. moveahead(r"C:\Documents and Settings\Mark Guzdial\My Documents\mediasources\barbshead")

26. My, isn’t that gorey! • Can’t we make it easier to read? • Can we just deal with the parts that we care about? • Maybe we could use sub-functions? • At least for the writing out of the frame.

27. def moveahead(directory): barbf=getMediaPath("barbara.jpg") barb = makePicture(barbf) for frame in range(0,100): #99 frames printNow("Frame number: "+str(frame)) canvas = makePicture(getMediaPath("640x480.jpg")) # Now, do the actual copying sourceX = 45 for targetX in range(frame*3,frame*3+((200-45)/2)): sourceY = 25 for targetY in range(frame*3,frame*3+((200-25)/2)): color = getColor(getPixel(barb,int(sourceX),int(sourceY))) setColor(getPixel(canvas,targetX,targetY), color) sourceY = sourceY + 2 sourceX = sourceX + 2 # Now, write out the frame writeFrame(frame,directory,canvas) def writeFrame(num,directory,framepict): # Have to deal with single digit vs. double digit frame numbers differently framenum=str(num) if num < 10: writePictureTo(framepict,directory+"//frame0"+framenum+".jpg") if num >= 10: writePictureTo(framepict,directory+"//frame"+framenum+".jpg") Using subfunctions

28. What if we have over 100 frames? def writeFrame(num,directory,framepict): # Have to deal with single digit vs. double digit frame numbers differently framenum=str(num) if num < 10: writePictureTo(framepict,directory+"//frame00"+framenum+".jpg") if num >= 10 and num<100: writePictureTo(framepict,directory+"//frame0"+framenum+".jpg") if num >= 100: writePictureTo(framepict,directory+"//frame0"+framenum+".jpg") This will work with moveahead() and other functions—it’s generally useful

29. Using real photographs • Of course, we can use any real photographs we want. • We can use any of the techniques we’ve learned previously for manipulating the photographs. • Even more, we can use the techniques in new ways to explore a range of effects.

30. Slowly making it (very) sunset • Remember this code? def makeSunset(picture): for p in getPixels(picture): value=getBlue(p) setBlue(p,value*0.7) value=getGreen(p) setGreen(p,value*0.7) • What if we applied this to create frames of a movie, but slowly increased the sunset effect?

31. SlowSunset Just one canvas repeatedly being manipulated def slowsunset(directory): canvas = makePicture(getMediaPath("beach-smaller.jpg")) #outside the loop! for frame in range(0,100): #99 frames printNow("Frame number: "+str(frame)) makeSunset(canvas) # Now, write out the frame writeFrame(frame,directory,canvas) def makeSunset(picture): for p in getPixels(picture): value=getBlue(p) setBlue(p,value*0.99) #Just 1% decrease! value=getGreen(p) setGreen(p,value*0.99) Not showing you writeFrame() because you know how that works.

32. SlowSunset frames

33. Fading by background subtraction • Remember background subtraction? def swapbg(person, bg, newbg,threshold): for x in range(1,getWidth(person)): for y in range(1,getHeight(person)): personPixel = getPixel(person,x,y) bgpx = getPixel(bg,x,y) personColor= getColor(personPixel) bgColor = getColor(bgpx) if distance(personColor,bgColor) < threshold: bgcolor = getColor(getPixel(newbg,x,y)) setColor(personPixel, bgcolor) One change here is that the threshold is now an input.

34. Use the frame number as the threshold def slowfadeout(directory): bg = makePicture(getMediaPath("wall.jpg")) jungle = makePicture(getMediaPath("jungle2.jpg")) for frame in range(0,100): #99 frames canvas = makePicture(getMediaPath("wall-two-people.jpg")) #outside the loop! printNow("Frame number: "+str(frame)) swapbg(canvas,bg,jungle,frame) # Now, write out the frame writeFrame(frame,directory,canvas)

35. SlowFadeout

36. Dealing with real video • We really can’t deal with real video. • Dealing with each frame takes a lot of processing. • If you were going to process each frame as fast as it was coming in (or going out), you’d have 1/30th of a second to process each frame! • We cheat by • Saving each frame as a JPEG image • Processing the JPEG images • Convert the frames back to a movie

37. The original kid-in-bg-seq movie

38. Let’s have Mommy “watching” • We’ll paste Barb’s head into each frame. • We’ll use os.listdir to process all the frames of the kid sequence.

39. MommyWatching def mommywatching(directory): kiddir=r"C:\Documents and Settings\Mark Guzdial\My Documents\mediasources\kid-in-bg-seq" barbf=getMediaPath("barbara.jpg") barb = makePicture(barbf) frame = 0 for file in os.listdir(kiddir): if file.endswith(".jpg"): frame = frame + 1 printNow("Frame number: "+str(frame)) framepic = makePicture(kiddir+"//"+file) # Now, do the actual copying sourceX = 45 for targetX in range(frame*3, frame*3+((200-45)/2) ): sourceY = 25 for targetY in range(frame*3, frame*3+((200-25)/2) ): color = getColor(getPixel(barb,sourceX,sourceY)) setColor(getPixel(framepic,targetX,targetY), color) sourceY = sourceY + 2 sourceX = sourceX + 2 # Now, write out the frame writeFrame(frame,directory,framepic) We process each frame, and copy Mommy’s head to the frame, just like we animated in a line before onto a blank canvas.

40. MommyWatching

41. Lightening a picture • I took some video of a puppet show in black light. • Very hard to see the puppets. • Your eye can pick them up, but the camera can’t. • Recall earlier discussion: Your eye can detect luminance changes that no media can replicate.

42. Dark-fish2 sequence

43. How I did the processing • First try, lighten every pixel. • Didn’t work. • Made all the black whiter as well as the colors • No improvement in contrast • Second try, explore under MediaTools first • Black parts are really black • Lighter parts have really low number values • So: • Look for any pixel less black than black (threshold=8) • Lighten it a couple values

44. Lightenfish import os def lightenFish(directory): framenum = 0 for framefile in os.listdir(getMediaPath("dark-fish2")): framenum = framenum + 1 printNow("Frame: "+str(framenum)) if framefile.endswith(".jpg"): frame=makePicture(getMediaPath("dark-fish2")+"//"+framefile) for p in getPixels(frame): color = getColor(p) if distance(color,black)>8: color=makeLighter(color) color=makeLighter(color) setColor(p,color) writeFrame(framenum,directory,frame)

45. Original sequence again

46. Same frames after lightening

47. That took forever! • On a 500 Mhz G4, this took almost a minute per frame to process! • Over 160 frames • I started it at 3 pm, checked to make sure that the first few frames looked good, went to a meeting, and collected the result after 4:30

48. Why? • Why does movie processing take so long? • Why does sound processing seem to go so fast? • Why can Photoshop do these things faster than we can in Python? • What makes software fast, or slow? • See next lecture…