Final Year Project 2004/2005 LYU0402 Augmented Reality Table for Interactive Card Games

1. Final Year Project 2004/2005LYU0402Augmented Reality Table for Interactive Card Games Supervisor: Prof Michael Lyu Presented by: Kevin Chow, Albert Lam

2. Overview • Introduction • Objective • Trading Card Game • Architecture of ART • Implementation in First Term • Experiment • Conclusion • Future Work

3. Introduction • What is Augmented Reality? • Mixed Reality combines the content from the real world with virtual imaginary • Augmented Reality is a subset of this where virtual content is overlaid into real objects of the world

4. Mixed Reality (MR) RealEnvironment AugmentedReality (AR) Augmented Virtuality (AV) VirtualEnvironment Reality-Virtuality (RV) Continuum Introduction (Con’t) • Paul Milgram characterized Mixed Reality interfaces on his “Reality-Virtuality Continuum” in 1994

5. Introduction (Con’t) • An AR system has the following three main characteristics: • Combines real and virtual objects in a real environment • Runs interactively, and in real time • Registers virtual objects onto the real world

6. Introduction (Con’t) • ART stands for Augmented Reality Table • User can play Card Games on the ART. • ART can enhance the visual effect during the playing of the card game.

7. Objective • Develop a generic Interactive ART Trading Card Game application. • Enhance the traditional card games: • Visual effect • Sound effect • Complex calculation

8. Trading Card Game • Trading Card Games • Two players play a match. • Players play cards to fight with each other. It includes summoning monster, casting magic, or setting traps, etc. • We choose “YU-GI-OH” as our implementation

9. Trading Card Game (Con’t) • Trading card game includes five kinds of actions • Restore • Draw cards • Play cards • Attack/Challenge • Discard cards

10. An Example - YU-GI-OH • In YU-GI-OH, Different information are stored on the card.

11. An Example - YU-GI-OH (Con’t) • Game Mat of YU-GI-OH • Different kinds of card must be put inside particular card zones

12. An Example - YU-GI-OH (Con’t) 1200 – 500 = 700 -700 ATK : 1200 ATK : 500 LP : 2000 LP : 1300 LP : 2000

13. An Example - YU-GI-OH (Con’t) 1300 – 1200 = 100 -100 ATK : 1200 DEF : 1300 LP : 1300 LP : 2000 LP : 1900

14. Architecture • Hardware Setup • Plasma monitor • Overhead camera • System architecture • Augmented Reality Perception • Game Core • Database • Game Enhancement

15. Hardware Setup

16. Hardware Setup (Con’t) • Computer • processes perceiving image • generates visual and audio enhancement • controls game flow • Plasma Monitor. • acts as the Game Table • displays computer-generated scene • Overhead mounted camera • captures the cards and the screen of the plasma. • only input of the system

17. System Architecture • Game Core Module • Perception Module • Database Module • Game Enhancement Module

18. System Architecture (Con’t)

19. System Architecture (Con’t) • Microsoft DirectX SDK • A set of low-level APIs for high-performance multimedia applications • directly access the hardware • provides device independent through HAL • We will use DirectShow, Direct3D and DirectSound in our implementation

20. Implementation • Perception Module • Calibration • Search window locator • Card locator • Card Recognizer • Database Module • Identify card type • Image retrieval • Image matching

21. Perception Module • Read and process the raw video perceived from the camera • Detect input card and command • Some assumptions • fixed camera, fixed table • camera is approximately right above the table • cards can only be placed in predefined region called card zones

22. Calibration • Why? • Captured image varies with environmental lighting condition • Take advantage of the assumptions made

23. Calibration (Con’t) • How? • Set the card zone position • Set the card area threshold for each card zone • Calibrate the colors • Calibrate card type colors

24. Search Windows Locator • Find region to search for cards • The screen is partitioned into some fixed small search windows • Search windows which just stop changing will be processed

25. Search Windows Locator (Con’t) • Located search windows that have changes • Image different • Pixel changed if • A Search window is said to be changed if • Activate search window from“change” to “unchange” state

26. Card Locator • Locate four corner card positions given a search window • Apply Canny Algorithm • Extract Contours • Search for contours which • contain 4 points • contain nearly right angle corners • have area within acertain threshold

27. Card Locator (Con’t) • Canny Algorithm • Remove noise by Gaussian Filter • Compute edge strength and edge direction bySobel Operator • Apply Nonmaximum Suppression to trace edges • Use Hystersis to eliminate streaking

28. Card Recognizer • Card orientation • Compare the lengths of the four edges • Compare color difference between top and bottom parts of the card

29. Card Recognizer (Con’t) • Get Card Image • Extract the undistorted card image from a distorted image

30. (x2,y2) (x3,y3) . (x’,y’) . (x,y) h (x1,y1) w (x0,y0) Card Recognizer (Con’t) • Pixel coordinate transformation • Map a point (x,y) on the distorted image to a point (x’,y’) on the undistorted image

31. Card Recognizer (Con’t) • Brightness interpolation • Assign brightness value to the point (x,y) • Apply Bilinear Interpolation

32. Card Recognizer (Con’t) • Identify the card uniquely • Query the Image Database with the card image for unique card Id • Query the Card Database with the card Id for card information and details

33. Database Module • Contains all game information • Identify card image by 3 steps: • Identify card type • Classify card by card type • Image retrieval • Retrieve several similar candidate cards • Image matching • Find the best match card

34. Identify Card Type • Compare the background color of the card to the calibrated color

35. Image Retrieval • Retrieve similar images of same card type. • Use color-based retrieval method due to low resolution • Color Histogram method select cards with largest color histogram intection

36. Image Retrieval (Con’t) • Color histogram intersection

37. Image Matching • Select the best matched card from the candidates • Split the captured image into 4 channels • Compare only inner image • Compare the different channels to the database separately

38. Image Matching • Find the pixel difference by • Reject the image if the pixel difference is larger than the threshold • Accept the image with minimum pixel difference

39. Experiment • Match 15 cards to a set of different cards with threshold set to 1500 • Here, we have chosen results of three cards for discussion

40. Experimental Result

41. Experimental Result (Con’t)

42. Experimental Result (Con’t)

43. Conclusion • We have implemented a simplified version of the ART system • Concerning low resolution of the image, we have develop an algorithm to recognize card with high accuracy but less efficiency

44. Future Work • Color Calibration • Develop an efficient card recognition algorithm while retaining high accuracy • Game Enhancement • Interactive input detector

45. End of Presentation Thank you very much

46. Q & A Session