LYU0503 Document Image Restoration on Mobile Phone Using Onboard Camera

1. LYU0503Document Image Restorationon Mobile PhoneUsing Onboard Camera Supervisor: Prof. Michael R.Lyu Group Member: Leung Man Kin, Stephen Ng Ying Kit, Tommy

2. Outline • Introduction • Approach • Stitching Algorithm • Geometric Transformation • Progress • Conclusion • Future Plan

3. Outline • Introduction • Approach • Stitching Algorithm • Geometric Transformation • Progress • Conclusion • Future Plan

4. Introduction • Use mobile phone to capture photo of document. • The document can be stored in the phone or sent to other people by several means.

5. Objective • Use camera to take photo of different parts of a document. • Use suitable algorithm to merge the photos back to a whole document. • Transmit the document by several means • e.g. Fax, MMS, etc.

6. Motivation • Most of phones nowadays have built-in camera. • Resolution of camera is not enough for document • The text on the document would be blurred • Symbian phone can provide several transmission function. E.g. Fax, MMS • Fax at any time!

7. Outline • Introduction • Approach • Stitching Algorithm • Geometric Transformation • Progress • Conclusion • Future Plan

8. Approach • Take photos of different part of document • Use stitching algorithm to merge the photos into a photo • Stitching algorithm is our major research area of our project

9. Assumption • The document consists mainly of text. • The camera has macro mode. • Sharp Close Up.

10. Approach • Procedure of our design • Take a photo of full document first. • Then ask the user to take photos of different parts • Overlay a rough picture on the screen of the phone.

11. Approach A4 document

12. Outline • Introduction • Approach • Stitching Algorithm • Geometric Transformation • Progress • Conclusion • Future Plan

13. Stitching Algorithm • Stitching needs to find corresponding points in two images • two different approaches • Direct alignment • Compare images pixel by pixel • e.g. • Feature-based registration • Use some methods to find feature points • Match features in two images

14. Stitching Algorithm • Feature-based registration is chosen. • Usually, direct alignment is less efficient. • Need to consider different scale, rotation, etc. • There are several methods to extract feature. • E.g. corner detector. • Scalable Invariant Feature Transform (SIFT) is used to extract the feature. • The features are invariant to image scale and rotation.

15. Stitching Algorithm • How to get the feature point? • Identifies key locations in scale space by looking for locations that are maxima or minima of a difference-of-Gaussian function.

16. Stitching Algorithm • Varying σ • Repeat the above process using subsampled image for several times • Decrease the resolution by 2.

17. Stitching Algorithm • Find the maxima or minima by comparing 26 neighbors. • They are the key feature points.

18. Stitching Algorithm • Property of the feature keypoints • Achieve partial invariance to local variations like affine or 3D projections. • Need to filter feature keypoints further • Reject keypoints that have low contrast.

19. Stitching Algorithm • How to store the feature? • For each keypoint, image gradient magnitudes and orientations are sampled to store the feature.

20. Stitching Algorithm • Compare the features in two images • Get the position to stitch • However, the photo taken may suffer from geometric distortion • e.g. Lens distortion, taking photo with different angles

21. Stitching Algorithm • Use the matched feature to calculate the transformation parameters. • Transform images to the correct orientation before merging

22. Stitching Algorithm SIFT Extract feature keypoint Matching keypoints in two images Find transformation parameter by matched keypoints Transform images using the parameter Merge all the images

23. Outline • Introduction • Approach • Stitching Algorithm • Geometric Transformation • Progress • Conclusion • Future Plan

24. Lens Correction Transformation

25. Image1 Image2 Transformation parameters ai Transformation parameters bi Image3

26. Take several overlapping images of the document……

27. High Resolution Document Image Restored!

28. Outline • Introduction • Approach • Stitching Algorithm • Geometric Transformation • Progress • Conclusion • Future Plan

29. Porting to Symbian Phone • On the PC, the stitching algorithm is implemented using C# and Matlab. • We should port the algorithm to the Symbian phone. • Symbian OS use C++ as the language.

30. Porting to Symbian Phone • Difficulty: • Their syntaxes are not the same. • e.g. set / get keyword in C# • Symbian and C# provide different API. • Math library, Array library. • Some of data types are different in Symbian C++.

31. Porting to Symbian Phone • Difficulty: • The processor on Symbian phone is not powerful. • The memory space on Symbian phone is not large. • We may need modify the algorithm so that it would suit the phone.

32. Outline • Introduction • Approach • Stitching Algorithm • Geometric Transformation • Progress • Conclusion • Future Plan

33. Conclusion • We have studied different stitching algorithms • We have implemented the stitching algorithm on PC • We are porting the stitching algorithm on Symbian phone

34. Outline • Introduction • Objective • Motivation • Approach • Stitching algorithm • Geometric Transformation • Progress • Conclusion • Future Plan

35. Future Plan • Implement the stitching on the Symbian phone. • Modify SIFT or design other algorithms for more efficient performance on mobile phone

36. Future Plan • Improve the algorithm of merging. • Study super-resolution to make text clearer.