Download
1 / 29
290 likes | 403 Views
Fast Image Replacement Using Multi-Resolution Approach. Chih-Wei Fang and Jenn-Jier James Lien Robotics Lab Department of Computer Science and Information Engineering National Cheng Kung University. Motivation. Photographs sometimes include unwanted objects.
E N D
Fast Image Replacement Using Multi-Resolution Approach Chih-Wei Fang andJenn-Jier James Lien Robotics Lab Department of Computer Science and Information Engineering National Cheng Kung University
Motivation • Photographs sometimes include unwanted objects. • Lose data when the network transmits media (ex. video) Robotics Lab.
System: Two modules • Texture Analysis (or Training) Module • Texture Synthesis Module
Processing for periodic pattern Processing for non-periodic pattern Input image for training (Size: W x H) ………… Is Periodic ………… ………… ………… No Yes There are Mn patches. ………… Non-periodic pattern Periodic pattern Mp > Mn There are Mp patches. M = Mp or Mn Processing for Γ-shaped pattern side-by-side side-by-side Principal Component Analysis (PCA)：Weight vector (W1…WN) for each Γ-shaped pattern Vector Quantization (VQ)： Speeding up to search the best patches. Texture Analysis Module
1. Sampling • Processing for non-periodic pattern • A pixel-by-pixel shifting method is used to divide the input texture into M patches. • M = (W–Wp+1)× (H–Hp+1) • W is the width of the input image (or texture) • H is the height of the input image • Wp is the width of the patch. • Hp is the height of the patch.
Processing for periodic pattern • The pattern is divided into M patches, where M=W×H. • Periodic patterns have continuous veins between two equivalent patterns when positioned side by side. • When the patch reaches the boundary of the input image, in order to form a complete patch, additional pixels are supplied from the opposite border.
ω pixels • Processing for the Γ-shaped pattern • Taking the whole pieces of the patch as training data may lead to an overestimation of the underlying structure of the patch and will certainly increase the length of the training time. • In addition, searching the matching patches by considering their whole contents generally produce unsatisfactory results since the whole contents tend to be quite different from the initial random values and it may cause the rim effect to become distinct.
2. Principal Component Analysis (PCA) • Each patch has K elements. During training, PCA is used to transform the original K×M matrix to an N×K matrix, where M>>K>N. And there are N eigenvectors. This can reduce most of the time complexity operations, and hence dramatically increases the performance. • Recombining the appearance of the features while maintaining the coherence of the characteristic content PCA
After the PCA process and a sort based on the eigenvalues with corresponding eigenvectors: • The first several eigenvectors control the global geometrical structure. • The middle eigenvectors control the local features. • The last few eigenvectors are controlled by some noises. • These noises cause the photograph to appear truer, but have no influence on the geometric structure. • Therefore a good matching structure need only compare the first several eigenvectors.
3. Vector Quantization (VQ) • Searching for the best matching pattern from the eigenspace Ψ during the synthesis process is a computationally expensive task. • In order to speed up the synthesis process, the training data is initially projected onto the eigenspace Ψ in order to retrieve the weight vectors. • The data is then separated into C clusters by means of VQ. Using this approach, the computational time can be reduced from O(M) to O( ).
Texture Synthesis Module 4. Result: Find the best patches to fill in the WoxHo-pixel output image. 1. Initialization: Each pixel is assigned a random value in the WoxHo-pixel output image area. 2. Γ-shaped pattern projectiong: Each WpxHp-pixel search window (or patch) is taken the Γ-shaped pattern and then projected onto the eigenspace Ψ to obtain one N-dimensional weight vector. Shifting this search window over this image in the scan-line order from the top-left order to the bottom-right order. 3. Similarity measure (based on SSD): Using the weight vector to find the closest cluster and then find the best matching patch from this cluster in the input image. Then this matching patch is called to fill the region of the search window. ……… ……… ………
Image Replacement Down-sampling with Texture Analysis β0 I0 Up-sampling with Texture Synthesis I1 β1 I2 β2 I3 β3 C4 C3 C2 C1 C0 I4 β4
1. Preprocessing Using Multi-resolution Approach • The input image I0 and the inverse matte β0 to do l times down-sampling ↓ , then to get each level of input image Ii and inverse matte βi, i=0~l • Bidenotes the background region and the foreground region Fi is then utilized to search the background Bi
K pixels ω … 2. Training Process Based on Background Region • If the same Γ-shaped pattern of the search window (or patch) is applied, the fragment between two different textures will be mapped by only one texture rather than being mapped by the mixing of two different textures. Hence, the search window is modified.
3. Completion Process • The reconstructed image Ci • First, the system reconstructs the level i (i=l) image Ii • The replaced part is reconstructed (or synthesized), called Ii’ • Only the reconstructed part of Ii’ is preserved at position Fi • The other parts are replaced by the original background Bi • The system applies the up-sampling ↑ technique • from the reconstructed image Ci at level i and proceeds to only replace the level i-1 image Ii-1 at position Fi-1
Experimental Results Table 1: Average training time in Section 3 for input images of various sizes. Units of time: milliseconds (ms). Table 2: Average synthesizing time in Section 4 for output images of various sizes.
Table 3: The replacement information of each level in Figure 9 and Figure 12 for multi-resolution approach. “~0” means close to 0 ms.
Table 4: The time in Figure 9 and Figure 12 for various processes. The total time of the training and synthesis processes include the processes of the multi-resolution approach, PCA, projection (the patches are projected onto eigenspace Ψ to obtain the weight vectors), VQ, and synthesis.
Conclusions • We developed a system including two modules • the texture analysis and synthesis modules. This system is able to be applied to the two different purposes: the synthesis of a large image, and the replacement of local removed region. • According to the training non-periodic or periodic pattern • we use different sampling methods to obtain different amount of patches in order to reduce the emergences of the seams of the output synthesized image. • Reduce most of the time complexity operations • the analysis module can reduce dimensions of the training data and cluster these data, so the synthesis module can synthesize a large output image very fast and keep geometrical structures and veins continuous.
Conclusions • Without needing to assign initial random values • The same process can also be used to replace the removed regions. Here, the multi-resolution approach is applied to the image replacement without needing to assign initial random values or approximate values. • Multi-resolution go through the image replacement process • The down-sampling step is used for the analysis process as compiling the training data, and the up-sampling step is used for the reconstructing process as assigning initial values. • Handle the large removed region • So this approach enables the system to handle the large removed region and obtain more realistic image (or textures) quickly.
