Image Renaissance Using Discrete Optimization

1. Image RenaissanceUsing Discrete Optimization

2. Outline • Theory: positions and patches • Finding positions with good similarities • Determining the patch sizes • Results and discussions

3. - 1 -Theory:positions and patches

4. Main idea: inpainting as a jigsaw Inpainting: • Reconstructing a destroyed part of an image (specified by a mask) • The process should be undetectable Our method: • Considering the reconstruction as a jigsaw which pieces are patches taken from the “good” part of the image 1 - Theory: positions and patches

5. Two steps • Finding positions/offsets in the “good” image having strong similarities with the content of the image at the boundary of the area to rebuild • For each offset found, finding which pixels should be copied to reconstruct the destroyed area 1 - Theory: positions and patches

6. - 2 -Finding positionswith good similarities

7. Positions and offsets • Such a problem consists in finding, given an origin point at the boundary of the region to inpaint, a number of candidate positions in the image domain that have similar content with the one at the boundary of the area to rebuild • A few remarks: • Missing content is not necessarily in the neighborhood of the destroyed region… • Testing each offset in the image for each position at the boundary of the destroyed area would be too expensive in computation time… • Our solution: using a probabilistic method, the particle filters 2 – Finding positions with good similarities

8. Particle filters theory • Considering a fixed number of samples, we have to repeat this steps: • Weight the samples with a Probability Density Function (pdf) • Generate a new set of samples (of equal number) from the existing ones with the biggest weights applying small perturbations (importance of perturbation decreases with the iterations to obtain a convergence) 2 – Finding positions with good similarities

9. Particle filters in positions research • A particle has an origin point (at the boundary of the area to inpaint, which is invariant), an offset and a size (width and height) • The perturbation is a displacement of the offset and a modification of the size • The weight is evaluated by a Sum of Square Differences (SSD) between the sized neighborhood around the origin point and the offset point balanced with the size 2 – Finding positions with good similarities

10. - 3 -Determiningthe patch sizes

11. How to find the optimal partition • For each pixel in the area to inpaint we have multiple offsets (and, as a consequence, multiple pixel values possible), so we have to find the best label partition to rebuild the image: • In that aim we use a combinatorial optimization: the α-expansion algorithm (using graph cuts) 3 – Determining the patch sizes

12. Term 1 to minimize:constraints to data • : region of the image to be reconstructed • : region of the image to be reconstructed and its close neighborhood (which contains data) • : distance (similarity) between the label and the data of the image at position 3 – Determining the patch sizes

13. Term 2 to minimize:constraints between patches • : region of the image to be reconstructed • : 4-neighborhood of • : distance (similarity) between the label L and the label at position • : Euclidian distance from to the boundary of the area to rebuild (to reconstruct firstly the borders of the region) 3 – Determining the patch sizes

14. Term 3 to minimize:smoothing term • : region of the image to be reconstructed • : 4-neighborhood of • : Potts model:penalty for neighbors with different 3 – Determining the patch sizes

15. Global energy function on partitionto minimize ω • Minimized through successive α-expansion (using graph cuts) 3 – Determining the patch sizes

16. Alpha-expansion principle • With a given partition, the algorithm expands the label α over all the other labels minimizing the energy function 3 – Determining the patch sizes "green"-expansion

17. Alpha-expansion algorithm • For each offset selected (explaining the interest to reduce their number), we: • Construct a graph representing the energy on the area to inpaint • Execute the graph-cut algorithm on the graph • Modify the partition • This is repeated until stability (but in practice, one iteration on each offset gives a result near of the final result) • Each iteration will so enhance the result by minimizing the global energy Y. Boykov, O. Veksler and R. Zabih.Fast Approximate Energy Minimization via Graph Cuts. PAMI 2001. 3 – Determining the patch sizes

18. - 4 -Results and discussions

19. Results 4 – Results and discussion

20. Results 4 – Results and discussion

21. Conclusion • Graph-based combinatorial approach for inpainting • Advantages: • Efficient for textured images • Quite fast: in general a couple of minutes (but the final result is nearly reached after the first series of α-expansion) • Drawbacks: • Polynomial cost depending on the number of offsets selected (so particular attention must be paid on this step) 4 – Results and discussion

22. Thank you for your attention!Any question?