VIPS: An Image Processing Library for Large, and Not So Large, Images

1. John Cupitt Imperial College London, UK VIPS: An Image Processing Library for Large, and Not So Large, Images Kirk Martinez University of Southampton Southampton, UK Presented by Nicolas Robidoux Université Laurentienne Sudbury ON http://www.vips.ecs.soton.ac.uk LGM Montréal 2009

2. Introduction • VIPS is a 2D scientific image processing system. • Needs little memory, runs quickly • Optimised for multi-CPU machines • Good support for large images and for colour • Mostly C with some C++. Python interface • Library runs on any Unix and on Windows. LGPL • Has an advanced GUI, nip2, a blend of a spreadsheet and a paint program. GPL

3. Speed and memory use 1.8 GHz PM laptop Load, crop, shrink, sharpen and save a 5k x 5k pixel RGB tiled TIFF image. Fastest real time of three runs on a quiet system. Tests run on a 1.8 GHz PM laptop. ImageMagick and GraphicsMagick compiled with Q16. FreeImage does not have a sharpening operation, so that part of that test was skipped. nip2 seems slow because it has a long start-up time: once it starts, it processes at the same speed as the Python and C++ versions. Source-code for the various implementations is on the VIPS website.

4. Speed and memory use 2 x 2.7 GHz Opteron server Load, crop, shrink, sharpen and save a 5k x 5k pixel RGB tiled TIFF image. Fastest real time of three runs on a quiet system. Tests run on a 2 x 2.7 GHz Opteron server. ImageMagick and GraphicsMagick compiled with Q16. FreeImage does not have a sharpening operation, so that part of that test was skipped. nip2 seems slow because it has a long start-up time: once it starts, it processes at the same speed as the Python and C++ versions. Source-code for the various implementations is on the VIPS website.

5. SMP scaling Load, crop, shrink, colour-correct, sharpen and save a 10k x 10k pixel CIELAB image in VIPS format. Number of CPUs on horizontal axis, speed-up on vertical. Run on a 64-CPU Itanium2 supercomputer (SGI Origin 2000). Details of the benchmark and source-code are on the VIPS website. This is after a substantial tuning effort on the SMP system as part of the development of the PARSEC benchmark.

6. History • VIPS was started in 1990 as the image processing system for the VASARI project (multispectral imaging of old-master paintings to detect long-term colour change). • 10,000 x 10,000 pixels, seven colour bands, 16 bits per band, up to 1.6 GB for the final image. Our Sun4 had 64 MB of RAM and a 25 Mhz processor. Ouch!

7. History • VIPS looked something like this: Several operations run at once and images are pulled through the pipeline by demand from the sink. A simple, lightweight system combines operations without the need for large intermediate images.

8. History • We added SMP support in 1993: Run a pipeline in each thread, sinks arrange synchronisation.

9. History • We added large file (>2GB) support in 2002: We map a small window into each input file. These windows are shared between threads when possible. Window positions are calculated using 64-bit arithmetic.

10. History • We improved SMP scaling in 2005: A buffer manager decouples workers from the image write library so they never have to wait. We also added a system for quickly sharing and recycling pixel buffers.

11. General principles • 2D colour images only: no video, no volumes • Any number of (pseudo-)colour bands • All band elements in an image must have the same pixel format (eg. 32-bit signed int) • All operations are non-destructive • Images are uninterpreted arrays. Alpha channels, CMYK, layers, etc. must be implemented on top of VIPS • Pipelines are static. Apps have to destroy and rebuild to make a change.

12. API • The C++ and Python APIs are very similar to PIL. • import sys • from vipsCC import * • im = VImage.VImage (sys.argv[1]) • im = im.extract_area (100, 100, im.Xsize () - 200, im.Ysize () - 200) • im = im.similarity (0.9, 0, 0, 0) • mask = VMask.VIMask (3, 3, 8, 0, • [-1, -1, -1, • -1, 16, -1, • -1, -1, -1]) • im = im.conv (mask) • im.write (sys.argv[2]) • Load, crop, shrink, sharpen, save. Python binding is generated automatically by SWIG.

13. Other features • Pixel can be 8/16/32-bit integer, signed and unsigned, 32/64-bit float, 64/128-bit complex • XYZ, Lab, Yuv, Yxy, Lch, RGB colour spaces; ICC colour management with lcms • Operation database, plugins, metadata, many file formats, memory, disc and screen sinks • ~350 operators, mostly simple filters: rank, Fourier, morphological operators, convolutions, histogram operations, colour, arithmetic, affine • Simple: ~10k lines of C for the core, 50k in operators

14. Comparing VIPS and GEGL • Low-level • VIPS images are uninterpreted 3D arrays. VIPS has no explicit support for alpha channels, no CMYK image type, no layers, ... Applications built on VIPS are responsible for presenting images to the user. Operators do set hints about the possible interpretation a pixel might have, for example ”this image is a histogram”. GEGL directly supports features like alpha channels. • Static • VIPS pipelines are fixed: you can't alter any settings after calling an operator, you can only evaluate pixels. If an application wants to change a parameter, it has to destroy and rebuild the pipeline. You only need to destroy and rebuild from the change onwards. GEGL has a dynamic, interactive graph. • No caching • VIPS does almost no caching for you. There is a cache operator, but you have to explicitly add it to the graph yourself. By contrast, GEGL automatically caches every pixel which contributes to the display.

15. Comparing VIPS and GEGL • Operators are polymorphic • VIPS operators have to each handle all 10 element types and any number of image bands. This is done with metaprogramming: usually, an operator will have 2 or 3 implementations and use macros or templates to generate the code for all the cases. GEGL operators only see float arrays and use BABL to convert to and from the real data type. • Analogy: • VIPS = Xlib, GEGL = canvas widget. • You can imagine a version of GEGL which uses VIPS as a backend.

16. GUI • nip2, the VIPS GUI, is a spreadsheet where each cell can be a complex object: an image, plot, widget, matrix, etc. • nip2 has its own lazy, higher-order, pure functional language with classes, somewhat like dynamically typed Haskell. Spreadsheet cells are class instances. Cells are joined with snippets of this language. • As the spreadsheet recalculates it builds optimised VIPS pipelines behind the scenes. Image generation is then pure VIPS. • Fast, low memory use, huge images.

17. GUI

18. GUI • You can use nip2 for quite large, complex applications. We have a set of linked workspaces which analyze four-dimensional images (volumes over time) from PET scanners to calculate tissue inflammation indices. • The workspaces read 300MB of image data, process 7,000 images, generate 60 GB of intermediate images and take 2 minutes to completely recalculate. They need only 400 MB of RSS to run. • Useful tool for technical users. Not aimed at general audience.

19. GUI

20. Bad things about VIPS • Somewhat old-fashioned, clunky C API • Uses manpages. We get a lot of complaints about hard-to-navigate docs. • Limited range of operators. It would be nice to have segmentation, for example. • Awkward to extend, despite a plugin system. Operators have a fixed set of arguments and it's difficult to add functionality.

21. TODO • We've started moving VIPS to GObject. • The current stable version has several GObject-based systems. We plan to move most of the VIPS types to GObject in the next version, then start rewriting operations in the version after. We will switch to gtk-doc for API docs. • This should give us a sane, extensible, well-documented, easy to bind API with hopefully similar performance to the current version. • We'll aim to have a vips7 compatibility layer.

22. For more information http://www.vips.ecs.soton.ac.uk Kirk Martinez University of Southampton Southampton, UK John Cupitt Imperial College London, UK