ImageMagick batch resizing performance

Question

convert \\
   original.jpg \\
  -quality 85 \\
  -colorspace rgb \\
  -profile /var/tmp/sRGB.icm \\
  -strip \\
  -profile /var/tmp/sRGB.icm \\
  -filter Lanczos \\
  -wr         


        

Answer 1

You CPU time is going to 3 tasks:

• JPEG decompression;
• resize;
• JPEG recompression

(Cropping itself takes maybe 1% of your time.)

To decode JPEG, just do it once, hold the result in RAM, and reuse for each output. (Details below.) That way, the cost will be insignificant.

To encode JPEG, use libjpeg-turbo; same API, but a 2-4x speedup if you use x86-{32,64} or ARM hardware.

To resize, ImageMagick is well-known for using ~100x as much CPU as any other software except PhotoShop and GIMP. That includes all photo viewers. It's doing multiple trigonometric functions per pixel, whereas everyone else just does one multiplication. Sometimes, if you look at pixels near a edge in the image, you can see ImageMagick choose a better color than its competitors. But if you think HTML5, Flash, Silverlight, Java, GD (popular with Perl, PHP, and Python web apps) etc look fine, then you don't need such pseudo-AI (artificial intelligence). You might be able to throw GPU (OpenCL) horsepower or more CPU (OpenMP) into ImageMagick, but why bother?

For high-efficiency, the equivalent of ImageMagick (de facto standard) is Imlib2. It is usable from almost as many OS/language environments as ImageMagick.

Your "convert" shell command is equivalent to 10-20 lines of a high level language calling Imlib2: decompress JPEG, and then repeatedly crop, resize, and compress JPEG.

An example without crop (or multiple output) is: Stretch, resize, or thumbnail an image using Perl

Let me know if you want other examples.

Answer 2

Late to the party but this is my current method if anybody has the same issue. If you're batch processing basic transforms but over many thousands of images, in my experience you don't get much benefit from openMP which seams to be good for 'multicoring' large complex transforms. My solution is a bash script to spawn the individual processes in parallel.

#!/bin/bash
counter=0
for i in tif/*.TIF;
do
    y=${i%.TIF}
    ((counter++))
    if [ -s gif$y.gif ];then
        :
    else
        gm convert $i gif$y.gif &
    fi
    if [ $counter -eq 30 ];then 
        ((counter =0))
        wait
    fi
done
wait  

This converts all TIF files in the 'tif' directory into gifs in the 'giftif' directory. If you have to stop this script, it will pick up where it left off next time. This chews up all 16 cores in my MBP and is around 12-14x faster than the single core method whilst i'm currently converting 150,000 images.

Answer 3

You should check if your ImageMagick installation comes with OpenCL support:

convert -list configure | grep FEATURES

If it does (like mine), you should see something like this:

FEATURES      HDRI OpenCL

This command

convert -version 

should also give info about supported features.

If it doesn't you should look after getting the most recent version of ImageMagick that has OpenCL support compiled in. Or if you build the package yourself from the sources, make sure OpenCL is used.

---

Update:

Oh wait. There's another feature that could help you, called OpenMP (for multi-processing).

When OpenMP is enabled, ImageMagick commands can execute in parallel on all the cores of your system. So if you have a quad-core system, and resize an image, the resizing happens on 4 cores (or even 8 if you have hyperthreading).

You can now also use the builtin -bench option to make ImageMagick run a benchmark for your command. For example:

convert logo: -resize 500% -bench 10 logo.png
  Performance[1]: 10i 0.689ips 1.000e 14.420u 0:14.510

This command with -resize 500% tells ImageMagick to run the convert command to scale the built-in IM logo: image by 500% in each direction. The -bench 10 part tells it to run that same command 10 times in a loop and then print the performance results:

• Since I don't have OpenMP enabled, I had only 1 thread (Performance[1]:).
• It reports that it ran 10 iterations (10i).
• The speed was nearly 0.7 iterations per second (0.689ips).
• Total user-alotted time was 14.420 seconds.

You should find out how your system is set up regarding resource limits with this command:

identify -list resource
  File       Area     Memory     Map       Disk    Thread         Time
  --------------------------------------------------------------------
   192    4.295GB       2GiB    4GiB  unlimited         1    unlimited

You can see my current system's settings (defaults -- I didn't tweak them). Each of the keywords in the column headers you can use pimp your system.

• The files defines the max concurrently opened files which ImageMagick will use.
• The memory, map, area and disk resource limits are defined in Bytes. For setting them to different values you can use SI prefixes, .e.g 500MB).

If I had OpenMP for ImageMagick on this system, I could run

convert -limit thread 2

in order to enable 2 parallel threads, re-run the benchmark and see if it really makes a difference, and if so how much. The I could set the limit to 4 or even 8 and repeat the excercise....

---

Finally, you could experiment with the internal format of ImageMagick's pixel cache, called MPC (Magick Pixel Cache). Some people say that for large operations the performance improves here, but I have no personal experience with it.

Convert your base picture to MPC first:

convert input.jpeg input.mpc

and only then run:

convert input.mpc [...your long-long-long list of crops...]

and see if this saves you significantly on time.

Most likely you can use this MPC format even "inline" (using the special mpr: notation), similar to how you applied the trick of using the mpr: format (memory program register) that reads the image into a named memory register. But I've never tried this technique to a real world problem, so I can't say how it works out in real life.

---

Update 2:

One more idea:

First check for your exact ImageMagick version: run convert -version.

In case your ImageMagick has a Q16 (or even Q32 or Q64) in its version string (meaning, its internal processes consider all images to have 16bit channel depth, which requires double memory as compared to Q8) -- this is the default nowadays -- you could test what performance benefits you'll achieve by switching to a Q8-build. (You'll pay your performance wins with quality losses, and you'll have to check if you can live with it or not....)