Interpreting a benchmark in C, Clojure, Python, Ruby, Scala and others [closed]

Answer 1

Here's a fast Clojure version, using the same basic algorithms:

(set! *unchecked-math* true)

(defn is-prime? [^long n]
  (loop [i 2]
    (if (zero? (unchecked-remainder-int n i))
      false
      (if (>= (inc i) n)
        true
        (recur (inc i))))))

(defn sexy-primes [m]
  (for [x (range 11 (inc m))
        :when (and (is-prime? x) (is-prime? (- x 6)))]
    [(- x 6) x]))

It runs about 20x faster than your original on my machine. And here's a version that leverages the new reducers library in 1.5 (requires Java 7 or JSR 166):

(require '[clojure.core.reducers :as r]) ;'

(defn sexy-primes [m]
  (->> (vec (range 11 (inc m)))
       (r/filter #(and (is-prime? %) (is-prime? (- % 6))))
       (r/map #(list (- % 6) %))
       (r/fold (fn ([] []) ([a b] (into a b))) conj)))

This runs about 40x faster than your original. On my machine, that's 100k in 1.5 seconds.

Answer 2

The answer to your question #1 is that Yes, the JVM is incredably fast and yes static typing helps.

The JVM should be faster than C in the long run, possibly even faster than "Normal" assembly language--Of course you can always hand optimize assembly to beat anything by doing manual runtime profiling and creating a separate version for each CPU, you just have to be amazingly good and knowledgable.

The reasons for Java's speed are:

The JVM can analyze your code while it runs and hand-optimize it--for instance, if you had a method that could be statically analyzed at compile time to be a true function and the JVM noticed that you were often calling it with the same parameters, it COULD actually eliminate the call completely and just inject the results from the last call (I'm not sure if Java actually does this exactly, but it doest a lot of stuff like this).

Due to static typing, the JVM can know a lot about your code at compile time, this lets it pre-optimize quite a bit of stuff. It also lets the compiler optimize each class individually without knowledge of how another class is planning to use it. Also Java doesn't have arbitrary pointers to memory location, it KNOWS what values in memory may and may not be changed and can optimize accordingly.

Heap allocation is MUCH more efficient than C, Java's heap allocation is more like C's stack allocation in speed--yet more versatile. A lot of time has gone into the different algroithims used here, it's an art--for instance, all the objects with a short lifespan (like C's stack variables) are allocated to a "known" free location (no searching for a free spot with enough space) and are all freed together in a single step (like a stack pop).

The JVM can know quirks about your CPU architecture and generate machine code specifically for a given CPU.

The JVM can speed your code long after you shipped it. Much like moving a program to a new CPU can speed it up, moving it to a new version of the JVM can also give you huge speed performances taylored to CPUs that didn't even exist when you initially compiled your code, something c physically cannot do without a recomiple.

By the way, most of the bad rep for java speed comes from the long startup time to load the JVM (Someday someone will build the JVM into the OS and this will go away!) and the fact that many developers are really bad at writing GUI code (especially threaded) which caused Java GUIs to often become unresponsive and glitchy. Simple to use languages like Java and VB have their faults amplified by the fact that the capibilities of the average programmer tends to be lower than more complicated languages.

Answer 3

Never mind the benchmarks; the problem got me interested and I made some fast tweaks. This uses the lru_cache decorator, which memoizes a function; so when we call is_prime(i-6) we basically get that prime check for free. This change cuts the work roughly in half. Also, we can make the range() calls step through just the odd numbers, cutting the work roughly in half again.

http://en.wikipedia.org/wiki/Memoization

http://docs.python.org/dev/library/functools.html

This requires Python 3.2 or newer to get lru_cache, but could work with an older Python if you install a Python recipe that provides lru_cache. If you are using Python 2.x you should really use xrange() instead of range().

http://code.activestate.com/recipes/577479-simple-caching-decorator/

from functools import lru_cache
import time as time_

@lru_cache()
def is_prime(n):
    return n%2 and all(n%i for i in range(3, n, 2))

def primes_below(x):
    return [(i-6, i) for i in range(9, x+1, 2) if is_prime(i) and is_prime(i-6)]

correct100 = [(5, 11), (7, 13), (11, 17), (13, 19), (17, 23), (23, 29),
        (31, 37), (37, 43), (41, 47), (47, 53), (53, 59), (61, 67), (67, 73),
        (73, 79), (83, 89)]
assert(primes_below(100) == correct100)

a = time_.time()
print(primes_below(30*1000))
b = time_.time()

elapsed = b - a
print("{} msec".format(round(elapsed * 1000)))

The above took only a very short time to edit. I decided to take it one step further, and make the primes test only try prime divisors, and only up to the square root of the number being tested. The way I did it only works if you check numbers in order, so it can accumulate all the primes as it goes; but this problem was already checking the numbers in order so that was fine.

On my laptop (nothing special; processor is a 1.5 GHz AMD Turion II "K625") this version produced an answer for 100K in under 8 seconds.

from functools import lru_cache
import math
import time as time_

known_primes = set([2, 3, 5, 7])

@lru_cache(maxsize=128)
def is_prime(n):
    last = math.ceil(math.sqrt(n))
    flag = n%2 and all(n%x for x in known_primes if x <= last)
    if flag:
        known_primes.add(n)
    return flag

def primes_below(x):
    return [(i-6, i) for i in range(9, x+1, 2) if is_prime(i) and is_prime(i-6)]

correct100 = [(5, 11), (7, 13), (11, 17), (13, 19), (17, 23), (23, 29),
        (31, 37), (37, 43), (41, 47), (47, 53), (53, 59), (61, 67), (67, 73),
        (73, 79), (83, 89)]
assert(primes_below(100) == correct100)

a = time_.time()
print(primes_below(100*1000))
b = time_.time()

elapsed = b - a
print("{} msec".format(round(elapsed * 1000)))

The above code is pretty easy to write in Python, Ruby, etc. but would be more of a pain in C.

You can't compare the numbers on this version against the numbers from the other versions without rewriting the others to use similar tricks. I'm not trying to prove anything here; I just thought the problem was fun and I wanted to see what sort of easy performance improvements I could glean.

Answer 4

I couldn't resist to do a few of the most obvious optimizations for the C version which made the 100k test now take 0.3s on my machine (5 times faster than the C version in the question, both compiled with MSVC 2010 /Ox).

int isprime( int x )
{
    int i, n;
    for( i = 3, n = x >> 1; i <= n; i += 2 )
        if( x % i == 0 )
            return 0;
    return 1;
}

void findprimes( int m )
{
    int i, s = 3; // s is bitmask of primes in last 3 odd numbers
    for( i = 11; i < m; i += 2, s >>= 1 ) {
        if( isprime( i ) ) {
            if( s & 1 )
                printf( "%d %d\n", i - 6, i );
            s |= 1 << 3;
        }
    }
}

main() {
    findprimes( 10 * 1000 );
}

Here is the identical implemention in Java:

public class prime
{
    private static boolean isprime( final int x )
    {
        for( int i = 3, n = x >> 1; i <= n; i += 2 )
            if( x % i == 0 )
                return false;
        return true;
    }

    private static void findprimes( final int m )
    {
        int s = 3; // s is bitmask of primes in last 3 odd numbers
        for( int i = 11; i < m; i += 2, s >>= 1 ) {
            if( isprime( i ) ) {
                if( ( s & 1 ) != 0 )
                    print( i );
                s |= 1 << 3;
            }
        }
    }

    private static void print( int i )
    {
        System.out.println( ( i - 6 ) + " " + i );
    }

    public static void main( String[] args )
    {
        // findprimes( 300 * 1000 ); // for some JIT training
        long time = System.nanoTime();
        findprimes( 10 * 1000 );
        time = System.nanoTime() - time;
        System.err.println( "time: " + ( time / 10000 ) / 100.0 + "ms" );
    }
}

With Java 1.7.0_04 this runs almost exactly as fast as the C version. Client or server VM doesn't show much difference, except that JIT training seems to help the server VM a bit (~3%) while it has almost no effect with the client VM. The output in Java seems to be slower than in C. If the output is replaced with a static counter in both versions, the Java version runs a little faster than the C version.

These are my times for the 100k run:

• 319ms C compiled with /Ox and output to >NIL:
• 312ms C compiled with /Ox and static counter
• 324ms Java client VM with output to >NIL:
• 299ms Java client VM with static counter

and the 1M run (16386 results):

• 24.95s C compiled with /Ox and static counter
• 25.08s Java client VM with static counter
• 24.86s Java server VM with static counter

While this does not really answer your questions, it shows that small tweaks can have a noteworthy impact on performance. So to be able to really compare languages you should try to avoid all algorithmic differences as much as possible.

It also gives a hint why Scala seems rather fast. It runs on the Java VM and thus benefits from its impressive performance.

Answer 5

I'll answer just #2, since it's the only one I've got anything remotely intelligent to say, but for your Python code, you're creating an intermediate list in is_prime, whereas you're using .map in your all in Ruby which is just iterating.

If you change your is_prime to:

def is_prime(n):
    return all((n%j > 0) for j in range(2, n))

they're on par.

I could optimize the Python further, but my Ruby isn't good enough to know when I've given more of an advantage (e.g., using xrange makes Python win on my machine, but I don't remember if the Ruby range you used creates an entire range in memory or not).

EDIT: Without being too silly, making the Python code look like:

import time

def is_prime(n):
    return all(n % j for j in xrange(2, n))

def primes_below(x):
    return [(j-6, j) for j in xrange(9, x + 1) if is_prime(j) and is_prime(j-6)]

a = int(round(time.time() * 1000))
print(primes_below(10*1000))
b = int(round(time.time() * 1000))
print(str((b-a)) + " mils")

which doesn't change much more, puts it at 1.5s for me, and, with being extra silly, running it with PyPy puts it at .3s for 10K, and 21s for 100K.

Answer 6

Rough answers:

1. Scala's static typing is helping it quite a bit here - this means that it uses the JVM pretty efficiently without too much extra effort.
2. I'm not exactly sure on the Ruby/Python difference, but I suspect that (2...n).all? in the function is-prime? is likely to be quite well optimised in Ruby (EDIT: sounds like this is indeed the case, see Julian's answer for more detail...)
3. Ruby 1.9.3 is just much better optimised
4. Clojure code can certainly be accelerated a lot! While Clojure is dynamic by default, you can use type hints, primitive maths etc. to get close to Scala / pure Java speed in many cases when you need to.

Most important optimisation in the Clojure code would be to use typed primitive maths within is-prime?, something like:

(set! *unchecked-math* true) ;; at top of file to avoid using BigIntegers

(defn ^:static is-prime? [^long n]
  (loop [i (long 2)] 
    (if (zero? (mod n i))
      false
      (if (>= (inc i) n) true (recur (inc i))))))

With this improvement, I get Clojure completing 10k in 0.635 secs (i.e. the second fastest on your list, beating Scala)

P.S. note that you have printing code inside your benchmark in some cases - not a good idea as it will distort the results, especially if using a function like print for the first time causes initialisation of IO subsystems or something like that!