efficiently checking that all the elements of a (big) list are the same

Question

Problem

Let us suppose that we have a list xs (possibly a very big one), and we want to check that all its elements are the same.

I came up wi

Answer 1

First of all, I don't think you want to be working with lists. A lot of your algorithms rely upon calculating the length, which is bad. You may want to consider the vector package, which will give you O(1) length compared to O(n) for a list. Vectors are also much more memory efficient, particularly if you can use Unboxed or Storable variants.

That being said, you really need to consider traversals and usage patterns in your code. Haskell's lists are very efficient if they can be generated on demand and consumed once. This means that you shouldn't hold on to references to a list. Something like this:

average xs = sum xs / length xs

requires that the entire list be retained in memory (by either sum or length) until both traversals are completed. If you can do your list traversal in one step, it'll be much more efficient.

Of course, you may need to retain the list anyway, such as to check if all the elements are equal, and if they aren't, do something else with the data. In this case, with lists of any size you're probably better off with a more compact data structure (e.g. vector).

Now that this is out of they way, here's a look at each of these functions. Where I show core, it was generated with ghc-7.0.3 -O -ddump-simpl. Also, don't bother judging Haskell code performance when compiled with -O0. Compile it with the flags you would actually use for production code, typically at least -O and maybe other options too.

Solution 0

allTheSame :: (Eq a) => [a] -> Bool
allTheSame xs = and $ map (== head xs) (tail xs)

GHC produces this Core:

Test.allTheSame
  :: forall a_abG. GHC.Classes.Eq a_abG => [a_abG] -> GHC.Bool.Bool
[GblId,
 Arity=2,
 Str=DmdType LS,
 Unf=Unf{Src=<vanilla>, TopLvl=True, Arity=2, Value=True,
         ConLike=True, Cheap=True, Expandable=True,
         Guidance=IF_ARGS [3 3] 16 0}]
Test.allTheSame =
  \ (@ a_awM)
    ($dEq_awN :: GHC.Classes.Eq a_awM)
    (xs_abH :: [a_awM]) ->
    case xs_abH of _ {
      [] ->
        GHC.List.tail1
        `cast` (CoUnsafe (forall a1_axH. [a1_axH]) GHC.Bool.Bool
                :: (forall a1_axH. [a1_axH]) ~ GHC.Bool.Bool);
      : ds1_axJ xs1_axK ->
        letrec {
          go_sDv [Occ=LoopBreaker] :: [a_awM] -> GHC.Bool.Bool
          [LclId, Arity=1, Str=DmdType S]
          go_sDv =
            \ (ds_azk :: [a_awM]) ->
              case ds_azk of _ {
                [] -> GHC.Bool.True;
                : y_azp ys_azq ->
                  case GHC.Classes.== @ a_awM $dEq_awN y_azp ds1_axJ of _ {
                    GHC.Bool.False -> GHC.Bool.False; GHC.Bool.True -> go_sDv ys_azq
                  }
              }; } in
        go_sDv xs1_axK
    }

This looks pretty good, actually. It will produce an error with an empty list, but that's easily fixed. This is the case xs_abH of _ { [] ->. After this GHC performed a worker/wrapper transformation, the recursive worker function is the letrec { go_sDv binding. The worker examines its argument. If [], it's reached the end of the list and returns True. Otherwise it compares the head of the remaining to the first element and either returns False or checks the rest of the list.

Three other features.

1. The map was entirely fused away and doesn't allocate a temporary list.
2. Near the top of the definition notice the Cheap=True statement. This means GHC considers the function "cheap", and thus a candidate for inlining. At a call site, if a concrete argument type can be determined, GHC will probably inline allTheSame and produce a very tight inner loop, completely bypassing the Eq dictionary lookup.
3. The worker function is tail-recursive.

Verdict: Very strong contender.

Solution 1

allTheSame' :: (Eq a) => [a] -> Bool
allTheSame' xs = (length xs) == (length $ takeWhile (== head xs) xs)

Even without looking at core I know this won't be as good. The list is traversed more than once, first by length xs then by length $ takeWhile. Not only do you have the extra overhead of multiple traversals, it means that the list must be retained in memory after the first traversal and can't be GC'd. For a big list, this is a serious problem.

Test.allTheSame'
  :: forall a_abF. GHC.Classes.Eq a_abF => [a_abF] -> GHC.Bool.Bool
[GblId,
 Arity=2,
 Str=DmdType LS,
 Unf=Unf{Src=<vanilla>, TopLvl=True, Arity=2, Value=True,
         ConLike=True, Cheap=True, Expandable=True,
         Guidance=IF_ARGS [3 3] 20 0}]
Test.allTheSame' =
  \ (@ a_awF)
    ($dEq_awG :: GHC.Classes.Eq a_awF)
    (xs_abI :: [a_awF]) ->
    case GHC.List.$wlen @ a_awF xs_abI 0 of ww_aC6 { __DEFAULT ->
    case GHC.List.$wlen
           @ a_awF
           (GHC.List.takeWhile
              @ a_awF
              (let {
                 ds_sDq :: a_awF
                 [LclId, Str=DmdType]
                 ds_sDq =
                   case xs_abI of _ {
                     [] -> GHC.List.badHead @ a_awF; : x_axk ds1_axl -> x_axk
                   } } in
               \ (ds1_dxa :: a_awF) ->
                 GHC.Classes.== @ a_awF $dEq_awG ds1_dxa ds_sDq)
              xs_abI)
           0
    of ww1_XCn { __DEFAULT ->
    GHC.Prim.==# ww_aC6 ww1_XCn
    }
    }

Looking at the core doesn't tell much beyond that. However, note these lines:

case GHC.List.$wlen @ a_awF xs_abI 0 of ww_aC6 { __DEFAULT ->
        case GHC.List.$wlen

This is where the list traversals happen. The first gets the length of the outer list and binds it to ww_aC6. The second gets the length of the inner list, but the binding doesn't happen until near the bottom, at

of ww1_XCn { __DEFAULT ->
GHC.Prim.==# ww_aC6 ww1_XCn

The lengths (both Ints) can be unboxed and compared by a primop, but that's a small consolation after the overhead that's been introduced.

Verdict: Not good.

Solution 2

allTheSame'' :: (Eq a) => [a] -> Bool
allTheSame'' xs
  | n == 0 = False
  | n == 1 = True
  | n == 2 = xs !! 0 == xs !! 1
  | otherwise = (xs !! 0 == xs !! 1) && (allTheSame'' $ snd $ splitAt 2 xs)
    where  n = length xs

This has the same problem as solution 1. The list is traversed multiple times, and it can't be GC'd. It's worse here though, because now the length is calculated for each sub-list. I'd expect this to have the worst performance of all on lists of any significant size. Also, why are you special-casing lists of 1 and 2 elements when you're expecting the list to be big?

Verdict: Don't even think about it.

Solution 3

allTheSame''' :: (Eq a) => [a] -> Bool
allTheSame''' xs
  | n == 0 = False
  | n == 1 = True
  | n == 2 = xs !! 0 == xs !! 1
  | n == 3 = xs !! 0 == xs !! 1 && xs !! 1 == xs !! 2
  | otherwise = allTheSame''' (fst split) && allTheSame''' (snd split)
    where n = length xs
          split = splitAt (n `div` 2) xs

This has the same problem as Solution 2. Namely, the list is traversed multiple times by length. I'm not certain a divide-and-conquer approach is a good choice for this problem, it could end up taking longer than a simple scan. It would depend on the data though, and be worth testing.

Verdict: Maybe, if you used a different data structure.

Solution 4

allTheSame'''' :: (Eq a) => [a] -> Bool
allTheSame'''' xs = all (== head xs) (tail xs)

This was basically my first thought. Let's check the core again.

Test.allTheSame''''
  :: forall a_abC. GHC.Classes.Eq a_abC => [a_abC] -> GHC.Bool.Bool
[GblId,
 Arity=2,
 Str=DmdType LS,
 Unf=Unf{Src=<vanilla>, TopLvl=True, Arity=2, Value=True,
         ConLike=True, Cheap=True, Expandable=True,
         Guidance=IF_ARGS [3 3] 10 0}]
Test.allTheSame'''' =
  \ (@ a_am5)
    ($dEq_am6 :: GHC.Classes.Eq a_am5)
    (xs_alK :: [a_am5]) ->
    case xs_alK of _ {
      [] ->
        GHC.List.tail1
        `cast` (CoUnsafe (forall a1_axH. [a1_axH]) GHC.Bool.Bool
                :: (forall a1_axH. [a1_axH]) ~ GHC.Bool.Bool);
      : ds1_axJ xs1_axK ->
        GHC.List.all
          @ a_am5
          (\ (ds_dwU :: a_am5) ->
             GHC.Classes.== @ a_am5 $dEq_am6 ds_dwU ds1_axJ)
          xs1_axK
    }

Ok, not too bad. Like solution 1, this will error on empty lists. The list traversal is hidden in GHC.List.all, but it will probably be expanded to good code at a call site.

Verdict: Another strong contender.

So between all of these, with lists I'd expect that Solutions 0 and 4 are the only ones worth using, and they are pretty much the same. I might consider Option 3 in some cases.

Edit: in both cases, the errors on empty lists can be simply fixed as in @augustss's answer.

The next step would be to do some time profiling with criterion.

Answer 2

A solution using consecutive pairs:

allTheSame xs = and $ zipWith (==) xs (tail xs)

Answer 3

I think I might just be implementing find and redoing this. I think it's instructive, though, to see the innards of it. (Note how the solution depends on equality being transitive, though note also how the problem requires equality to be transitive to be coherent.)

sameElement x:y:xs = if x /= y then Nothing else sameElement y:xs
sameElement [x] = Just x
allEqual [] = True
allEqual xs = isJust $ sameElement xs

I like how sameElement peeks at the first O(1) elements of the list, then either returns a result or recurses on some suffix of the list, in particular the tail. I don't have anything smart to say about that structure, I just like it :-)

I think I do the same comparisons as this. If instead I had recursed with sameElement x:xs, I would compare the head of the input list to each element like in solution 0.

Tangent: one could, if one wanted, report the two mismatching elements by replacing Nothing with Left (x, y) and Just x with Right x and isJust with either (const False) (const True).

Answer 4

This implementation is superior.

allSame [ ] = True
allSame (h:t) = aux h t

aux x1 [ ]                 = True
aux x1 (x2:xs) | x1==x2    = aux x2 xs 
               | otherwise = False

Given the transitivity of the (==) operator, assuming the instance of Eq is well implemented if you wish to assure the equality of a chain of expressions, eg a = b = c = d, you will only need to assure that a=b, b=c, c=d, and that d=a, Instead of the provided techniques above, eg a=b, a=c, a=d, b=c , b=d, c=d.

The solution I proposed grows linearly with the number of elements you wish to test were's the latter is quadratic even if you introduce constant factors in hopes of improving its efficiency.

It's also superior to the solution using group since you don't have to use length in the end.

You can also write it nicely in pointwise fashion but I won't bore you with such trivial details.

Answer 5

Here's another fun way:

{-# INLINABLE allSame #-}
allSame :: Eq a => [a] -> Bool
allSame xs = foldr go (`seq` True) xs Nothing where
  go x r Nothing = r (Just x)
  go x r (Just prev) = x == prev && r (Just x)

By keeping track of the previous element, rather than the first one, this implementation can easily be changed to implement increasing or decreasing. To check all of them against the first instead, you could rename prev to first, and replace Just x with Just first.

---

How will this be optimized? I haven't checked in detail, but I'm going to tell a good story based on some things I know about GHC's optimizations.

Suppose first that list fusion does not occur. Then foldr will be inlined, giving something like

allSame xs = allSame' xs Nothing where
  allSame' [] = (`seq` True)
  allSame' (x : xs) = go x (allSame' xs)

Eta expansion then yields

allSame' [] acc = acc `seq` True
allSame' (x : xs) acc = go x (allSame' xs) acc

Inlining go,

allSame' [] acc = acc `seq` True
allSame' (x : xs) Nothing = allSame' xs (Just x)
allSame' (x : xs) (Just prev) =
  x == prev && allSame' xs (Just x)

Now GHC can recognize that the Maybe value is always Just on the recursive call, and use a worker-wrapper transformation to take advantage of this:

allSame' [] acc = acc `seq` True
allSame' (x : xs) Nothing = allSame'' xs x
allSame' (x : xs) (Just prev) = x == prev && allSame'' xs x

allSame'' [] prev = True
allSame'' (x : xs) prev = x == prev && allSame'' xs x

Remember now that

allSame xs = allSame' xs Nothing

and allSame' is no longer recursive, so it can be beta-reduced:

allSame [] = True
allSame (x : xs) = allSame'' xs x

allSame'' [] _ = True
allSame'' (x : xs) prev = x == prev && allSame'' xs x

So the higher-order code has turned into efficient recursive code with no extra allocation.

Compiling the module defining allSame using -O2 -ddump-simpl -dsuppress-all -dno-suppress-type-signatures yields the following (I've cleaned it up a bit):

allSame :: forall a. Eq a => [a] -> Bool
allSame =
  \ (@ a) ($dEq_a :: Eq a) (xs0 :: [a]) ->
    let {
      equal :: a -> a -> Bool
      equal = == $dEq_a } in
    letrec {
      go :: [a] -> a -> Bool
      go =
        \ (xs :: [a]) (prev :: a) ->
          case xs of _ {
            [] -> True;
            : y ys ->
              case equal y prev of _ {
                False -> False;
                True -> go ys y
              }
          }; } in
    case xs0 of _ {
      [] -> True;
      : x xs -> go xs x
    }

As you can see, this is essentially the same as the result I described. The equal = == $dEq_a bit is where the equality method is extracted from the Eq dictionary and saved in a variable so it only needs to be extracted once.

---

What if list fusion does occur? Here's a reminder of the definition:

allSame xs = foldr go (`seq` True) xs Nothing where
  go x r Nothing = r (Just x)
  go x r (Just prev) = x == prev && r (Just x)

If we call allSame (build g), the foldr will fuse with the build according to the rule foldr c n (build g) = g c n, yielding

allSame (build g) = g go (`seq` True) Nothing

That doesn't get us anywhere interesting unless g is known. So let's choose something simple:

replicate k0 a = build $ \c n ->
  let
    rep 0 = n
    rep k = a `c` rep (k - 1)
  in rep k0

So if h = allSame (replicate k0 a), h becomes

let
  rep 0 = (`seq` True)
  rep k = go a (rep (k - 1))
in rep k0 Nothing

Eta expanding,

let
  rep 0 acc = acc `seq` True
  rep k acc = go a (rep (k - 1)) acc
in rep k0 Nothing

Inlining go,

let
  rep 0 acc = acc `seq` True
  rep k Nothing = rep (k - 1) (Just a)
  rep k (Just prev) = a == prev && rep (k - 1) (Just a)
in rep k0 Nothing

Again, GHC can see the recursive call is always Just, so

let
  rep 0 acc = acc `seq` True
  rep k Nothing = rep' (k - 1) a
  rep k (Just prev) = a == prev && rep' (k - 1) a
  rep' 0 _ = True
  rep' k prev = a == prev && rep' (k - 1) a
in rep k0 Nothing

Since rep is no longer recursive, GHC can reduce it:

let
  rep' 0 _ = True
  rep' k prev = a == prev && rep' (k - 1) a
in
  case k0 of
    0 -> True
    _ -> rep' (k - 1) a

As you can see, this can run with no allocation whatsoever! Obviously, it's a silly example, but something similar will happen in many more interesting cases. For example, if you write an AllSameTest module importing the allSame function and defining

foo :: Int -> Bool
foo n = allSame [0..n]

and compile it as described above, you'll get the following (not cleaned up).

$wfoo :: Int# -> Bool
$wfoo =
  \ (ww_s1bY :: Int#) ->
    case tagToEnum# (># 0 ww_s1bY) of _ {
      False ->
        letrec {
          $sgo_s1db :: Int# -> Int# -> Bool
          $sgo_s1db =
            \ (sc_s1d9 :: Int#) (sc1_s1da :: Int#) ->
              case tagToEnum# (==# sc_s1d9 sc1_s1da) of _ {
                False -> False;
                True ->
                  case tagToEnum# (==# sc_s1d9 ww_s1bY) of _ {
                    False -> $sgo_s1db (+# sc_s1d9 1) sc_s1d9;
                    True -> True
                  }
              }; } in
        case ww_s1bY of _ {
          __DEFAULT -> $sgo_s1db 1 0;
          0 -> True
        };
      True -> True
    }

foo :: Int -> Bool
foo =
  \ (w_s1bV :: Int) ->
    case w_s1bV of _ { I# ww1_s1bY -> $wfoo ww1_s1bY }

That may look disgusting, but you'll note that there are no : constructors anywhere, and that the Ints are all unboxed, so the function can run with zero allocation.

Answer 6

Q1 -- Yeah, I think your simple solution is fine, there is no memory leak. Q4 -- Solution 3 is not log(n), via the very simple argument that you need to look at all list elements to determine whether they are the same, and looking at 1 element takes 1 time step. Q5 -- yes. Q6, see below.

The way to go about this is to type it in and run it

main = do
    print $ allTheSame (replicate 100000000 1)

then run ghc -O3 -optc-O3 --make Main.hs && time ./Main. I like the last solution best (you can also use pattern matching to clean it up a little),

allTheSame (x:xs) = all (==x) xs

Open up ghci and run ":step fcn" on these things. It will teach you a lot about what lazy evaluation is expanding. In general, when you match a constructor, e.g. "x:xs", that's constant time. When you call "length", Haskell needs to compute all of the elements in the list (though their values are still "to-be-computed"), so solution 1 and 2 are bad.

edit 1

Sorry if my previous answer was a bit shallow. It seems like expanding things manually does help a little (though compared to the other options, it's a trivial improvement),

{-# LANGUAGE BangPatterns #-}
allTheSame [] = True
allTheSame ((!x):xs) = go x xs where
    go !x [] = True
    go !x (!y:ys) = (x == y) && (go x ys)

It seems that ghc is specializing the function already, but you can look at the specialize pragma too, in case it doesn't work for your code [ link ].