Unique values from 1D-array, without iteration

Question

At risk of being of topic, I decided to share some code, Q&A-style. If the general opinion is such that this would be off-topic I\'ll be happy to delete if need be.

Answer 1

Approach via FilterXML()

Just to enrich the variety of fine solutions above, I demonstrate an approach via the new worksheet function FilterXML().

Sub testUniqueItems()
'   Purp: list unique items
'   Site: https://stackoverflow.com/questions/59683363/unique-values-from-1d-array-without-iteration
    Dim arr As Variant: arr = Array("A", "A", "C", "D", "A", "E", "G")
'[1]get uniques
    Dim uniques
    uniques = UniqueXML(arr)
'[2]display in Immediate Window: A,A,C,D,A,E,G => A,C,D,E,G
    Debug.Print Join(arr, ",") & " => " & _
                Join(uniques, ",")
End Sub

Function UniqueXML(arr, Optional Delim As String = ",")
  ' Purp: return unique list of array items
  ' Note: optional argument Delim defaulting to colon (",")
  ' Help: https://docs.microsoft.com/de-de/office/vba/api/excel.worksheetfunction.filterxml
  ' [1] get array data to xml node structure (including root element)
    Dim wellformed As String
    wellformed = "<root><i>" & Join(arr, "</i><i>") & "</i></root>"
  ' ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  ' [2] define XPath string searching unique item values
  ' Note: c.f. udf: https://stackoverflow.com/questions/58677041/vba-excel-how-to-display-non-equal-values-in-an-excel-array/58685756#58685756
  ' ------------------------------------------------
  ' //i                    ... all <i> node values after the DocumentElement
  ' [not( .=preceding::i)] ... only if not preceded by siblings of the same node value
  ' ------------------------------------------------
    Dim myXPath As String
    myXPath = "//i[not( .=preceding::i)]"
  ' [3a] get (delimiter separated) unique list
    UniqueXML = Evaluate("=TEXTJOIN(""" & Delim & """,,FILTERXML(""" & wellformed & """, """ & myXPath & """))")
  ' [3b] return array
    UniqueXML = Split(UniqueXML, Delim)
End Function

Related links

MS Help

Display non equal values in an Excel array

Caveat

Note that the Worksheet function FilterXML() can be used from vers. 2016+, but TextJoin only in vers. 2019+ (thx @FaneDuru for comments:-)

Furthermore you would have to be aware of the limits of evaluate. 255 chars only (thx @JvDv).

To overcome both obstacles I reworked above function to work in versions 2016+, too.

Modified function /Edit as of 2020-08-20

Function UniqueXML(arr, Optional Delim As String = ",")
  ' Purp: return unique list of array items
  ' Note: optional argument Delim defaulting to colon (",")
  ' Help: https://docs.microsoft.com/de-de/office/vba/api/excel.worksheetfunction.filterxml
  ' [1] get array data to xml node structure (including root element)
    Dim wellformed As String
    wellformed = "<root><i>" & Join(arr, "</i><i>") & "</i></root>"
  ' ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  ' [2] define XPath string searching unique item values
  ' Note: c.f. udf: https://stackoverflow.com/questions/58677041/vba-excel-how-to-display-non-equal-values-in-an-excel-array/58685756#58685756
  ' ------------------------------------------------
  ' //i                    ... all <i> node values after the DocumentElement
  ' [not( .=preceding::i)] ... only if not preceded by siblings of the same node value
  ' ------------------------------------------------
    Dim myXPath As String
    myXPath = "//i[not( .=preceding::i)]"
   
  ' [3] get "flat" 1-dim array (~> one-based!)
    Dim tmp As Variant
    tmp = Application.Transpose(WorksheetFunction.FilterXML(wellformed, myXPath))
'  ' [3a] optional redim as zero-based array
'    ReDim Preserve tmp(LBound(tmp) - 1 To UBound(tmp) - 1)
        
  ' [4] return function result
    UniqueXML = tmp
End Function

Answer 2

With the new Dynamic Array functions it can be simplified to:

Sub test()

Dim arr As Variant: arr = Array("A", "A", "C", "D", "A", "E", "G")
With Application
    Dim uniques as variant
    uniques = .Transpose(.Unique(.Transpose(arr)))
End With

End Sub

The new Uniques Formula needs a vertical array, and it can be 2d. It acts like Range.RemoveDuplicate without the ability to choose columns.

Answer 3

Applying the Unique() function without double transposition (Office 365)

As addition to @ScottCraner 's Office 365 solution an alternative without need to transpose twice:

Sub testUniques()
Dim arr:     arr = Array("A", "A", "C", "D", "A", "E", "G")  ' example data
Dim uniques: uniques = Application.Unique(arr, True)         ' return function result
'optional display in VB Editor's immediate window
    Debug.Print Join(arr, ",") & " ~> " & Join(uniques, ",") ' A,A,C,D,A,E,G ~> A,C,D,E,G
End Sub

Explanation to additional argument by_col

Due to the Unique function reference its syntax is UNIQUE(array,[by_col],[exactly_once]), where

"the by_col argument is a logical value indicating how to compare. TRUE will compare columns against each other and return the unique columns."

Setting the by_col argument to True allows to compare the array items against each other as they are considered as "columns" in a "flat" 1-dimensional array.

Answer 4

Really all code needed are just a few lines:

Sub test()

Dim arr As Variant: arr = Array("A", "A", "C", "D", "A", "E", "G")
With Application
    uniques = .Index(arr, 1, Filter(.IfError(.Match(.Transpose(.Evaluate("ROW(1:" & UBound(.Match(arr, arr, 0)) & ")")), .Match(arr, arr, 0), 0), "|"), "|", False))
End With

End Sub

The above will return a 1D-array, returning all unique elements in our original array:

---

_{Explaination:}

The line that retrieves all these values looks intense, so let's break it into pieces:

---

Application.Match has the ability to work with arrays within it's parameters. So basically we are looking at: .Match({"A","A","C","D","A","E","G"},{"A","A","C","D","A","E","G"},0). The returned array would then be: {1,1,3,4,1,6,7}, and what this really is are the first positions where each value is found at. This result will be the basis on where we build on further.

---

We can see a third .Match in our code which we need to basically tell the following: .Match({1,2,3,4,5,6,7},{1,1,3,4,1,6,7},0). The first parameter is what is retrieved by the above higlighted code.

Where .Evaluate("ROW(1:" & UBound(.Match(arr, arr, 0)) & ")") will return an array of values from 1-7, the Application.Transpose will return it such that it's a 1D-array.

---

The last step will return an array holding errors, however the code won't break since we are using Application instead of WorksheetFunction. The resulting array will look like {1,Error 2042,3,4,Error 2042,6,7}. Now the whole point is to get rid of the Error values.

The way to do so is through Application.IfError, which will evaluate the array and change all error values into a give string value. In our case I used the pipe symbol. It's up to the user to decide on a symbol unique enough it won't appear in any of the elements in the original array. So after evaluation. Our current array will look like: {1,|,3,4,|,6,7}.

---

Now we retrieved an array with pipe symbols we would want them out! A quick way to do so is with Filter function. Filter returns an array with or without the elements that fit our criteria (depending on the TRUE or FALSE in it's third paramter).

So basically we want to return an array like so: Filter(<array>, "|", False). The resulting 1D-array now looks like: {1,3,4,6,7}.

---

We kind of have it at this point. We just need to slice out the correct values from our original array. To do so we can use Application.Index. We just want to tell .Index which rows we are interested in. And to do so we can load our previously found 1D-array. So the code will look like: .Index(arr1, <array>, 1) which will result in a 1D-array: {"A","C","D","E","G"}

---

_Conclusion:

There you have it. One line (with more than just a single operation) to retrieve a 1D-array of unique values from another 1D-array without iteration. This code is ready to be used on any 1D-array declared with arr.

Usefull? I'm not 100% sure, but I finally reached what I was trying in my project. The resulting array can be used immediately in whichever task you need to use unique values in.

_{Comparison: Dictionary vs Application.Methods:}

Doing a comparison on random items in the Range(A1:A50000), the performance really takes a hit. Hereby a time-comparison between the iterative Dictionary against the non-iterative Application.Methods approach in 1000 items steps. Below the result of a 1000 items and each 10000 items mark (in seconds):

| Items     | Dictionary    | Methods       |
|-------    |------------   |-------------  |
| 1000      | 0,02          | 0,03          |
| 10000     | 0             | 0,88          |
| 20000     | 0,02          | 3,31          |
| 30000     | 0,02          | 7,3           |
| 40000     | 0,02          | 12,84         |
| 50000     | 0,03          | 20,2          |

The Dictionary approach used:

Sub Test()

Dim arr As Variant: arr = Application.Transpose(Range("A1:A50000"))
Dim dict As Object: Set dict = CreateObject("Scripting.Dictionary")

Dim x As Long

For x = LBound(arr) To UBound(arr)
    dict(arr(x)) = 1
Next x

Dim uniques As Variant: uniques = dict.Keys

End Sub

Conclusion: Up to a 1000 items this method would be about equal in processing time compared to a more common Dictionary practice. On anything larger, iteration (through memory) will always beat the methods approach!

I'm sure processing time would be more limited with the new Dynamic Array functions as shown by @ScottCraner.