Is it faster to union sets or check the whole list for a duplicate?

Question

Sorry for the poorly worded title but I asked a question earlier about getting a unique list of items from two lists. People told me to make the list -> sets and then union.

Answer 1

All depends of what you have as input and want as output.

If you have a list li at the beginning and want to get a modified list in the end, then the faster method is if not elt in li: li.append(elt) the problem is converting initial list to set, then converting back to list which is way too slow.

But if you can work with a set s at all time (you don't care about the order of the list, and methods receiving it just need some iterable), just doing s.add(elt) is faster.

If in the beginning you have to lists and want a list in the end, even with final conversion from list set to list, it is faster to manage unicity of items using sets, but you can easily check looking at the exemple provided by @virhilo in it's answer, than concatenating the two lists using extend, then converting the result to set is faster than converting the two lists to sets and performing an union.

I don't know exactly what are the constraints of your programs, but if unicity is as important as it seems, and if keeping insertion order is not necessary, you would be well advised to use sets at all time, never changing them to lists. Most algorithms will work for both anyway, thanks to Duck Typing as they both are different kinds of iterables.