If by “array” you mean a regular Python list, use list(dict.fromkeys(items)) to remove duplicates while keeping each value’s first position. If order does not matter, list(set(items)) is shorter. Both approaches require hashable elements; for nested lists or other unhashable values, use an equality-based loop.
Python’s official FAQ recommends a list for a general-purpose sequence. The separate array module is intended for compact arrays of fixed-type values, not as a synonym for every list.
Choose based on order and element type
Before choosing a method, decide whether the result must retain the input’s order and whether its elements are hashable. Numbers, strings, and tuples containing only hashable values can generally be used as set members or dictionary keys. Lists and dictionaries cannot.
| Approach | Keeps first-seen order? | Requires hashable elements? | Best suited to |
|---|---|---|---|
list(set(items)) |
No | Yes | Unordered results |
list(dict.fromkeys(items)) |
Yes | Yes | Concise ordered deduplication |
| Loop with a set | Yes | Yes | Clear, explicit ordered logic |
| Comprehension with a seen set | Yes | Yes | Compact code when the idiom is familiar |
| Equality-based loop | Yes | No | Unhashable values such as nested lists |
1. Convert to a set when order does not matter
items = ["pear", "apple", "pear", "plum"]
unique = list(set(items))A set contains no duplicate elements, so converting the list to a set and back removes repeats. But sets are unordered: do not rely on the output matching the input’s order. This method also fails if an element is unhashable, such as a list. Python’s tutorial describes a set as an unordered collection with no duplicate elements.
#1 Best Overall
2. Use dictionary keys to preserve first-seen order
items = ["pear", "apple", "pear", "plum"]
unique = list(dict.fromkeys(items))
# ['pear', 'apple', 'plum']dict.fromkeys(items) creates dictionary keys from the values, and repeated keys appear only once. Converting those keys back to a list produces the values in first-occurrence order. Dictionary insertion order is guaranteed in Python 3.7 and later; the keys must be hashable.
3. Use a loop and a set for readable ordered logic
items = ["pear", "apple", "pear", "plum"]
seen = set()
unique = []
for item in items:
if item not in seen:
seen.add(item)
unique.append(item)The separate seen set checks whether a value has already appeared, while unique records accepted values in order. This makes the ordering rule explicit and is often easier to adapt than a one-liner. Like the dictionary method, it requires hashable elements.
4. Use a seen-set comprehension when the idiom is clear
items = ["pear", "apple", "pear", "plum"]
seen = set()
unique = [item for item in items if item not in seen and not seen.add(item)]This retains the first occurrence, but it works by calling seen.add(item) as a side effect inside the filter. Since set.add returns None, not seen.add(item) is true when the item is first encountered. The expression is compact, but less obvious than the loop; use it only when readers will recognize the pattern. Elements still must be hashable.
5. Use equality checks for unhashable values
items = [[1, 2], [3, 4], [1, 2]]
unique = []
for item in items:
if item not in unique:
unique.append(item)
# [[1, 2], [3, 4]]Membership in a list compares values for equality, so this works for lists and other equality-comparable values that cannot be set members or dictionary keys. It keeps the first equal value and preserves order. As the number of retained values grows, each new item may need to be compared with many earlier ones, making this approach potentially quadratic in the number of unique values.
Do these 3 things before closing this tab:
1Repair Windows errors before they cause bigger problems2Fix the driver behind crashes, sound loss and screen glitches3Clear out junk files and repair common Windows errorsIf you can define a hashable key that represents the equivalence you intend, you can instead deduplicate by that key. For example, when records should be considered duplicates by an ID field, track the IDs in a set while appending the original records to a result list. That preserves the records’ first-seen order without requiring the records themselves to be hashable.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.What about sorting first?
Sorting and scanning for adjacent equal values is another option when reordering is acceptable and all elements can be compared with one another. Sorting changes the original order, and it may fail for values that cannot be ordered together, such as a mixture of integers and strings. The Python FAQ includes sorting and scanning among possible approaches.
Quick Recap
Which method should you use?
- Need first-occurrence order and have hashable values: use
list(dict.fromkeys(items))for concise code or the explicit set loop when clarity and customization matter. - Order is irrelevant and values are hashable: use
list(set(items)). - Values are unhashable: use equality checks, or deduplicate using a suitable hashable key.
- Performance matters: consider the data type and workload, then benchmark under the Python version and input conditions that match your use case. Hash-based membership and repeated equality comparisons have different costs, but the documentation does not establish a universal speed ranking for these five implementations.
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
