Recommended Free Tools
Use set(values) to remove duplicates from a Python list when its elements are hashable and you do not need to preserve their order. If you need a deduplicated list in first-seen order, use list(dict.fromkeys(values)). For NumPy arrays, use numpy.unique(); its default output is sorted.
Convert a Python list to a set
A set keeps distinct hashable values. Pass the list to set():
values = [3, 1, 3, 2, 1]
unique_set = set(values) # {1, 2, 3}If you need a list as the result, convert the set back:
unique_list = list(set(values))Neither set iteration nor this round trip guarantees the original list order. The Python tutorial describes a set as “an unordered collection with no duplicate elements” (Python tutorial).
Quick wins for a faster PC:
Clear out junk files and repair common Windows errorsFree Scan →Fix the driver behind crashes, sound loss and screen glitchesFind Drivers →#1 Best Overall
What “fast” means here
The Python FAQ says the set-based approach is often faster when every list element is hashable, but it gives no timing figure or universal performance guarantee (Python FAQ). Actual speed depends on the data and workload; benchmark the relevant alternatives if performance is important.
Keep the first-seen order
When output order matters, use an insertion-ordered dictionary to discard later repeats while keeping each value’s first occurrence:
values = [3, 1, 3, 2, 1]
unique_in_order = list(dict.fromkeys(values)) # [3, 1, 2]This requires hashable values, just like a set. If you want the membership test to be explicit, or you are processing an iterable one value at a time, build the output list alongside a seen set:
seen = set()
unique_in_order = []
for value in values:
if value not in seen:
seen.add(value)
unique_in_order.append(value)The list records encounter order; the set makes it possible to check whether each value has appeared before.
Convert a NumPy array to unique values
For a NumPy array, call numpy.unique() (usually imported as np.unique()):
import numpy as np
array = np.array([3, 1, 3, 2, 1])
unique_values = np.unique(array) # array([1, 2, 3])By default, the result contains sorted unique values, not values in first-occurrence order. The function can also return first-occurrence indices, inverse indices, counts, or unique subarrays along an axis. See the NumPy reference for numpy.unique for the full options and examples.
Restore first-occurrence order
Request the indices of each unique value’s first occurrence, then sort those indices to put the corresponding values back in input order:
unique_values, first_indices = np.unique(array, return_index=True)
unique_in_input_order = array[np.sort(first_indices)]np.unique() flattens its input by default. To find unique rows or other subarrays instead, pass an appropriate axis, such as axis=0 for rows. The axis option does not support object arrays or structured arrays containing objects. NumPy 2.3 added sorted=False, but its documentation cautions that elements may still be sorted in practice and that behavior can change; do not rely on that flag to preserve encounter order.
Choose the right method for your data
| Method | Result | Order behavior | Important condition |
|---|---|---|---|
set(values) |
Python set | Does not preserve input order | All elements must be hashable |
list(set(values)) |
Python list | Does not preserve input order | All elements must be hashable |
list(dict.fromkeys(values)) or a seen set and output list |
Python list | Preserves first-seen order | Values must be hashable |
np.unique(array) |
NumPy array | Sorted by default | Use axis for row-like subarrays |
Handle unhashable elements
Lists and other mutable containers cannot be inserted directly into a set. For example, set([[1, 2], [1, 2]]) raises a TypeError because the inner lists are unhashable. A set’s elements must be hashable (Python built-in types documentation).
If each inner list can faithfully be treated as a tuple, convert it to a tuple before deduplicating:
rows = [[1, 2], [1, 2], [3, 4]]
unique_rows = [list(row) for row in dict.fromkeys(map(tuple, rows))]Choose a transformation only when it preserves the equality you intend. If the elements are arbitrary unhashable objects, use a comparison-based approach that checks each item against previously kept items; that avoids hashing but may require comparing an item with many earlier items.
Quick Recap
Common mistakes
- Expecting set conversion to retain order: use
dict.fromkeys()or theseen-set loop when first-seen order matters. - Passing nested lists directly to
set(): transform them to an appropriate hashable key or use comparison-based deduplication. - Writing
{}for an empty set: that creates an empty dictionary. Useset()instead (Python tutorial). - Assuming
np.unique()keeps encounter order: its default output is sorted; use its first-occurrence indices and sort those indices when you need input order.
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
