remove duplicates from list python assignment expert

• Learn Python
• Python Lists
• Python Dictionaries
• Python Strings
• Python Functions
• Learn Pandas & NumPy
• Pandas Tutorials
• Numpy Tutorials
• Learn Data Visualization
• Python Seaborn
• Python Matplotlib

Python: Remove Duplicates From a List (7 Ways)

• October 17, 2021 December 19, 2022

In this tutorial, you’ll learn how to use Python to remove duplicates from a list . Knowing how to working with Python lists is an important skill for any Pythonista. Being able to remove duplicates can be very helpful when working with data where knowing frequencies of items is not important.

You’ll learn how to remove duplicates from a Python list while maintaining order or when order doesn’t matter. You’ll learn how to do this using naive methods, list comprehensions, sets, dictionaries, the collections library, numpy , and pandas .

The Quick Answer :

Table of Contents

Remove Duplicates from a Python List Using For Loops

The most naive implementation of removing duplicates from a Python list is to use a for loop method. Using this method involves looping over each item in a list and seeing if it already exists in another list.

Let’s see what this looks like in Python:

Let’s explore what we did here:

• We instantiated a new, empty list to hold de-duplicated items
• We looped over each item in our duplicated list and checked if it existed in the deduplicated list
• If it didn’t, we appended the item to our list. If it did exist, then we did nothing

In the next section, you’ll learn how to deduplicate a list in Python using a list comprehension.

Want to learn more about Python for-loops? Check out my in-depth tutorial that takes your from beginner to advanced for-loops user! Want to watch a video instead? Check out my YouTube tutorial here .

Remove Duplicates from a Python List Using a List Comprehension

Similar to the method using for loops, you can also use Python list comprehensions to deduplicate a list. The process involved here is a little different than a normal list comprehension, as we’ll be using the comprehension more for looping over the list.

Let’s see what this looks like:

This approach is a little bit awkward as the list comprehension sits by itself. This can make the code less intuitive to follow, as list comprehensions are often used to create new lists. However, since we’re only looping over the list and appending to another list, no new list is instantiated using the comprehension.

In the next section, you’ll learn how to use Python dictionaries to deduplicate a Python list.

Want to learn more about Python list comprehensions? Check out this in-depth tutorial that covers off everything you need to know, with hands-on examples. More of a visual learner, check out my YouTube tutorial here .

Use Python Dictionaries to Remove Duplicates from a List

Since Python 3.7, Python dictionaries maintain the original order of items passed into them. While this method will work for versions earlier than Python 3.7, the resulting deduplicated list will not maintain the order of the original list.

The reason that converting a list to a dictionary works is that dictionary keys must be unique. One important thing to note is that Python dictionaries require their keys to be hashable, meaning that they must be immutable. if your list contains mutable elements, then this approach will not work.

Let’s take a look at how we can use Python dictionaries to deduplicate a list:

Let’s take a look at what we’ve done here:

• We created a dictionary using the .fromkeys() method, which uses the items passed into it to create a dictionary with the keys from the object
• We then turned the dictionary into a list using the list() function, which creates a list from the keys in the dictionary.

In the next section, you’ll learn how to use Python sets to deduplicate a list.

Need to check if a key exists in a Python dictionary? Check out this tutorial , which teaches you five different ways of seeing if a key exists in a Python dictionary, including how to return a default value.

Use Python Sets to Remove Duplicates from a List

Sets are unique Python data structures that are generated using the curly braces {} . They contain only unique items and are unordered and unindexed.

Because Python sets are unique, when we create a set based off of another object, such as a list, then duplicate items are removed.

What we can do is first convert our list to a set, then back to a list.

What we did here was:

• We passed our original list into the set() function, which created a set and removed all duplicate items,
• We then passed that set into the list() function, to produce another list

In the next section, you’ll learn how to use the collections library to remove duplicates from a Python list.

Want to learn how to use the Python zip() function to iterate over two lists? This tutorial teaches you exactly what the zip() function does and shows you some creative ways to use the function.

Remove Duplicates from a Python List Using Collections

If you’re working with an older version of Python that doesn’t support ordered dictionaries (prior to Python 3.6), you can also use the collections library to accomplish a very similar approach.

We use the collections library to create an ordered dictionary and then convert it back to a list.

Let’s see how this works:

In the next section, you’ll learn how to use numpy to remove duplicates from a list.

Check out some other Python tutorials on datagy, including our complete guide to styling Pandas and our comprehensive overview of Pivot Tables in Pandas !

Use Numpy to Remove Duplicates from a Python List

The popular Python library numpy has a list-like object called arrays . What’s great about these arrays is that they have a number of helpful methods built into them.

One of these functions is the unique() function, which finds, well, unique items in an array.

Let’s see how we can use numpy to remove duplicates from a Python list.

Here, we first create an array out of our list, pass it into the unique() function. Finally, we use the .tolist() method to create a list out of the array.

In the final section, you’ll learn how to use Pandas to deduplicate a Python list.

Want to learn more about calculating the square root in Python? Check out my tutorial here , which will teach you different ways of calculating the square root, both without Python functions and with the help of functions.

Use Pandas to Remove Duplicates from a Python List

In this final section, you’ll learn how to use the popular pandas library to de-duplicate a Python list.

Pandas uses a numpy array and creates a Pandas series object. These objects are also similar to Python lists, but are extended by a number of functions and methods that can be applied to them.

Let’s see how we can do this in Python and Pandas:

In this, we first created a pd.Series() object, then apply the .unique() method, and finally use the .tolist() method to return a list.

Need to automate renaming files? Check out this in-depth guide on using pathlib to rename files. More of a visual learner, the entire tutorial is also available as a video in the post!

In this tutorial, you learned a number of different ways to remove duplicates from a Python list. You learned a number of naive methods of doing this, including using for loops and list comprehensions. You also learned how to use sets and dictionaries to remove duplicates, as well as using other libraries, such as collections , numpy , and pandas to do this.

To learn more about the collections library, check out the official documentation here .

Nik Piepenbreier

Nik is the author of datagy.io and has over a decade of experience working with data analytics, data science, and Python. He specializes in teaching developers how to use Python for data science using hands-on tutorials. View Author posts

1 thought on “Python: Remove Duplicates From a List (7 Ways)”

the list comprehension is not correct over here. If someone copy that code they will have an abusement of list comprehension method and that’s not good. Better to use set comprehension.

Here is the better readable code without any deduplicate var: duplicated_list = [1,1,2,1,3,4,1,2,3,4] duplicated_list = list({i for i in duplicated_list})

Leave a Reply Cancel reply

Your email address will not be published. Required fields are marked *

Save my name, email, and website in this browser for the next time I comment.

Python Tutorial

File handling, python modules, python numpy, python pandas, python matplotlib, python scipy, machine learning, python mysql, python mongodb, python reference, module reference, python how to, python examples, how to remove duplicates from a python list.

Learn how to remove duplicates from a List in Python.

Remove any duplicates from a List:

Example Explained

First we have a List that contains duplicates:

A List with Duplicates

Create a dictionary, using the List items as keys. This will automatically remove any duplicates because dictionaries cannot have duplicate keys.

Create a Dictionary

Then, convert the dictionary back into a list:

Convert Into a List

Now we have a List without any duplicates, and it has the same order as the original List.

Print the List to demonstrate the result

Print the List

Advertisement

Create a Function

If you like to have a function where you can send your lists, and get them back without duplicates, you can create a function and insert the code from the example above.

Create a function that takes a List as an argument.

Create a dictionary, using this List items as keys.

Convert the dictionary into a list.

Return the list

Return List

Call the function, with a list as a parameter:

Call the Function

Print the result:

Print the Result

COLOR PICKER

Report Error

If you want to report an error, or if you want to make a suggestion, do not hesitate to send us an e-mail:

[email protected]

Top Tutorials

Top references, top examples, get certified.

Remove/extract duplicate elements from list in Python

This article describes how to generate a new list in Python by removing and extracting duplicate elements from a list. Note that removing duplicate elements is equivalent to extracting only unique elements.

Do not keep the order of the original list: set()

Keep the order of the original list: dict.fromkeys() , sorted(), for a two-dimensional list (list of lists), do not keep the order of the original list, keep the order of the original list.

The same idea can be applied to tuples instead of lists.

To check if a list or tuple has duplicate elements, or to extract elements that are common or non-common among multiple lists, see the following articles.

• Check if a list has duplicates in Python
• Extract common/non-common/unique elements from multiple lists in Python

Remove duplicate elements (Extract unique elements) from a list

Use set() if you don't need to keep the order of the original list.

By passing a list to set() , it returns set , which ignores duplicate values and keeps only unique values as elements.

• Built-in Functions - set() — Python 3.11.3 documentation

set can be converted back to a list or tuple with list() or tuple() .

Of course, you can use set as it is. See the following article for more information about set .

• Set operations in Python (union, intersection, symmetric difference, etc.)

If you want to keep the order of the original list, use dict.fromkeys() or sorted() .

dict.fromkeys() creates a new dictionary with keys from the iterable. If the second argument is omitted, the default value is set to None .

• Built-in Types - dict.fromkeys() — Python 3.11.3 documentation

Since a dictionary key cannot have duplicate elements, duplicate values are ignored like set() . Passing a dictionary to list() returns a list with dictionary keys as elements.

Starting from Python 3.7 (3.6 for CPython), dict.fromkeys() guarantees that the sequence order is preserved. In earlier versions, you can use the built-in function sorted() as described below.

The index() method returns the index of a value. By specifying it as the key parameter in sorted() , the list can be sorted based on the order of the original list.

• How to use a key parameter in Python (sorted, max, etc.)

For a two-dimensional list (list of lists), using set() or dict.fromkeys() will raise a TypeError .

This is because unhashable objects such as lists cannot be set type elements or dict type keys.

Define the following function to resolve this issue. This function preserves the order of the original list and works for both one-dimensional lists and tuples.

• python - How do you remove duplicates from a list whilst preserving order? - Stack Overflow

List comprehension is used.

• List comprehensions in Python

Extract duplicate elements from a list

If you want to extract only duplicate elements from the original list, use collections.Counter() that returns collections.Counter (dictionary subclass) whose key is an element and whose value is its count.

• Count elements in a list with collections.Counter in Python

Since it is a subclass of a dictionary, you can retrieve keys and values with items() . You can extract keys with more than two counts by list comprehension.

• Iterate dictionary (key and value) with for loop in Python

As in the above example, since Python 3.7, the key of collections.Counter preserves the order of the original list.

In earlier versions, you can sort by sorted() as in the example to remove duplicate elements.

If you want to extract elements in their duplicated state, simply include elements with two or more occurrences from the original list. The order is also preserved.

For a two-dimensional list (list of lists):

Note that count() requires O(n) , so the function that repeatedly executes count() shown above is very inefficient. There may be more efficient and optimized approaches available.

Since collections.Counter is a subclass of the dictionary, an error is raised if you pass a list or tuple whose elements are unhashable, such as a list, to collections.Counter() .

Related Categories

Related articles.

• Find the index of an item in a list in Python
• The in operator in Python (for list, string, dictionary, etc.)
• Sort a list of dictionaries by the value of the specific key in Python
• Use enumerate() and zip() together in Python
• Get the n-largest/smallest elements from a list in Python
• Convert between a list and a tuple in Python
• Transpose 2D list in Python (swap rows and columns)
• Extract and replace elements that meet the conditions of a list of strings in Python
• Filter (extract/remove) items of a list with filter() in Python
• Compare lists in Python
• How to slice a list, string, tuple in Python
• Convert between NumPy array and Python list
• Convert a list of strings and a list of numbers to each other in Python

Remove Duplicates from List Python

Working with lists is a common task in Python. Sometimes we need to remove duplicate elements from a list.

Whether you are ensuring unique elements in a list or clearing data for further processing, removing duplicates from a list is a common task.

Here you will learn multiple different ways to remove duplicates from a list in Python.

1. Using set() method

set() method is the easiest way to remove duplicates from a list. It removes all the duplicate elements from the list and returns a set of unique elements.

You can later convert this set back to a list using the list() method.

Output: (It can be in any order)

As you can see, all the duplicate elements are removed from the list.

But there is a problem with this method. It changes the order of elements in the list. If you want to preserve the order of elements, you can use the OrderedDict method.

2. Using OrderedDict method

OrderedDict is a dictionary subclass that remembers the order in which the elements were added. It is available in the collections module.

So, we can use this property of OrderedDict to remove duplicates from a list while preserving the order of elements.

First, we will convert the list to an OrderedDict and then convert it back to a list.

As you can see, the order of elements is preserved.

3. Using list comprehension

List comprehension is a single tool that can be used to perform multiple operations on a list.

You can also use this to remove duplicates from a list.

Here, we are iterating over the list and checking if the current element is not present in the list before the current element. If it is not present, we are adding it to the new list.

4. Using for loop

Using a for loop is the most basic way to remove duplicates from a list.

Here, we will iterate over the list and check if the current element is not present in the new list. If it is not present, we will add it to the new list.

5. Using dict.fromkeys() method

dict.fromkeys() method is used to create a new dictionary from a given sequence of elements.

Since a dictionary cannot have duplicate keys, we can use this property to remove duplicates from a list.

Later we can convert this dictionary back to a list.

6. Using numpy.unique() method

numpy.unique() method returns the unique elements from a list.

It also returns the indices of the elements in the original list.

So, we can use this method to remove duplicates from a list.

Here, we are also returning the indices of the elements in the original list. You can use these indices to get the elements from the original list.

7. Using pandas.drop_duplicates() method

pandas.drop_duplicates() method is used to remove duplicate rows from a DataFrame .

This can be a handy method if you are working with dataframes.

These are some of the ways to remove duplicates from a list in Python. You can use any of these methods depending on your use case.

But if you are working with a large list, you should use the set() method as it is the fastest method to remove duplicates from a list.

Here is a speed comparison of all the methods discussed above.

You can clearly see fastest way to remove duplicates from list python is using set() method.

Learn Python practically and Get Certified .

Popular Tutorials

Popular examples, reference materials, learn python interactively, python examples.

• Add Two Matrices
• Transpose a Matrix
• Multiply Two Matrices
• Check Whether a String is Palindrome or Not
• Remove Punctuations From a String
• Sort Words in Alphabetic Order
• Illustrate Different Set Operations
• Count the Number of Each Vowel

Python Tutorials

• Python List remove()
• Python List pop()

Python List

• Python List extend()
• Python list()

Python Sets

Python Program to Remove Duplicate Element From a List

To understand this example, you should have the knowledge of the following Python programming topics:

Example 1: Using set()

In the above example, we first convert the list into a set, then we again convert it into a list. Set cannot have a duplicate item in it, so set() keeps only an instance of the item.

Example 2: Remove the items that are duplicated in two lists

In the above example, the items that are present in both lists are removed.

• Firstly, both lists are converted to two sets to remove the duplicate items from each list.
• Then, ^ gets the symmetric difference of two lists (excludes the overlapping elements of two sets).

Sorry about that.

Related Examples

Python Example

Iterate Through Two Lists in Parallel

Concatenate Two Lists

Python Tutorial

• [email protected]

What’s New ?

The Top 10 favtutor Features You Might Have Overlooked

• Don’t have an account Yet? Sign Up

Remember me Forgot your password?

• Already have an Account? Sign In

Lost your password? Please enter your email address. You will receive a link to create a new password.

Back to log-in

By Signing up for Favtutor, you agree to our Terms of Service & Privacy Policy.

How to Remove Duplicates from List in Python? (with code)

• Nov 13, 2023
• 7 Minutes Read
• Why Trust Us We uphold a strict editorial policy that emphasizes factual accuracy, relevance, and impartiality. Our content is crafted by top technical writers with deep knowledge in the fields of computer science and data science, ensuring each piece is meticulously reviewed by a team of seasoned editors to guarantee compliance with the highest standards in educational content creation and publishing.
• By Shivali Bhadaniya

In Python, a list is a versatile collection of objects that allows for duplicates.  But sometimes it is necessary to make the list unique to streamline our data or perform certain operations. Here, we are going to study the multiple ways to remove duplicates from the list in Python. So, let's get started!

Why Remove Duplicates from the List?

A Python list is a built-in data structure to store a collection of items. It is written as the list of comma-separated values inside the square bracket. The most important advantage of it is that the elements inside the list are not compulsorily of the same data type. Learn more about printing lists in Python to understand the concept better.

Now there are several reasons to remove duplicates from a list. Duplicates in a list can take up unnecessary space and decrease performance. Additionally, it can lead to confusion and errors if you're using the list for certain operations. So, removing duplicates will make the data more accurate for better analysis.

For example, if you're trying to find the unique elements in a list, duplicates can give you incorrect results. In general, it's a good idea to remove duplicates from a list to make it more organized and easier to work with.

If you are still confused between an array and a list, read Array vs List in Python.

7 Ways to Remove Duplicates from a List in Python

There are many ways to remove duplicates from a list in Python. Let’s check them out one by one:

1) Using set()

  A set is a data structure that is very similar to lists. It is a collection of items that can be accessed using a single variable name.

The simplest way to remove duplicates from a list in Python is by converting the list into a set. It will automatically remove similar entries because set has a property that it cannot have duplicate values.

If a list is typecasted to a set, that is it is passed as an argument to the set() method, it will automatically create a set consisting of all elements in the list but it will not keep duplicate values. The resultant set can be converted back to a list using the list() method.

This approach makes use of Python sets which are implemented as hash tables, allowing for very quick validity checks. It is quite quick, especially for larger lists, but the only drawback is that we will lose the order that exists in the original list.

2) Using a For Loop

In this method, we will iterate over the whole list using a 'for' loop. We will create a new list to keep all the unique values and use the "not in" operator in Python to find out if the current element that we are checking exists in the new list that we have created. If it does not exist, we will add it to the new list and if it does exist we will ignore it.

3) Using collections.OrderedDict.fromkeys()

This is the fastest method to solve the problem. We will first remove the duplicates and return a dictionary that has been converted to a list. In the below code when we use the fromkeys() method it will create keys of all the elements in the list. But keys in a dictionary cannot be duplicated, therefore, the fromkeys() method will remove duplicate values on its own.

We used OrderedDict from the collections module to preserve the order.

4) Using a list comprehension

List comprehension refers to using a for loop to create a list and then storing it under a variable name. The method is similar to the naive approach that we have discussed above but instead of using an external for loop, it creates a for loop inside the square braces of a list. This method is called list comprehension.

We use the for loop inside the list braces and add the if condition allowing us to filter out values that are duplicates.

5) Using list comprehension & enumerate()

List comprehensive when merged with enumerate function we can remove the duplicate from the python list. Basically in this method, the already occurred elements are skipped, and also the order is maintained. This is done by the enumerate function.

In the code below, the variable n keeps track of the index of the element being checked, and then it can be used to see if the element already exists in the list up to the index specified by n. If it does exist, we ignore it else we add it to a new list and this is done using list comprehensions too as we discussed above.

 6) Using the ‘pandas’ module

Using the pd.Series() method, a Pandas Series object is constructed from the orginal list. The Series object is then invoked using the drop duplicates() function to eliminate any duplicate values. Lastly, using the tolist() function, the resulting Series object is transformed back into a list.

7) Using the ‘pandas’ module

Using the pd.Series() method, a Pandas Series object is constructed from the original list. The Series object is then invoked using the drop duplicates() function to eliminate any duplicate values. Lastly, using the tolist() function, the resulting Series object is transformed back into a list.

How to remove duplicate words from a list?  

To remove duplicate words from a list in Python, you can use the set( ) function, consider the example below :

Keep in mind, that this method does not maintain the order of the original list. If you need to keep the order, you can filter out duplicates with a loop and an empty list:

Key Considerations

While the methods described above provide effective ways to remove duplicates from a list, there are a few additional considerations to keep in mind:

• Data Type Compatibility: Some methods may have limitations when working with specific data types. Ensure that the chosen method is compatible with the elements in your list.
• Order Preservation: If maintaining the original order of elements is crucial, consider using methods such as list comprehension with enumeration or the collections module's OrderedDict class.
• Performance: For large lists or time-sensitive operations, it's important to choose a method that offers optimal performance. Consider conducting performance tests to determine the most efficient approach.

Removing duplicates from a list in Python is a common task, and this article has provided you with various methods to accomplish it. We learned different methods to remove duplicate elements from the list in Python from the naive approach to utilizing set, list comprehension, and specialized modules.

FavTutor - 24x7 Live Coding Help from Expert Tutors!

About The Author

Shivali Bhadaniya

More by favtutor blogs, monte carlo simulations in r (with examples), aarthi juryala.

The unlist() Function in R (with Examples)

Paired Sample T-Test using R (with Examples)

• Free Python 3 Tutorial
• Control Flow
• Exception Handling
• Python Programs
• Python Projects
• Python Interview Questions
• Python Database
• Data Science With Python
• Machine Learning with Python
• Flatten A List of Lists in Python
• Print a List in Python Horizontally
• How To Combine Multiple Lists Into One List Python
• Appending To One List In A List Of Lists
• Python Program to Split a List into Two Halves
• Check If Value Exists in Python List of Objects
• Python List Comprehension Using If-Else
• Access Two Elements at a Time using Python
• Split Elements of a List in Python
• Add Values into an Empty List from Python For Loop
• Python Initialize List of Lists
• Merge Multiple Lists into one List
• Append Element to an Empty List In Python
• Implementation of Stack in Python using List
• Remove Dictionary from List If Key is Equal to Value in Python
• Sort List of Lists Ascending and then Descending in Python
• Indexing Lists Of Lists In Python
• Python Program to Square Each Odd Number in a List using List Comprehension
• Cloning Row and Column Vectors in Python

Python List: Remove Duplicates And Keep The Order

Python lists are versatile data structures that allow the storage of multiple elements in a single variable. While lists provide a convenient way to manage collections of data, duplicates within a list can sometimes pose challenges. In this article, we will explore five different methods to remove duplicates from a Python list while preserving the original order.

Remove Duplicates And Keep The Order in Python List

Below are the examples of Remove Duplicates And Keep The Order in Python .

• Using a For Loop
• Using a Set Method
• Using List Comprehension
• Using OrderedDict.fromkeys()
• Using numpy.unique() Method

Remove Duplicates And Keep The Order Using Loop

In this example, the below code initializes a list named `original_list` with integers and prints it. It then creates an empty list `unique_list` and uses a loop to iterate through `original_list`, appending elements to `unique_list` only if they haven’t been added before. This results in an updated list without duplicates.

Remove Duplicates And Keep The Order Using Set

In this example, the below code initializes a list named ` original_list ` with various integers. It then prints the original list. Subsequently, it creates a ` unique_list ` by converting the original list to a set to remove duplicates and prints the updated list.

Remove Duplicates And Keep The Order Using List Comprehension

In this example, below code initializes a list named `original_list` with various integers and prints it. It then creates an empty list `unique_list` and uses a list comprehension to append elements from the original list to `unique_list` only if they haven’t been added before, effectively removing duplicates.

Remove Duplicates And Keep The Order Using OrderedDict.fromkeys()

The code utilizes the ` OrderedDict ` from the `collections` module to remove duplicates from the list `original_list` while preserving the order. It prints the original list and then creates a `unique_list` by converting the keys of an ordered dictionary created from the original list. The result is an updated list without duplicates.

Remove Duplicates And Keep The Order Using numpy.unique() Method

In this example, below code employs NumPy to remove duplicates from the list original_list while preserving the order. It prints the original list and then uses np.unique() to obtain unique elements and their indices. The resulting unique_list represents the updated list without duplicates.

In conclusion, removing duplicates from a Python list while preserving the original order can be achieved through various methods. Whether using the simplicity of sets, concise list comprehensions, OrderedDict, traditional loops, or the NumPy library, developers have a range of efficient options. The choice of method depends on specific requirements and preferences, allowing for flexibility in managing lists with unique elements.

Please Login to comment...

Similar reads.

• python-list
• How to Use ChatGPT with Bing for Free?
• 7 Best Movavi Video Editor Alternatives in 2024
• How to Edit Comment on Instagram
• 10 Best AI Grammar Checkers and Rewording Tools
• 30 OOPs Interview Questions and Answers (2024)

Improve your Coding Skills with Practice

What kind of Experience do you want to share?

How to Remove Duplicates from a List in Python

Python remove Duplicates from a List

A list is a container that contains different Python objects, which could be integers, words, values, etc. It is the equivalent of an array in other programming languages.

So here will go through different ways in which we can remove duplicates from a given list in Python.

Method 1) Remove duplicates from list using Set

To remove the duplicates from a list, you can make use of the built-in function set() . The specialty of set() method is that it returns distinct elements.

Method 2) Using the Temporary List

To remove duplicates from a given list , you can make use of an empty temporary list. For that first, you will have to loop through the list having duplicates and add the unique items to the temporary list. Later the temporary list is assigned to the main list.

Example Here is a working example using temporary list.

Method 3) Using Dict

We can remove duplicates from the given list by importing OrderedDict from collections. It is available from python2.7 onwards. OrderedDict takes care of returning you the distinct elements in an order in which the key is present.

Let us make use of a list and use fromkeys() method available in OrderedDict to get the unique elements from the list.

To make use of OrderedDict.fromkey() method, you have to import OrderedDict from collections, as shown below:

Here is an example to remove duplicates using OrderedDict.fromkeys() method.

From Python 3.5+ onwards, we can make use of the regular dict.fromkeys() to get the distinct elements from the list. The dict.fromkeys() methods return keys that are unique and helps to get rid of the duplicate values.

An example that shows the working of dict.fromkeys() on a list to give the unique items is as follows:

Method 4) Using for-loop

Using for-loop , we will traverse the list of items to remove duplicates.

First initialize array to empty i.e myFinallist = [] .Inside the for-loop, add check if the items in the list exist in the array myFinallist . If the items do not exist, add the item to the array myFinallist using the append() method.

So whenever the duplicate item is encountered it will be already present in the array myFinallist and will not be inserted. Let us now check the same in the example below:

Method 5) Using list comprehension

List comprehensions are Python functions that are used for creating new sequences (such as lists, dictionaries, etc.) using sequences that have already been created. This helps you to reduce longer loops and make your code easier to read and maintain.

Let us make use of list comprehension to remove duplicates from the list given.

Method 6) Using Numpy unique() method.

• The method unique() from Numpy module can help us remove duplicate from the list given.

To work with Numpy first import numpy module, you need to follow these steps:

Step 1 ) Import Numpy module

Step 2) Use your list with duplicates inside unique method as shown below. The output is converted back to a list format using tolist() method.

Step 3) Finally print the list as shown below:

The final code with output is as follows:

Method 7) Using Pandas methods

• The Pandas module has a unique() method that will give us the unique elements from the list given.

To work with Pandas module, you need to follow these steps:

Step 1) Import Pandas module

Step 2) Use your list with duplicates inside unique() method as shown below:

Step 3) Print the list as shown below:

Method 8) Using enumerate() and list comprehension

Here the combination of list comprehension and enumerate to remove the duplicate elements. Enumerate returns an object with a counter to each element in the list. For example (0,1), (1,2) etc. Here the first value is the index, and the second value is the list item. W

Each element is checked if it exists in the list, and if it does, it is removed from the list.

• To remove the duplicates from a list, you can make use of the built-in function set() . The specialty of the set() method is that it returns distinct elements.
• You can remove duplicates from the given list by importing OrderedDictfrom collections. It is available from python2.7 onwards. OrderedDictdict takes care of returning you the distinct elements in an order in which the key is present.
• You can make use of a for-loop that we will traverse the list of items to remove duplicates.
• The combination of list comprehension and enumerate is used to remove the duplicate elements from the list. Enumerate returns an object with a counter to each element in the list.
• Online Python Compiler (Editor / Interpreter / IDE) to Run Code
• PyUnit Tutorial: Python Unit Testing Framework (with Example)
• How to Install Python on Windows [Pycharm IDE]
• Hello World: Create your First Python Program
• Python Variables: How to Define/Declare String Variable Types
• Python Strings: Replace, Join, Split, Reverse, Uppercase & Lowercase
• Python TUPLE – Pack, Unpack, Compare, Slicing, Delete, Key
• Dictionary in Python with Syntax & Example

5 Best Ways to Remove Duplicate Entries in a Python List

💡 Problem Formulation: When working with lists in Python, it is quite common to encounter duplicates which can skew results or affect performance. Suppose you have a list: [3, 5, 2, 3, 8, 5] and you want to eliminate the duplicates to get [3, 5, 2, 8] . This article provides multiple solutions to tackle such scenarios.

Method 1: Using a for Loop

This method involves iterating over the original list and appending unique elements to a new list. It is simple and intuitive but not the most efficient for large lists due to its O(n^2) complexity.

Here’s an example:

Output: [3, 5, 2, 8]

This function iterates through the input list and appends each item to the result only if it is not already present in the result, thus ensuring all elements are unique.

Method 2: Using set()

The set() data structure in Python is designed to store unique elements. This method converts the list to a set to remove duplicates and then back to a list. It is efficient and has a lower complexity of O(n), but the original order of elements is not guaranteed.

Output: [2, 3, 5, 8] (Note: Order might vary)

Converting the list to a set removes duplicates and converting it back to a list provides a duplicate-free list, but without preserving the original order of elements.

Method 3: Using a Dictionary

Similar to sets, dictionaries in Python cannot have duplicate keys. This method involves using a dictionary to remove duplicates, preserving the order of first occurrences, thereby combining efficiency with ordered results.

This code uses the dict.fromkeys() function which creates a dictionary with unique elements from the list as keys (preserving their order) and then converts the keys back into a list.

Method 4: Using List Comprehensions

List comprehensions provide a succinct and readable way to create lists. This method removes duplicates by iterating through the list and including only the first occurrence of each element, maintaining the original order.

The list comprehension here includes an item in the result only if it has not appeared before in the list up to the current index.

Bonus One-Liner Method 5: Using sorted() and itertools.groupby()

This method is perfect for when you need a one-liner that removes duplicates while preserving order. It requires the list to be sorted and then uses `itertools.groupby()` to group the list into individual unique elements. Note that sorting the list can change the original order of elements.

Output: [2, 3, 5, 8]

Sorting the list and then using groupby() allows us to iterate over grouped items, picking out only the unique ones, though the original order is not preserved.

Summary/Discussion

• Method 1: Using a for Loop. Simple and intuitive. Not the best performance for large lists.
• Method 2: Using set(). Efficient and simple. Doesn’t preserve the order of elements.
• Method 3: Using a Dictionary. Preserves order. Efficient for any size list.
• Method 4: Using List Comprehensions. Concise and preserves order. Could be less efficient for large lists.
• Method 5: Using sorted() and itertools.groupby(). One-liner solution. Does not preserve original order and requires elements to be sortable.

Emily Rosemary Collins is a tech enthusiast with a strong background in computer science, always staying up-to-date with the latest trends and innovations. Apart from her love for technology, Emily enjoys exploring the great outdoors, participating in local community events, and dedicating her free time to painting and photography. Her interests and passion for personal growth make her an engaging conversationalist and a reliable source of knowledge in the ever-evolving world of technology.

• How it works
• Homework answers

Answer to Question #274794 in Python for Naveen

a=[10, 20,30,40,50,30]

a. remove(30)

Need a fast expert's response?

and get a quick answer at the best price

for any assignment or question with DETAILED EXPLANATIONS !

Leave a comment

Ask your question, related questions.

• 1. Write a Python program that get a number using keyboard input. (Remember to use input for Python 3 b
• 2. n distributed systems, most of the time the processing is based on client/server computing- theclien
• 3. def countdown (n):If n<=0:print('Blastoff!')else:print(n)countdown(n-1)Write a new recu
• 4. Assume, you have been given a tuple with details about books that won the Good Reads Choice Awards.b
• 5. Write a source code of object detection from any of the videos.
• 6. Write a function solution that given a three digit integer N and Integer K, returns the maximum poss
• 7. Write a Python program that takes a dictionary as an input from the user and then prints the average
• Programming
• Engineering