How to Find and Delete Duplicate Files in Mac OS X

SpotlightDuplicate files such as documents and photos can occupy a surprising amount of space on your Mac’s hard disk and you might not even realise they are there. Perhaps you’ve copied files into several folders and external USB drives, and additional copies could exist in the Mail downloads folder. You might even find a few gigabytes of duplicate content in iTunes, normally because several albums from the same artist might contain the same tracks (e.g. an album and a greatest hits).

It’s therefore a good idea to get rid of those unwanted duplicates which are just clogging up your system, especially if you’re a bit low on disk space.

There are several ways to find and delete duplicate files in Mac OS X:

iTunes also has the ability to find duplicate items, which is covered later in this tutorial.


find and delete mac apps

METHOD 1: Use a Third-Party Application

There are dozens of apps available in the Mac App Store which can find and remove duplicate files. Some are free whereas others charge a small fee. They all do pretty much the same thing but some have much more comprehensive features and allow you to filter by filename only or the actual content. For example, it’s possible that multiple identical files have different filenames – utilities such as those mentioned below can help to analyse the contents of files on a byte level to determine if they really are duplicates.

The apps below are just a few examples that can be found in the Mac App Store. Many similar apps can be found on the Internet, but it’s safe to assume that applications in the App Store are legitimate and don’t contain any malware.


METHOD 2: Use Finder to Locate Duplicates

Duplicate files can be found quite easily using Finder. The process is rather slow because you need to manually decide which ones to keep and which to delete, but with just a little time and patience you can locate most duplicate items.

Follow the method below:

  1. Open a new Finder window
  2. In the search field, enter the wild card asterisk symbol
  3. Make sure the Size and Kind columns are displayed
  4. Filter the results by Kind, and you will see the list of similar items

This method is fairly accurate because it shows identical files listed by name, type, and size. Just delete any files that you don’t want to keep but make sure they really are duplicates first. Unfortunately, this method doesn’t let you determine whether the files are true byte-level copies but there is enough information presented to be able to make an informed decision. The main drawback with this method is the amount of time it takes to go through the items one at a time.

Using Finder to find duplicate files



itunes logo icon

A Note on How to Remove Duplicate Items in iTunes

Fortunately iTunes provides the ability to check for duplicate files, as follows:

  1. Make sure you’ve highlighted the relevant item in your iTunes library (such as Music, Films, Podcasts, etc)
  2. In the File menu, select Display Duplicates

Any duplicate items found will be displayed in the main iTunes window, allowing you to simply delete them as normal.

How to find duplicate items in iTunes



The Terminal Icon

METHOD 3: Use a Terminal Command or Script

It’s possible (but certainly not recommended for novices) to use “shell commands” in Terminal to list and delete duplicate files. Generally speaking, it’s considerably easier to use a third-party application but some people prefer the flexibility that Terminal provides.

Follow the steps below to view the list of duplicate files.

  1. Open Terminal from Spotlight or the Utilities folder
  2. Change to the directory (folder) you want to search from (including sub-folders) using the cd command. At the command prompt type cd <folder name> for example cd ~/Documents to change directory to your home Documents folder
  3. At the command prompt, type the following command: find . -size 20 \! -type d -exec cksum {} \; | sort | tee /tmp/f.tmp | cut -f 1,2 -d ‘ ‘ | uniq -d | grep -hif – /tmp/f.tmp > duplicates.txt
  4. This method uses a simple checksum to determine whether files are identical. The names of duplicate items will be listed in a file named duplicates.txt in the current directory. Open this to view the names of identical files

There are now various ways to delete the duplicates. To delete all the files in the text file, at the command prompt type: while read file; do rm “$file”; done < duplicates.txt. Note: with this method, you’ll need to copy one of each duplicate item to another folder before you enter the command because it deletes every file listed.

There are more complex but useful Terminal commands and scripts available on the Internet which can automate the process and ensure that for every duplicate the original is kept, but it’s only really recommended for experts as there is a lot of scope for things to go wrong.

Comments

  1. Ian says

    Hold down the ALT key while selecting duplicates in iTunes to find EXACT duplicates and tracks that just match name.

  2. Meltzbr says

    Great Product, Terrible Product Name (Chipmunk??)

    Thanks for the great review, Chris. After having used Remove Duplicate Items in iTunes I decided I needed a more robust application. I checked out all 5 applications reviewed above:
    ACDSee Duplicate Finder
    Araxis Find Duplicate Files
    Chipmunk
    Gemini: The Duplicate Finder.
    Search Duplicate Files plus Tidy Up
    I decided to go with Chipmunk and am very pleased with the result. Even tho more expensive than Gemini or Araxis it is far more robust. For example, Chipmunk found iTunes music files that had been misnamed but, were the same song, i.e., Scarlet Begonias by Grateful Dead had a duplicate mislabeled ‘Radio Head at Bank of America’ Using Chipmunk’s ‘Quick Look’ function i could play both versions and determine that they were, in fact, duplicates. This is also true for photos. I am very glad I bought Chipmunk. It is fast, easy to use and very useful at cleaning up duplicates.

  3. Lou says

    Gemini no longer works. I downloaded last week (april 2013) it only allows manual deletion, and even then kept warning me about deleting the original. I got a refund and am now searching for a program that works in 2013.

  4. Lou says

    Gemini no longer works. I downloaded last week (april 2013) it only allows manual deletion, and even then kept warning me about deleting the original. I got a refund and am now searching for a program that works in 2013.

  5. Kathleen says

    iTunes doesn’t have the Find Duplicates option under the File menu on my computer. It is the latest version, and the system is 10.6.8 Any ideas if that option exists somewhere else?
    Also, when trying to use the Finder search to locate duplicate items, I cannot find how to add a Size column to the menu, only to the search criteria. It must be simple, help, please. The Help lookup doesn’t, in fact, help.

  6. Kathleen says

    iTunes doesn’t have the Find Duplicates option under the File menu on my computer. It is the latest version, and the system is 10.6.8 Any ideas if that option exists somewhere else?
    Also, when trying to use the Finder search to locate duplicate items, I cannot find how to add a Size column to the menu, only to the search criteria. It must be simple, help, please. The Help lookup doesn’t, in fact, help.

  7. Ricktor says

    Thanks for the information. Also try Long Path Tool. It helped me with Error 1320 on Win 7. :)

  8. says

    When I try this, I modified the line to remove the fancy quotes and replace them with simple ‘ quotes. Also, there is an em dash instead of a minus sign, so I changed that. Both before and after though, it errors: ‘grep: -: No such file or directory’

Leave a Reply