Removing duplicate files with Powershell

后端 未结 4 2054
渐次进展
渐次进展 2021-02-14 21:39

I have several thousand duplicate files (jar files as an example) that I\'d like to use powershell to

  1. Search through the file system recursively
  2. Find the
相关标签:
4条回答
  • 2021-02-14 21:57

    Even though the question is old, I have been in a need to clean up all duplicate files based on content. The idea is simple, the algorithm for this is not straightforward. Here is the code which accepts a parameter of "path" to delete duplicates from.

     Function Delete-Duplicates {
        param(
        [Parameter(
        Mandatory=$True,
        ValueFromPipeline=$True,
        ValueFromPipelineByPropertyName=$True
        )]
        [string[]]$PathDuplicates)
    
        $DuplicatePaths = 
            Get-ChildItem $PathDuplicates | 
            Get-FileHash |
            Group-Object -Property Hash |
            Where-Object -Property Count -gt 1 |
            ForEach-Object {
                $_.Group.Path |
                Select -First ($_.Count -1)}
        $TotalCount = (Get-ChildItem $PathDuplicates).Count
     Write-Warning ("You are going to delete {0} files out of {1} total. Please confirm the prompt" -f $DuplicatePaths.Count, $TotalCount)    
     $DuplicatePaths | Remove-Item -Confirm
    
        }
    

    The script

    a) Lists all ChildItems

    b) Retrieves FileHash from them

    c) Groups them by Hash Property (so all the same files are in the single group)

    d) Filters out the already-unique files (count of group -eq 1)

    e) Loops through each group and lists all but last paths - ensuring one file of each "Hash" always stays

    f) Warns before preceding, saying how many files are there in total and how many are going to be deleted.

    Probably not the most performance-wise option (SHA1-ing every file) but ensures the file is a duplicate. Works perfectly fine for me :)

    0 讨论(0)
  • 2021-02-14 22:04

    Keep a dictionary of files, delete when the next file name was already encountered before:

    $dict = @{};
    dir c:\admin -Recurse | foreach {
      $key = $_.Name #replace this with your checksum function
      $find = $dict[$key];
      if($find -ne $null) {
        #current file is a duplicate
        #Remove-Item -Path $_.FullName ?    
      }
      $dict[$key] = 0; #dummy placeholder to save memory
    }
    

    I used file name as a key, but you can use a checksum if you want (or both) - see code comment.

    0 讨论(0)
  • 2021-02-14 22:16

    try this:

    ls *.txt -recurse | get-filehash | group -property hash | where { $_.count -gt 1 } | % { $_.group | select -skip 1 } | del
    

    from: http://n3wjack.net/2015/04/06/find-and-delete-duplicate-files-with-just-powershell/

    0 讨论(0)
  • 2021-02-14 22:23

    Instead of just remove your duplicates files, you can replace by a shortcut

    #requires -version 3
    <#
        .SYNOPSIS
        Script de nettoyage des doublons
        .DESCRIPTION
        Cherche les doublons par taille, compare leur CheckSum MD5 et les regroupes par Taille et MD5
        peut remplacer chacun des doubles par un lien vers le 1er fichier, l'original
    
        .PARAMETER Path
        Chemin ou rechercher les doublon
    
        .PARAMETER ReplaceByShortcut
        si specifier alors les doublons seront remplacé
    
        .PARAMETER MinLength
        ignore les fichiers inferieure a cette taille (en Octets)
    
        .EXAMPLE
        .\Clean-Duplicate '\\dfs.adds\donnees\commun'
    
        .EXAMPLE
        recherche les doublon de 10Ko et plus
        .\Clean-Duplicate '\\dfs.adds\donnees\commun' -MinLength 10000
    
        .EXAMPLE
        .\Clean-Duplicate '\\dpm1\d$\Coaxis\Logiciels' -ReplaceByShortcut
    #>
    [CmdletBinding()]
    param (
        [string]$Path = '\\Contoso.adds\share$\path\data',
        [switch]$ReplaceByShortcut = $false,
        [int]$MinLength = 10*1024*1024 # 10 Mo
    )
    
    $version = '1.0'
    
    function Create-ShortCut ($ShortcutPath, $shortCutName, $Target) {
        $link = "$ShortcutPath\$shortCutName.lnk"
        $WshShell = New-Object -ComObject WScript.Shell
        $Shortcut = $WshShell.CreateShortcut($link)
        $Shortcut.TargetPath = $Target
        #$Shortcut.Arguments ="shell32.dll,Control_RunDLL hotplug.dll"
        #$Shortcut.IconLocation = "hotplug.dll,0"
        $Shortcut.Description ="Copy Doublon"
        #$Shortcut.WorkingDirectory ="C:\Windows\System32"
        $Shortcut.Save()
        # write-host -fore Cyan $link -nonewline; write-host -fore Red ' >> ' -nonewline; write-host -fore Yellow $Target 
        return $link
    }
    
    function Replace-ByShortcut {
        Param(
            [Parameter(ValueFromPipeline=$true,ValueFromPipelineByPropertyName=$true)]
                $SameItems
        )
        begin{
            $result = [pscustomobject][ordered]@{
                Replaced = @()
                Gain = 0
                Count = 0
            }
        }
        Process{
            $Original = $SameItems.group[0]
            foreach ($doublon in $SameItems.group) {
                if ($doublon -ne $Original) {
                    $result.Replaced += [pscustomobject][ordered]@{
                        lnk = Create-Shortcut -ShortcutPath $doublon.DirectoryName -shortCutName $doublon.BaseName -Target $Original.FullName
                        target = $Original.FullName
                        size = $doublon.Length
                    }
                    $result.Gain += $doublon.Length
                    $result.Count++
                    Remove-item $doublon.FullName -force
                }
            }
        }
        End{
            $result
        }
    }
    
    function Get-MD5 {
        param (
            [Parameter(Mandatory)]
                [string]$Path
        )
        $HashAlgorithm = New-Object -TypeName System.Security.Cryptography.MD5CryptoServiceProvider
        $Stream = [System.IO.File]::OpenRead($Path)
        try {
            $HashByteArray = $HashAlgorithm.ComputeHash($Stream)
        } finally {
            $Stream.Dispose()
        }
    
        return [System.BitConverter]::ToString($HashByteArray).ToLowerInvariant() -replace '-',''
    }
    
    if (-not $Path) {
        if ((Get-Location).Provider.Name -ne 'FileSystem') {
            Write-Error 'Specify a file system path explicitly, or change the current location to a file system path.'
            return
        }
        $Path = (Get-Location).ProviderPath
    }
    
    $DuplicateFiles = Get-ChildItem -Path $Path -Recurse -File |
        Where-Object { $_.Length -gt $MinLength } |
            Group-Object -Property Length |
                Where-Object { $_.Count -gt 1 } |
                    ForEach-Object {
                        $_.Group |
                            ForEach-Object {
                                $_ | Add-Member -MemberType NoteProperty -Name ContentHash -Value (Get-MD5 -Path $_.FullName)
                            }
                        $_.Group |
                            Group-Object -Property ContentHash |
                                Where-Object { $_.Count -gt 1 }
                    }
    
    $somme = ($DuplicateFiles.group | Measure-Object length -Sum).sum
    write-host "$($DuplicateFiles.group.count) doublons, soit $($somme/1024/1024) Mo" -fore cyan
    
    if ($ReplaceByShortcut) {
        $DuplicateFiles | Replace-ByShortcut
    } else {
        $DuplicateFiles
    }
    
    0 讨论(0)
提交回复
热议问题