From f8018096c575bc3c9fd41be629db9141c24dd165 Mon Sep 17 00:00:00 2001 From: Oliver Booth Date: Tue, 16 Apr 2024 20:15:07 +0100 Subject: [PATCH] feat: initial commit --- .github/CODE_OF_CONDUCT.md | 133 +++++++++++++++++++++++++++ .github/FUNDING.yml | 2 + .github/workflows/dotnet.yml | 53 +++++++++++ .gitignore | 37 ++++++++ FindDuplicates.sln | 16 ++++ FindDuplicates/FindDuplicates.csproj | 15 +++ FindDuplicates/ListCommand.cs | 108 ++++++++++++++++++++++ FindDuplicates/ListSettings.cs | 16 ++++ FindDuplicates/Program.cs | 5 + LICENSE.md | 21 +++++ README.md | 57 ++++++++++++ global.json | 7 ++ 12 files changed, 470 insertions(+) create mode 100644 .github/CODE_OF_CONDUCT.md create mode 100644 .github/FUNDING.yml create mode 100644 .github/workflows/dotnet.yml create mode 100644 .gitignore create mode 100644 FindDuplicates.sln create mode 100644 FindDuplicates/FindDuplicates.csproj create mode 100644 FindDuplicates/ListCommand.cs create mode 100644 FindDuplicates/ListSettings.cs create mode 100644 FindDuplicates/Program.cs create mode 100644 LICENSE.md create mode 100644 README.md create mode 100644 global.json diff --git a/.github/CODE_OF_CONDUCT.md b/.github/CODE_OF_CONDUCT.md new file mode 100644 index 0000000..593e231 --- /dev/null +++ b/.github/CODE_OF_CONDUCT.md @@ -0,0 +1,133 @@ +# Contributor Covenant Code of Conduct + +## Our Pledge + +We as members, contributors, and leaders pledge to make participation in our +community a harassment-free experience for everyone, regardless of age, body +size, visible or invisible disability, ethnicity, sex characteristics, gender +identity and expression, level of experience, education, socio-economic status, +nationality, personal appearance, race, caste, color, religion, or sexual +identity and orientation. + +We pledge to act and interact in ways that contribute to an open, welcoming, +diverse, inclusive, and healthy community. + +## Our Standards + +Examples of behavior that contributes to a positive environment for our +community include: + +* Demonstrating empathy and kindness toward other people +* Being respectful of differing opinions, viewpoints, and experiences +* Giving and gracefully accepting constructive feedback +* Accepting responsibility and apologizing to those affected by our mistakes, + and learning from the experience +* Focusing on what is best not just for us as individuals, but for the overall + community + +Examples of unacceptable behavior include: + +* The use of sexualized language or imagery, and sexual attention or advances of + any kind +* Trolling, insulting or derogatory comments, and personal or political attacks +* Public or private harassment +* Publishing others' private information, such as a physical or email address, + without their explicit permission +* Other conduct which could reasonably be considered inappropriate in a + professional setting + +## Enforcement Responsibilities + +Community leaders are responsible for clarifying and enforcing our standards of +acceptable behavior and will take appropriate and fair corrective action in +response to any behavior that they deem inappropriate, threatening, offensive, +or harmful. + +Community leaders have the right and responsibility to remove, edit, or reject +comments, commits, code, wiki edits, issues, and other contributions that are +not aligned to this Code of Conduct, and will communicate reasons for moderation +decisions when appropriate. + +## Scope + +This Code of Conduct applies within all community spaces, and also applies when +an individual is officially representing the community in public spaces. +Examples of representing our community include using an official e-mail address, +posting via an official social media account, or acting as an appointed +representative at an online or offline event. + +## Enforcement + +Instances of abusive, harassing, or otherwise unacceptable behavior may be +reported to the community leaders responsible for enforcement at +yiliansource@gmail.com or me@olivr.me. +All complaints will be reviewed and investigated promptly and fairly. + +All community leaders are obligated to respect the privacy and security of the +reporter of any incident. + +## Enforcement Guidelines + +Community leaders will follow these Community Impact Guidelines in determining +the consequences for any action they deem in violation of this Code of Conduct: + +### 1. Correction + +**Community Impact**: Use of inappropriate language or other behavior deemed +unprofessional or unwelcome in the community. + +**Consequence**: A private, written warning from community leaders, providing +clarity around the nature of the violation and an explanation of why the +behavior was inappropriate. A public apology may be requested. + +### 2. Warning + +**Community Impact**: A violation through a single incident or series of +actions. + +**Consequence**: A warning with consequences for continued behavior. No +interaction with the people involved, including unsolicited interaction with +those enforcing the Code of Conduct, for a specified period of time. This +includes avoiding interactions in community spaces as well as external channels +like social media. Violating these terms may lead to a temporary or permanent +ban. + +### 3. Temporary Ban + +**Community Impact**: A serious violation of community standards, including +sustained inappropriate behavior. + +**Consequence**: A temporary ban from any sort of interaction or public +communication with the community for a specified period of time. No public or +private interaction with the people involved, including unsolicited interaction +with those enforcing the Code of Conduct, is allowed during this period. +Violating these terms may lead to a permanent ban. + +### 4. Permanent Ban + +**Community Impact**: Demonstrating a pattern of violation of community +standards, including sustained inappropriate behavior, harassment of an +individual, or aggression toward or disparagement of classes of individuals. + +**Consequence**: A permanent ban from any sort of public interaction within the +community. + +## Attribution + +This Code of Conduct is adapted from the [Contributor Covenant][homepage], +version 2.1, available at +[https://www.contributor-covenant.org/version/2/1/code_of_conduct.html][v2.1]. + +Community Impact Guidelines were inspired by +[Mozilla's code of conduct enforcement ladder][Mozilla CoC]. + +For answers to common questions about this code of conduct, see the FAQ at +[https://www.contributor-covenant.org/faq][FAQ]. Translations are available at +[https://www.contributor-covenant.org/translations][translations]. + +[homepage]: https://www.contributor-covenant.org +[v2.1]: https://www.contributor-covenant.org/version/2/1/code_of_conduct.html +[Mozilla CoC]: https://github.com/mozilla/diversity +[FAQ]: https://www.contributor-covenant.org/faq +[translations]: https://www.contributor-covenant.org/translations + diff --git a/.github/FUNDING.yml b/.github/FUNDING.yml new file mode 100644 index 0000000..86dc0fe --- /dev/null +++ b/.github/FUNDING.yml @@ -0,0 +1,2 @@ +ko_fi: oliverbooth +custom: ['https://buymeacoffee.com/oliverbooth'] diff --git a/.github/workflows/dotnet.yml b/.github/workflows/dotnet.yml new file mode 100644 index 0000000..d42292f --- /dev/null +++ b/.github/workflows/dotnet.yml @@ -0,0 +1,53 @@ +name: .NET + +on: + push: + branches: + - '*' + - '*/*' + pull_request: + branches: + - '*' + - '*/*' + +jobs: + build: + name: "Build & Test" + runs-on: ubuntu-latest + + steps: + - name: Checkout + uses: actions/checkout@v3 + + - name: Setup .NET + uses: actions/setup-dotnet@v3 + with: + dotnet-version: | + 6.0.x + 7.0.x + 8.0.x + + - name: Add NuGet source + run: dotnet nuget add source --username oliverbooth --password ${{ secrets.GITHUB_TOKEN }} --store-password-in-clear-text --name github "https://nuget.pkg.github.com/oliverbooth/index.json" + + - name: Restore dependencies + run: dotnet restore + + - name: Build + run: dotnet build --no-restore --configuration Release + + - name: Test .NET 6 + run: dotnet test --no-build --verbosity normal --configuration Release --framework net6.0 --collect:"XPlat Code Coverage" --results-directory test-results/net6.0 + + - name: Test .NET 7 + run: dotnet test --no-build --verbosity normal --configuration Release --framework net7.0 --collect:"XPlat Code Coverage" --results-directory test-results/net7.0 + + - name: Test .NET 8 + run: dotnet test --no-build --verbosity normal --configuration Release --framework net8.0 --collect:"XPlat Code Coverage" --results-directory test-results/net8.0 + + - name: Upload coverage reports to Codecov + uses: codecov/codecov-action@v4.0.0 + with: + directory: test-results + token: ${{ secrets.CODECOV_TOKEN }} + slug: oliverbooth/X10D diff --git a/.gitignore b/.gitignore new file mode 100644 index 0000000..a437a65 --- /dev/null +++ b/.gitignore @@ -0,0 +1,37 @@ +*.swp +*.*~ +project.lock.json +.DS_Store +*.pyc +nupkg/ + +# Visual Studio Code +.vscode + +# Rider +.idea + +# User-specific files +*.suo +*.user +*.userosscache +*.sln.docstates + +# Build results +[Dd]ebug/ +[Dd]ebugPublic/ +[Rr]elease/ +[Rr]eleases/ +x64/ +x86/ +build/ +bld/ +[Bb]in/ +[Oo]bj/ +[Oo]ut/ +msbuild.log +msbuild.err +msbuild.wrn + +# Visual Studio 2015 +.vs/ diff --git a/FindDuplicates.sln b/FindDuplicates.sln new file mode 100644 index 0000000..56d446a --- /dev/null +++ b/FindDuplicates.sln @@ -0,0 +1,16 @@ + +Microsoft Visual Studio Solution File, Format Version 12.00 +Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "FindDuplicates", "FindDuplicates\FindDuplicates.csproj", "{543F0776-01CD-420B-973A-BCE11F47C758}" +EndProject +Global + GlobalSection(SolutionConfigurationPlatforms) = preSolution + Debug|Any CPU = Debug|Any CPU + Release|Any CPU = Release|Any CPU + EndGlobalSection + GlobalSection(ProjectConfigurationPlatforms) = postSolution + {543F0776-01CD-420B-973A-BCE11F47C758}.Debug|Any CPU.ActiveCfg = Debug|Any CPU + {543F0776-01CD-420B-973A-BCE11F47C758}.Debug|Any CPU.Build.0 = Debug|Any CPU + {543F0776-01CD-420B-973A-BCE11F47C758}.Release|Any CPU.ActiveCfg = Release|Any CPU + {543F0776-01CD-420B-973A-BCE11F47C758}.Release|Any CPU.Build.0 = Release|Any CPU + EndGlobalSection +EndGlobal diff --git a/FindDuplicates/FindDuplicates.csproj b/FindDuplicates/FindDuplicates.csproj new file mode 100644 index 0000000..cc24914 --- /dev/null +++ b/FindDuplicates/FindDuplicates.csproj @@ -0,0 +1,15 @@ + + + + Exe + net8.0 + enable + enable + fdup + + + + + + + diff --git a/FindDuplicates/ListCommand.cs b/FindDuplicates/ListCommand.cs new file mode 100644 index 0000000..01f1f38 --- /dev/null +++ b/FindDuplicates/ListCommand.cs @@ -0,0 +1,108 @@ +using System.Diagnostics; +using System.Security.Cryptography; +using System.Text; +using Spectre.Console; +using Spectre.Console.Cli; + +namespace FindDuplicates; + +internal sealed class ListCommand : AsyncCommand +{ + private readonly Dictionary> _fileHashMap = new(); + + public override async Task ExecuteAsync(CommandContext context, ListSettings settings) + { + var inputDirectory = new DirectoryInfo(settings.InputPath); + if (!inputDirectory.Exists) + { + AnsiConsole.MarkupLine($"[red]{inputDirectory} does not exist![/]"); + return -1; + } + + AnsiConsole.MarkupLineInterpolated($"Searching [cyan]{inputDirectory.FullName}[/]"); + AnsiConsole.MarkupLine($"Recursive mode is {(settings.Recursive ? "[green]ON" : "[red]OFF")}[/]"); + + await SearchAsync(inputDirectory, settings); + + AnsiConsole.WriteLine(); + + int duplicates = 0; + foreach ((string hash, List files) in _fileHashMap) + { + int fileCount = files.Count; + + if (fileCount > 1) + { + duplicates += fileCount; + AnsiConsole.MarkupLineInterpolated($"Found [cyan]{fileCount}[/] identical files"); + AnsiConsole.MarkupLineInterpolated($"SHA512 [green]{hash}[/]:"); + + foreach (FileInfo file in files) + AnsiConsole.MarkupLineInterpolated($"- {file.FullName}"); + + AnsiConsole.WriteLine(); + } + } + + if (duplicates == 0) + AnsiConsole.MarkupLine("[green]No duplicates found![/]"); + else + AnsiConsole.MarkupLineInterpolated($"[yellow]Found [cyan]{duplicates}[/] duplicates![/]"); + + return 0; + } + + private async Task SearchAsync(DirectoryInfo inputDirectory, ListSettings settings) + { + var tasks = new List(); + var directoryStack = new Stack([inputDirectory]); + while (directoryStack.Count > 0) + { + DirectoryInfo currentDirectory = directoryStack.Pop(); + string relativePath = Path.GetRelativePath(inputDirectory.FullName, currentDirectory.FullName); + if (relativePath != ".") + AnsiConsole.MarkupLineInterpolated($"Searching [cyan]{relativePath}[/]"); + + if (settings.Recursive) + { + foreach (DirectoryInfo childDirectory in currentDirectory.EnumerateDirectories()) + directoryStack.Push(childDirectory); + } + + foreach (FileInfo file in currentDirectory.EnumerateFiles()) + { + string relativeFilePath = Path.GetRelativePath(inputDirectory.FullName, file.FullName); + AnsiConsole.MarkupLineInterpolated($"Checking hash for [cyan]{relativeFilePath}[/]"); + tasks.Add(Task.Run(() => ProcessFile(file))); + } + } + + await Task.WhenAll(tasks); + } + + private void ProcessFile(FileInfo file) + { + Span buffer = stackalloc byte[64]; + using FileStream stream = file.OpenRead(); + using BufferedStream bufferedStream = new BufferedStream(stream, 1048576 /* 1MB */); + SHA512.HashData(bufferedStream, buffer); + string hash = ByteSpanToString(buffer); + Trace.WriteLine($"{file.FullName}: {hash}"); + + if (!_fileHashMap.TryGetValue(hash, out List? cache)) + _fileHashMap[hash] = cache = new List(); + + lock (cache) + cache.Add(file); + } + + private static string ByteSpanToString(ReadOnlySpan buffer) + { + var builder = new StringBuilder(); + + foreach (byte b in buffer) + builder.Append($"{b:X2}"); + + return builder.ToString(); + } +} diff --git a/FindDuplicates/ListSettings.cs b/FindDuplicates/ListSettings.cs new file mode 100644 index 0000000..fd9ba40 --- /dev/null +++ b/FindDuplicates/ListSettings.cs @@ -0,0 +1,16 @@ +using System.ComponentModel; +using Spectre.Console.Cli; + +namespace FindDuplicates; + +internal sealed class ListSettings : CommandSettings +{ + [CommandArgument(0, "")] + [Description("The path to search.")] + public string InputPath { get; set; } = string.Empty; + + [CommandOption("-r|--recursive")] + [Description("When this flag is set, the directory will be scanned recursively. This may take longer.")] + [DefaultValue(false)] + public bool Recursive { get; set; } = false; +} diff --git a/FindDuplicates/Program.cs b/FindDuplicates/Program.cs new file mode 100644 index 0000000..4b99467 --- /dev/null +++ b/FindDuplicates/Program.cs @@ -0,0 +1,5 @@ +using FindDuplicates; +using Spectre.Console.Cli; + +var app = new CommandApp(); +await app.RunAsync(args).ConfigureAwait(false); diff --git a/LICENSE.md b/LICENSE.md new file mode 100644 index 0000000..c63c8b3 --- /dev/null +++ b/LICENSE.md @@ -0,0 +1,21 @@ +MIT License + +Copyright (c) 2024 Oliver Booth + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. diff --git a/README.md b/README.md new file mode 100644 index 0000000..3bf8dc2 --- /dev/null +++ b/README.md @@ -0,0 +1,57 @@ +# Find Duplicates (fdup) + +![GitHub Actions Workflow Status](https://img.shields.io/github/actions/workflow/status/oliverbooth/fdup/dotnet.yml?style=flat-square) +![GitHub Issues or Pull Requests](https://img.shields.io/github/issues/oliverbooth/fdup?style=flat-square) +![GitHub License](https://img.shields.io/github/license/oliverbooth/fdup?style=flat-square) + +## About +fdup is a small command-line utility written in C# to quickly and easily find duplicate files. It can also search recursively to find duplicate files in child directories. + +## Usage +```bash +$ fdup --help +USAGE: + fdup [OPTIONS] + +ARGUMENTS: + The path to search + +OPTIONS: + -h, --help Prints help information + -v, --version Prints version information + -r, --recursive When this flag is set, the directory will be scanned recursively. This may take longer +``` + +## Example +```bash +$ echo "Hello World" > file1 +$ echo "Goodbye World" > file2 +$ fdup . +Searching /home/user/example +Recursive mode is OFF +Checking hash for file2 +Checking hash for file1 + +No duplicates found! +$ echo "Hello World" > file2 +$ fdup . +Searching /home/user/example +Recursive mode is OFF +Checking hash for file2 +Checking hash for file1 + +Found 2 identical files +SHA512 E1C112FF908FEBC3B98B1693A6CD3564EAF8E5E6CA629D084D9F0EBA99247CACDD72E369FF8941397C2807409FF66BE64BE908DA17AD7B8A49A2A26C0E8086AA: +- /home/user/example/file1 +- /home/user/example/file2 + +Found 2 duplicates! +``` + +## Contributing + +Contributions are welcome. See [CONTRIBUTING.md](CONTRIBUTING.md). + +## License + +X10D is released under the MIT License. See [here](https://github.com/oliverbooth/X10D/blob/main/LICENSE.md) for more details. diff --git a/global.json b/global.json new file mode 100644 index 0000000..2ddda36 --- /dev/null +++ b/global.json @@ -0,0 +1,7 @@ +{ + "sdk": { + "version": "8.0.0", + "rollForward": "latestMinor", + "allowPrerelease": false + } +} \ No newline at end of file