how to find duplicate text exist in shared path .txt file

you have a large text file (~5.4 million lines) in a shared path, and you want to find duplicate records (lines) and their counts using C#.NET, while keeping performance in mind.

Key Considerations

• File size: 5.4 million lines can be hundreds of MBs ? must avoid loading entire file into memory at once.

• Performance: Use streaming (line by line) instead of File.ReadAllLines().

• Data structure: Use a Dictionary or ConcurrentDictionary to count occurrences.

• Duplicates: After processing, filter entries where count > 1.

Efficient C# Example

using System;
using System.Collections.Generic;
using System.IO;

class Program
{
    static void Main()
    {
        string filePath = @"\\sharedpath\yourfile.txt"; // shared path

        var counts = new Dictionary(StringComparer.OrdinalIgnoreCase);

        // Read line by line (streaming)
        using (var reader = new StreamReader(filePath))
        {
            string line;
            while ((line = reader.ReadLine()) != null)
            {
                if (counts.ContainsKey(line))
                    counts[line]++;
                else
                    counts[line] = 1;
            }
        }

        // Find duplicates
        foreach (var kvp in counts)
        {
            if (kvp.Value > 1)
            {
                Console.WriteLine($"Text: {kvp.Key}, Count: {kvp.Value}");
            }
        }
    }
}

Performance Tips

• Streaming: StreamReader.ReadLine() avoids loading the whole file into memory.

• Dictionary: O(1) average lookup ? efficient for millions of records.

• Case sensitivity: Use StringComparer.OrdinalIgnoreCase if you want "Anand" and "anand" treated as duplicates.

• Memory: Each unique line is stored once. If lines are very long or unique, memory usage can grow.

  • If memory is a concern, consider writing counts to a temporary database (SQL Server, SQLite) instead of keeping everything in memory.

Example Output

If file contains:

anand
kumar
anand
raj

Output:

Text: anand, Count: 2

Alternative Approaches

• LINQ (simpler, less memory efficient):

  var duplicates = File.ReadLines(filePath)
                       .GroupBy(line => line)
                       .Where(g => g.Count() > 1)
                       .Select(g => new { Text = g.Key, Count = g.Count() });
  

  Loads more into memory, not ideal for 5.4M records.

• Database approach: Bulk insert into a table, then run:

  SELECT Text, COUNT(*) 
  FROM Records 
  GROUP BY Text 
  HAVING COUNT(*) > 1;
  

  Handles huge files efficiently.

Hello Anand

Easiest way to use LINQ

string[] lines = File.ReadAllLines(path);
lines = lines.GroupBy(x => x).Where(g => g.Count() > 1).Select(g => g.Key).ToArray();
File.WriteAllLines(newPath, lines):

try to use the HashSet approach which overcomes the performance issue. Please find the reference link 

https://stackoverflow.com/questions/43016773/how-to-remove-duplicate-lines-from-a-large-text-file-efficiently