在2个文件的字节数组中相交和并集

Question

I have 2 files. 1 is Source File and 2nd is Destination file.

Below is my code for Intersect and Union two file using byte array.

FileStream frsrc = new FileStream("Src.bin", FileMode.Open);
FileStream frdes = new FileStream("Des.bin", FileMode.Open);
int length = 24; // get file length
byte[] src = new byte[length];
byte[] des = new byte[length]; // create buffer
int Counter = 0;   // actual number of bytes read
int subcount = 0;

while (frsrc.Read(src, 0, length) > 0)
{
    try
    {
        Counter = 0;
        frdes.Position = subcount * length;
        while (frdes.Read(des, 0, length) > 0)
        {                               
                var  data = src.Intersect(des);                          
                var data1 = src.Union(des);                               
                Counter++;                               
        }        
        subcount++;
        Console.WriteLine(subcount.ToString());
        }
    }
    catch (Exception ex)
    {                          
    }
}

It is works fine with fastest speed. but Now the problem is that I want count of it and when I Use below code then it becomes very slow.

  var  data = src.Intersect(des).Count();                          
  var  data1 = src.Union(des).Count();

So, Is there any solution for that ? If yes,then please lete me know as soon as possible. Thanks

Answer 1

Intersect and Union are not the fastest operations. The reason you see it being fast is that you never actually enumerate the results!

Both return an enumerable, not the actual results of the operation. You're supposed to go through that and enumerate the enumerable, otherwise nothing happens - this is called "deferred execution". Now, when you do Count , you actually enumerate the enumerable, and incur the full cost of the Intersect and Union - believe me, the Count itself is relatively trivial (though still an O(n) operation!).

You'll need to make your own methods, most likely. You want to avoid the enumerable overhead, and more importantly, you'll probably want a lookup table.

Answer 2

A few points: the comment // get file length is misleading as it is the buffer size. Counter is not the number of bytes read, it is the number of blocks read. data and data1 will end up with the result of the last block read, ignoring any data before them. That is assuming that nothing goes wrong in the while loop - you need to remove the try structure to see if there are any errors.

What you can do is count the number of occurences of each byte in each file, then if the count of a byte in any file is greater than one then it is is a member of the intersection of the files, and if the count of a byte in all the files is greater than one then it is a member of the union of the files.

It is just as easy to write the code for more than two files as it is for two files, whereas LINQ is easy for two but a little bit more fiddly for more than two. (I put in a comparison with using LINQ in a naïve fashion for only two files at the end.)

using System;
using System.Collections.Generic;
using System.IO;
using System.Linq;

namespace ConsoleApplication1
{
    class Program
    {
        static void Main(string[] args)
        {

            var file1 = @"C:\Program Files (x86)\Electronic Arts\Crysis 3\Bin32\Crysis3.exe"; // 26MB
            var file2 = @"C:\Program Files (x86)\Electronic Arts\Crysis 3\Bin32\d3dcompiler_46.dll"; // 3MB
            List<string> files = new List<string> { file1, file2 };

            var sw = System.Diagnostics.Stopwatch.StartNew();

            // Prepare array of counters for the bytes
            var nFiles = files.Count;
            int[][] count = new int[nFiles][];
            for (int i = 0; i < nFiles; i++)
            {
                count[i] = new int[256];
            }

            // Get the counts of bytes in each file
            int bufLen = 32768;
            byte[] buffer = new byte[bufLen];

            int bytesRead;

            for (int fileNum = 0; fileNum < nFiles; fileNum++)
            {
                using (var sr = new FileStream(files[fileNum], FileMode.Open, FileAccess.Read))
                {
                    bytesRead = bufLen;
                    while (bytesRead > 0)
                    {
                        bytesRead = sr.Read(buffer, 0, bufLen);
                        for (int i = 0; i < bytesRead; i++)
                        {
                            count[fileNum][buffer[i]]++;
                        }
                    }
                }
            }

            // Find which bytes are in any of the files or in all the files
            var inAny = new List<byte>(); // union
            var inAll = new List<byte>(); // intersect

            for (int i = 0; i < 256; i++)
            {
                Boolean all = true;
                for (int fileNum = 0; fileNum < nFiles; fileNum++)
                {
                    if (count[fileNum][i] > 0)
                    {
                        if (!inAny.Contains((byte)i)) // avoid adding same value more than once
                        {
                            inAny.Add((byte)i);
                        }
                    }
                    else
                    {
                        all = false;
                    }
                };

                if (all)
                {
                    inAll.Add((byte)i);
                };

            }

            sw.Stop();

            Console.WriteLine(sw.ElapsedMilliseconds);

            // Display the results
            Console.WriteLine("Union: " + string.Join(",", inAny.Select(x => x.ToString("X2"))));
            Console.WriteLine();
            Console.WriteLine("Intersect: " + string.Join(",", inAll.Select(x => x.ToString("X2"))));
            Console.WriteLine();

            // Compare to using LINQ.
            // N/B. Will need adjustments for more than two files.

            var srcBytes1 = File.ReadAllBytes(file1);
            var srcBytes2 = File.ReadAllBytes(file2);

            sw.Restart();

            var intersect = srcBytes1.Intersect(srcBytes2).ToArray().OrderBy(x => x);
            var union = srcBytes1.Union(srcBytes2).ToArray().OrderBy(x => x);

            Console.WriteLine(sw.ElapsedMilliseconds);

            Console.WriteLine("Union: " + String.Join(",", union.Select(x => x.ToString("X2"))));
            Console.WriteLine();
            Console.WriteLine("Intersect: " + String.Join(",", intersect.Select(x => x.ToString("X2"))));

            Console.ReadLine();

        }
    }
}

The counting-the-byte-occurences method is roughly five times faster than the LINQ method on my computer, even without the latter loading the files and on a range of file sizes (a few KB to a few MB).