Fast copying of data to a file

Question

hello,

        i wish to know if there is a way to copy data to a file faster then the following approach used below ? What i am doing is to write matrix/array data to a .cvs file. there are on an average 150,000 elements in the array/matrix with each element having three double data type members.The code is here: 


           var path = Path.Combine(myData, "KinectData-" + time + ".txt");

         

            // start writing the data to file

            for (int i = 0; i < maxLen; i++)

            {

                System.IO.File.AppendAllText(@path, Convert.ToString(p1[i].X) + " ," +                         Convert.ToString(p1[i].Y) + " ," + Convert.ToString(p1[i].Z) +                                                                                                 Environment.NewLine);

                System.IO.File.AppendAllText(@path, Convert.ToString(p2[i].X) + " ," +                 Convert.ToString(p2[i].Y) + " ," + Convert.ToString(p2[i].Z) +                                                                                                 Environment.NewLine);

           
             }

The time taken for this process is close to 3 min which i wish to minimize.any solution will be  of great value to me.

Thank you

Answer 1

I'd see if this is any faster:

var path = Path.Combine(myData, "KinectData-" + time + ".txt");

         

// start writing the data to file

int batchSize = 1000;

int numBatches = maxLen/batchSize;

int remainder = maxLen % batchSize;

if (remainder > 0) ++numBatches;

var sb = new System.Text.StringBuilder(132 * batchSize);

using(var sw = new System.IO.StreamWriter(path, true))

{

   for(int b = 0; b < numBatches; b++)

   {  

      if( (b == numBatches - 1) && remainder > 0) batchSize = remainder;  
 

      for (int j = 0; j < batchSize; j++)

      {

         int i = b * 1000 + j;

         sb.Append(p1[i].X);

         sb.Append(" ,"); 

         sb.Append(p1[i].Y);

         sb.Append(" ,");

         sb.Append(p1[i].Z);

         sb.AppendLine();

         sb.Append(p2[i].X);

         sb.Append(" ,"); 

         sb.Append(p2[i].Y);

         sb.Append(" ,");

         sb.Append(p2[i].Z);

         sb.AppendLine();    

      }

      sw.Write(sb.ToString());

      if (b < numBatches - 1) sb.Length = 0;

   }

}

Answer 2

There were two points which struck me about your original code:

1. You were doing a lot of string concatenations.

2. You were appending to the file after every iteration and - worse still - opening and closing it each time.

The first problem can be solved by using a StringBuilder and the second by only appending to the file after (say) every 1000 iterations and by opening and closing the file just the once.

Also, if we make the capacity of the StringBuilder big enough then it won't need to keep increasing the size of its internal buffer which slows things down. So, I figured a maximum of 20 characters for each  of the 6 doubles, 8 characters for the spaces/commas and 4 characters for the new lines (\r\n), making 132 in total, multiply by 1000 to give 132,000.

Also, if you just clear the StringBuilder after each batch (by setting its Length to zero), one instance of it will suffice :)

Answer 3

wonderful! thanks for the quick answers Vulpes.

Answer 4

hello Vulpes. I am sorry for the last comment here. the method you have suggested here works perfectly fine and the results are amazing. it took close to 1 sec to do the same task ! Thanks for the wonderful technique. I would like to know why i was taking so long for doing the same task? Is it the no of times write is done which made the difference ? thanks

Answer 5

hello Vulpes. i tried this method.A file does get created but the data it contain is either NaN OR O only like this
NaN,NaN,NaN
0,0,0
NaN,NaN,NaN
0,0,0
NaN,NaN,NaN
0,0,0 N
aN,NaN,NaN
0,0,0 N
aN,NaN,NaN
0,0,0
NaN,NaN,NaN
0,0,0


      and so on.