Hi Folks,
PFA.
I have a task to push 2 lakhs records to AMAZON storage.
I have no problem when I push the record one by one to Amazon storage.
To speed up the process, have wrote threading code to push multiple record parallel.
I have created 2 thread & run the window app, after pushing 4 records its throw an "Out of memory exception".
In the code;
1) I kept the 2 lakhs records in List collection.
2) Pass 1 lakh record to each thread.
3) PFA
Its throw an exception when it reach the below line
documentInfoList = documentClient.GetDocumentsInfo(applicationId, arrayOfFileName);
Line No 247 on FileUpload.cs class.
Kindly help me.
Thnaks & Regards,
Murugan Krishnan
Loading
Dhirendra MisraPosted Dec 26, 2013, 11:16 PM
Each new thread or fiber receives its own stack space consisting of both reserved and initially committed memory. The default stack reservation size used by the linker is 1 MB. Hence a thread uses 1 MB of memory.
More details can be found at the following link.
http://msdn.microsoft.com/en-us/library/windows/desktop/ms686774(v=vs.85).aspx
You should need to use TPL instead.
http://msdn.microsoft.com/en-us/library/dd537609(v=vs.110).aspx.
OR
You may go with approach suggested by Robbinson.
Thanks
RobbinsonPosted Dec 26, 2013, 7:35 AM
Uploading large data over internet may not be good idea. it may also impact the performance.
I suggest you to divide data in chunks and upload it on the amazon storage, it will also helps you improve the performance and get rid of such errors.
Lets divide data in optimum size based on the complexity say, 1000 records at a time.
Hope, this will help you out. Feel free to connect in case of query.
Please don't forget to mark as answer if this post helps you.