My requirement is like this. I will be retrieving lets say around 1million or more records using ADO.NET into my database. In case there is some failure in the data retrieval, how do i handle this scenario? For eg. is there a way that I can bring the data till the failure occured, for eg, say 0.1 million records.. and resume the action once again from the point where it got failed.
Regards
Raja
Loading
Andrew FensterPosted Jan 18, 2010, 2:12 AM
(1) You could use a DataReader and process records one at a time:
while (reader.Read( ))
{
save the current row
}
This makes it easy to pick up where you left off. There may be performance problems. You would have to test this.
(2) You could use a DataAdapter and make several calls. In each call you get the next 100,000 records and move it to your new database. If you blow up, you pick up again where you left off.
raja ramanPosted Jan 20, 2010, 10:11 AM
Andrew FensterPosted Jan 20, 2010, 7:58 AM
If you have two SQL Server databases running on the same server, you can write a stored procedure where one database selects data from the other database. I don't remember how to do it. I haven't got time to look it up right now, but it's possible. I believe it's possible in DB2 as well. I don't know about other databases. Doing it this way may have performance benefits.
As with ADO.net, you might want to write a stored procedure that gets the data a bit at a time. For example, it might keep selecting 5000 rows at a time until there are no more rows.
As an aside, it still seems unusual to have to move so much data on the fly. It should be avoided altogether if possible.
raja ramanPosted Jan 18, 2010, 9:41 PM
Andrew FensterPosted Jan 18, 2010, 12:29 PM
Your real issue here is that you're trying to move a HUGE amount of data on the fly. A million records is a lot of data to copy. It's processor intensive. It could cause transaction timeouts if you're not careful. It makes sense to break it into several smaller updates.
raja ramanPosted Jan 18, 2010, 10:51 AM
raja ramanPosted Jan 17, 2010, 9:24 PM
Now let me put this as a combination of 2 small processes..
1. I will intermediately save the 0.1 million (thats till my failure point)
2. I will restart my process from the 0.1 million instead of the entire contents..
My question is more related to point 1, how will i code in such a manner so that the intermediate data gets saved.
Sam HobbsPosted Jan 17, 2010, 12:07 PM
You can get the maximum value for the key in the output table, then select all records in the input table with a key greater than that value.
raja ramanPosted Jan 17, 2010, 8:24 AM
If data retrieval from Table A fails or the whole process stops at 0.1 million at the input stage, What will be my possible alternatives.
1. Import Table A again into my app DB (unnecessary overhead)
2. My app saves till 0.1 million (i.e) till the failure occured. Then I will import Table A again but this time i will move the rest of the 0.9 million rows only instead of the entire 1 million.
I like option 2, and hence my question is how to code in such a manner that the data till the failure point is saved in my App DB.
Sam HobbsPosted Jan 16, 2010, 10:57 AM
raja ramanPosted Jan 15, 2010, 11:27 PM
To make my question more clear,
I am using ado.net data adapter to fill out a dataset lets say from table A, which contains 1 million rows. Once it fills out, I am saving this initial copy somewhere in my app database. If some failure occurs at around 0.1 million records, how do i move the 0.1 million rows that has already been transferred to the dataset to move into my app database, so that i can resume from the point it failed.
Regards
Raja
Sam HobbsPosted Jan 15, 2010, 10:52 PM