Dear All,
I want to upload huge amount of data from gridview to database but fast way i already did in below code but its taking time like 300 rows data taking 5 to 6 mints. please need your supports how can i upload fastest to database from girdivew using c#.
below are my codes which i am using i used for loop and foreach both are same taking times.
cs.query = @"insert into Table (Account,Currency,Posted,Posted_Hijrah,Value_date,Value_date_Hijrah,
Debit_amount,Credit_amount,Running_balance,Reference,Description,Narrative_1,Narrative_2,Narrative_3,Narrative_4,created_date,
created_by,status)
Values (@Account,@Currency,@Posted,@Posted_Hijrah,@Value_date,@Value_date_Hijrah,
@Debit_amount,@Credit_amount,@Running_balance,@Reference,@Description,@Narrative_1,@Narrative_2,@Narrative_3,@Narrative_4,@created_date,
@created_by,@status)";
cs.cmd = new SqlCommand(cs.query, cs.conn);
cs.cmd.Parameters.AddWithValue("@Account", row.Cells[0].Value.ToString().Trim());
cs.cmd.Parameters.AddWithValue("@Currency", row.Cells[1].Value.ToString().Trim());
cs.cmd.Parameters.AddWithValue("@Posted", row.Cells[2].Value.ToString().Trim());
cs.cmd.Parameters.AddWithValue("@Posted_Hijrah", row.Cells[3].Value.ToString().Trim());
cs.cmd.Parameters.AddWithValue("@Value_date", row.Cells[4].Value.ToString().Trim());
cs.cmd.Parameters.AddWithValue("@Value_date_Hijrah", row.Cells[5].Value.ToString().Trim());
cs.cmd.Parameters.AddWithValue("@Debit_amount", row.Cells[6].Value.ToString().Trim());
cs.cmd.Parameters.AddWithValue("@Credit_amount", row.Cells[7].Value.ToString().Trim());
cs.cmd.Parameters.AddWithValue("@Running_balance", row.Cells[8].Value.ToString().Trim());
cs.cmd.Parameters.AddWithValue("@Reference", row.Cells[9].Value.ToString().Trim());
cs.cmd.Parameters.AddWithValue("@Description", row.Cells[10].Value.ToString().Trim());
cs.cmd.Parameters.AddWithValue("@Narrative_1", row.Cells[11].Value.ToString().Trim());
cs.cmd.Parameters.AddWithValue("@Narrative_2", row.Cells[12].Value.ToString().Trim());
cs.cmd.Parameters.AddWithValue("@Narrative_3", row.Cells[13].Value.ToString().Trim());
cs.cmd.Parameters.AddWithValue("@Narrative_4", row.Cells[14].Value.ToString().Trim());
cs.cmd.Parameters.AddWithValue("@created_date", System.DateTime.Now.ToString("dd-MMM-yyyy"));
cs.cmd.Parameters.AddWithValue("@created_by", System.Net.Dns.GetHostName());
cs.cmd.Parameters.AddWithValue("@status", "Yes");
cs.cmd.ExecuteNonQuery();
Saravanan GanesanPosted Aug 14, 2023, 8:58 AM
Hi Feroz Khan ,
It looks like you are inserting data from a
GridViewinto a database table using C# and SQL Server. The code you provided seems to be using parameterized queries, which is a good practice for security and performance.However, there are a few optimizations you can consider to potentially improve the performance of your data insertion process:
Batch Insertion: Instead of inserting each row individually, you can perform batch insertions. Collect a certain number of rows and insert them together in a single query. This reduces the overhead of opening and closing connections for each row.
Use
SqlBulkCopy: If you are dealing with a large amount of data, using theSqlBulkCopyclass can be significantly faster than individual insertions. It is optimized for bulk insert operations.Open and Close Connections Smartly: Opening and closing connections for each row can be resource-intensive. Open the database connection before starting the insertion process and close it only after all the data has been inserted.
Find the code snippet below :
// Assuming cs.conn is your SqlConnection object
cs.conn.Open();
using (SqlBulkCopy bulkCopy = new SqlBulkCopy(cs.conn))
{
bulkCopy.DestinationTableName = "Table"; // Set your destination table name
DataTable dataTable = new DataTable();
// Add DataColumn objects to dataTable.Columns for each column in your GridView
foreach (GridViewRow row in yourGridView.Rows)
{
// Populate the dataTable.Rows with data from the GridViewRow
// Make sure the order of data matches the column order in the destination table
}
bulkCopy.WriteToServer(dataTable);
}
cs.conn.Close();
Thank you .
Feroz KhanPosted Aug 16, 2023, 4:01 PM
Sorry Guys i did not shared my complete code with below are my codes. now i am facing another issue null refrence because i am using my code in backgroundWorker1_DoWork and issue is same maybe alot of calculation and contains thats why also slow.
i am reading multiple excel files from one of location and extracting data from excel to database but did some split as well to take some values. its runing but some time showing error like null refrence.
private void backgroundWorker1_DoWork(object sender, System.ComponentModel.DoWorkEventArgs e)
{
try
{
for (int i = 0; i < listBox1.Items.Count; i++)
{
dataGridView1.DataSource = null;
string filepath = listBox1.Items[i].ToString();
string filename = Path.GetFileName(filepath).ToString();
string con = "Provider=Microsoft.ACE.OLEDB.12.0;Data Source={0}; Extended Properties='Excel 8.0;HDR={1}'";
con = string.Format(con, filepath, "yes");
OleDbConnection Econ = new OleDbConnection(con);
Econ.Open();
DataTable Edt = Econ.GetOleDbSchemaTable(OleDbSchemaGuid.Tables, null);
string excelsheet = Edt.Rows[0]["Table_Name"].ToString();
OleDbCommand Ecmd = new OleDbCommand("Select * from [" + excelsheet + "]", Econ);
OleDbDataAdapter oda = new OleDbDataAdapter(Ecmd);
DataTable dt = new DataTable();
oda.Fill(dt);
Econ.Close();
Control.CheckForIllegalCrossThreadCalls = false;
dataGridView1.DataSource = dt;
TXT_Count.Text = dataGridView1.Rows.Count.ToString();
if (cs.conn.State == ConnectionState.Closed)
{
cs.conn.Open();
}
foreach (DataGridViewRow row in dataGridView1.Rows)
{
try
{
cs.query = @"insert into Bank_Statement (Account,Currency,Posted,Posted_Hijrah,Value_date,Value_date_Hijrah,
Debit_amount,Credit_amount,Running_balance,Reference,Description,Narrative_1,Narrative_2,Narrative_3,Narrative_4,created_date,
created_by,status,filepath,Channel,TranDate,GrandValue,Feesvalue,PDGNumber,Branch)
Values (@Account,@Currency,@Posted,@Posted_Hijrah,@Value_date,@Value_date_Hijrah,
@Debit_amount,@Credit_amount,@Running_balance,@Reference,@Description,@Narrative_1,@Narrative_2,@Narrative_3,@Narrative_4,@created_date,
@created_by,@status,@filepath, @Channel, @TranDate, @GrandValue, @FeesValue, @PDQNumber, @Branch)";
cs.cmd = new SqlCommand(cs.query, cs.conn);
cs.cmd.Parameters.AddWithValue("@Account", row.Cells[0].Value.ToString().Trim());
cs.cmd.Parameters.AddWithValue("@Currency", row.Cells[1].Value.ToString().Trim());
cs.cmd.Parameters.AddWithValue("@Posted", row.Cells[2].Value.ToString().Trim());
cs.cmd.Parameters.AddWithValue("@Posted_Hijrah", row.Cells[3].Value.ToString().Trim());
cs.cmd.Parameters.AddWithValue("@Value_date", row.Cells[4].Value.ToString().Trim());
cs.cmd.Parameters.AddWithValue("@Value_date_Hijrah", row.Cells[5].Value.ToString().Trim());
cs.cmd.Parameters.AddWithValue("@Debit_amount", row.Cells[6].Value.ToString().Trim());
cs.cmd.Parameters.AddWithValue("@Credit_amount", row.Cells[7].Value.ToString().Trim());
cs.cmd.Parameters.AddWithValue("@Running_balance", row.Cells[8].Value.ToString().Trim());
cs.cmd.Parameters.AddWithValue("@Reference", row.Cells[9].Value.ToString().Trim());
cs.cmd.Parameters.AddWithValue("@Description", row.Cells[10].Value.ToString().Trim());
cs.cmd.Parameters.AddWithValue("@Narrative_1", row.Cells[11].Value.ToString());
cs.cmd.Parameters.AddWithValue("@Narrative_2", row.Cells[12].Value.ToString().Trim());
cs.cmd.Parameters.AddWithValue("@Narrative_3", row.Cells[13].Value.ToString().Trim());
cs.cmd.Parameters.AddWithValue("@Narrative_4", row.Cells[14].Value.ToString().Trim());
cs.cmd.Parameters.AddWithValue("@created_date", System.DateTime.Now.ToString("dd-MMM-yyyy"));
cs.cmd.Parameters.AddWithValue("@created_by", System.Net.Dns.GetHostName());
cs.cmd.Parameters.AddWithValue("@status", "Yes");
cs.cmd.Parameters.AddWithValue("@filepath", filepath.ToString());
//--------------NAR#1----Channel-----------------------------------------
string nar1T = row.Cells[11].Value.ToString().Trim();
string Ref = row.Cells[9].Value.ToString().Trim();
string nar3 = row.Cells[13].Value.ToString().Trim();
string desc = row.Cells[10].Value.ToString().Trim();
bool Mada = nar1T.Contains("???");
bool Visa = nar1T.Contains("????");
bool Master = nar1T.Contains("?????");
bool Amex = nar3.Contains("Amex");
//bool Modern = nar1T.Contains("Modern");
//bool Local = nar1T.Contains("Local");
// bool Blank = string.IsNullOrWhiteSpace(nar1T);
// bool VAT = nar1T.Contains("VAT");
// bool SAR = nar1T.Contains("SAR");
string GCC = Ref.Substring(0, 1);
bool GCCRef = GCC.Contains("G");
bool des = desc.Contains("Credit");
if (Mada == true)
{
cs.cmd.Parameters.AddWithValue("@Channel", "Mada");
}
else if (Visa == true)
{
cs.cmd.Parameters.AddWithValue("@Channel", "Visa");
}
else if (Master == true)
{
cs.cmd.Parameters.AddWithValue("@Channel", "Master");
}
else if (GCCRef == true && des == true)
{
cs.cmd.Parameters.AddWithValue("@Channel", "GCC");
}
else if (Amex == true)
{
cs.cmd.Parameters.AddWithValue("@Channel", "Amex");
}
else
{
cs.cmd.Parameters.AddWithValue("@Channel", row.Cells[11].Value.ToString().Trim());
}
//------------------NAR#3------GrandValue----------------------------------
string[] nara3;
nar3 = row.Cells[13].Value.ToString().Trim();
// string[] nara3 = nar3.Split(' ');
bool Value = nar3.Contains("???");
//bool Amex3 = nar3.Contains("Amex");
//bool Blank3 = string.IsNullOrWhiteSpace(nar3);
if (Value == true)
{
nara3 = nar3.Split(' ');
int index = 4;
if (index > nara3.Length)
{
cs.cmd.Parameters.AddWithValue("@GrandValue", row.Cells[13].Value.ToString().Trim());
cs.cmd.Parameters.AddWithValue("@FeesValue", "0").ToString().Trim();
}
else
{
nar3 = nara3[4].ToString().Trim();
cs.cmd.Parameters.AddWithValue("@GrandValue", nar3.ToString().Trim());
//------------------------@FeesValue----------------------------------
if (nar3 != "")
{
decimal Grand = Convert.ToDecimal(nar3);
decimal CrediAmnt = Convert.ToDecimal(row.Cells[7].Value.ToString().Trim());
decimal feescal = Convert.ToDecimal(1.15);
decimal feesresult = (Grand - CrediAmnt) / feescal;
cs.cmd.Parameters.AddWithValue("@FeesValue", feesresult.ToString().Trim());
}
else
{
cs.cmd.Parameters.AddWithValue("@FeesValue", "0").ToString().Trim();
}
}
}
else
{
cs.cmd.Parameters.AddWithValue("@GrandValue", row.Cells[13].Value.ToString().Trim());
cs.cmd.Parameters.AddWithValue("@FeesValue", "0").ToString().Trim();
}
//------------------------TranDate----------------------------------
bool trandate = nar1T.Contains("??????");
if (trandate == true)
{
string[] trand = nar1T.Split(' ');
string trandD = trand[1];
DateTime Dtime = DateTime.Parse(trandD);
bool time12 = nar1T.Contains("12,");
bool time01 = nar1T.Contains("01,");
bool time02 = nar1T.Contains("02,");
string finald;
if (time12 == true || time01 == true || time02 == true)
{
DateTime DfinalDate = Dtime.AddDays(-1);
finald = DfinalDate.ToString("dd-MMM-yyyy");
//Dtime.ToString("dd-MMM-yyyy");
//string finald = Convert.ToString(DfinalDate);
// finald.ToString();
// int indexT = 4;
cs.cmd.Parameters.AddWithValue("@TranDate", finald.ToString().Trim());
}
else
{
finald = Dtime.ToString("dd-MMM-yyyy");
// int indexT = 4;
cs.cmd.Parameters.AddWithValue("@TranDate", finald.ToString().Trim());
}
}
else
{
cs.cmd.Parameters.AddWithValue("@TranDate", "None");
}
//------------------------@PDQNumber----------------------------------
string PDQ = row.Cells[12].Value.ToString().Trim();
bool pdqno = PDQ.Contains("?????");
// bool BlankPDQ = string.IsNullOrWhiteSpace(PDQ);
//bool SARPDQ = PDQ.Contains("SAR");
if (pdqno == true)
{
string[] PDQA = PDQ.Split(' ');
string PDQN = PDQA[4];
cs.cmd.Parameters.AddWithValue("@PDQNumber", PDQN).ToString().Trim();
// cs.cmd.Parameters.AddWithValue("@POSNumber", PDQN).ToString().Trim();
//------------------------@Branch----------------------------------
cs.query = @"select branch, Account_Number from PDQ_Details where Account_Number = '" + row.Cells[0].Value.ToString().Trim() + "'";
cs.sda = new SqlDataAdapter(cs.query, cs.conn);
cs.dt = new DataTable();
cs.sda.Fill(cs.dt);
if (cs.dt.Rows.Count > 0)
{
cs.cmd.Parameters.AddWithValue("@Branch", cs.dt.Rows[0][0].ToString().Trim());
}
else
{
cs.cmd.Parameters.AddWithValue("@Branch", "Branch Name not Found..").ToString().Trim();
}
}
else
{
cs.query = @"select branch, Account_Number from PDQ_Details where Account_Number = '" + row.Cells[0].Value.ToString().Trim() + "'";
cs.sda = new SqlDataAdapter(cs.query, cs.conn);
cs.dt = new DataTable();
cs.sda.Fill(cs.dt);
cs.cmd.Parameters.AddWithValue("@PDQNumber", row.Cells[12].Value).ToString().Trim();
cs.cmd.Parameters.AddWithValue("@Branch", cs.dt.Rows[0][0].ToString().Trim());
}
cs.cmd.ExecuteNonQuery();
}
catch (Exception ex)
{
//MessageBox.Show(ex.Message, "File not correct format", MessageBoxButtons.OK, MessageBoxIcon.Error);
MessageBox.Show(ex.Message + " - " + filename, "Wrong File..", MessageBoxButtons.OK, MessageBoxIcon.Error);
if (this.backgroundWorker1.IsBusy)
{
e.Cancel = true;
return;
}
Btn_Import.PerformClick();
}
}
// MessageBox.Show("All Files Successfully imported...", "Information", MessageBoxButtons.OK, MessageBoxIcon.Information);
System.IO.File.Move(filepath, filepath + ".done");
string selectPath = "select count(filepath) from Bank_Statement where filepath = '" + filepath.ToString() + "'";
SqlDataAdapter sda = new SqlDataAdapter(selectPath, cs.conn);
DataTable dtt = new DataTable();
sda.Fill(dtt);
if (!File.Exists(filename + ".txt"))
{
using (FileStream fs = File.Create(filepath + ".txt"))
{
// Add some text to file
Byte[] title = new UTF8Encoding(true).GetBytes("Logs:");
fs.Write(title, 0, title.Length);
byte[] author = new UTF8Encoding(true).GetBytes("Excel File Count: " + TXT_Count.Text + " - Database Count: " + dtt.Rows[0][0].ToString());
fs.Write(author, 0, author.Length);
}
}
else
{
}
if (cs.conn.State == ConnectionState.Open)
{
cs.conn.Close();
}
}
}
catch (Exception)
{
//MessageBox.Show(ex.Message,"Error", MessageBoxButtons.OK, MessageBoxIcon.Error);
//if (this.backgroundWorker1.IsBusy)
//{
// e.Cancel = true;
// return;
//}
//Btn_Import.PerformClick();
}
}
Tahir AnsariPosted Aug 14, 2023, 9:09 AM
1. Batch Insertion:
Instead of inserting rows one by one in a loop, consider using batch insertion. You can prepare a batch of records to be inserted and then use a single INSERT statement to insert them all at once. This can significantly reduce the overhead of sending multiple individual queries to the database.
2. Parameterized Queries:
You're already using parameterized queries, which is great for security and performance. However, you can further improve performance by preparing the parameterized query outside the loop, so you only need to prepare it once. Then, within the loop, you just update the parameter values and execute the query.
3. Transaction Management:
Use transactions to group multiple insert statements into a single transaction. This can improve performance and ensure data consistency. Begin a transaction before your loop and commit it after the loop completes successfully. If an error occurs during the loop, you can roll back the transaction to avoid partial data insertion.