Read directory of files and write data into single file.

Hi Team,

Goal:

Read 6,00,000 xml files and write into text file.

here is my code

List<string> xmlFiles = new List<string>(Directory.GetFiles(sourceFolder, "*.xml", SearchOption.AllDirectories));
LoadReadWriteSettings(); // loading xml reder ,writer settings and xsl load settings.
foreach (string file in xmlFiles)
{
if (IsValidFile(file))
{
Task task = Task.Run(() => ReadXml(file));
tasks.Add(task);
if (tasks.Count == 200)
{
Task.WaitAll(tasks.ToArray());
tasks.Clear();
LogData(content, outputFlie);// after reading 200 files, writing into text file.
content = new StringBuilder();
}
}
}
internal async void ReadXml(string xmlFile)
{
try
{
StringBuilder lines = new StringBuilder();
XmlReader reader = System.Xml.XmlReader.Create(xmlFile, readerSettings);
XmlWriter results = XmlWriter.Create((TextWriter)new StringWriter(lines), writerSettings);
transform.Transform(reader, null, results);
if (String.IsNullOrWhiteSpace(lines.ToString()))
emptyFiles.Add(Path.GetFileName(xmlFile));
else
content.Append(lines.ToString()); // content here StringBuilder
}
catch (Exception ex)
{
errorFiles.Add(Path.GetFileName(xmlFile));
LogException(ex);
}
}

its working fine. but taking too much time. almost 3:00 hours.

is there any other way reduce the time.

i read about semaphore, am failing to how to use? getting error at semaphore realse();

how many max instance can i give to seamphore? (above 16 also can i give. i.e Semaphore(0, 16))

Thanks for the advance.

J. Sateesh kumar

1 Reply

Know the answer? Post it — somebody with the same question will find it here.

Guest User6y ago
What does your code actually do (in words)?

Before trying all kinds of stuff, find out using a profiler what the actual bottleneck is. You are running 200 tasks, perhaps they all end up waiting because they're task switching between each other.