Hello, I am working on a performance engineering project. The application is a C# web application using ASP.NET and IIS. I have found many things that I know can be refactored to enhance performance. For each one I first write a test driver and test it thoroughly in a stand alone test harness. The refactored version always runs anywhere from 4 to 80 times faster. Not the entire app, the pattern that I am micro-benchmarking. I then refactor the application code to put in this pattern and the application runs SLOWER.
I stopped working with the code itself and started capturing metrics at the OS level. One thing I saw was context switching of 8,000 to 9,000 per second. I found several places in the code that are likely culprits for this. A lock inside a while loop inside another lock was one. A method that gets called all the time which creates a new thread each time was another.
My theory is that the reason that changes which should be performance enhancements actually make the code run slower is that any increase of throughput just makes whatever contention is causing the context switching worse.
Has anyone ever seen behavior like this before?
My next step is to use some tools to find out where the contention is.
Thanks