We have a bunch of multi-threaded time-critical C# applications communicating with the outside world via TCP/IP sockets. UI of any of these applications can occasionally get frozen for a few seconds (in extreme cases up to 30 seconds). This does not happen often and can perhaps be explained by some unusual conditions that occur from time to time on any Windows system. As far as we could ascertain from the logs, these delays only occur on the UI thread, which is usually running with lower priority than worker threads of the application.
Since the applications in question are time-critical as I said, even those rare delays are subject of our investigation; however their random character greatly hinders any analysis. It is possible that there is no miracle solution to the problem, but we need at least to be able to determine in each particular case, what was the cause of the delay.
Now my question: is there any way to:
1) Detect an "abnormal" delay (i.e. determine that the application is not just idle, it's waiting for something).
2) Determine the cause of such delay.
Obviously either some internal tracing/debugging/hooking or/and usage of an external observing tool is needed. Any ideas and suggestions will be greatly welcomed.
TIA