Here is another C# version of the Java StringTokenizer Class from the version posted August last year on your site. Given a string and a set of delimiters, the class return an enumeration of substrings based on the delimeters passed in. The StringTokenizer class implements the IEnumerable so that the foreach keyword can be used. It also has a Count property that returns the total number of words that are in the string based on the delimeters passed in.
Added by Kunal Johar on April 08, 2003.
If the string does not end a eliminator a token is lost in the attached code. You can fix this problem by adding the following code:
if(tokenizedWord.Length > 0)
{
resultStringArray.Add(tokenizedWord.ToString());
tokenizedWord = new StringBuilder(stringToTokenize.Length);
}