Stream Tokenizer Class in Java

Introduction

In this article, we will learn about the Stream tokenizer class in Java with code examples and output.

What is Stream Tokenizer Class in Java?

The Stream Tokenizer class helps in identifying the patterns in the input stream. It is responsible for breaking up the InputStream into tokens, which are delimited by a set of characters. The best use of this class is to identify the number of words or lines within a file. A stream is tokenized by creating a StreamTokenizer with a Reader object as its source and then setting parameters for the screen. A scanner loop invokes nextToken, which returns the token type of the next token on the screen.

When nextToken recognizes a token, it returns the token type as its value and also set the type field to the same value. 

Types of Token Types

• TT_WORD:  A word is scanned. The string field value contains the word that is scanned.
• TT_NUMBER: A number is scanned. The double field Value contains the value of the number. Only decimal floating numbers are recognized.
• TT_EOL: An end-of-line is found.
• TT_EOF: The end-of-file is reached.

The example below shows how to use a Stream Tokenizer to count the number of words in a  file.

Source Code

import java.io.*;
public class wordcounter
{
public static void main(String args[])throws IOException
{
FileReader fr=new FileReader("d:\\ashish.txt");
StreamTokenizer input=new StreamTokenizer(fr);
int tok;
int count=0;
while((tok=input.nextToken())!=StreamTokenizer.TT_EOF)
if(tok==StreamTokenizer.TT_WORD)
{
System.out.println("Word Found: "+input.sval);
count++;
}
System.out.println("Found "+count+ " words in ashish.txt");
}
}

Output

Summary

To recognize patterns in the input stream, use the Stream Tokenizer class.