# Java: Tokenizing Delimiters

## Purpose

When tokenizing a string, the delimiters are ignored by default. However, it may be useful to get them returned as tokens too. Each delimiter will be returned as a one-character string.

## Constructor

Specify that you want the delimiters returned by specifying `true` in the third parameter of the constructor.

`StringTokenizer st = new StringTokenizer(s, d, true);`
  Creates a StringTokenizer for the String s using delimiters from the String d

## Example: Tokenizing an expression

Here is an example method that tokenizes a string, making all delimiters except whitespace into tokens.

     private String tokenizeEx(String expression, String delims) {
        StringBuffer sb = new StringBuffer(100);
        StringTokenizer scanner;
        scanner = new StringTokenizer(expression, delims, true);

        while (scanner.hasMoreTokens()) {
           String aWord = scanner.nextToken();
           if (!Character.isWhitespace(aWord.charAt(0))) {
              sb.append(aWord + "\n");
           }
        }
        return sb.toString();
     }//end tokenizeEx

## Example Applet

The following applet uses the above method. TokenizeExpr.java. Unfortunately, most browsers don't run Java 2 applets correctly yet, so it's doubtful that you can see it working here. You may have to compile and run it yourself.