Java StringTokenizer.nextToken() skips over empty fields

Question

I am using a tab (/t) as delimiter and I know there are some empty fields in my data e.g.:

one->two->->three

Where -> equals the tab. As you can see an empty field is still correctly surrounded by tabs. Data is collected using a loop :

 while ((strLine = br.readLine()) != null) {
    StringTokenizer st = new StringTokenizer(strLine, "\t");
    String test = st.nextToken();
    ...
    }

Yet Java ignores this "empty string" and skips the field.

Is there a way to circumvent this behaviour and force java to read in empty fields anyway?

Answer 1

There is a RFE in the Sun's bug database about this StringTokenizer issue with a status Will not fix.

The evaluation of this RFE states, I quote:

With the addition of the java.util.regex package in 1.4.0, we have basically obsoleted the need for StringTokenizer. We won't remove the class for compatibility reasons. But regex gives you simply what you need.

And then suggests using String#split(String) method.

Answer 2

Thank you at all. Due to the first comment I was able to find a solution: Yes you are right, thank you for your reference:

 Scanner s = new Scanner(new File("data.txt"));
 while (s.hasNextLine()) {
      String line = s.nextLine();
      String[] items= line.split("\t", -1);
      System.out.println(items[5]);
      //System.out.println(Arrays.toString(cols));
 }

Answer 3

You can use Apache Commons StringUtils.splitPreserveAllTokens(). It does exactly what you need.

Answer 4

I would use Guava's Splitter, which doesn't need all the big regex machinery, and is more well-behaved than String's split() method:

Iterable<String> parts = Splitter.on('\t').split(string);

Answer 5

As you can see in the Java Doc http://docs.oracle.com/javase/6/docs/api/java/util/StringTokenizer.html you can use the Constructor public StringTokenizer(String str, String delim, boolean returnDelims) with returnDelims true

So it returns each Delimiter as a seperate string!

Edit:

DON'T use this way, as @npe already typed out, StringTokenizer shouldn't be used any more! See JavaDoc:

StringTokenizer is a legacy class that is retained for compatibility reasons although its use is discouraged in new code. It is recommended that anyone seeking this functionality use the split method of String or the java.util.regex package instead.