Using StringTokenizer in Java

By: Emiley J  

The processing of text often consists of parsing a formatted input string. Parsing is the division of text into a set of discrete parts, or tokens, which in a certain sequence can convey a semantic meaning. The StringTokenizer class provides the first step in this parsing process, often called the lexer (lexical analyzer) or scanner. StringTokenizer implements the Enumeration interface. Therefore, given an input string, you can enumerate the individual tokens contained in it using StringTokenizer.

To use StringTokenizer, you specify an input string and a string that contains delimiters. Delimiters are characters that separate tokens. Each character in the delimiters string is considered a valid delimiter—for example, ",;:" sets the delimiters to a comma, semicolon, and colon. The default set of delimiters consists of the whitespace characters: space, tab, newline, and carriage return.

The StringTokenizer constructors are shown here:

StringTokenizer(String str)
StringTokenizer(String str, String delimiters)
StringTokenizer(String str, String delimiters, boolean delimAsToken)

In all versions, str is the string that will be tokenized. In the first version, the default delimiters are used. In the second and third versions, delimiters is a string that specifies the delimiters. In the third version, if delimAsToken is true, then the delimiters are also returned as tokens when the string is parsed. Otherwise, the delimiters are not returned. 

Delimiters are not returned as tokens by the first two forms. Once you have created a StringTokenizer object, the nextToken( ) method is used to extract consecutive tokens. The hasMoreTokens( ) method returns true while there are more tokens to be extracted. Since StringTokenizer implements Enumeration, the hasMoreElements( ) and nextElement( ) methods are also implemented, and they act the same as hasMoreTokens( ) and nextToken( ), respectively.

Here is an example that creates a StringTokenizer to parse "key=value" pairs. Consecutive sets of "key=value" pairs are separated by a semicolon.

// Demonstrate StringTokenizer.
import java.util.StringTokenizer;
class STDemo {
static String in = "title=Java-Samples;" +
"author=Emiley J;" +
";" +
public static void main(String args[]) {
StringTokenizer st = new StringTokenizer(in, "=;");
while(st.hasMoreTokens()) {
String key = st.nextToken();
String val = st.nextToken();
System.out.println(key + "\t" + val);

The output from this program is shown here:

title Java-samples
author Emiley J
copyright 2007

Archived Comments

1. I see you don't monetize your website, don't waste your traffic, you can earn extra cash every

View Tutorial          By: 86Julio at 2017-07-26 23:06:03

2. Thanks i am looking for that how to use stringTokenizer very helpful
View Tutorial          By: waqas at 2015-06-01 18:34:48

3. Thanks a lot for that excellent explanation. It has helped me a great deal. @ Emiley
View Tutorial          By: Gregory at 2014-12-30 05:49:21

4. Very good example .!

View Tutorial          By: Mprogrammer at 2014-11-25 12:10:21

5. pls help how to fetch vowels as a first letter of d name using jdbc
View Tutorial          By: anjali at 2014-09-18 20:45:49

6. give some good examples very bad i dont lke dis example worst behaviour
View Tutorial          By: abinaya banu at 2014-07-14 06:04:28

7. Thanks for this explanation. I'm not sure whether this is completely explained in Java API !! The de
View Tutorial          By: Venkatesh Challa at 2013-03-29 04:35:21

8. what if the delimiter is "(quotation mark)?
View Tutorial          By: hiakoto at 2013-02-18 16:15:20

9. it's emergency
plz explain delimiter in tokenizer

View Tutorial          By: sajad at 2013-02-03 05:56:44

10. i have a program in which we have to find the summation of 2 whole number entered in following forma
View Tutorial          By: Navjyot at 2013-01-28 10:38:55

11. date span progarm sorce code in jsp by using string tokenizer concept for ex:if u have entered one d
View Tutorial          By: prudhvi at 2013-01-27 13:55:09

12. good explanation with sample example makes me to understand the concept easily ., thanks for your wo
View Tutorial          By: at 2013-01-04 09:10:21

13. Thanks for Your valuable Explanation.
View Tutorial          By: Srikanth at 2012-12-24 10:49:02

14. Thanks. What datatype can we give to a variable Student number lets say for example student # 199300
View Tutorial          By: Bafokeng Lebesa at 2012-11-27 12:35:40

15. how can we split 12+12 into three tokens like 12,+and 12.or there is any alternate way to split thes
View Tutorial          By: arjun at 2012-11-26 11:07:24

16. thanks buddy is's very nice and useful example
View Tutorial          By: mayur at 2012-11-05 05:00:11

17. it worked n this case but could not find some more cases...however good explanation ..
View Tutorial          By: abhilash koleti at 2012-09-14 16:21:58

18. Nice explanation and a valuable example. Awesome!!!!!
View Tutorial          By: kesariJena at 2012-07-20 13:05:35

19. @pooja: this method should work, although i'm sure it isn't the most efficient one. str is your stri
View Tutorial          By: Anush at 2012-07-16 08:30:58

20. thanx nice explanation...
View Tutorial          By: vignesh at 2012-07-15 20:12:22

21. hi....
can u please show me the code for finding the frequency of the tokens ie tokens that a

View Tutorial          By: pooja at 2012-07-08 16:08:08

22. Awesome explanation....

View Tutorial          By: Abhishek Patel at 2012-03-27 05:29:32

23. shes a hot chick programmer!
View Tutorial          By: pola at 2012-01-16 02:33:21

24. great solution and brief explanation, love it.
View Tutorial          By: habesha at 2011-12-02 16:14:45

25. Very good example.
It's very useful articles

View Tutorial          By: elangovan at 2011-11-16 12:33:10

26. You just save ma arse, had a SQLite database, and returns a Cursor which I printed as a string using
View Tutorial          By: codeRealm at 2011-11-13 12:58:05

27. Nice example to understand what does the string Tokennizer do.
View Tutorial          By: Spha at 2011-07-14 08:00:03

28. Nice example to understand what does the string Tokennizer do.
Great job.

View Tutorial          By: vikash from india at 2011-06-23 06:39:03

29. its a very very good artical
View Tutorial          By: xyz at 2011-06-17 08:12:26

30. Dear Samantha, Thx for pointing the extra \. I have removed the extra \ now.
View Tutorial          By: Emiley at 2011-06-06 04:50:40

31. Hi.....u have put an extra backslash in the last for \t......
u have put \\'s o

View Tutorial          By: Samantha at 2011-05-12 12:54:39

32. Create a lexical analyzer of the C- programming language using Java program. The following statement
View Tutorial          By: lucky at 2011-05-06 23:05:22

33. Create a lexical analyzer of the C- programming language using Java program. The following statement
View Tutorial          By: lucky at 2011-05-06 23:01:43

34. how to use tokenizer "No.1 hello,No.2 world,No.3 NO"
to get
1 hello,

View Tutorial          By: saanu at 2011-03-04 00:41:21

35. thanx 4 these info......
View Tutorial          By: rajkamal at 2011-02-18 23:39:35

36. huahahaha, ya the ";" is forgotten at the end of the "author=Emiley J" and also
View Tutorial          By: g at 2011-01-07 22:08:26

37. Wonderful...
It was Really Useful...
Thank u so much..Gr8..
Keep Goin On..!!

View Tutorial          By: Gift Lee at 2010-12-02 23:27:15

38. the best explaination
View Tutorial          By: kingh at 2010-09-17 04:06:46

39. thanks for your explanation, but for desired output following changes is required

View Tutorial          By: bismillah at 2010-09-02 10:16:09

40. thanks for your explanation, but for desired output following changes is required

View Tutorial          By: bismillah at 2010-09-02 10:15:35

41. nice example!!
View Tutorial          By: gayathri subramanian at 2010-08-18 09:29:35

42. nice example!!
View Tutorial          By: gayathri subramanian at 2010-08-18 09:27:59

43. Simplified explanation with perfect Examples
View Tutorial          By: Sushant Chaudhary at 2010-07-12 00:32:18

44. it was very help ful
i got it when i really wanted it
thank you

View Tutorial          By: anand harshan at 2010-07-07 12:18:22

45. @:some1

The input should be

String in = "title=Java-Samples;"

View Tutorial          By: Sam at 2010-06-16 07:18:22

46. Thank you very much for explaning the things in such a very good manner and with good example.
View Tutorial          By: Sunil Yadav at 2010-05-27 04:04:31

47. It gives strange result when we use == as delimiter.
String myStr = "This==is==correct=b

View Tutorial          By: Amit at 2010-05-07 05:03:41

48. thanks emily, this was fantastic... just saved me a huge headache.

also, my apologies

View Tutorial          By: philip at 2010-04-10 20:39:08

49. We need an example to tokenize a xml file
View Tutorial          By: Todo at 2010-03-03 03:27:01

50. there will be a semi-colon(;) after Emiley J and
to produce the desire outpu

View Tutorial          By: Manojit at 2010-02-21 01:37:47

51. cse exalters
View Tutorial          By: jayasimha at 2010-02-20 02:12:52

52. awesome example
View Tutorial          By: jagjyot singh at 2010-01-12 20:39:04

53. thank you sir
View Tutorial          By: ramya at 2010-01-11 00:42:05

54. prog was understandable wat u gave there,but it not gave the particular o/p sir.

View Tutorial          By: sailaja at 2009-07-03 02:53:20

55. thnx, 4 ur explanation, that was usefl
but this input wont give that output

View Tutorial          By: some1 at 2009-04-28 03:26:28

56. very good explanation... thnx...
View Tutorial          By: gobu at 2009-04-05 23:43:19

57. f**k
View Tutorial          By: stupid head at 2009-03-11 04:37:07

Most Viewed Articles (in Java )

Latest Articles (in Java)

Comment on this tutorial