strtok_s behaviour with consecutive delimiters

Question

I\'m parsing 3 values in parallel which are separated with a specific separator.

token1 = strtok_s(str1, separator, &nextToken1);
token2 = strtok_s(str2,         


        

Answer 1

Unfortunately, strtok() ignores empty tokens. Even though you said you wish to avoid doing that, there is no other way but to parse it yourself, using for example strchr() to find the next delimiter and then copying the token to a temporary variable for processing. This way you can handle empty tokens whichever way you please.

Answer 2

Yes, that's the way this function works. It's more appropriate for tasks like parsing words where multiple whitespace characters should not be treated as empty words.

I've done a lot of parsing. I would simply write my own parser here, where the code examines one character at a time. It's not that difficult and you can make it behave exactly how you need. As an example, I've posted some C++ code to parse a CSV file in my article Reading and Writing CSV Files in MFC