Why does my nom parser not consume the entire input, leaving the last piece unparsed?

Question

I\'m trying to split a log line on space and commas in order to create a Vector of Tokens of Field and Separator as shown

Answer 1

For completeness, I implemented the following changes according to @Zarenor's answer and the parser now consumes the entire input.

changes to main.rs

use nom::types::CompleteByteSlice;
use nom::IResult;

named!(separator<CompleteByteSlice, CompleteByteSlice>, is_a!(" ,"));
named!(not_separator<CompleteByteSlice, CompleteByteSlice>, is_not!(" ,"));

fn token<'a>(input: CompleteByteSlice<'a>) -> IResult<CompleteByteSlice<'a>, Token<'a>> {
    alt!(input,
        separator =>     { | s: CompleteByteSlice<'a> | Token::Separator(s.0) } |
        not_separator => { | n: CompleteByteSlice<'a> | Token::Field(n.0)     }
    )
}

named!(sequence<CompleteByteSlice, Vec<Token>>, many1!(token));

pub fn scan(input: &[u8]) -> Vec<Token> {
    let (_, seq) = sequence(CompleteByteSlice(input)).unwrap();
    seq
}

Answer 2

The issue you're running into is that Nom is designed to always assume that there may be more input, unless you tell it otherwise. Since you know your input here is complete, you need to feed the parsers your literal wrapped in a CompleteByteSlice (or if you used a &str, a CompleteStr). These types are thin wrappers that Nom uses to indicate that we know there isn't more input coming. It will make it so a parser that fails to make a complete match returns an Error instead of an Incomplete, and in this case, will instruct the the parser to consume that final token, rather than ask for more characters.