Recognizing repeat_till0 to () without the suffix

[yuriks]

Hi,

The docs for repeat_till0 say:

To recognize a series of tokens, Accumulate into a () and then Parser::recognize.

However, I can't figure out a way to do that while separating the parse of the parser f from g. This is what I have right now: Playground Link

use winnow::prelude::*; // 0.5.15
use winnow::ascii::space0;
use winnow::combinator::{eof, preceded, repeat_till0};
use winnow::token::any;

fn parse_test<'i>(i: &mut &'i str) -> PResult<&'i str> {
    preceded(
        ";;; ",
        repeat_till0::<_, _, (), _, _, _, _>(any, (space0, ";;;", space0, eof)).recognize(),
    )
    .parse_next(i)
}

fn main() {
    // Value should be "test text"
    let _ = dbg!(parse_test.parse(";;; test text ;;;"));
    // Value should be "test ;;; with more semicolon"
    let _ = dbg!(parse_test.parse(";;; test ;;; with more semicolon ;;;  "));
}

Is there some way to do this that I'm overlooking, or is this an oversight in the API? Also, is there a more ergonomic way to specify that I'm collecting to a ()? The turbofish there looks unsightly with the multitude of unused parameters.

[epage]

What I tend to do is repeat_till0(...).map(|()| ()). I've tried to find a way to represent this as a trait function but not been able to so far.

[yuriks]

I ended up writing my own replacement for repeat_till0 to be able to have both parts of the output separated:

fn lazy_quantifier<
    I: Stream,
    O,
    E: ParserError<I>,
    P: ContainsToken<I::Token>,
    F: Parser<I, O, E>,
>(
    predicate: P,
    mut terminator: F,
) -> impl Parser<I, (I::Slice, O), E> {
    move |i: &mut I| {
        let match_start = i.checkpoint();
        loop {
            let potential_match_end = i.checkpoint();
            let match_length = i.offset_from(&match_start);
            match terminator.parse_next(i) {
                Ok(o) => {
                    let final_pos = i.checkpoint();
                    i.reset(match_start);
                    let matched = i.next_slice(match_length);
                    i.reset(final_pos);
                    return Ok((matched, o));
                }
                Err(ErrMode::Backtrack(_)) => i.reset(potential_match_end),
                Err(e) => return Err(e),
            }

            let c = i.next_token();
            if !c.is_some_and(|c| predicate.contains_token(c)) {
                return Err(ErrMode::from_error_kind(i, ErrorKind::Slice));
            }
        }
    }
}

pub fn parse_sub_comment<'i>(i: &mut &'i str) -> PResult<(u16, &'i str)> {
    let addr = delimited((";;;", space1, '$'), hex4, (':', space0)).parse_next(i)?;
    let (description, ()) =
        lazy_quantifier(|_| true, (opt((space0, ";;;")), space0, eof).void()).parse_next(i)?;
    Ok((addr, description))
}

Would still love to hear if there's a way to do this with one of the built-in combinators.

[epage]

Oh, right, you want to recognize only the accumulated portion and ignore the other part. That wouldn't work with the approach I mentioned.

The options I can think of

• Accumulate into a String and take the performance hit
• use a closure to capture a checkpoint at the start of the space0, ";;;", ... section and then manually recognize the preceding string
• Require location tracking on I and capture indices for the space0, ";;;", ... section and use that to get the preceding string.

[yuriks]

The second suggestion is interesting, I hadn't though of using closures with mutable state to keep track of things "out of band" like that! I had another situation where I wanted to accumulate a varying number of elements per parsed item. I tried using Stateful to thread the output structure through the specific rule I needed to mutate it from, but ended up finding the dance of wrapping and unwrapping the stream with stateful to be a bit verbose, and so ended up returning iterators from each item (which I could then .flatten()), and it sounds like this could be a good approach for that. Even if I don't replace my custom combinator with it, definitely something to keep in mind!

Anyway, I've solved my immediate problem, but if you want to, you can consider this a soft feature request for an API that allows you to do this. :) I'm pretty happy with winnow otherwise. I used to use nom but the ergonomics improvements are definitely very appreciated. I've used it to replaced regex-based parsing I had previously and I think the code is easier to read and modify, and it doubled the performance of my tool!

[Nahor]

Another solution, at the cost of parsing the terminator twice (playground):

fn parse_test<'i>(i: &mut &'i str) -> PResult<&'i str> {
    let terminator = (space0, ";;;", space0, eof);
    preceded(
        ";;; ",
        terminated(repeat_till::<_, _, (), _, _, _, _>(.., any, peek(terminator)).recognize(), terminator),
    )
    .parse_next(i)
}