===== The "Up-to-here" Marker =====

The //up-to-here// marker ''=>||'' and the //up-to-here-plus// marker ''=>|+n'' (where n is an integer) were introduced to be able to express [[lookahead]] in a more succinct and less err-prone manner than using the ''LOOKAHEAD'' from legacy JavaCC or even the terser ''SCAN'' statement.

Where in legacy JavaCC, you would write:

<code>
    LOOKAHEAD(Foo() Bar()) Foo() Bar() Baz()
</code>    

or in CongoCC (//without up-to-here//) you could write:

<code>
    SCAN Foo Bar => Foo Bar Baz
</code>

but now, with the //up-to-here// syntax, you can now write the above simply as:

<code>
     Foo Bar =>|| Baz
</code>

The latter construct can be read as meaning that we enter the expansion ''Foo Bar Baz'' and yes, we scan ahead to check when deciding whether to enter the expansion. However, if we get to the end of the Bar, we can stop scanning. 

Now, I would point out that, if we knew for a fact that ''Foo Bar'' consumes //exactly// 7 tokens, say, then the above would //effectively// be the same as writing:

<code>
    LOOKAHEAD(7) Foo() Bar() Baz()
</code>

However, in the general case, the expansions ''Foo'' and ''Bar'' above will consume a variable number of tokens. Also, as the grammar evolves over time, the maximum number of tokens that these productions consume may change and you would then have to change the number specified in the ''LOOKAHEAD''. The //up-to-here// marker dispenses with that.

==== The up-to-here marker in parsing productions ====

The //up-to-here// marker can also be used in grammatical productions to indicate that the production requires scanahead up to a certain point.

For example, suppose we have the following two simple productions:

<code>
    Foobar : "foo" bar" "baz" ;
</code>

<code>
    Foobaz : "foo" "baz" "bat" ;
</code>

(I refer to these as //simple// because both of them are just a sequence of three string literal tokens. I use these as examples, but the same concept applies to much more complex expansions, of course.)

Suppose further that we have a choice construct:

<code>
   ( Foobar | Foobaz )
</code>

Now, if no ''LOOKAHEAD'' (or ''SCAN'') is specified, then the ''Foobaz'' production is going to be unreachable. This is because, by default, CongoCC looks ahead exactly one token to decide which of the two expansions to enter, and will simply go with the first one that matches. So, assuming that the next token is a "foo", then it will //always// enter ''Foobar'' and the second choice, ''Foobaz'', is unreachable.

The classic solution would be to specify two tokens of lookahead, so that we need to match "foo" followed by "bar" to enter ''Foobar''. Like so:

<code>
   ( LOOKAHEAD (2) Foobar() | Foobaz() )
</code>

In CongoCC, you have the option of putting an //up-to-here// marker in the ''Foobar'' production like so:

<code>
    Foobar : "foo" "bar" =>|| "baz" ;
</code>

This means that we scan up to (and including) the "bar" token when the expansion at a [[choice points|choice point]] starts with ''Foobar''.

Now, the choice construct above can be written simply as:

<code>
   ( Foobar | FooBaz )
</code>

Since the //up-to-here// marker is specified inside the ''Foobar'' production, the code generated for the choice construct takes that into account and scans the necessary two tokens, i.e. up to and including the "bar" token.

==== The "up-to-here-plus" marker ====

Not long after implementing the above //up-to-here// syntax, I realized that there was a common problem that it did not address. Frequently, you want to scan up to a given point and then an extra token (or two... or three...)

For example, suppose you have a production that represents the declaration of a function in some C-ish sort of language. 

<code>
    FunctionDeclaration : [ReturnType] <IDENTIFIER> Args Block ;
</code>

Suppose that you decide that, to decide that something is indeed a function declaration, you need to scan to the end of the Args and //then one more token// which would be the opening curly brace of the ''Block'', "{". 

Regardless, what we certainly don't want to do is scan all the way to the end of the Block that could be very very long and costly. We reason that it is enough to hit the "{" that starts the ''Block''. So, we could write:

<code>
    FunctionDeclaration : [ReturnType] <IDENTIFIER> Args =>|+1 Block ;
</code>

The ''=>|+1'' is the //up-to-here-plus// marker and means that once we scan past Args, we set a numerical limit of 1 more token. In this case, that would mean that we scan up to and including the initial "{" of the ''Block'' production.

Then, for example, if the ''FunctionDeclaration'' was used at some [[choice point]] in your grammar, say:

<code>
   ( FunctionDeclaration | VariableDeclaration | StructDeclaration | EnumDeclaration )
</code>

without the ability to put the up-to-here (or in this case, //up-to-here-plus// marker in  ''FunctionDeclaration'' we would have to write something like this, as in legacy JavaCC:

<code>
   ( LOOKAHEAD (ReturnType() <IDENTIFIER> Args() "{" ) FunctionDeclaration
     | VariableDeclaration | StructDeclaration | EnumDeclaration )
</code>

Of course, most likely, the other productions, such as ''VariableDeclaration'' etcetera would also requires some complicated (complicated looking) syntactic lookahead and the whole thing would end up looking very hairy indeed!    
 
==== Niggles (Annoying Details) ====

The above basically summarizes how the new //up-to-here// construct works. There are, however, a few little annoying details to keep in mind.
  * If an expansion has an //up-to-here// marker then any nested //up-to-here// is ignored.
  * If an expansion has an explicit //numerical// or //syntactic// lookahead, then any //up-to-here// marker in a that expansion or in a nested production is ignored.

What this means, for example, is that if we have:

<code>
   Foo Bar =>|| Baz
</code>

then any //up-to-here// inside of the ''Foo'' production is ignored because, at a higher level, we specified that we want to scan past it up to and including the ''Bar'' that follows.

By the same token, if we specify a specific numerical limit, then that overrides any nested //up-to-here// as well. So, in the following:

<code>
  [ SCAN 1 Foo Bar Baz ]
</code>

We are just going to scan ahead one token, regardless of whether ''Foo'' contains an //up-to-here// saying that we should scan more than 1, because the ''SCAN'' statement is assumed to mean that we override that.

Note also that the up-to-here marker in a production only applies at a maximum of one level of nesting. Thus, for example, if you have:

<code>
    Foo : Bar Baz ;
</code>

and:

<code>
    Bar : "baz" "bat" =>|| "bam";
</code>

Then using ''Foo'' at a choice point, like :

<code>
    ( Foo )*
</code>

Though it could conceivably be anti-intuitive (I'm not sure really...) the //up-to-here// marker in the ''Bar'' above is //not// used. Or, in other words, the ''( Foo )*'' expansion above only scans //one// token ahead on each iteration of the loop. This is the current implementation and it is possible that it will be revisited at a later point. Perhaps not, since respecting the //up-to-here// marker when recursing more than one level deep seems like it could be problematic, both in the implementation and in terms of grammars being easily readable. 

Note also that the only //nested// up-to-here that applies to an expansion is in a NonTerminal that //starts// the expansion. Thus, if we have the expansion:

<code>
   ( Foo Bar Baz )*
</code>

The //up-to-here// marker inside the ''Bar'' production is not used, only one inside of ''Foo'', assuming it does have one! In the above case, if we did want to scan past Foo and into Bar, stopping after two tokens, we could write:

<code>
   ( Foo =>|+2 Bar Baz )*
</code>

As of this writing, there is no way to get the lookahead machinery to respect the //up-to-here// marker inside of a production (like ''Bar'' above) if it is not at the very beginning of the expansion. This may be addressed later, but it is probably not at all urgent. In most cases, where you want to scan past a production and then somewhat more into the next one, it is exactly one token. You will typically have something like:

<code>
   MethodDeclaration : [ Modifiers ] ReturnType <IDENTIFIER> =>||+1 Args Block ;
</code>

What the above means is that we scan to the end of the <IDENTIFIER> (which would be the method name) and then one more token to check for the initial parenthesis "("in the Args production. We anticipate that the vast majority of //up-to-here-plus// markers will indicate one extra token. (Though that remains to be seen in actual praxis, admittedly...)