Fix parse timing by separating consume() into fetch() and consume() (#2054)

* Fix parse timing by separating consume() into fetch() and consume()

Fix #1989, and generally cleanup parse timing (many fewer hoops to jump
through) by defining two methods on parser:

1. `consume()` marks the current token (`nextToken`) as "done", but
   doesn't yet fetch the next token (setting `nextToken` to `null`).
2. `fetch()` fetches the next token if we don't already have one
   (e.g., if last token was `consume()`d).

Before this change, `consume()` did both actions.  By separating them,
and allowing the parser to live in a state for a while where `nextToken`
is `null`, it is far easier to change settings (in particular, math/text
mode and catcodes) before reading the next token, in a way that depends
on what we're parsing.  For example, if an argument should be treated in
text mode, we can just set the mode in the argument parser, instead of
when the previous token was consumed.  Similarly, if an argument should
be treated as a URL, we can just set the catcode of `%` in the URL
argument parser, and reset it after.  We no longer have to take care to
reset things before calling `consume()`.

This change mostly involves changing `this.nextToken` to `this.fetch()`.
In a perfect world, we could use slightly fewer calls to `this.fetch()`,
but Flow doesn't realize that `this.nextToken` will be non-null after a
call to `this.fetch()`, so we need to use a few more calls to
`this.nextToken()` or a few more local `nextToken` variables.

* Remove now-unnecessary consumeMath

* Update Parser.js

test/katex-spec.js

@@ -3079,7 +3079,7 @@ describe("A macro expander", function() {
     // which doesn't treat all four letters as an argument.
     //it("\\TextOrMath should work in a macro passed to \\text", function() {
     //    expect`\text\mode`.toParseLike(r`\text{text}`, new Settings({macros:
-    //        {"\\mode": "\\TextOrMath{text}{math}"});
+    //        {"\\mode": "\\TextOrMath{text}{math}"}});
     //});
 
     it("\\gdef defines macros", function() {
@@ -3144,20 +3144,12 @@ describe("A macro expander", function() {
     });
 
     it("array cells generate groups", () => {
-        expect`\def\x{1}\begin{matrix}\x&\def\x{2}\x&\x\end{matrix}`
-            .toParseLike`\begin{matrix}1&2&1\end{matrix}`;
+        expect`\def\x{1}\begin{matrix}\x&\def\x{2}\x&\x\end{matrix}\x`
+            .toParseLike`\begin{matrix}1&2&1\end{matrix}1`;
+        expect`\def\x{1}\begin{matrix}\def\x{2}\x&\x\end{matrix}\x`
+            .toParseLike`\begin{matrix}2&1\end{matrix}1`;
     });
 
-    // TODO: This doesn't yet work; before the environment gets called,
-    // {matrix} gets consumed which means that the \def gets executed, before
-    // we can create a group. :-(  Issue #1989
-    /*
-    it("array cells generate groups", () => {
-        expect`\def\x{1}\begin{matrix}\def\x{2}&\x\end{matrix}`
-            .toParseLike`\begin{matrix}&1\end{matrix}`;
-    });
-    */
-
     it("\\gdef changes settings.macros", () => {
         const macros = {};
         expect`\gdef\foo{1}`.toParse(new Settings({macros}));