* Remove redundant consumeSpaces()
- Spaces after command sequence are ignored in Lexer
- parseExpression consumes spaces in the math mode
* Add catcode to Lexer, move comment parsing back to Lexer
- Fix parsing a comment before a sup/subscript argument
- Fix parsing a comment before an expression
- Fix parsing a comment before or between \hline
- Fix parsing a comment in the macro definition
- Fix parsing a comment including a command sequence
* Update Lexer.js
* Update Parser.js
* catcode -> catcodes
* Comments without terminating newlines in nonstrict mode
Fix#1506 by allowing single-line comments (`%` without terminating newline)
in nonstrict mode. `Lexer` and `MacroExpander` now store the `Settings`
object, so the `Lexer` can complain about missing newline according to the
`strict` setting. I filtered this out from the snapshot tests with a slightly
different `replacer`.
* Reimplement \href like \verb, add \url
Major restructuring to lex URL arguments differently, e.g. to support
`\href%{hello}` and `\href{http://foo.com/#test%}{hello}`. The new URL
parsing code is simpler, but involves a special case in `parseSymbol`
like `\verb`.
Also add support for `\url` while we're here.
* Cleanup
* Fix flow errors and improve error messages
* Add \url to documentation
* Improve doc formatting
* Strict setting controls \newline display-mode behavior
* Bug-fix space handling in macros
Whitespace after a \controlWord is now handled within the lexer, not by the
MacroExpander. This way, \\ expanding to \newline doesn't accidentally
cause spaces to get consumed.
* Rename nonstrict -> reportNonstrict; strictBehavior -> useStrictBehavior
* Second category of errorCodes
* Unicode accents
* Lexer now looks for combining dicritical marks and adds them to the same character
* Parser's `parseSymbol` now recognizes both combined and uncombined forms of Unicode accents, and builds accent objects just like the accent functions
* Added CJK support to math mode (not just text mode)
* Add invalid combining character test
* Add MathML test
* Add weak support for other Latin-1 characters
This maintains backwards compatibility, but it uses the wrong font.
There's a TODO to fix this later.
Also refactor symbol code to use for..of
* Update Unicode screenshot
* Remove dot from accented i and j (in math mode)
Also add dotless Unicode characters to support some accented i's and j's
* Fix \imath, \jmath, \pounds, and more tests
* Switch from for..of to .split().forEach()
Save around 800 bytes in minified code
* Fix split
* normalize() detection
* Convert back to vanilla for loops
* Fix merge
* Move normalize dependency to unicodeMake.js
* Make unicodeSymbols into a lookup table instead of macros
This is important for multi-accented characters.
* Add comments about when to run
* Move symbols definition into unicodeMake/Symbols.js
* Remove CJK support in text mode
* Add missing semicolon
* Refactor unicodeAccents to its own file
* Dotless i/j support in text mode
* Remove excess character mappings
* Fix Åå in math mode (still via Times)
* Update to support #1030
* Add accented Greek letter support (for supported Greek symbols)
* Update screenshot
* remove Æ, æ, Ø, ø, and ß from math mode test
Fixes several issues with space handling: (fix#910)
1. "Control symbols" (as they're called in the TeXbook), such as `\\`, should
not have spaces eaten after them (only "control words" such as `\foo`).
2. In math mode, spaces should be consumed at the parser level, not the
gullet level. This enables `\\ [x]` to parse differently from `\\[x]`
3. Eat spaces between arguments, so `\frac x y` still works.
(This used to work only because math mode ate all spaces.
The analog in text mode wouldn't have worked.)
Also eat spaces in initial arguments in math mode, and before ^ and _ in atoms.
* Add SourceLocation to encapsulate Token/ParseNode debug information.
* Specify concrete Token text type as it captures type mismatches.
* Responded to comments.
* Implement \verb
* Implement @gagern's comments
* \verb: look up characters one at a time.
* Add screenshot test for \verb
* Add error tests for \verb
* Include space symbol in typewriter font, and fix single quotes
This is based on https://github.com/Khan/MathJax-dev/pull/2
which hasn't been accepted yet at the time this commit is made.
* Add \verb* tests
* \verb should use Typewriter-Regular font!
* Switch \verb to use text mode and no-break space.
* Screenshot update with Typewriter-Regular
* \verb test: fix *, add commas to make spaces clear
* Fix spaces and style handling
* Implement @kevinbarabash's comments
* Make error clearly an assertion failure
* verb screenshot for Chrome
* Advanced macro support and magic \dots
* Fix \relax behavior
* Use \DOTSB in \iff, \implies, \impliedby
* Add multiple expansion test
* Implement some of @kevinbarash's comments
* More @kevinbarabash comments
* Token moved from merge
* Add type to defineMacro
* @flow
* Add babel transform-class-properties to have static class properties
* Upgrade Lexer and Parser files to use ES6 classes
* Update eslint max line length to 90 character (more indent because of using ES6 classes)
* Upgrade eslint and jasmin to support ES stage-2 features
* Use static properties to place constants near their functions
* Migrate all remaining sources to ES6 syntax
* Increase eslint max line length to 84
* Remove non-babelified endpoint in dev server.js
* Clean up server.js functions after removing browserified
* Make screenshotter not to use babel endpoint as we babelify everything now
This adds support for the following input sequences:
-- --- ` ' `` '' \degree \pounds \maltese
resulting in – — ‘ ’ “ ” ° £ ✠ symbols already present in our fonts.
As part of this modification, the recognition of multiple dashes was moved
from the lexer to the parser.
This is neccessary since in math mode a sequence of hyphens is just a
sequence of minus signs. Just like a pair of apostrophes in math mode is a
double prime not a right double quotation mark.
To make this easier, parseGroup and parseOptionalGroup have been merged.
* Introduce MacroExpander
The job of the MacroExpander is turning a stream of possibly expandable
tokens, as obtained from the Lexer, into a stream of non-expandable tokens
(in KaTeX, even though they may well be expandable in TeX) which can be
processed by the Parser. The challenge here is that we don't have
mode-specific lexer implementations any more, so we need to do everything on
the token level, including reassembly of sizes and colors.
* Make macros available in development server
Now one can specify macro definitions like \foo=bar as part of the query
string and use these macros in the formula being typeset.
* Add tests for macro expansions
* Handle end of input in special groups
This avoids an infinite loop if input ends prematurely.
* Simplify parseSpecialGroup
The parseSpecialGroup methos now returns a single token spanning the whole
special group, and leaves matching that string against a suitable regular
expression to whoever is calling the method. Suggested by @cbreeden.
* Incorporate review suggestions
Add improvements suggested by Kevin Barabash during review.
* Input range sanity checks
Ensure that both tokens of a token range come from the same lexer,
and that the range has a non-negative length.
* Improved wording of two comments
Summary
We'd like contributors to use the same linter and lint rules that we use
internally. This diff swaps out eslint for jshint and fixes all lint failures
except for the max-len failures in the test suites.
Test Plan:
- ka-lint src
- make lint
- make test
Reviewers: emily
There are two main motivations for this commit. One is unicode input, which
requires unicode characters to get past the lexer. See discussion in #261.
The second is in preparation for #266, where we'd deal with one token of
look-ahead but might be lexing that token in an unknown mode in some cases.
The unit test shipped with this commit addresses the latter concern, since
it checks that a math-mode-only token may immediately follow some text mode
content group.
In this new implementation, all the various things that could get matched
have been collected into a single regular expression. The hope is that
this will be beneficial for performance and keep the code simpler.
The code was written with Unicode input in mind, including non-BMP codepoints.
The role of the lexer as a gate keeper, keeping out invalid TeX syntax, has
been abandoned. That role is still fulfilled by the symbols and functions
tables, though, since any input which is neither a symbol nor a command is
still considered invalid input, even though it lexes successfully.
This commit introduces environments, and implements the parser
infrastructure to handle them, even including arguments after the
“\begin{name}” construct. It also offers a way to turn array-like data
structures, i.e. delimited by “&” and “\\”, into nested arrays of groups.
Environments are essentially functions which call back to the parser to
parse their body. It is their responsibility to stop at the next “\end”,
while the parser takes care of verifing that the names match between
“\begin” and “\end”. The environment has to return a ParseResult, to
provide the position that goes with the resulting node.
One application of this is the “array” environment. So far, it supports
column alignment, but no column separators, and no multi-column shorthands
using “*{…}”. Building on the same infrastructure, there are “matrix”,
“pmatrix”, “bmatrix”, “vmatrix” and “Vmatrix” environments. Internally
these are just “\left..\right” wrapped around an array with no margins at
its ends. Spacing for arrays and matrices was derived from the LaTeX
sources, and comments indicate the appropriate references.
Now we have hard-wired breaks in parseExpression, to always break on “}”,
“\end”, “\right”, “&”, “\\” and “\cr”. This means that these symbols are
never PART of an expression, at least not without some nesting. They may
follow AFTER an expression, and the caller of parseExpression should be
expecting them. The implicit groups for sizing or styling don't care what
ended the expression, which is all right for them. We still have support
for breakOnToken, but now it is only used for “]” since that MAY be used to
terminate an optional argument, but otherwise it's an ordinary symbol.
Summary: Theoretically this allocates way less. In practice it seems to be exactly the same speed.
Test Plan: make test
Reviewers: emily
Reviewed By: emily
Differential Revision: https://phabricator.khanacademy.org/D16621
Summary:
Add correct parsing of optional arguments. Now, things like `\rule` can shift
based on its argument, and parsing of `\sqrt[3]{x}` fails (correctly) because we
don't support that yet.
Also, cleaned up the lexing code a bit. There was a vestige of the old types in
the lexer (they have now been completely moved to symbols.js). As a byproduct,
this made it hard to call `expect("]")`, because it would look at the type of
the Token and the type for "]" was "close". Now, all functions just look at the
text of the parsed token, and in special occasions (like in the dimension lexer)
it can return some data along with it.
Test Plan:
- Make sure tests still work, and new tests work
- Make sure no huxley screenshots changed
- Make EXTRA SURE `\sqrt[3]{x}` fails.
Reviewers: alpert
Reviewed By: alpert
Differential Revision: http://phabricator.khanacademy.org/D13505
