Comments without terminating newlines, \href fixes, \url support (#1529)

* Comments without terminating newlines in nonstrict mode

Fix #1506 by allowing single-line comments (`%` without terminating newline)
in nonstrict mode.  `Lexer` and `MacroExpander` now store the `Settings`
object, so the `Lexer` can complain about missing newline according to the
`strict` setting.  I filtered this out from the snapshot tests with a slightly
different `replacer`.

* Reimplement \href like \verb, add \url

Major restructuring to lex URL arguments differently, e.g. to support
`\href%{hello}` and `\href{http://foo.com/#test%}{hello}`.  The new URL
parsing code is simpler, but involves a special case in `parseSymbol`
like `\verb`.

Also add support for `\url` while we're here.

* Cleanup

* Fix flow errors and improve error messages

* Add \url to documentation

* Improve doc formatting

test/setup.js

@@ -1,6 +1,7 @@
 /* global expect: false */
 
 import stringify from 'json-stable-stringify';
+import Lexer from "../src/Lexer";
 import ParseError from "../src/ParseError";
 import {
     Mode, ConsoleWarning,
@@ -19,8 +20,16 @@ const typeFirstCompare = (a, b) => {
     }
 };
 
-const regExpReplacer = (key, value) => {
-    return value instanceof RegExp ? {lastIndex: value.lastIndex} : value;
+const replacer = (key, value) => {
+    if (value instanceof Lexer) {
+        return {
+            input: value.input,
+            // omit value.settings
+            lastIndex: value.tokenRegex.lastIndex,
+        };
+    } else {
+        return value;
+    }
 };
 
 const serializer = {
@@ -28,7 +37,7 @@ const serializer = {
         return stringify(val, {
             cmp: typeFirstCompare,
             space: '  ',
-            replacer: regExpReplacer,
+            replacer: replacer,
         });
     },
     test(val) {