Menu

- Learn
- Documentation
  - Language Reference
  - Library Reference
  - Command-line Reference
  - Feature Overview
  - Articles
- Downloads
- Packages
- Community
  - Blog
  - Orgs using D
  - Twitter
  - Calendar
  - Forums
  - IRC
  - Wiki
  - GitHub
  - Issues
  - Get involved
  - Contributors
  - Foundation
  - Security Team
  - Donate
  - Sponsors
- Resources
  - Tour
  - Books
  - Tutorials
  - Tools
  - Editors
  - IDEs
  - run.dlang.io
  - Visual D
  - Acknowledgments
  - D Style
  - Glossary
  - Sitemap

Search
Report a bug
If you spot a problem with this page, click here to create a Bugzilla issue.
Improve this page
Quickly fork, edit online, and submit a pull request for this page. Requires a signed-in GitHub account. This works well for small changes. If you'd like to make larger changes you may want to consider using a local clone.

# dmd.lexer

Compiler implementation of the D programming language.
Authors:
Walter Bright
License:
Boost License 1.0

Source lexer.d

Documentation https://dlang.org/phobos/dmd_lexer.html

Coverage https://codecov.io/gh/dlang/dmd/src/master/src/dmd/lexer.d

static immutable ubyte[256] `cmtable`;
  Do our own char maps
abstract class `ErrorHandler`;
  Handles error messages
  abstract void `error`(const(char)* `format`, ...);
    Report an error message.
    Parameters:

    const(char)* `format` | format string for error
    ... | format string arguments

  abstract void `error`(Loc `loc`, const(char)* `format`, ...);
    Report an error message.
    Parameters:

    Loc `loc` | Location of error
    const(char)* `format` | format string for error
    ... | format string arguments

class `Lexer`: dmd.lexer.ErrorHandler;
  this(const(char)* `filename`, const(char)* `base`, size_t `begoffset`, size_t `endoffset`, bool `doDocComment`, bool `commentToken`);
    Creates a Lexer for the source code base[begoffset..endoffset+1]. The last character, base[endoffset], must be null (0) or EOF (0x1A).
    Parameters:

    const(char)* `filename` | used for error messages
    const(char)* `base` | source code, must be terminated by a null (0) or EOF (0x1A) character
    size_t `begoffset` | starting offset into base[]
    size_t `endoffset` | the last offset to read into base[]
    bool `doDocComment` | handle documentation comments
    bool `commentToken` | comments become TOK.comment's

  final TOK `peekNext`();
    Look ahead at next token's value.
  final TOK `peekNext2`();
    Look 2 tokens ahead at value.
  final void `scan`(Token* `t`);
    Turn next token in buffer into a token.
  final Token* `peekPastParen`(Token* `tk`);
    tk is on the opening (. Look ahead and return token that is past the closing ).
  final uint `escapeSequence`();
    Parse escape sequence.
  static dchar `escapeSequence`(ErrorHandler `handler`, ref const(char)* `sequence`);
    Parse the given string literal escape sequence into a single character.
    Parameters:

    ErrorHandler `handler` | the error handler object
    const(char)* `sequence` | pointer to string with escape sequence to parse. this is a reference variable that is also used to return the position after the sequence

    Returns:
    the escaped sequence as a single character
  final void `wysiwygStringConstant`(Token* `result`);
    Lex a wysiwyg string. p must be pointing to the first character before the contents of the string literal. The character pointed to by p will be used as the terminating character (i.e. backtick or double-quote).
    Parameters:

    Token* `result` | pointer to the token that accepts the result

  final TOK `hexStringConstant`(Token* `t`);
    Lex hex strings: x"0A ae 34FE BD"
  final void `delimitedStringConstant`(Token* `result`);
    Lex a delimited string. Some examples of delimited strings are:

        q"(foo(xxx))"      // "foo(xxx)"
        q"[foo$(LPAREN)]"  // "foo$(LPAREN)"
        q"/foo]/"          // "foo]"
        q"HERE
        foo
        HERE"              // "foo\n"

    It is assumed that p points to the opening double-quote '"'.
    Parameters:

    Token* `result` | pointer to the token that accepts the result

  final void `tokenStringConstant`(Token* `result`);
    Lex a token string. Some examples of token strings are:

        q{ foo(xxx) }    // " foo(xxx) "
        q{foo$(LPAREN)}  // "foo$(LPAREN)"
        q{{foo}"}"}      // "{foo}"}""

    It is assumed that p points to the opening curly-brace '{'.
    Parameters:

    Token* `result` | pointer to the token that accepts the result

  final void `escapeStringConstant`(Token* `t`);
    Scan a double-quoted string while building the processed string value by handling escape sequences. The result is returned in the given `t` token. This function assumes that p currently points to the opening double-quote of the string.
    Parameters:

    Token* `t` | the token to set the resulting string to

  final TOK `charConstant`(Token* `t`);
  final void `stringPostfix`(Token* `t`);
    Get postfix of string literal.
  final TOK `number`(Token* `t`);
    Read in a number. If it's an integer, store it in tok.TKutok.Vlong. integers can be decimal, octal or hex Handle the suffixes U, UL, LU, L, etc. If it's double, store it in tok.TKutok.Vdouble.
    Returns:
    TKnum TKdouble,...
  final TOK `inreal`(Token* `t`);
    Read in characters, converting them to real.
    Bugs:
    Exponent overflow not detected. Too much requested precision is not detected.
  final void `poundLine`();
    parse: #line linnum [filespec] also allow __LINE__ for linnum, and __FILE__ for filespec
  final uint `decodeUTF`();
    Decode UTF character. Issue error messages for invalid sequences. Return decoded character, advance p to last character in UTF sequence.
  final void `getDocComment`(Token* `t`, uint `lineComment`, bool `newParagraph`);
    Parse doc comment embedded between t.ptr and p. Remove trailing blanks and tabs from lines. Replace all newlines with \n. Remove leading comment character from each line. Decide if it's a lineComment or a blockComment. Append to previous one for this token.
    If newParagraph is true, an extra newline will be added between adjoining doc comments.
  static const(char)* `combineComments`(const(char)* `c1`, const(char)* `c2`, bool `newParagraph`);
    Combine two document comments into one, separated by an extra newline if newParagraph is true.

Copyright © 1999-2018 by the D Language Foundation | Page generated by Ddoc on Sat Dec 8 15:29:47 2018