Menu

- Learn
- Documentation
  - Language Reference
  - Library Reference
  - Command-line Reference
  - Feature Overview
  - Articles
- Downloads
- Packages
- Community
  - Blog
  - Orgs using D
  - Twitter
  - Forums
  - IRC
  - Wiki
  - GitHub
  - Issues
  - Get involved
  - Contributors
  - Foundation
  - Donate
  - Sponsors
- Resources
  - Tour
  - Books
  - Tutorials
  - Tools
  - Editors
  - IDEs
  - Visual D
  - Acknowledgments
  - D Style
  - Glossary
  - Sitemap

Search
Report a bug
If you spot a problem with this page, click here to create a Bugzilla issue.
Improve this page
Quickly fork, edit online, and submit a pull request for this page. Requires a signed-in GitHub account. This works well for small changes. If you'd like to make larger changes you may want to consider using a local clone.

# dmd.lexer

Compiler implementation of the D programming language.
Authors:
Walter Bright
License:
Boost License 1.0

Source lexer.d

Documentation https://dlang.org/phobos/dmd_lexer.html

Coverage https://codecov.io/gh/dlang/dmd/src/master/src/dmd/lexer.d

static immutable ubyte[256] `cmtable`;
  Do our own char maps
class `Lexer`;
  this(const(char)* `filename`, const(char)* `base`, size_t `begoffset`, size_t `endoffset`, bool `doDocComment`, bool `commentToken`);
    Creates a Lexer for the source code `base`[`begoffset`..`endoffset`+1]. The last character, `base`[`endoffset`], must be `null` (0) or EOF (0x1A).
    Parameters:

    const(char)* `filename` | used for error messages
    const(char)* `base` | source code, must be terminated by a `null` (0) or EOF (0x1A) character
    size_t `begoffset` | starting offset into `base`[]
    size_t `endoffset` | the last offset to read into `base`[]
    bool `doDocComment` | handle documentation comments
    bool `commentToken` | comments become TOK.comment's

  final TOK `peekNext`();
    Look ahead at next token's value.
  final TOK `peekNext2`();
    Look 2 tokens ahead at value.
  final void `scan`(Token* `t`);
    Turn next token in buffer into a token.
  final Token* `peekPastParen`(Token* `tk`);
    `tk` is on the opening (. Look ahead and return token that is past the closing ).
  final uint `escapeSequence`();
    Parse escape sequence.
  final TOK `wysiwygStringConstant`(Token* `t`, int `tc`);
  final TOK `hexStringConstant`(Token* `t`);
    Lex hex strings: x"0A ae 34FE BD"
  final TOK `delimitedStringConstant`(Token* `t`);
    Lex delimited strings: q"(foo(xxx))" // "foo(xxx)" q"[foo(]" // "foo(" q"/foo]/" // "foo]" q"HERE foo HERE" // "foo\n"

    Input p is on the "

  final TOK `tokenStringConstant`(Token* `t`);
    Lex delimited strings: q{ foo(xxx) } // " foo(xxx) " q{foo(} // "foo(" q{{foo}"}"} // "{foo}"}""

    Input p is on the q

  final TOK `escapeStringConstant`(Token* `t`, int `wide`);
  final TOK `charConstant`(Token* `t`);
  final void `stringPostfix`(Token* `t`);
    Get postfix of string literal.
  final TOK `number`(Token* `t`);
    Read in a `number`. If it's an integer, store it in tok.TKutok.Vlong. integers can be decimal, octal or hex Handle the suffixes U, UL, LU, L, etc. If it's double, store it in tok.TKutok.Vdouble.
    Returns:
    TKnum TKdouble,...
  final TOK `inreal`(Token* `t`);
    Read in characters, converting them to real.
    Bugs:
    Exponent overflow not detected. Too much requested precision is not detected.
  final void `poundLine`();
    parse: #line linnum [filespec] also allow __LINE__ for linnum, and __FILE__ for filespec
  final uint `decodeUTF`();
    Decode UTF character. Issue error messages for invalid sequences. Return decoded character, advance p to last character in UTF sequence.
  final void `getDocComment`(Token* `t`, uint `lineComment`, bool `newParagraph`);
    Parse doc comment embedded between `t`.ptr and p. Remove trailing blanks and tabs from lines. Replace all newlines with \n. Remove leading comment character from each line. Decide if it's a `lineComment` or a blockComment. Append to previous one for this token.
    If `newParagraph` is `true`, an extra newline will be added between adjoining doc comments.
  static const(char)* `combineComments`(const(char)* `c1`, const(char)* `c2`, bool `newParagraph`);
    Combine two document comments into one, separated by an extra newline if `newParagraph` is `true`.

Copyright © 1999-2018 by the D Language Foundation | Page generated by Ddoc on Fri Mar 2 18:16:35 2018