Lexical - D Programming Language
Description: D Programming Language
Keywords: D programming language

Menu

- Learn
- Documentation
  - Language Reference
  - Library Reference
  - Command-line Reference
  - Feature Overview
  - Articles
- Downloads
- Packages
- Community
  - Blog
  - Orgs using D
  - Twitter
  - Calendar
  - Forums
  - IRC
  - Community Discord
  - Wiki
  - GitHub
  - Issues
  - Get involved
  - Contributors
  - Foundation
  - Security Team
  - Donate
  - Sponsors
- Resources
  - Tour
  - Books
  - Tutorials
  - Tools
  - Editors
  - IDEs
  - run.dlang.io
  - Visual D
  - Acknowledgments
  - D Style
  - Glossary
  - Sitemap

Search
Report a bug
If you spot a problem with this page, click here to create a Bugzilla issue.
Improve this page
Quickly fork, edit online, and submit a pull request for this page. Requires a signed-in GitHub account. This works well for small changes. If you'd like to make larger changes you may want to consider using a local clone.

# Lexical

Contents

1. Source Text
2. Character Set
3. End of File
4. End of Line
5. White Space
6. Comments
7. Tokens
8. Identifiers
9. String Literals
   1. Wysiwyg Strings
   2. Double Quoted Strings
   3. Hex Strings
   4. Delimited Strings
   5. Token Strings
   6. String Postfix
10. Escape Sequences
11. Character Literals
12. Integer Literals
13. Floating Point Literals
14. Keywords
15. Special Tokens
16. Special Token Sequences

The lexical analysis is independent of the syntax parsing and the semantic analysis. The lexical analyzer splits the source text into tokens. The lexical grammar describes the syntax of these tokens. The grammar is designed to be suitable for high-speed scanning and to facilitate the implementation of a correct scanner. It has a minimum of special case rules and there is only one phase of translation.

## Source Text

    SourceFile:
        ByteOrderMark Moduleopt
        Shebang Moduleopt
        Moduleopt

    ByteOrderMark:
        \uFEFF

    Shebang:
        #! Charactersopt EndOfShebang

    EndOfShebang:
        \u000A
        EndOfFile

Source text can be encoded as any one of the following:

- ASCII (strictly, 7-bit ASCII)
- UTF-8
- UTF-16BE
- UTF-16LE
- UTF-32BE
- UTF-32LE

One of the following UTF BOMs (Byte Order Marks) can be present at the beginning of the source text:

UTF Byte Order Marks
Format | BOM
--- | ---
UTF-8 | EF BB BF
UTF-16BE | FE FF
UTF-16LE | FF FE
UTF-32BE | 00 00 FE FF
UTF-32LE | FF FE 00 00
ASCII | no BOM

If the source file does not begin with a BOM, then the first character must be less than or equal to U+0000007F.

The source text is decoded from its source representation into Unicode Characters. The Characters are further divided into: WhiteSpace, EndOfLine, Comments, SpecialTokenSequences, and Tokens, with the source terminated by an EndOfFile.

The source text is split into tokens using the maximal munch algorithm, i.e., the lexical analyzer assumes the longest possible token. For example, >> is a right-shift token rather than two greater-than tokens. There are two exceptions to this rule:

- A .. embedded between what appear to be two floating point literals, as in 1..2, is interpreted as if the .. were separated by a space from the first integer.
- A 1.a is interpreted as the three tokens 1, ., and a, whereas 1. a is interpreted as the two tokens 1. and a.

## Character Set

    Character:
        any Unicode character

## End of File

    EndOfFile:
        physical end of the file
        \u0000
        \u001A

The source text is terminated by whichever comes first.

## End of Line

    EndOfLine:
        \u000D
        \u000A
        \u000D \u000A
        \u2028
        \u2029
        EndOfFile

## White Space

    WhiteSpace:
        Space
        Space WhiteSpace

    Space:
        \u0020
        \u0009
        \u000B
        \u000C

## Comments

    Comment:
        BlockComment
        LineComment
        NestingBlockComment

    BlockComment:
        /* Charactersopt */

    LineComment:
        // Charactersopt EndOfLine

    NestingBlockComment:
        /+ NestingBlockCommentCharactersopt +/

    NestingBlockCommentCharacters:
        NestingBlockCommentCharacter
        NestingBlockCommentCharacter NestingBlockCommentCharacters

    NestingBlockCommentCharacter:
        Character
        NestingBlockComment

    Characters:
        Character
        Character Characters

There are three kinds of comments:

1. Block comments can span multiple lines, but do not nest.
2. Line comments terminate at the end of the line.
3. Nesting block comments can span multiple lines and can nest.

The contents of strings and comments are not tokenized. Consequently, comment openings occurring within a string do not begin a comment, and string delimiters within a comment do not affect the recognition of comment closings and nested /+ comment openings. With the exception of /+ occurring within a /+ comment, comment openings within a comment are ignored.

    a = /+ // +/ 1;    // parses as if 'a = 1;'
    a = /+ "+/" +/ 1"; // parses as if 'a = " +/ 1";'
    a = /+ /* +/ */ 3; // parses as if 'a = */ 3;'

Comments cannot be used as token concatenators, for example, abc/**/def is two tokens, abc and def, not one abcdef token.

## Tokens

    Tokens:
        Token
        Token Tokens

    Token:
        {
        }
        TokenNoBraces

    TokenNoBraces:
        Identifier
        StringLiteral
        CharacterLiteral
        IntegerLiteral
        FloatLiteral
        Keyword
        /
        /=
        .
        ..
        ...
        &
        &=
        &&
        |
        |=
        ||
        -
        -=
        --
        +
        +=
        ++
        <
        <=
        <<
        <<=
        >
        >=
        >>=
        >>>=
        >>
        >>>
        !
        !=
        (
        )
        [
        ]
        ?
        ,
        ;
        :
        $
        =
        ==
        *
        *=
        %
        %=
        ^
        ^=
        ^^
        ^^=
        ~
        ~=
        @
        =>
        #

## Identifiers

    Identifier:
        IdentifierStart
        IdentifierStart IdentifierChars

    IdentifierChars:
        IdentifierChar
        IdentifierChar IdentifierChars

    IdentifierStart:
        _
        Letter
        UniversalAlpha

    IdentifierChar:
        IdentifierStart
        0
        NonZeroDigit

Identifiers start with a letter, _, or universal alpha, and are followed by any number of letters, _, digits, or universal alphas. Universal alphas are as defined in ISO/IEC 9899:1999(E) Appendix D of the C99 Standard. Identifiers can be arbitrarily long, and are case sensitive.

Implementation Defined: Identifiers starting with __ (two underscores) are reserved.

## String Literals

    StringLiteral:
        WysiwygString
        AlternateWysiwygString
        DoubleQuotedString

        HexString
        DelimitedString
        TokenString

    WysiwygString:
        r" WysiwygCharactersopt " StringPostfixopt

    AlternateWysiwygString:
        ` WysiwygCharactersopt ` StringPostfixopt

    WysiwygCharacters:
        WysiwygCharacter
        WysiwygCharacter WysiwygCharacters

    WysiwygCharacter:
        Character
        EndOfLine

    DoubleQuotedString:
        " DoubleQuotedCharactersopt " StringPostfixopt

    DoubleQuotedCharacters:
        DoubleQuotedCharacter
        DoubleQuotedCharacter DoubleQuotedCharacters

    DoubleQuotedCharacter:
        Character
        EscapeSequence
        EndOfLine

    EscapeSequence:
        \'
        \"
        \?
        \\
        \0
        \a
        \b
        \f
        \n
        \r
        \t
        \v
        \x HexDigit HexDigit
        \ OctalDigit
        \ OctalDigit OctalDigit
        \ OctalDigit OctalDigit OctalDigit
        \u HexDigit HexDigit HexDigit HexDigit
        \U HexDigit HexDigit HexDigit HexDigit HexDigit HexDigit HexDigit HexDigit
        \ NamedCharacterEntity

    HexString:
        x" HexStringCharsopt " StringPostfixopt

    HexStringChars:
        HexStringChar
        HexStringChar HexStringChars

    HexStringChar:
        HexDigit
        WhiteSpace
        EndOfLine

    StringPostfix:
        c
        w
        d

    DelimitedString:
        q" Delimiter WysiwygCharactersopt MatchingDelimiter " StringPostfixopt
        q"( ParenDelimitedCharactersopt )" StringPostfixopt
        q"[ BracketDelimitedCharactersopt ]" StringPostfixopt
        q"{ BraceDelimitedCharactersopt }" StringPostfixopt
        q"< AngleDelimitedCharactersopt >" StringPostfixopt

    Delimiter:
        Identifier

    MatchingDelimiter:
        Identifier

    ParenDelimitedCharacters:
        WysiwygCharacter
        WysiwygCharacter ParenDelimitedCharacters
        ( ParenDelimitedCharactersopt )

    BracketDelimitedCharacters:
        WysiwygCharacter
        WysiwygCharacter BracketDelimitedCharacters
        [ BracketDelimitedCharactersopt ]

    BraceDelimitedCharacters:
        WysiwygCharacter
        WysiwygCharacter BraceDelimitedCharacters
        { BraceDelimitedCharactersopt }

    AngleDelimitedCharacters:
        WysiwygCharacter
        WysiwygCharacter AngleDelimitedCharacters
        < AngleDelimitedCharactersopt >

    TokenString:
        q{ TokenStringTokensopt } StringPostfixopt

    TokenStringTokens:
        TokenStringToken
        TokenStringToken TokenStringTokens

    TokenStringToken:
        TokenNoBraces
        { TokenStringTokensopt }

A string literal is either a double quoted string, a wysiwyg quoted string, a delimited string, a token string, or a hex string.

In all string literal forms, an EndOfLine is regarded as a single \n character.

String literals are read only.

Undefined Behavior: Writes to string literals cannot always be detected, but cause undefined behavior.

### Wysiwyg Strings

Wysiwyg ("what you see is what you get") quoted strings can be defined using either of two syntaxes.

In the first form, they are enclosed between r" and ". All characters between the r" and " are part of the string. There are no escape sequences inside wysiwyg strings.

    r"I am Oz"
    r"c:\games\Sudoku.exe"
    r"ab\n" // string is 4 characters,
            // 'a', 'b', '\', 'n'

Alternatively, wysiwyg strings can be enclosed by backquotes, using the ` character.

    `the Great and Powerful.`
    `c:\games\Empire.exe`
    `The "lazy" dog`
    `a"b\n`  // string is 5 characters,
             // 'a', '"', 'b', '\', 'n'

### Double Quoted Strings

Double quoted strings are enclosed by "". EscapeSequences can be embedded in them.

    "Who are you?"
    "c:\\games\\Doom.exe"
    "ab\n"   // string is 3 characters,
             // 'a', 'b', and a linefeed
    "ab
    "        // string is 3 characters,
             // 'a', 'b', and a linefeed

### Hex Strings

Hex strings allow string literals to be created using hex data. The hex data need not form valid UTF characters.

    x"0A"              // same as "\x0A"
    x"00 FBCD 32FD 0A" // same as
                       // "\x00\xFB\xCD\x32\xFD\x0A"

Whitespace and newlines are ignored, so the hex data can be easily formatted. The number of hex characters must be a multiple of 2.

Note: Hex Strings are deprecated. Please use std.conv.hexString instead.

### Delimited Strings

Delimited strings use various forms of delimiters. The delimiter, whether a character or identifier, must immediately follow the " without any intervening whitespace. The terminating delimiter must immediately precede the closing " without any intervening whitespace. A nesting delimiter nests, and is one of the following characters:

Nesting Delimiters
Delimiter | Matching Delimiter
--- | ---
[ | ]
( | )
< | >
{ | }

    q"(foo(xxx))"   // "foo(xxx)"
    q"[foo{]"       // "foo{"

If the delimiter is an identifier, the identifier must be immediately followed by a newline, and the matching delimiter must be the same identifier starting at the beginning of the line:

    writeln(q"EOS
    This
    is a multi-line
    heredoc string
    EOS"
    );

The newline following the opening identifier is not part of the string, but the last newline before the closing identifier is part of the string. The closing identifier must be placed on its own line at the leftmost column.

Otherwise, the matching delimiter is the same as the delimiter character:

    q"/foo]/"          // "foo]"
    // q"/abc/def/"    // error

### Token Strings

Token strings open with the characters q{ and close with the token }. In between must be valid D tokens. The { and } tokens nest. The string is formed of all the characters between the opening and closing of the token string, including comments.

    q{this is the voice of} // "this is the voice of"
    q{/*}*/ }               // "/*}*/ "
    q{ world(q{control}); } // " world(q{control}); "
    q{ __TIME__ }           // " __TIME__ "
                            // i.e. it is not replaced with the time
    // q{ __EOF__ }         // error
                            // __EOF__ is not a token, it's end of file

### String Postfix

The optional StringPostfix character gives a specific type to the string, rather than it being inferred from the context. The types corresponding to the postfix characters are:

String Literal Postfix Characters
Postfix | Type | Alias
--- | --- | ---
c | immutable(char)[] | string
w | immutable(wchar)[] | wstring
d | immutable(dchar)[] | dstring

    "hello"c  // string
    "hello"w  // wstring
    "hello"d  // dstring

The string literals are assembled as UTF-8 char arrays, and the postfix is applied to convert to wchar or dchar as necessary as a final step.

## Escape Sequences

The escape sequences listed in EscapeSequence are:

Escape Sequences
Sequence | Meaning
--- | ---
' | Literal single-quote: '
" | Literal double-quote: "
? | Literal question mark: ?
\ | Literal backslash:
\0 | Binary zero (NUL, U+0000).
\a | BEL (alarm) character (U+0007).
\b | Backspace (U+0008).
\f | Form feed (FF) (U+000C).
\n | End-of-line (U+000A).
\r | Carriage return (U+000D).
\t | Horizontal tab (U+0009).
\v | Vertical tab (U+000B).
\xnn | Byte value in hexadecimal, where nn is specified as two hexadecimal digits. For example: \xFF represents the character with the value 255.
n nn nnn | Byte value in octal. For example: \101 represents the character with the value 65 ('A'). Analogous to hexadecimal characters, the largest byte value is \377 (= \xFF in hexadecimal or 255 in decimal)
\unnnn | Unicode character U+nnnn, where nnnn are four hexadecimal digits. For example, \u03B3 represents the Unicode character γ (U+03B3 - GREEK SMALL LETTER GAMMA).
\Unnnnnnnn | Unicode character U+nnnnnnnn, where nnnnnnnn are 8 hexadecimal digits. For example, \U0001F603 represents the Unicode character U+1F603 (SMILING FACE WITH OPEN MOUTH).
name | Named character entity from the HTML5 specification. These names begin with & and end with ; | e.g., &euro;. See NamedCharacterEntity.

## Character Literals

    CharacterLiteral:
        ' SingleQuotedCharacter '

    SingleQuotedCharacter:
        Character
        EscapeSequence

Character literals are a single character or escape sequence enclosed by single quotes.

    'h'   // the letter h
    '\n'  // newline
    '\\'  // the backslash character

## Integer Literals

    IntegerLiteral:
        Integer
        Integer IntegerSuffix

    Integer:
        DecimalInteger
        BinaryInteger
        HexadecimalInteger

    IntegerSuffix:
        L
        u
        U
        Lu
        LU
        uL
        UL

    DecimalInteger:
        0
        NonZeroDigit
        NonZeroDigit DecimalDigitsUS

    BinaryInteger:
        BinPrefix BinaryDigitsNoSingleUS

    BinPrefix:
        0b
        0B

    HexadecimalInteger:
        HexPrefix HexDigitsNoSingleUS

    NonZeroDigit:
        1
        2
        3
        4
        5
        6
        7
        8
        9

    DecimalDigits:
        DecimalDigit
        DecimalDigit DecimalDigits

    DecimalDigitsUS:
        DecimalDigitUS
        DecimalDigitUS DecimalDigitsUS

    DecimalDigitsNoSingleUS:
        DecimalDigit
        DecimalDigit DecimalDigitsUS
        DecimalDigitsUS DecimalDigit

    DecimalDigitsNoStartingUS:
        DecimalDigit
        DecimalDigit DecimalDigitsUS

    DecimalDigit:
        0
        NonZeroDigit

    DecimalDigitUS:
        DecimalDigit
        _

    BinaryDigitsNoSingleUS:
        BinaryDigit
        BinaryDigit BinaryDigitsUS
        BinaryDigitsUS BinaryDigit
        BinaryDigitsUS BinaryDigit BinaryDigitsUS

    BinaryDigitsUS:
        BinaryDigitUS
        BinaryDigitUS BinaryDigitsUS

    BinaryDigit:
        0
        1

    BinaryDigitUS:
        BinaryDigit
        _

    OctalDigit:
        0
        1
        2
        3
        4
        5
        6
        7

    HexDigits:
        HexDigit
        HexDigit HexDigits

    HexDigitsUS:
        HexDigitUS
        HexDigitUS HexDigitsUS

    HexDigitsNoSingleUS:
        HexDigit
        HexDigit HexDigitsUS
        HexDigitsUS HexDigit

    HexDigitsNoStartingUS:
        HexDigit
        HexDigit HexDigitsUS

    HexDigit:
        DecimalDigit
        HexLetter

    HexDigitUS:
        HexDigit
        _

    HexLetter:
        a
        b
        c
        d
        e
        f
        A
        B
        C
        D
        E
        F

Integers can be specified in decimal, binary, or hexadecimal.

Decimal integers are a sequence of decimal digits.

Binary integers are a sequence of binary digits preceded by a ‘0b’ or ‘0B’.

C-style octal integer notation was deemed too easy to mix up with decimal notation; it is only fully supported in string literals. D still supports octal integer literals interpreted at compile time through the std.conv.octal template, as in octal!167.

Hexadecimal integers are a sequence of hexadecimal digits preceded by a ‘0x’ or ‘0X’.

Integers can have embedded ‘_’ characters to improve readability, and which are ignored.

    20_000        // leagues under the sea
    867_5309      // number on the wall
    1_522_000     // thrust of F1 engine (lbf sea level)

Integers can be immediately followed by one ‘L’ or one of ‘u’ or ‘U’ or both. Note that there is no ‘l’ suffix.

The type of the integer is resolved as follows:

Decimal Literal Types
Literal | Type
--- | ---
| Usual decimal notation
0 .. 2_147_483_647 | int
2_147_483_648 .. 9_223_372_036_854_775_807 | long
9_223_372_036_854_775_808 .. 18_446_744_073_709_551_615 | ulong
| Explicit suffixes
0L .. 9_223_372_036_854_775_807L | long
0U .. 4_294_967_295U | uint
4_294_967_296U .. 18_446_744_073_709_551_615U | ulong
0UL .. 18_446_744_073_709_551_615UL | ulong
| Hexadecimal notation
0x0 .. 0x7FFF_FFFF | int
0x8000_0000 .. 0xFFFF_FFFF | uint
0x1_0000_0000 .. 0x7FFF_FFFF_FFFF_FFFF | long
0x8000_0000_0000_0000 .. 0xFFFF_FFFF_FFFF_FFFF | ulong
| Hexadecimal notation with explicit suffixes
0x0L .. 0x7FFF_FFFF_FFFF_FFFFL | long
0x8000_0000_0000_0000L .. 0xFFFF_FFFF_FFFF_FFFFL | ulong
0x0U .. 0xFFFF_FFFFU | uint
0x1_0000_0000U .. 0xFFFF_FFFF_FFFF_FFFFU | ulong
0x0UL .. 0xFFFF_FFFF_FFFF_FFFFUL | ulong

An integer literal may not exceed these values.

Best Practices: Octal integer notation is not supported for integer literals. However, octal integer literals can be interpreted at compile time through the std.conv.octal template, as in octal!167.

## Floating Point Literals

    FloatLiteral:
        Float
        Float Suffix
        Integer FloatSuffix
        Integer ImaginarySuffix
        Integer FloatSuffix ImaginarySuffix
        Integer RealSuffix ImaginarySuffix

    Float:
        DecimalFloat
        HexFloat

    DecimalFloat:
        LeadingDecimal .
        LeadingDecimal . DecimalDigits
        DecimalDigits . DecimalDigitsNoStartingUS DecimalExponent
        . DecimalInteger
        . DecimalInteger DecimalExponent
        LeadingDecimal DecimalExponent

    DecimalExponent:
        DecimalExponentStart DecimalDigitsNoSingleUS

    DecimalExponentStart:
        e
        E
        e+
        E+
        e-
        E-

    HexFloat:
        HexPrefix HexDigitsNoSingleUS . HexDigitsNoStartingUS HexExponent
        HexPrefix . HexDigitsNoStartingUS HexExponent
        HexPrefix HexDigitsNoSingleUS HexExponent

    HexPrefix:
        0x
        0X

    HexExponent:
        HexExponentStart DecimalDigitsNoSingleUS

    HexExponentStart:
        p
        P
        p+
        P+
        p-
        P-

    Suffix:
        FloatSuffix
        RealSuffix
        ImaginarySuffix
        FloatSuffix ImaginarySuffix
        RealSuffix ImaginarySuffix

    FloatSuffix:
        f
        F

    RealSuffix:
        L

    ImaginarySuffix:
        i

    LeadingDecimal:
        DecimalInteger
        0 DecimalDigitsNoSingleUS

Floats can be in decimal or hexadecimal format.

Hexadecimal floats are preceded by a 0x or 0X and the exponent is a p or P followed by a decimal number serving as the exponent of 2.

Floating literals can have embedded ‘_’ characters to improve readability, and which are ignored.

    2.645_751
    6.022140857E+23
    6_022.140857E+20
    6_022_.140_857E+20_

Floating literals with no suffix are of type double. Floating literals followed by f or F are of type float, and those followed by L are of type real.

Examples:

    0x1.FFFFFFFFFFFFFp1023 // double.max
    0x1p-52                // double.epsilon
    1.175494351e-38F       // float.min

The literal may not exceed the range of the type. The literal is rounded to fit into the significant digits of the type.

If a floating literal has a . and a type suffix, at least one digit must be in-between:

    1f;  // OK, float
    1.f; // error
    1.;  // OK, double

NOTE: Floating literals followed by i to denote imaginary floating point values have been deprecated.

## Keywords

Keywords are reserved identifiers.

    Keyword:
        abstract
        alias
        align
        asm
        assert
        auto

        body
        bool
        break
        byte

        case
        cast
        catch
        cdouble
        cent
        cfloat
        char
        class
        const
        continue
        creal

        dchar
        debug
        default
        delegate
        delete
        deprecated
        do
        double

        else
        enum
        export
        extern

        false
        final
        finally
        float
        for
        foreach
        foreach_reverse
        function

        goto

        idouble
        if
        ifloat
        immutable
        import
        in
        inout
        int
        interface
        invariant
        ireal
        is

        lazy
        long

        macro
        mixin
        module

        new
        nothrow
        null

        out
        override

        package
        pragma
        private
        protected
        public
        pure

        real
        ref
        return

        scope
        shared
        short
        static
        struct
        super
        switch
        synchronized

        template
        this
        throw
        true
        try
        typeid
        typeof

        ubyte
        ucent
        uint
        ulong
        union
        unittest
        ushort

        version
        void

        wchar
        while
        with

        __FILE__
        __FILE_FULL_PATH__
        __MODULE__
        __LINE__
        __FUNCTION__
        __PRETTY_FUNCTION__

        __gshared
        __traits
        __vector
        __parameters

## Special Tokens

These tokens are replaced with other tokens according to the following table:

Special Tokens
Special Token | Replaced with
--- | ---
__DATE__ | string literal of the date of compilation "mmm dd yyyy"
__EOF__ | tells the scanner to ignore everything after this token
__TIME__ | string literal of the time of compilation "hh:mm:ss"
__TIMESTAMP__ | string literal of the date and time of compilation "www mmm dd hh:mm:ss yyyy"
__VENDOR__ | Compiler vendor string
__VERSION__ | Compiler version as an integer

Implementation Defined: The replacement string literal for __VENDOR__ and the replacement integer value for __VERSION__.

## Special Token Sequences

    SpecialTokenSequence:
        # line IntegerLiteral Filespecopt EndOfLine
        # line __LINE__ Filespecopt EndOfLine

    Filespec:
        " Charactersopt "

Special token sequences are processed by the lexical analyzer, may appear between any other tokens, and do not affect the syntax parsing.

There is currently only one special token sequence, #line.

This sets the current source line number to IntegerLiteral, and optionally the current source file name to Filespec, beginning with the next line of source text.

The backslash character is not treated specially inside Filespec strings.

For example:

    int #line 6 "pkg/mod.d"
    x;  // this is now line 6 of file pkg/mod.d

Implementation Defined: The source file and line number is typically used for printing error messages and for mapping generated code back to the source for the symbolic debugging output.
Introduction
Grammar
Copyright © 1999-2021 by the D Language Foundation | Page generated by Ddoc on Sun Oct 10 11:25:53 2021