D Programming Language 2.0

- Comments
- D
- Search
- Home

Last update Tue Dec 13 15:30:34 2011

# std.regex

Regular expressions are commonly used method of pattern matching on strings, with regex being a catchy word for a pattern in this domain specific language. Typical problems usually solved by regular expressions include validation of user input and ubiquitous find & replace in text processing utilities.

Synposis:

      import std.regex;
      import std.stdio;
      void main()
      {
          //print out all possible dd/mm/yy(yy) dates found in user input
          //g - global, find all matches
          auto r = regex(r"\b[0-9][0-9]?/[0-9][0-9]?/[0-9][0-9](?:[0-9][0-9])?\b", "g");
          foreach(line; stdin.byLine)
          {
            //match returns a range that can be iterated
            //to get all of subsequent matches
            foreach(c; match(line, r))
                writeln(c.hit);
          }
      }
      ...

      //create static regex at compile-time, contains fast native code
      enum ctr = ctRegex!(`^.*/([^/]+)/?$`);

      //works just like normal regex:
      auto m2 = match("foo/bar", ctr);   //first match found here if any
      assert(m2);   // be sure to check if there is a match, before examining contents!
      assert(m2.captures[1] == "bar");//captures is a range of submatches, 0 - full match

      ...

      //result of match is directly testable with if/assert/while
      //e.g. test if a string consists of letters:
      assert(match("Letter", `^\p{L}+$`));

The general usage guideline is keeping regex complexity on the side of simplicity, as its capabilities reside in purely character-level manipulation, and as such are ill suited for tasks involving higher level invariants like matching an integer number bounded in [a,b] interval. Checks of this sort of are better addressed by additional post-processing.

The basic syntax shouldn't surprize experienced users of regular expressions. Thankfully, nowdays the web is bustling with resources to help newcomers, and a good reference with tutorial on regular expressions could be found.

This library uses ECMAScript syntax flavor with the following extensions:

- Named subexpressions, with Python syntax.
- Unicode properties such as Scripts, Blocks and common binary properties e.g Alphabetic, White_Space, Hex_Digit etc.
- Arbitrary length and complexity lookbehind, including lookahead in lookbehind and vise-versa.

###
Pattern syntax

std.regex operates on codepoint level, 'character' in this table denotes single unicode codepoint.

Pattern element | Semantics
Atoms | Match single characters
any character except [|*+?() | Matches the character itself.
. | In single line mode matches any charcter. Otherwise it matches any character except '\n' and '\r'.
[class] | Matches single character that belongs to this character class.
[^class] | Matches single character that does not belong to this character class.
\cC | Matches the control character corresponding to letter C
\xXX | Matches a character with hexadecimal value of XX.
\uXXXX | Matches a character with hexadecimal value of XXXX.
\U00YYYYYY | Matches a character with hexadecimal value of YYYYYY.
\f | Matches a formfeed character.
\n | Matches a linefeed character.
\r | Matches a carriage return character.
\t | Matches a tab character.
\v | Matches a vertical tab character.
\d | Matches any unicode digit.
\D | Matches any character but unicode digit.
\w | Matches any word character (note: this includes numbers).
\W | Matches any non-word character.
\s | Matches whitespace, same as \p{White_Space}.
\S | Matches any character but these recognized as \s .
\\ | Matches \ character.
\c where c is one of [|*+?() | Matches the character c itself.
\p{PropertyName} | Matches character that belongs to unicode PropertyName set. Single letter abreviations could be used without surrounding {,}.
\P{PropertyName} | Matches character that does not belong to unicode PropertyName set. Single letter abreviations could be used without surrounding {,}.
\p{InBasicLatin} | Matches any character that is part of BasicLatin unicode block.
\P{InBasicLatin} | Matches any character except ones in BasicLatin unicode block.
\p{Cyrilic} | Matches any character that is part of Cyrilic script.
\P{Cyrilic} | Matches any character except ones in Cyrilic script.
Quantifiers | Specify repetition of other elements
* | Matches previous character/subexpression 0 or more times. Greedy version - tries as many times as possible.
*? | Matches previous character/subexpression 0 or more times. Lazy version - stops as early as possible.
+ | Matches previous character/subexpression 1 or more times. Greedy version - tries as many times as possible.
+? | Matches previous character/subexpression 1 or more times. Lazy version - stops as early as possible.
{n} | Matches previous character/subexpression n exactly times.
{n,} | Matches previous character/subexpression n times or more. Greedy version - tries as many times as possible.
{n,}? | Matches previous character/subexpression n times or more. Lazy version - stops as early as possible.
{n,m} | Matches previous character/subexpression n to m times. Greedy version - tries as many times as possible.
{n,m}? | Matches previous character/subexpression n to m times. Lazy version - stops as early as possible, but no less then n times.
Other | Subexpressions & alternations
(regex) | Matches subexpression regex, saving matched portion of text for later retrival.
(?:regex) | Matches subexpression regex, not saving matched portion of text. Useful to speed up matching.
A|B | Matches subexpression A, failing that matches B.
(?P<name>regex) | Matches named subexpression regex labeling it with name 'name'. When refering to matched portion of text, names work like aliases in addition to direct numbers.
Assertions | Match position rather then character
^ | Matches at the begining of input or line (in multiline mode).
$ | Matches at the end of input or line (in multiline mode).
\b | Matches at word boundary.
\B | Matches when not at word boundary.
(?=regex) | Zero-width lookahead assertion. Matches at a point where the subexpression regex could be matched starting from current position.
(?!regex) | Zero-width negative lookahead assertion. Matches at a point where the subexpression regex could not be matched starting from current position.
(?<=regex) | Zero-width lookbehind assertion. Matches at a point where the subexpression regex could be matched ending at current position (matching goes backwards).
(?<!regex) | Zero-width negative lookbehind assertion. Matches at a point where the subexpression regex could not be matched ending at current position (matching goes backwards).

###
Character classes

Pattern element | Semantics
Any atom | Have the same meaning as outside of character class.
a-z | Includes characters a, b, c, ..., z.
[a||b], [a--b], [a~~b], [a&&b] | Where a, b are arbitrary classes, means union, set difference, symmetric set difference, and intersection respectively. Any sequence of character class elements implicitly forms union.

###
Regex flags

Flag | Semantics
g | Global regex, repeat over the whole input.
i | Case insensitive matching.
m | Multi-line mode, match ^, $ on start and end line separators as well as start and end of input.
s | Single-line mode, makes . match '\n' and '\r' as well.
x | Free-form syntax, ignores whitespace in pattern, useful for formating complex regular expressions.

Unicode support

This library provides full Level 1 support* according to UTS 18. Specifically:

- 1.1 Hex notation via any of \uxxxx, \U00YYYYYY, \xZZ.
- 1.2 Unicode properties.
- 1.3 Character classes with set operations.
- 1.4 Word boundaries use full set of "word" characters.
- 1.5 Using simple casefolding to match case insensitevely across full range of codepoints.
- 1.6 Respecting line breaks as any of \u000A | \u000B | \u000C | \u000D | \u0085 | \u2028 | \u2029 | \u000D\u000A.
- 1.7 Operating on codepoint level.

*With exception of point 1.1.1, as of yet, normalization of input is expected to be enforced by user.

All matches returned by pattern matching functionality in this library are slices of original input. Notable exception being replace family of functions that generate new string from input.

License:
Boost License 1.0.

Authors:
Dmitry Olshansky,

API and utility constructs are based on original std.regex by Walter Bright and Andrei Alexandrescu

struct Regex(Char);
  Regex object holds regular expression pattern in compiled form. Instances of this object are constructed via calls to regex. This is an intended form for caching and storage of frequently used regular expressions.

  const nothrow bool empty();
    Test if this object doesn't contain any compiled pattern.

    Example:

                Regex!char r;
                assert(r.empty);
                r = regex("");//note: "" is a valid regex pattern
                assert(!r.empty);

struct StaticRegex(Char);
  A StaticRegex is Regex object that contains specially generated machine code to speed up matching. Implicitly convertible to normal Regex, however doing so will result in loosing this additional capability.

struct Captures(R,DIndex) if (isSomeString!(R));
  Captures object contains submatches captured during a call to match or iteration over RegexMatch range.

  First element of range is the whole match.

  Example, showing basic operations on Captures:

          import std.regex;
          import std.range;

          void main()
          {
              auto m = match("@abc#", regex(`(\w)(\w)(\w)`));
              auto c = m.captures;
              assert(c.pre == "@");// part of input preceeding match
              assert(c.post == "#"); // immediately after match
              assert(c.hit == c[0] && c.hit == "abc");// the whole match
              assert(c[2] =="b");
              assert(c.front == "abc");
              c.popFront();
              assert(c.front == "a");
              assert(c.back == "c");
              c.popBack();
              assert(c.back == "b");
              popFrontN(c, 2);
              assert(c.empty);
          }

  R pre();
    Slice of input prior to the match.

  R post();
    Slice of input immediately after the match.

  R hit();
    Slice of matched portion of input.

  R front();
  R back();
  void popFront();
  void popBack();
  const bool empty();
  R opIndex()(size_t i);
    Range interface.

  R opIndex(String)(String i);
    Lookup named submatch.

                import std.regex;
                import std.range;

                auto m = match("a = 42;", regex(`(?P<var>\w+)\s*=\s*(?P<value>\d+);`));
                auto c = m.captures;
                assert(c["var"] == "a");
                assert(c["value"] == "42");
                popFrontN(c, 2);
                //named groups are unaffected by range primitives
                assert(c["var"] =="a");
                assert(c.front == "42");

  const size_t length();
    Number of matches in this object.

  @property ref auto captures();
    A hook for compatibility with original std.regex.

struct RegexMatch(R,alias Engine = ThompsonMatcher) if (isSomeString!(R));
  A regex engine state, as returned by match family of functions.

  Effectively it's a forward range of Captures!R, produced by lazily searching for matches in a given input.

  alias Engine specifies an engine type to use during matching, and is automatically deduced in a call to match/bmatch.

  R pre();
  R post();
  R hit();
    Shorthands for front.pre, front.post, front.hit.

  @property auto front();
  void popFront();
  auto save();
    Functionality for processing subsequent matches of global regexes via range interface:

                import std.regex;
                auto m = match("Hello, world!", regex(`\w+`, "g"));
                assert(m.front.hit == "Hello");
                m.popFront();
                assert(m.front.hit == "world");
                m.popFront();
                assert(m.empty);

  bool empty();
    Test if this match object is empty.

  T opCast(T : bool)();
    Same as !(x.empty), provided for its convenience in conditional statements.

  @property auto captures();
    Same as .front, provided for compatibility with original std.regex.

auto regex(S)(S pattern, const(char)[] flags = "");
  Compile regular expression pattern for the later execution.

  Returns:
  Regex object that works on inputs having the same character width as pattern.

  Parameters:

  pattern | Regular expression
  flags | The attributes (g, i, m and x accepted)

  Throws:
  RegexException if there were any errors during compilation.

template ctRegex(alias pattern,alias flags = [])
  Experimental feature.

  Compile regular expression using CTFE and generate optimized native machine code for matching it.

  Returns:
  StaticRegex object for faster matching.

  Parameters:

  pattern | Regular expression
  flags | The attributes (g, i, m and x accepted)

auto match(R, RegEx)(R input, RegEx re);
auto match(R, String)(R input, String re);
  Start matching input to regex pattern re, using Thompson NFA matching scheme.

  This is the recommended method for matching regular expression.

  re parameter can be one of three types:
  - Plain string, in which case it's compiled to bytecode before matching.
  - Regex!char (wchar/dchar) that contains pattern in form of precompiled bytecode.
  - StaticRegex!char (wchar/dchar) that contains pattern in form of specially crafted native code.

  Returns:
  a RegexMatch object holding engine state after first match.

auto bmatch(R, RegEx)(R input, RegEx re);
auto bmatch(R, String)(R input, String re);
  Start matching input to regex pattern re, using traditional backtracking matching scheme.

  re parameter can be one of three types:
  - Plain string, in which case it's compiled to bytecode before matching.
  - Regex!char (wchar/dchar) that contains pattern in form of precompiled bytecode.
  - StaticRegex!char (wchar/dchar) that contains pattern in form of specially crafted native code.

  Returns:
  a RegexMatch object holding engine state after first match.

R replace(alias scheme = match, R, RegEx)(R input, RegEx re, R format);
  Construct a new string from input by replacing each match with a string generated from match according to format specifier.

  To replace all occurances use regex with "g" flag, otherwise only first occurrence gets replaced.

  Parameters:

  input | string to search
  re | compiled regular expression to use
  format | format string to generate replacements from

  Example:

          //Comify a number
          auto com = regex(r"(?<=\d)(?=(\d\d\d)+\b)","g");
          assert(replace("12000 + 42100 = 56000", com, ",") == "12,000 + 42,100 = 56,100");

  The format string can reference parts of match using the following notation.

  Format specifier | Replaced by
  $& | the whole match.
  $` | part of input preceding the match.
  $' | part of input following the match.
  $$ | '$' character.
  \c , where c is any character | the character c itself.
  \\ | '\' character.
  $1 .. $99 | submatch number 1 to 99 respectively.

          assert(replace("noon", regex("^n"), "[$&]") == "[n]oon");

R replace(alias fun, R, RegEx, alias scheme = match)(R input, RegEx re);
  Search string for matches using regular expression pattern re and pass captures for each match to user-defined functor fun.

  To replace all occurrances use regex with "g" flag, otherwise only first occurrence gets replaced.

  Returns:
  new string with all matches replaced by return values of fun.

  Parameters:

  s | string to search
  re | compiled regular expression
  fun | delegate to use

  Example:
  Capitalize the letters 'a' and 'r':

          string baz(Captures!(string) m)
          {
              return std.string.toUpper(m.hit);
          }
          auto s = replace!(baz)("Strap a rocket engine on a chicken.",
                  regex("[ar]", "g"));
          assert(s == "StRAp A Rocket engine on A chicken.");

struct Splitter(Range,alias Engine = ThompsonMatcher) if (isSomeString!(Range));
  Range that splits a string using a regular expression as a separator.

  Example:

      auto s1 = ", abc, de,  fg, hi, ";
      assert(equal(splitter(s1, regex(", *")),
          ["", "abc", "de", "fg", "hi", ""]));

  Range front();
  bool empty();
  void popFront();
  @property auto save();
    Forward range primitives.

Splitter!(Range) splitter(Range, RegEx)(Range r, RegEx pat);
  A helper function, creates a Spliiter on range r separated by regex pat.

String[] split(String, RegEx)(String input, RegEx rx);
  An eager version of splitter that creates an array with splitted slices of input.

class RegexException: object.Exception;
  Exception object thrown in case of errors during regex compilation.

  @trusted this(string msg, string file = __FILE__, uint line = cast(uint)__LINE__);

Copyright Dmitry Olshansky, 2011 | Page generated by Ddoc.