Variable: semantic-lex-tokens

semantic-lex-tokens is a variable defined in lex.el.gz.

Value

((bol) (charquote) (close-paren) (comment) (newline) (open-paren)
 (punctuation) (semantic-list) (string) (symbol) (whitespace))

Documentation

An alist of semantic token types.

As of December 2001 (semantic 1.4beta13), this variable is not used in any code. The only use is to refer to the doc-string from elsewhere.

The key to this alist is the symbol representing token type that M-x semantic-flex (semantic-flex) returns. These are

  - bol: Empty string matching a beginning of line.
                   This token is produced with
                   semantic-lex-beginning-of-line(var)/semantic-lex-beginning-of-line(fun).

  - charquote: String sequences that match \s\+ regexp.
                   This token is produced with semantic-lex-charquote(var)/semantic-lex-charquote(fun).

  - close-paren: Characters that match \s) regexp.
                   These are typically ), }, ], etc.
                   This token is produced with
                   semantic-lex-close-paren(var)/semantic-lex-close-paren(fun).

  - comment: A comment chunk. These token types are not
                   produced by default.
                   This token is produced with semantic-lex-comments(var)/semantic-lex-comments(fun).
                   Comments are ignored with semantic-lex-ignore-comments(var)/semantic-lex-ignore-comments(fun).
                   Comments are treated as whitespace with
                   semantic-lex-comments-as-whitespace(var)/semantic-lex-comments-as-whitespace(fun).

  - newline Characters matching \s-*\(
\|\s>\) regexp.
                   This token is produced with semantic-lex-newline(var)/semantic-lex-newline(fun).

  - open-paren: Characters that match \s( regexp.
                   These are typically (, {, [, etc.
                   If semantic-lex-paren-or-list(var)/semantic-lex-paren-or-list(fun) is used,
                   then open-paren is not usually generated unless
                   the depth argument to M-x semantic-lex (semantic-lex) is
                   greater than 0.
                   This token is always produced if the analyzer
                   semantic-lex-open-paren(var)/semantic-lex-open-paren(fun) is used.

  - punctuation: Characters matching {\(\s.\|\s$\|\s\)'
                   regexp.
                   This token is produced with semantic-lex-punctuation(var)/semantic-lex-punctuation(fun).
                   Always specify this analyzer after the comment
                   analyzer.

  - semantic-list: String delimited by matching parenthesis, braces,
                   etc. that the lexer skipped over, because the
                   depth parameter to M-x semantic-flex (semantic-flex) was not high
                   enough.
                   This token is produced with semantic-lex-paren-or-list(var)/semantic-lex-paren-or-list(fun).

  - string: Quoted strings, i.e., string sequences that start
                   and end with characters matching \s"
                   regexp. The lexer relies on @code{forward-sexp} to
                   find the matching end.
                   This token is produced with semantic-lex-string(var)/semantic-lex-string(fun).

  - symbol: String sequences that match \(\sw\|\s_\)+
                   regexp.
                   This token is produced with
                   semantic-lex-symbol-or-keyword(var)/semantic-lex-symbol-or-keyword(fun). Always add this analyzer
                   after semantic-lex-number(var)/semantic-lex-number(fun), or other analyzers that
                   match its regular expression.

  - whitespace: Characters that match \s-+ regexp.
                   This token is produced with semantic-lex-whitespace(var)/semantic-lex-whitespace(fun).

Source Code

;; Defined in /usr/src/emacs/lisp/cedet/semantic/lex.el.gz
(defvar semantic-lex-tokens
  '(
    (bol)
    (charquote)
    (close-paren)
    (comment)
    (newline)
    (open-paren)
    (punctuation)
    (semantic-list)
    (string)
    (symbol)
    (whitespace)
    )
  "An alist of semantic token types.
As of December 2001 (semantic 1.4beta13), this variable is not used in
any code.  The only use is to refer to the doc-string from elsewhere.

The key to this alist is the symbol representing token type that
\\[semantic-flex] returns.  These are

  - bol:           Empty string matching a beginning of line.
                   This token is produced with
                   `semantic-lex-beginning-of-line'.

  - charquote:     String sequences that match `\\s\\+' regexp.
                   This token is produced with `semantic-lex-charquote'.

  - close-paren:   Characters that match `\\s)' regexp.
                   These are typically `)', `}', `]', etc.
                   This token is produced with
                   `semantic-lex-close-paren'.

  - comment:       A comment chunk.  These token types are not
                   produced by default.
                   This token is produced with `semantic-lex-comments'.
                   Comments are ignored with `semantic-lex-ignore-comments'.
                   Comments are treated as whitespace with
                   `semantic-lex-comments-as-whitespace'.

  - newline        Characters matching `\\s-*\\(\n\\|\\s>\\)' regexp.
                   This token is produced with `semantic-lex-newline'.

  - open-paren:    Characters that match `\\s(' regexp.
                   These are typically `(', `{', `[', etc.
                   If `semantic-lex-paren-or-list' is used,
                   then `open-paren' is not usually generated unless
                   the `depth' argument to \\[semantic-lex] is
                   greater than 0.
                   This token is always produced if the analyzer
                   `semantic-lex-open-paren' is used.

  - punctuation:   Characters matching `{\\(\\s.\\|\\s$\\|\\s'\\)'
                   regexp.
                   This token is produced with `semantic-lex-punctuation'.
                   Always specify this analyzer after the comment
                   analyzer.

  - semantic-list: String delimited by matching parenthesis, braces,
                   etc.  that the lexer skipped over, because the
                   `depth' parameter to \\[semantic-flex] was not high
                   enough.
                   This token is produced with `semantic-lex-paren-or-list'.

  - string:        Quoted strings, i.e., string sequences that start
                   and end with characters matching `\\s\"'
                   regexp.  The lexer relies on @code{forward-sexp} to
                   find the matching end.
                   This token is produced with `semantic-lex-string'.

  - symbol:        String sequences that match `\\(\\sw\\|\\s_\\)+'
                   regexp.
                   This token is produced with
                   `semantic-lex-symbol-or-keyword'.  Always add this analyzer
                   after `semantic-lex-number', or other analyzers that
                   match its regular expression.

  - whitespace:    Characters that match `\\s-+' regexp.
                   This token is produced with `semantic-lex-whitespace'.")