Function: ietf-drums-date--tokenize-string

ietf-drums-date--tokenize-string is a byte-compiled function defined in ietf-drums-date.el.gz.

Signature

(ietf-drums-date--tokenize-string STRING &optional COMMENT-EOF)

Documentation

Turn STRING into tokens, separated only by whitespace and commas.

Multiple commas are ignored. Pure digit sequences are turned into integers. If COMMENT-EOF is true, then a comment as defined by RFC5322 (strictly, the CFWS production that also accepts comments) is treated as an end-of-file, and no further tokens are recognized, otherwise we strip out all comments and treat them as whitespace (per RFC822).

Source Code

;; Defined in /usr/src/emacs/lisp/mail/ietf-drums-date.el.gz
(defun ietf-drums-date--tokenize-string (string &optional comment-eof)
  "Turn STRING into tokens, separated only by whitespace and commas.
Multiple commas are ignored.  Pure digit sequences are turned
into integers.  If COMMENT-EOF is true, then a comment as
defined by RFC5322 (strictly, the CFWS production that also
accepts comments) is treated as an end-of-file, and no further
tokens are recognized, otherwise we strip out all comments and
treat them as whitespace (per RFC822)."
  (let ((index 0)
	(end (length string))
	(list ()))
    (cl-flet ((skip-ignored ()
                ;; Skip ignored characters at index (the scan
                ;; position).  Skip RFC822 comments in matched parens,
                ;; but do not complain about unterminated comments.
                (let ((char nil)
                      (nest 0))
                  (while (and (< index end)
                              (setq char (aref string index))
                              (or (> nest 0)
                                  (ietf-drums-date--ignore-char-p char)
                                  (and (not comment-eof) (eql char ?\())))
                    (incf index)
                    ;; FWS bookkeeping.
                    (cond ((and (eq char ?\\)
                                (< (1+ index) end))
                            ;; Move to the next char but don't check
                            ;; it to see if it might be a paren.
                            (incf index))
                          ((eq char ?\() (incf nest))
                          ((eq char ?\)) (decf nest)))))))
      (skip-ignored)		;; Skip leading whitespace.
      (while (and (< index end)
                  (not (and comment-eof
                            (eq (aref string index) ?\())))
        (let* ((start index)
               (char (aref string index))
               (all-digits (<= ?0 char ?9)))
          ;; char is valid; look for more valid characters.
          (when (and (eq char ?\\)
                     (< (1+ index) end))
            ;; Escaped character, which might be a "(".  If so, we are
            ;; correct to include it in the token, even though the
            ;; caller is sure to barf.  If not, we violate RFC2?822 by
            ;; not removing the backslash, but no characters in valid
            ;; RFC2?822 dates need escaping anyway, so it shouldn't
            ;; matter that this is not done strictly correctly.  --
            ;; rgr, 24-Dec-21.
            (incf index))
          (while (and (< (incf index) end)
                      (setq char (aref string index))
                      (not (or (ietf-drums-date--ignore-char-p char)
                               (eq char ?\())))
            (unless (<= ?0 char ?9)
              (setq all-digits nil))
            (when (and (eq char ?\\)
                       (< (1+ index) end))
              ;; Escaped character, see above.
              (incf index)))
          (push (if all-digits
                    (cl-parse-integer string :start start :end index)
                  (substring string start index))
                list)
          (skip-ignored)))
      (nreverse list))))