File: org-persist.el.html

This file implements persistent cache storage across Emacs sessions. Both global and buffer-local data can be stored. This implementation is not meant to be used to store important data - all the caches should be safe to remove at any time.

Entry points are org-persist-register, org-persist-write, org-persist-read, and org-persist-load.

org-persist-register will mark the data to be stored. By default, the data is written on disk before exiting Emacs session. Optionally, the data can be written immediately.

org-persist-write will immediately write the data onto disk.

org-persist-read will read the data and return its value or list of values for each requested container.

org-persist-load will read the data with side effects. For example, loading elisp container will assign the values to variables.

Example usage:

1. Temporarily cache Elisp symbol value to disk. Remove upon
   closing Emacs:
   (org-persist-write 'variable-symbol)
   (org-persist-read 'variable-symbol) ;; read the data later

2. Temporarily cache a remote URL file to disk. Remove upon
   closing Emacs:
   (org-persist-write 'url "https://static.fsf.org/common/img/logo-new.png")
   (org-persist-read 'url "https://static.fsf.org/common/img/logo-new.png")
   org-persist-read will return the cached file location or nil if cached file
   has been removed.

3. Temporarily cache a file, including TRAMP path to disk:
   (org-persist-write 'file "/path/to/file")

4. Cache file or URL while some other file exists.
   (org-persist-register '(url "https://static.fsf.org/common/img/logo-new.png") '(:file "/path to the other file") :expiry 'never :write-immediately t)
   or, if the other file is current buffer file
   (org-persist-register '(url "https://static.fsf.org/common/img/logo-new.png") (current-buffer) :expiry 'never :write-immediately t)

5. Cache value of a Elisp variable to disk. The value will be
   saved and restored automatically (except buffer-local
   variables).
   ;; Until org-persist-default-expiry
   (org-persist-register 'variable-symbol)
   ;; Specify expiry explicitly
   (org-persist-register 'variable-symbol :expiry 'never)
   ;; Save buffer-local variable (buffer-local will not be
   ;; autoloaded!)
   (org-persist-register 'org-element--cache (current-buffer))
   ;; Save several buffer-local variables preserving circular links
   ;; between:
   (org-persist-register 'org-element--headline-cache (current-buffer)
              :inherit 'org-element--cache)

6. Load variable by side effects assigning variable symbol:
   (org-persist-load 'variable-symbol (current-buffer))

7. Version variable value:
   (org-persist-register '((elisp variable-symbol) (version "2.0")))

8. Define a named container group:

   (let ((info1 "test")
         (info2 "test 2"))
     (org-persist-register
        ("Named data" (elisp info1 local) (elisp info2 local))
        nil :write-immediately t))
   (org-persist-read
      "Named data"
      nil nil nil :read-related t) ; => ("Named data" "test" "test2")

9. Cancel variable persistence:
   (org-persist-unregister variable-symbol 'all) ; in all buffers
   (org-persist-unregister 'variable-symbol) ;; global variable
   (org-persist-unregister 'variable-symbol (current-buffer)) ;; buffer-local

Most common data type is variable data. However, other data types can also be stored.

Persistent data is stored in individual files. Each of the files can contain a collection of related data, which is particularly useful when, say, several variables cross-reference each-other's data-cells and we want to preserve their circular structure.

Each data collection can be associated with a local or remote file, its inode number, contents hash. The persistent data collection can later be accessed using either file buffer, file, inode, or contents hash.

The data collections can be versioned and removed upon expiry.

In the code below, I will use the following naming conventions:

1. Container :: a type of data to be stored
   Containers can store elisp variables, files, and version
   numbers. Each container can be customized with container
   options. For example, elisp container is customized with
   variable symbol. (elisp variable) is a container storing
   Lisp variable value. Similarly, (version "2.0") container
   will store version number.

   Container can also refer to a group of containers:

   ;; Three containers stored together.
   '((elisp variable) (file "/path") (version "x.x"))

   Providing a single container from the list to org-persist-read
   is sufficient to retrieve all the containers (with appropriate
   optional parameter).

   Example:

   (org-persist-register '((version "My data") (file "/path/to/file")) '(:key "key") :write-immediately t)
   (org-persist-read '(version "My data") '(:key "key") :read-related t) ;; => '("My data" "/path/to/file/copy")

   Individual containers can also take a short form (not a list):

   '("String" file '(quoted elisp "value") :keyword)
   is the same with
   '((elisp-data "String") (file nil)
     (elisp-data '(quoted elisp "value")) (elisp-data :keyword))

   Note that '(file "String" (elisp value)) would be interpreted as
   file container with "String" path and extra options. See
   org-persist--normalize-container.

2. Associated :: an object the container is associated with. The
   object can be a buffer, file, inode number, file contents hash,
   a generic key, or multiple of them. Associated can also be nil.

   Example:

   '(:file "/path/to/file" :inode number :hash buffer-hash :key arbitrary-key)

   When several objects are associated with a single container, it
   is not necessary to provide them all to access the container.
   Just using a single :file/:inode/:hash/:key is sufficient. This
   way, one can retrieve cached data even when the file has moved -
   by contents hash.

3. Data collection :: a list of containers, the associated
   object/objects, expiry, access time, and information about where
   the cache is stored. Each data collection can also have
   auxiliary records. Their only purpose is readability of the
   collection index.

   Example:

   (:container
    ((index "2.7"))
    :persist-file "ba/cef3b7-e31c-4791-813e-8bd0bf6c5f9c"
    :associated nil :expiry never
    :last-access 1672207741.6422956 :last-access-hr "2022-12-28T09:09:01+0300")

4. Index file :: a file listing all the stored data collections.

5. Persist file :: a file holding data values or references to
   actual data values for a single data collection. This file
   contains an alist associating each data container in data
   collection with its value or a reference to the actual value.

   Example (persist file storing two elisp container values):

   (((elisp org-element--headline-cache) . #s(avl-tree- ...))
    ((elisp org-element--cache) . #s(avl-tree- ...)))

All the persistent data is stored in org-persist-directory. The data collections are listed in org-persist-index-file and the actual data is stored in UID-style subfolders.

The org-persist-index-file stores the value of org-persist--index.

Each collection is represented as a plist containing the following properties:

- :container : list of data containers to be stored in single
                   file;
- :persist-file: data file name;
- :associated : list of associated objects;
- :last-access : last date when the container has been accessed;
- :expiry : list of expiry conditions.
- all other keywords are ignored

The available types of data containers are:
1. (elisp variable-symbol scope) or just variable-symbol :: Storing
   elisp variable data. SCOPE can be

   - nil :: Use buffer-local value in associated :file or global
                value if no :file is associated.
   - string :: Use buffer-local value in buffer named STRING or
               with STRING buffer-file-name(var)/buffer-file-name(fun).
   - local :: Use symbol value in current scope.
                Note: If local scope is used without writing the
                value immediately, the actual stored value is
                undefined.

2. (file) :: Store a copy of the associated file preserving the
   extension.

   (file "/path/to/a/file") :: Store a copy of the file in path.

3. (version "version number") :: Version the data collection.
    If the stored collection has different version than "version
    number", disregard it.

4. (url) :: Store a downloaded copy of URL object given by
            associated :file.
   (url "path") :: Use "path" instead of associated :file.

The data collections can expire, in which case they will be removed from the persistent storage at the end of Emacs session. The expiry condition can be set when saving/registering data containers. The expirty condition can be never - data will never expire; nil - data will expire at the end of current Emacs session; a number - data will expire after the number days from last access; a function - data will expire if the function, called with a single argument - collection, returns non-nil.

Data collections associated with files will automatically expire when the file is removed. If the associated file is remote, the expiry is controlled by org-persist-remote-files instead.

Data loading/writing can be more accurately controlled using org-persist-before-write-hook, org-persist-before-read-hook, and org-persist-after-read-hook.

Defined variables (19)

Defined functions (64)

Defined faces (0)