# repos

node_id: R_kgDOIJsqeg
name: datasette-external-links-new-tabs
full_name: ocdtrekkie/datasette-external-links-new-tabs
html_url: https://github.com/ocdtrekkie/datasette-external-links-new-tabs
description: Datasette plugin to open external links in new tabs
created_at: 2022-10-07T03:34:09Z
updated_at: 2022-10-07T05:25:44Z
pushed_at: 2022-10-07T04:51:51Z
language: Python
license: apache-2.0
topics: []
visibility: public
default_branch: main
permissions: {"admin": false, "maintain": false, "push": false, "triage": false, "pull": true}
template_repository:
[cell repeats an earlier one in this file: 6377 characters]

readme:
  # datasette-external-links-new-tabs

  [![PyPI](https://img.shields.io/pypi/v/datasette-external-links-new-tabs.svg)](https://pypi.org/project/datasette-external-links-new-tabs/)
  [![Changelog](https://img.shields.io/github/v/release/ocdtrekkie/datasette-external-links-new-tabs?include_prereleases&label=changelog)](https://github.com/ocdtrekkie/datasette-external-links-new-tabs/releases)
  [![Tests](https://github.com/ocdtrekkie/datasette-external-links-new-tabs/workflows/Test/badge.svg)](https://github.com/ocdtrekkie/datasette-external-links-new-tabs/actions?query=workflow%3ATest)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/ocdtrekkie/datasette-external-links-new-tabs/blob/main/LICENSE)

  Datasette plugin to open external links in new tabs

  ## Installation

  Install this plugin in the same environment as Datasette.

      datasette install datasette-external-links-new-tabs

  ## Usage

  There are no usage instructions, it simply opens external links in a new tab.

  ## Development

  To set up this plugin locally, first checkout the code. Then create a new virtual environment:

      cd datasette-external-links-new-tabs
      python3 -m venv venv
      source venv/bin/activate

  Now install the dependencies and test dependencies:

      pip install -e '.[test]'

  To run the tests:

      pytest

readme_html:
  <div id="readme" class="md" data-path="README.md"><article class="markdown-body entry-content container-lg" itemprop="text"><div class="markdown-heading" dir="auto"><h1 class="heading-element" dir="auto">datasette-external-links-new-tabs</h1><a id="user-content-datasette-external-links-new-tabs" class="anchor" aria-label="Permalink: datasette-external-links-new-tabs" href="#datasette-external-links-new-tabs"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto"><a href="https://pypi.org/project/datasette-external-links-new-tabs/" rel="nofollow"><img src="https://img.shields.io/pypi/v/datasette-external-links-new-tabs.svg" alt="PyPI" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/1e0a8ff321f205dd577f02bc6ee770510b774349788c70d6a776bb7771c0dae6/68747470733a2f2f696d672e736869656c64732e696f2f707970692f762f6461746173657474652d65787465726e616c2d6c696e6b732d6e65772d746162732e737667"></a>
  <a href="https://github.com/ocdtrekkie/datasette-external-links-new-tabs/releases"><img src="https://img.shields.io/github/v/release/ocdtrekkie/datasette-external-links-new-tabs?include_prereleases&label=changelog" alt="Changelog" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/c11954467a6cd322d7d19def157f637c32a2993662d65394bcdb0947386f6025/68747470733a2f2f696d672e736869656c64732e696f2f6769746875622f762f72656c656173652f6f63647472656b6b69652f6461746173657474652d65787465726e616c2d6c696e6b732d6e65772d746162733f696e636c7564655f70726572656c6561736573266c6162656c3d6368616e67656c6f67"></a>
  <a href="https://github.com/ocdtrekkie/datasette-external-links-new-tabs/actions?query=workflow%3ATest"><img src="https://github.com/ocdtrekkie/datasette-external-links-new-tabs/workflows/Test/badge.svg" alt="Tests" style="max-width: 100%;"></a>
  <a href="https://github.com/ocdtrekkie/datasette-external-links-new-tabs/blob/main/LICENSE"><img src="https://img.shields.io/badge/license-Apache%202.0-blue.svg" alt="License" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/c355f200ea90fddaa407b6eaab303663a669248ea3ca7b1fcf77dbe04ff5f48c/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6c6963656e73652d417061636865253230322e302d626c75652e737667"></a></p>
  <p dir="auto">Datasette plugin to open external links in new tabs</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Installation</h2><a id="user-content-installation" class="anchor" aria-label="Permalink: Installation" href="#installation"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">Install this plugin in the same environment as Datasette.</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="datasette install datasette-external-links-new-tabs"><pre class="notranslate"><code>datasette install datasette-external-links-new-tabs
  </code></pre></div>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Usage</h2><a id="user-content-usage" class="anchor" aria-label="Permalink: Usage" href="#usage"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">There are no usage instructions, it simply opens external links in a new tab.</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Development</h2><a id="user-content-development" class="anchor" aria-label="Permalink: Development" href="#development"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">To set up this plugin locally, first checkout the code. Then create a new virtual environment:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="cd datasette-external-links-new-tabs
  python3 -m venv venv
  source venv/bin/activate"><pre class="notranslate"><code>cd datasette-external-links-new-tabs
  python3 -m venv venv
  source venv/bin/activate
  </code></pre></div>
  <p dir="auto">Now install the dependencies and test dependencies:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="pip install -e '.[test]'"><pre class="notranslate"><code>pip install -e '.[test]'
  </code></pre></div>
  <p dir="auto">To run the tests:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="pytest"><pre class="notranslate"><code>pytest
  </code></pre></div>
  </article></div>


node_id: R_kgDOISPElA
name: datasette-gunicorn
full_name: simonw/datasette-gunicorn
html_url: https://github.com/simonw/datasette-gunicorn
description: Plugin for running Datasette using Gunicorn
created_at: 2022-10-22T20:23:57Z
updated_at: 2023-01-31T20:42:01Z
pushed_at: 2022-12-18T06:46:37Z
homepage: https://datasette.io/plugins/datasette-gunicorn
language: Python
license: apache-2.0
topics: ["datasette", "datasette-plugin"]
visibility: public
default_branch: main
permissions: {"admin": false, "maintain": false, "push": false, "triage": false, "pull": true}
readme:
  # datasette-gunicorn

  [![PyPI](https://img.shields.io/pypi/v/datasette-gunicorn.svg)](https://pypi.org/project/datasette-gunicorn/)
  [![Changelog](https://img.shields.io/github/v/release/simonw/datasette-gunicorn?include_prereleases&label=changelog)](https://github.com/simonw/datasette-gunicorn/releases)
  [![Tests](https://github.com/simonw/datasette-gunicorn/workflows/Test/badge.svg)](https://github.com/simonw/datasette-gunicorn/actions?query=workflow%3ATest)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/datasette-gunicorn/blob/main/LICENSE)

  Run a [Datasette](https://datasette.io/) server using [Gunicorn](https://gunicorn.org/)

  ## Installation

  Install this plugin in the same environment as Datasette.

      datasette install datasette-gunicorn

  ## Usage

  The plugin adds a new `datasette gunicorn` command. This takes most of the same options as `datasette serve`, plus one more option for setting the number of Gunicorn workers to start:

  `-w/--workers X` - set the number of workers. Defaults to 1.

  To start serving a database using 4 workers, run the following:

      datasette gunicorn fixtures.db -w 4

  It is advisable to switch your datasette [into WAL mode](https://til.simonwillison.net/sqlite/enabling-wal-mode) to get the best performance out of this configuration:

      sqlite3 fixtures.db 'PRAGMA journal_mode=WAL;'

  Run `datasette gunicorn --help` for a full list of options (which are the same as `datasette serve --help`, with the addition of the new `-w` option).

  ## datasette gunicorn --help

  Not all of the options to `datasette serve` are supported. Here's the full list of available options:

  <!-- [[[cog
  import cog
  from datasette import cli
  from click.testing import CliRunner
  runner = CliRunner()
  result = runner.invoke(cli.cli, ["gunicorn", "--help"])
  help = result.output.replace("Usage: cli", "Usage: datasette")
  cog.out(
      "```\n{}\n```".format(help)
  )
  ]]] -->
  ```
  Usage: datasette gunicorn [OPTIONS] [FILES]...

    Start a Gunicorn server running to serve Datasette

  Options:
    -i, --immutable PATH            Database files to open in immutable mode
    -h, --host TEXT                 Host for server. Defaults to 127.0.0.1 which
                                    means only connections from the local machine
                                    will be allowed. Use 0.0.0.0 to listen to all
                                    IPs and allow access from other machines.
    -p, --port INTEGER RANGE        Port for server, defaults to 8001. Use -p 0 to
                                    automatically assign an available port.
                                    [0<=x<=65535]
    --cors                          Enable CORS by serving Access-Control-Allow-
                                    Origin: *
    --load-extension PATH:ENTRYPOINT?
                                    Path to a SQLite extension to load, and
                                    optional entrypoint
    --inspect-file TEXT             Path to JSON file created using "datasette
                                    inspect"
    -m, --metadata FILENAME         Path to JSON/YAML file containing
                                    license/source metadata
    --template-dir DIRECTORY        Path to directory containing custom templates
    --plugins-dir DIRECTORY         Path to directory containing custom plugins
    --static MOUNT:DIRECTORY        Serve static files from this directory at
                                    /MOUNT/...
    --memory                        Make /_memory database available
    --config CONFIG                 Deprecated: set config option using
                                    configname:value. Use --setting instead.
    --setting SETTING...            Setting, see
                                    docs.datasette.io/en/stable/settings.html
    --secret TEXT                   Secret used for signing secure values, such as
                                    signed cookies
    --version-note TEXT             Additional note to show on /-/versions
    --help-settings                 Show available settings
    --create                        Create database files if they do not exist
    --crossdb                       Enable cross-database joins using the /_memory
                                    database
    --nolock                        Ignore locking, open locked files in read-only
                                    mode
    -w, --workers INTEGER           Number of Gunicorn workers  [default: 1]
    --help                          Show this message and exit.

  ```
  <!-- [[[end]]] -->

  ## Development

  To set up this plugin locally, first checkout the code. Then create a new virtual environment:

      cd datasette-gunicorn
      python3 -m venv venv
      source venv/bin/activate

  Now install the dependencies and test dependencies:

      pip install -e '.[test]'

  To run the tests:

      pytest

readme_html:
  <div id="readme" class="md" data-path="README.md"><article class="markdown-body entry-content container-lg" itemprop="text"><div class="markdown-heading" dir="auto"><h1 class="heading-element" dir="auto">datasette-gunicorn</h1><a id="user-content-datasette-gunicorn" class="anchor" aria-label="Permalink: datasette-gunicorn" href="#datasette-gunicorn"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto"><a href="https://pypi.org/project/datasette-gunicorn/" rel="nofollow"><img src="https://img.shields.io/pypi/v/datasette-gunicorn.svg" alt="PyPI" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/93768d0b429a0e5b6f23d7bbf60c4dcb7df4e49b656db4f3fb0dd9075980f105/68747470733a2f2f696d672e736869656c64732e696f2f707970692f762f6461746173657474652d67756e69636f726e2e737667"></a>
  <a href="https://github.com/simonw/datasette-gunicorn/releases"><img src="https://img.shields.io/github/v/release/simonw/datasette-gunicorn?include_prereleases&label=changelog" alt="Changelog" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/a934c5a43c57b766d7e98ace63c78cfcc1d64011ad69b81eab70aa56898ceafc/68747470733a2f2f696d672e736869656c64732e696f2f6769746875622f762f72656c656173652f73696d6f6e772f6461746173657474652d67756e69636f726e3f696e636c7564655f70726572656c6561736573266c6162656c3d6368616e67656c6f67"></a>
  <a href="https://github.com/simonw/datasette-gunicorn/actions?query=workflow%3ATest"><img src="https://github.com/simonw/datasette-gunicorn/workflows/Test/badge.svg" alt="Tests" style="max-width: 100%;"></a>
  <a href="https://github.com/simonw/datasette-gunicorn/blob/main/LICENSE"><img src="https://img.shields.io/badge/license-Apache%202.0-blue.svg" alt="License" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/c355f200ea90fddaa407b6eaab303663a669248ea3ca7b1fcf77dbe04ff5f48c/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6c6963656e73652d417061636865253230322e302d626c75652e737667"></a></p>
  <p dir="auto">Run a <a href="https://datasette.io/" rel="nofollow">Datasette</a> server using <a href="https://gunicorn.org/" rel="nofollow">Gunicorn</a></p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Installation</h2><a id="user-content-installation" class="anchor" aria-label="Permalink: Installation" href="#installation"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">Install this plugin in the same environment as Datasette.</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="datasette install datasette-gunicorn"><pre class="notranslate"><code>datasette install datasette-gunicorn
  </code></pre></div>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Usage</h2><a id="user-content-usage" class="anchor" aria-label="Permalink: Usage" href="#usage"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">The plugin adds a new <code>datasette gunicorn</code> command. This takes most of the same options as <code>datasette serve</code>, plus one more option for setting the number of Gunicorn workers to start:</p>
  <p dir="auto"><code>-w/--workers X</code> - set the number of workers. Defaults to 1.</p>
  <p dir="auto">To start serving a database using 4 workers, run the following:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="datasette gunicorn fixtures.db -w 4"><pre class="notranslate"><code>datasette gunicorn fixtures.db -w 4
  </code></pre></div>
  <p dir="auto">It is advisable to switch your datasette <a href="https://til.simonwillison.net/sqlite/enabling-wal-mode" rel="nofollow">into WAL mode</a> to get the best performance out of this configuration:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="sqlite3 fixtures.db 'PRAGMA journal_mode=WAL;'"><pre class="notranslate"><code>sqlite3 fixtures.db 'PRAGMA journal_mode=WAL;'
  </code></pre></div>
  <p dir="auto">Run <code>datasette gunicorn --help</code> for a full list of options (which are the same as <code>datasette serve --help</code>, with the addition of the new <code>-w</code> option).</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">datasette gunicorn --help</h2><a id="user-content-datasette-gunicorn---help" class="anchor" aria-label="Permalink: datasette gunicorn --help" href="#datasette-gunicorn---help"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">Not all of the options to <code>datasette serve</code> are supported. Here's the full list of available options:</p>

  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="Usage: datasette gunicorn [OPTIONS] [FILES]...

    Start a Gunicorn server running to serve Datasette

  Options:
    -i, --immutable PATH            Database files to open in immutable mode
    -h, --host TEXT                 Host for server. Defaults to 127.0.0.1 which
                                    means only connections from the local machine
                                    will be allowed. Use 0.0.0.0 to listen to all
                                    IPs and allow access from other machines.
    -p, --port INTEGER RANGE        Port for server, defaults to 8001. Use -p 0 to
                                    automatically assign an available port.
                                    [0<=x<=65535]
    --cors                          Enable CORS by serving Access-Control-Allow-
                                    Origin: *
    --load-extension PATH:ENTRYPOINT?
                                    Path to a SQLite extension to load, and
                                    optional entrypoint
    --inspect-file TEXT             Path to JSON file created using "datasette
                                    inspect"
    -m, --metadata FILENAME         Path to JSON/YAML file containing
                                    license/source metadata
    --template-dir DIRECTORY        Path to directory containing custom templates
    --plugins-dir DIRECTORY         Path to directory containing custom plugins
    --static MOUNT:DIRECTORY        Serve static files from this directory at
                                    /MOUNT/...
    --memory                        Make /_memory database available
    --config CONFIG                 Deprecated: set config option using
                                    configname:value. Use --setting instead.
    --setting SETTING...            Setting, see
                                    docs.datasette.io/en/stable/settings.html
    --secret TEXT                   Secret used for signing secure values, such as
                                    signed cookies
    --version-note TEXT             Additional note to show on /-/versions
    --help-settings                 Show available settings
    --create                        Create database files if they do not exist
    --crossdb                       Enable cross-database joins using the /_memory
                                    database
    --nolock                        Ignore locking, open locked files in read-only
                                    mode
    -w, --workers INTEGER           Number of Gunicorn workers  [default: 1]
    --help                          Show this message and exit.
  "><pre class="notranslate"><code>Usage: datasette gunicorn [OPTIONS] [FILES]...

    Start a Gunicorn server running to serve Datasette

  Options:
    -i, --immutable PATH            Database files to open in immutable mode
    -h, --host TEXT                 Host for server. Defaults to 127.0.0.1 which
                                    means only connections from the local machine
                                    will be allowed. Use 0.0.0.0 to listen to all
                                    IPs and allow access from other machines.
    -p, --port INTEGER RANGE        Port for server, defaults to 8001. Use -p 0 to
                                    automatically assign an available port.
                                    [0<=x<=65535]
    --cors                          Enable CORS by serving Access-Control-Allow-
                                    Origin: *
    --load-extension PATH:ENTRYPOINT?
                                    Path to a SQLite extension to load, and
                                    optional entrypoint
    --inspect-file TEXT             Path to JSON file created using "datasette
                                    inspect"
    -m, --metadata FILENAME         Path to JSON/YAML file containing
                                    license/source metadata
    --template-dir DIRECTORY        Path to directory containing custom templates
    --plugins-dir DIRECTORY         Path to directory containing custom plugins
    --static MOUNT:DIRECTORY        Serve static files from this directory at
                                    /MOUNT/...
    --memory                        Make /_memory database available
    --config CONFIG                 Deprecated: set config option using
                                    configname:value. Use --setting instead.
    --setting SETTING...            Setting, see
                                    docs.datasette.io/en/stable/settings.html
    --secret TEXT                   Secret used for signing secure values, such as
                                    signed cookies
    --version-note TEXT             Additional note to show on /-/versions
    --help-settings                 Show available settings
    --create                        Create database files if they do not exist
    --crossdb                       Enable cross-database joins using the /_memory
                                    database
    --nolock                        Ignore locking, open locked files in read-only
                                    mode
    -w, --workers INTEGER           Number of Gunicorn workers  [default: 1]
    --help                          Show this message and exit.

  </code></pre></div>

  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Development</h2><a id="user-content-development" class="anchor" aria-label="Permalink: Development" href="#development"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">To set up this plugin locally, first checkout the code. Then create a new virtual environment:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="cd datasette-gunicorn
  python3 -m venv venv
  source venv/bin/activate"><pre class="notranslate"><code>cd datasette-gunicorn
  python3 -m venv venv
  source venv/bin/activate
  </code></pre></div>
  <p dir="auto">Now install the dependencies and test dependencies:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="pip install -e '.[test]'"><pre class="notranslate"><code>pip install -e '.[test]'
  </code></pre></div>
  <p dir="auto">To run the tests:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="pytest"><pre class="notranslate"><code>pytest
  </code></pre></div>
  </article></div>


node_id: R_kgDOIX3UzQ
name: sqlite-regex
full_name: asg017/sqlite-regex
html_url: https://github.com/asg017/sqlite-regex
description: A fast regular expression SQLite extension, written in Rust
created_at: 2022-11-04T18:31:01Z
updated_at: 2025-04-29T05:56:12Z
pushed_at: 2024-04-22T07:46:31Z
language: Rust
license: apache-2.0
topics: ["sqlite", "sqlite-extension"]
visibility: public
default_branch: main
permissions: {"admin": false, "maintain": false, "push": false, "triage": false, "pull": true}
readme:
  # sqlite-regex

  A fast and performant SQLite extension for regular expressions. Based on [`sqlite-loadable-rs`](https://github.com/asg017/sqlite-loadable-rs), and the [regex crate](https://crates.io/crates/regex).

  See [_Introducing sqlite-regex: The fastest Regular Expression Extension for SQLite_](https://observablehq.com/@asg017/introducing-sqlite-regex) (Jan 2023) for more details!

  If your company or organization finds this library useful, consider [supporting my work](#supporting)!

  ![](./benchmarks/dates.png)

  ## Usage

  ```sql
  .load ./regex0
  select 'foo' regexp 'f';

  ```

  **Find all occurrences of a pattern in a string**

  ```sql
  select regex_find(
    '[0-9]{3}-[0-9]{3}-[0-9]{4}',
    'phone: 111-222-3333'
  );
  -- '111-222-3333'

  select rowid, *
  from regex_find_all(
    '\b\w{13}\b',
    'Retroactively relinquishing remunerations is reprehensible.'
  );
  /*
  ┌───────┬───────┬─────┬───────────────┐
  │ rowid │ start │ end │     match     │
  ├───────┼───────┼─────┼───────────────┤
  │ 0     │ 0     │ 13  │ Retroactively │
  │ 1     │ 14    │ 27  │ relinquishing │
  │ 2     │ 28    │ 41  │ remunerations │
  │ 3     │ 45    │ 58  │ reprehensible │
  └───────┴───────┴─────┴───────────────┘
  */
  ```

  **Extract capture group values by index or name**

  ```sql
  select
    regex_capture(captures, 0)        as entire_match,
    regex_capture(captures, 'title')  as title,
    regex_capture(captures, 'year')   as year
  from regex_captures(
    regex("'(?P<title>[^']+)'\s+\((?P<year>\d{4})\)"),
    "'Citizen Kane' (1941), 'The Wizard of Oz' (1939), 'M' (1931)."
  );
  /*
  ┌───────────────────────────┬──────────────────┬──────┐
  │       entire_match        │      title       │ year │
  ├───────────────────────────┼──────────────────┼──────┤
  │ 'Citizen Kane' (1941)     │ Citizen Kane     │ 1941 │
  │ 'The Wizard of Oz' (1939) │ The Wizard of Oz │ 1939 │
  │ 'M' (1931)                │ M                │ 1931 │
  └───────────────────────────┴──────────────────┴──────┘
  */
  ```

  **Use RegexSets to match a string on multiple patterns in linear time**

  ```sql
  select regexset_is_match(
    regexset(
      "bar",
      "foo",
      "barfoo"
    ),
    'foobar'
  )
  ```

  **Split the string on the given pattern delimiter**

  ```sql
  select rowid, *
  from regex_split('[ \t]+', 'a b     c d    e');
  /*
  ┌───────┬──────┐
  │ rowid │ item │
  ├───────┼──────┤
  │ 0     │ a    │
  │ 1     │ b    │
  │ 2     │ c    │
  │ 3     │ d    │
  │ 4     │ e    │
  └───────┴──────┘
  */
  ```

  **Replace occurrences of a pattern with another string**

  ```sql
  select regex_replace(
    '(?P<last>[^,\s]+),\s+(?P<first>\S+)',
    'Springsteen, Bruce',
    '$first $last'
  );
  -- 'Bruce Springsteen'

  select regex_replace_all('a', 'abc abc', '');
  -- 'bc bc'
  ```

  ## Documentation

  See [`docs.md`](./docs.md) for a full API reference.

  ## Installing

  | Language       | Install                                                        |                                                                                                                                                                                               |
  | -------------- | -------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
  | Python         | `pip install sqlite-regex`                                     | [![PyPI](https://img.shields.io/pypi/v/sqlite-regex.svg?color=blue&logo=python&logoColor=white)](https://pypi.org/project/sqlite-regex/)                                                      |
  | Datasette      | `datasette install datasette-sqlite-regex`                     | [![Datasette](https://img.shields.io/pypi/v/datasette-sqlite-regex.svg?color=B6B6D9&label=Datasette+plugin&logoColor=white&logo=python)](https://datasette.io/plugins/datasette-sqlite-regex) |
  | Node.js        | `npm install sqlite-regex`                                     | [![npm](https://img.shields.io/npm/v/sqlite-regex.svg?color=green&logo=nodedotjs&logoColor=white)](https://www.npmjs.com/package/sqlite-regex)                                                |
  | Deno           | [`deno.land/x/sqlite_regex`](https://deno.land/x/sqlite_regex) | [![deno.land/x release](https://img.shields.io/github/v/release/asg017/sqlite-regex?color=fef8d2&include_prereleases&label=deno.land%2Fx&logo=deno)](https://deno.land/x/sqlite_regex)        |
  | Ruby           | `gem install sqlite-regex`                                     | ![Gem](https://img.shields.io/gem/v/sqlite-regex?color=red&logo=rubygems&logoColor=white)                                                                                                     |
  | Github Release |                                                                | ![GitHub tag (latest SemVer pre-release)](https://img.shields.io/github/v/tag/asg017/sqlite-regex?color=lightgrey&include_prereleases&label=Github+release&logo=github)                       |
  | Rust           | `cargo add sqlite-regex`                                       | [![Crates.io](https://img.shields.io/crates/v/sqlite-regex?logo=rust)](https://crates.io/crates/sqlite-regex)                                                                                 |

  <!--
  | Elixir         | [`hex.pm/packages/sqlite_regex`](https://hex.pm/packages/sqlite_regex) | [![Hex.pm](https://img.shields.io/hexpm/v/sqlite_regex?color=purple&logo=elixir)](https://hex.pm/packages/sqlite_regex)                                                                       |
  | Go             | `go get -u github.com/asg017/sqlite-regex/bindings/go`               | [![Go Reference](https://pkg.go.dev/badge/github.com/asg017/sqlite-regex/bindings/go.svg)](https://pkg.go.dev/github.com/asg017/sqlite-regex/bindings/go)                                     |
  -->

  The [Releases page](https://github.com/asg017/sqlite-regex/releases) contains pre-built binaries for Linux x86_64, MacOS, and Windows.

  ### As a loadable extension

  If you want to use `sqlite-regex` as a [Runtime-loadable extension](https://www.sqlite.org/loadext.html), Download the `regex0.dylib` (for MacOS), `regex0.so` (Linux), or `regex0.dll` (Windows) file from a release and load it into your SQLite environment.

  > **Note:**
  > The `0` in the filename (`regex0.dylib`/ `regex0.so`/`regex0.dll`) denotes the major version of `sqlite-regex`. Currently `sqlite-regex` is pre v1, so expect breaking changes in future versions.

  For example, if you are using the [SQLite CLI](https://www.sqlite.org/cli.html), you can load the library like so:

  ```sql
  .load ./regex0
  select regex_version();
  -- v0.1.0
  ```

  Or in Python, using the builtin [sqlite3 module](https://docs.python.org/3/library/sqlite3.html):

  ```python
  import sqlite3
  con = sqlite3.connect(":memory:")
  con.enable_load_extension(True)
  con.load_extension("./regex0")
  print(con.execute("select regex_version()").fetchone())
  # ('v0.1.0',)
  ```

  Or in Node.js using [better-sqlite3](https://github.com/WiseLibs/better-sqlite3):

  ```javascript
  const Database = require("better-sqlite3");
  const db = new Database(":memory:");
  db.loadExtension("./regex0");
  console.log(db.prepare("select regex_version()").get());
  // { 'regex_version()': 'v0.1.0' }
  ```

  Or with [Datasette](https://datasette.io/):

  ```
  datasette data.db --load-extension ./regex0
  ```

  ## Supporting

  I (Alex 👋🏼) spent a lot of time and energy on this project and [many other open source projects](https://github.com/asg017?tab=repositories&q=&type=&language=&sort=stargazers). If your company or organization uses this library (or you're feeling generous), then please [consider supporting my work](https://alexgarcia.regex/work.html), or share this project with a friend!

  ## See also

  - [sqlite-xsv](https://github.com/asg017/sqlite-xsv), A SQLite extension for working with CSVs
  - [sqlite-loadable](https://github.com/asg017/sqlite-loadable-rs), A framework for writing SQLite extensions in Rust
  - [sqlite-http](https://github.com/asg017/sqlite-http), A SQLite extension for making HTTP requests

readme_html:
  <div id="readme" class="md" data-path="README.md"><article class="markdown-body entry-content container-lg" itemprop="text"><div class="markdown-heading" dir="auto"><h1 class="heading-element" dir="auto">sqlite-regex</h1><a id="user-content-sqlite-regex" class="anchor" aria-label="Permalink: sqlite-regex" href="#sqlite-regex"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">A fast and performant SQLite extension for regular expressions. Based on <a href="https://github.com/asg017/sqlite-loadable-rs"><code>sqlite-loadable-rs</code></a>, and the <a href="https://crates.io/crates/regex" rel="nofollow">regex crate</a>.</p>
  <p dir="auto">See <a href="https://observablehq.com/@asg017/introducing-sqlite-regex" rel="nofollow"><em>Introducing sqlite-regex: The fastest Regular Expression Extension for SQLite</em></a> (Jan 2023) for more details!</p>
  <p dir="auto">If your company or organization finds this library useful, consider <a href="#supporting">supporting my work</a>!</p>
  <p dir="auto"><a target="_blank" rel="noopener noreferrer" href="./benchmarks/dates.png"><img src="./benchmarks/dates.png" alt="" style="max-width: 100%;"></a></p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Usage</h2><a id="user-content-usage" class="anchor" aria-label="Permalink: Usage" href="#usage"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <div class="highlight highlight-source-sql notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content=".load ./regex0
  select 'foo' regexp 'f';
  "><pre>.load .<span class="pl-k">/</span>regex0
  <span class="pl-k">select</span> <span class="pl-s"><span class="pl-pds">'</span>foo<span class="pl-pds">'</span></span> regexp <span class="pl-s"><span class="pl-pds">'</span>f<span class="pl-pds">'</span></span>;
  </pre></div>
  <p dir="auto"><strong>Find all occurrences of a pattern in a string</strong></p>
  <div class="highlight highlight-source-sql notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="select regex_find(
    '[0-9]{3}-[0-9]{3}-[0-9]{4}',
    'phone: 111-222-3333'
  );
  -- '111-222-3333'

  select rowid, *
  from regex_find_all(
    '\b\w{13}\b',
    'Retroactively relinquishing remunerations is reprehensible.'
  );
  /*
  ┌───────┬───────┬─────┬───────────────┐
  │ rowid │ start │ end │     match     │
  ├───────┼───────┼─────┼───────────────┤
  │ 0     │ 0     │ 13  │ Retroactively │
  │ 1     │ 14    │ 27  │ relinquishing │
  │ 2     │ 28    │ 41  │ remunerations │
  │ 3     │ 45    │ 58  │ reprehensible │
  └───────┴───────┴─────┴───────────────┘
  */"><pre><span class="pl-k">select</span> regex_find(
    <span class="pl-s"><span class="pl-pds">'</span>[0-9]{3}-[0-9]{3}-[0-9]{4}<span class="pl-pds">'</span></span>,
    <span class="pl-s"><span class="pl-pds">'</span>phone: 111-222-3333<span class="pl-pds">'</span></span>
  );
  <span class="pl-c"><span class="pl-c">--</span> '111-222-3333'</span>

  <span class="pl-k">select</span> rowid, <span class="pl-k">*</span>
  <span class="pl-k">from</span> regex_find_all(
    <span class="pl-s"><span class="pl-pds">'</span><span class="pl-cce">\b\w</span>{13}<span class="pl-cce">\b</span><span class="pl-pds">'</span></span>,
    <span class="pl-s"><span class="pl-pds">'</span>Retroactively relinquishing remunerations is reprehensible.<span class="pl-pds">'</span></span>
  );
  <span class="pl-c"><span class="pl-c">/*</span></span>
  <span class="pl-c">┌───────┬───────┬─────┬───────────────┐</span>
  <span class="pl-c">│ rowid │ start │ end │     match     │</span>
  <span class="pl-c">├───────┼───────┼─────┼───────────────┤</span>
  <span class="pl-c">│ 0     │ 0     │ 13  │ Retroactively │</span>
  <span class="pl-c">│ 1     │ 14    │ 27  │ relinquishing │</span>
  <span class="pl-c">│ 2     │ 28    │ 41  │ remunerations │</span>
  <span class="pl-c">│ 3     │ 45    │ 58  │ reprehensible │</span>
  <span class="pl-c">└───────┴───────┴─────┴───────────────┘</span>
  <span class="pl-c"><span class="pl-c">*/</span></span></pre></div>
  <p dir="auto"><strong>Extract capture group values by index or name</strong></p>
  <div class="highlight highlight-source-sql notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="select
    regex_capture(captures, 0)        as entire_match,
    regex_capture(captures, 'title')  as title,
    regex_capture(captures, 'year')   as year
  from regex_captures(
    regex("'(?P<title>[^']+)'\s+\((?P<year>\d{4})\)"),
    "'Citizen Kane' (1941), 'The Wizard of Oz' (1939), 'M' (1931)."
  );
  /*
  ┌───────────────────────────┬──────────────────┬──────┐
  │       entire_match        │      title       │ year │
  ├───────────────────────────┼──────────────────┼──────┤
  │ 'Citizen Kane' (1941)     │ Citizen Kane     │ 1941 │
  │ 'The Wizard of Oz' (1939) │ The Wizard of Oz │ 1939 │
  │ 'M' (1931)                │ M                │ 1931 │
  └───────────────────────────┴──────────────────┴──────┘
  */"><pre><span class="pl-k">select</span>
    regex_capture(captures, <span class="pl-c1">0</span>)        <span class="pl-k">as</span> entire_match,
    regex_capture(captures, <span class="pl-s"><span class="pl-pds">'</span>title<span class="pl-pds">'</span></span>)  <span class="pl-k">as</span> title,
    regex_capture(captures, <span class="pl-s"><span class="pl-pds">'</span>year<span class="pl-pds">'</span></span>)   <span class="pl-k">as</span> year
  <span class="pl-k">from</span> regex_captures(
    regex(<span class="pl-s"><span class="pl-pds">"</span>'(?P<title>[^']+)'\s+\((?P<year>\d{4})\)<span class="pl-pds">"</span></span>),
    <span class="pl-s"><span class="pl-pds">"</span>'Citizen Kane' (1941), 'The Wizard of Oz' (1939), 'M' (1931).<span class="pl-pds">"</span></span>
  );
  <span class="pl-c"><span class="pl-c">/*</span></span>
  <span class="pl-c">┌───────────────────────────┬──────────────────┬──────┐</span>
  <span class="pl-c">│       entire_match        │      title       │ year │</span>
  <span class="pl-c">├───────────────────────────┼──────────────────┼──────┤</span>
  <span class="pl-c">│ 'Citizen Kane' (1941)     │ Citizen Kane     │ 1941 │</span>
  <span class="pl-c">│ 'The Wizard of Oz' (1939) │ The Wizard of Oz │ 1939 │</span>
  <span class="pl-c">│ 'M' (1931)                │ M                │ 1931 │</span>
  <span class="pl-c">└───────────────────────────┴──────────────────┴──────┘</span>
  <span class="pl-c"><span class="pl-c">*/</span></span></pre></div>
  <p dir="auto"><strong>Use RegexSets to match a string on multiple patterns in linear time</strong></p>
  <div class="highlight highlight-source-sql notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="select regexset_is_match(
    regexset(
      "bar",
      "foo",
      "barfoo"
    ),
    'foobar'
  )"><pre><span class="pl-k">select</span> regexset_is_match(
    regexset(
      <span class="pl-s"><span class="pl-pds">"</span>bar<span class="pl-pds">"</span></span>,
      <span class="pl-s"><span class="pl-pds">"</span>foo<span class="pl-pds">"</span></span>,
      <span class="pl-s"><span class="pl-pds">"</span>barfoo<span class="pl-pds">"</span></span>
    ),
    <span class="pl-s"><span class="pl-pds">'</span>foobar<span class="pl-pds">'</span></span>
  )</pre></div>
  <p dir="auto"><strong>Split the string on the given pattern delimiter</strong></p>
  <div class="highlight highlight-source-sql notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="select rowid, *
  from regex_split('[ \t]+', 'a b     c d    e');
  /*
  ┌───────┬──────┐
  │ rowid │ item │
  ├───────┼──────┤
  │ 0     │ a    │
  │ 1     │ b    │
  │ 2     │ c    │
  │ 3     │ d    │
  │ 4     │ e    │
  └───────┴──────┘
  */"><pre><span class="pl-k">select</span> rowid, <span class="pl-k">*</span>
  <span class="pl-k">from</span> regex_split(<span class="pl-s"><span class="pl-pds">'</span>[ <span class="pl-cce">\t</span>]+<span class="pl-pds">'</span></span>, <span class="pl-s"><span class="pl-pds">'</span>a b     c d    e<span class="pl-pds">'</span></span>);
  <span class="pl-c"><span class="pl-c">/*</span></span>
  <span class="pl-c">┌───────┬──────┐</span>
  <span class="pl-c">│ rowid │ item │</span>
  <span class="pl-c">├───────┼──────┤</span>
  <span class="pl-c">│ 0     │ a    │</span>
  <span class="pl-c">│ 1     │ b    │</span>
  <span class="pl-c">│ 2     │ c    │</span>
  <span class="pl-c">│ 3     │ d    │</span>
  <span class="pl-c">│ 4     │ e    │</span>
  <span class="pl-c">└───────┴──────┘</span>
  <span class="pl-c"><span class="pl-c">*/</span></span></pre></div>
  <p dir="auto"><strong>Replace occurrences of a pattern with another string</strong></p>
  <div class="highlight highlight-source-sql notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="select regex_replace(
    '(?P<last>[^,\s]+),\s+(?P<first>\S+)',
    'Springsteen, Bruce',
    '$first $last'
  );
  -- 'Bruce Springsteen'

  select regex_replace_all('a', 'abc abc', '');
  -- 'bc bc'"><pre><span class="pl-k">select</span> regex_replace(
    <span class="pl-s"><span class="pl-pds">'</span>(?P<last>[^,<span class="pl-cce">\s</span>]+),<span class="pl-cce">\s</span>+(?P<first><span class="pl-cce">\S</span>+)<span class="pl-pds">'</span></span>,
    <span class="pl-s"><span class="pl-pds">'</span>Springsteen, Bruce<span class="pl-pds">'</span></span>,
    <span class="pl-s"><span class="pl-pds">'</span>$first $last<span class="pl-pds">'</span></span>
  );
  <span class="pl-c"><span class="pl-c">--</span> 'Bruce Springsteen'</span>

  <span class="pl-k">select</span> regex_replace_all(<span class="pl-s"><span class="pl-pds">'</span>a<span class="pl-pds">'</span></span>, <span class="pl-s"><span class="pl-pds">'</span>abc abc<span class="pl-pds">'</span></span>, <span class="pl-s"><span class="pl-pds">'</span><span class="pl-pds">'</span></span>);
  <span class="pl-c"><span class="pl-c">--</span> 'bc bc'</span></pre></div>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Documentation</h2><a id="user-content-documentation" class="anchor" aria-label="Permalink: Documentation" href="#documentation"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">See <a href="./docs.md"><code>docs.md</code></a> for a full API reference.</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Installing</h2><a id="user-content-installing" class="anchor" aria-label="Permalink: Installing" href="#installing"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <markdown-accessiblity-table><table>
  <thead>
  <tr>
  <th>Language</th>
  <th>Install</th>
  <th></th>
  </tr>
  </thead>
  <tbody>
  <tr>
  <td>Python</td>
  <td><code>pip install sqlite-regex</code></td>
  <td><a href="https://pypi.org/project/sqlite-regex/" rel="nofollow"><img src="https://img.shields.io/pypi/v/sqlite-regex.svg?color=blue&logo=python&logoColor=white" alt="PyPI" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/7bf95741f64da97cb4a08a3438c02a788ef937655db66de9dea9ec37d129568f/68747470733a2f2f696d672e736869656c64732e696f2f707970692f762f73716c6974652d72656765782e7376673f636f6c6f723d626c7565266c6f676f3d707974686f6e266c6f676f436f6c6f723d7768697465"></a></td>
  </tr>
  <tr>
  <td>Datasette</td>
  <td><code>datasette install datasette-sqlite-regex</code></td>
  <td><a href="https://datasette.io/plugins/datasette-sqlite-regex" rel="nofollow"><img src="https://img.shields.io/pypi/v/datasette-sqlite-regex.svg?color=B6B6D9&label=Datasette+plugin&logoColor=white&logo=python" alt="Datasette" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/9d6617551dc0b41361b66ebfd87594fa345584429fe8c94595d2ba3c29e89846/68747470733a2f2f696d672e736869656c64732e696f2f707970692f762f6461746173657474652d73716c6974652d72656765782e7376673f636f6c6f723d423642364439266c6162656c3d4461746173657474652b706c7567696e266c6f676f436f6c6f723d7768697465266c6f676f3d707974686f6e"></a></td>
  </tr>
  <tr>
  <td>Node.js</td>
  <td><code>npm install sqlite-regex</code></td>
  <td><a href="https://www.npmjs.com/package/sqlite-regex" rel="nofollow"><img src="https://img.shields.io/npm/v/sqlite-regex.svg?color=green&logo=nodedotjs&logoColor=white" alt="npm" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/83c2fc543cc0a4cda1f259e6998331a52f37164f53ea2008b0d3977cebeb6e7c/68747470733a2f2f696d672e736869656c64732e696f2f6e706d2f762f73716c6974652d72656765782e7376673f636f6c6f723d677265656e266c6f676f3d6e6f6465646f746a73266c6f676f436f6c6f723d7768697465"></a></td>
  </tr>
  <tr>
  <td>Deno</td>
  <td><a href="https://deno.land/x/sqlite_regex" rel="nofollow"><code>deno.land/x/sqlite_regex</code></a></td>
  <td><a href="https://deno.land/x/sqlite_regex" rel="nofollow"><img src="https://img.shields.io/github/v/release/asg017/sqlite-regex?color=fef8d2&include_prereleases&label=deno.land%2Fx&logo=deno" alt="deno.land/x release" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/eadb6fb2a2bd03c17eccb5bff91eb559d8fd63112fe6f9562dc40973a79459f1/68747470733a2f2f696d672e736869656c64732e696f2f6769746875622f762f72656c656173652f6173673031372f73716c6974652d72656765783f636f6c6f723d66656638643226696e636c7564655f70726572656c6561736573266c6162656c3d64656e6f2e6c616e6425324678266c6f676f3d64656e6f"></a></td>
  </tr>
  <tr>
  <td>Ruby</td>
  <td><code>gem install sqlite-regex</code></td>
  <td><a target="_blank" rel="noopener noreferrer nofollow" href="https://camo.githubusercontent.com/eaed247131a4fd8505d2a197e88116ed7a17049c001164f74b3d1adf63c0b29e/68747470733a2f2f696d672e736869656c64732e696f2f67656d2f762f73716c6974652d72656765783f636f6c6f723d726564266c6f676f3d7275627967656d73266c6f676f436f6c6f723d7768697465"><img src="https://img.shields.io/gem/v/sqlite-regex?color=red&logo=rubygems&logoColor=white" alt="Gem" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/eaed247131a4fd8505d2a197e88116ed7a17049c001164f74b3d1adf63c0b29e/68747470733a2f2f696d672e736869656c64732e696f2f67656d2f762f73716c6974652d72656765783f636f6c6f723d726564266c6f676f3d7275627967656d73266c6f676f436f6c6f723d7768697465"></a></td>
  </tr>
  <tr>
  <td>Github Release</td>
  <td></td>
  <td><a target="_blank" rel="noopener noreferrer nofollow" href="https://camo.githubusercontent.com/a4dc58f470a7b6ea7ae96716d8efc5005255df259c956373c1f0cc0d45494e8a/68747470733a2f2f696d672e736869656c64732e696f2f6769746875622f762f7461672f6173673031372f73716c6974652d72656765783f636f6c6f723d6c696768746772657926696e636c7564655f70726572656c6561736573266c6162656c3d4769746875622b72656c65617365266c6f676f3d676974687562"><img src="https://img.shields.io/github/v/tag/asg017/sqlite-regex?color=lightgrey&include_prereleases&label=Github+release&logo=github" alt="GitHub tag (latest SemVer pre-release)" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/a4dc58f470a7b6ea7ae96716d8efc5005255df259c956373c1f0cc0d45494e8a/68747470733a2f2f696d672e736869656c64732e696f2f6769746875622f762f7461672f6173673031372f73716c6974652d72656765783f636f6c6f723d6c696768746772657926696e636c7564655f70726572656c6561736573266c6162656c3d4769746875622b72656c65617365266c6f676f3d676974687562"></a></td>
  </tr>
  <tr>
  <td>Rust</td>
  <td><code>cargo add sqlite-regex</code></td>
  <td><a href="https://crates.io/crates/sqlite-regex" rel="nofollow"><img src="https://img.shields.io/crates/v/sqlite-regex?logo=rust" alt="Crates.io" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/d86899311dda3effe0153ed78921c0723f9d2f3a004999b573d7dd7f402bc4cd/68747470733a2f2f696d672e736869656c64732e696f2f6372617465732f762f73716c6974652d72656765783f6c6f676f3d72757374"></a></td>
  </tr>
  </tbody>
  </table></markdown-accessiblity-table>

  <p dir="auto">The <a href="https://github.com/asg017/sqlite-regex/releases">Releases page</a> contains pre-built binaries for Linux x86_64, MacOS, and Windows.</p>
  <div class="markdown-heading" dir="auto"><h3 class="heading-element" dir="auto">As a loadable extension</h3><a id="user-content-as-a-loadable-extension" class="anchor" aria-label="Permalink: As a loadable extension" href="#as-a-loadable-extension"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">If you want to use <code>sqlite-regex</code> as a <a href="https://www.sqlite.org/loadext.html" rel="nofollow">Runtime-loadable extension</a>, Download the <code>regex0.dylib</code> (for MacOS), <code>regex0.so</code> (Linux), or <code>regex0.dll</code> (Windows) file from a release and load it into your SQLite environment.</p>
  <blockquote>
  <p dir="auto"><strong>Note:</strong>
  The <code>0</code> in the filename (<code>regex0.dylib</code>/ <code>regex0.so</code>/<code>regex0.dll</code>) denotes the major version of <code>sqlite-regex</code>. Currently <code>sqlite-regex</code> is pre v1, so expect breaking changes in future versions.</p>
  </blockquote>
  <p dir="auto">For example, if you are using the <a href="https://www.sqlite.org/cli.html" rel="nofollow">SQLite CLI</a>, you can load the library like so:</p>
  <div class="highlight highlight-source-sql notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content=".load ./regex0
  select regex_version();
  -- v0.1.0"><pre>.load .<span class="pl-k">/</span>regex0
  <span class="pl-k">select</span> regex_version();
  <span class="pl-c"><span class="pl-c">--</span> v0.1.0</span></pre></div>
  <p dir="auto">Or in Python, using the builtin <a href="https://docs.python.org/3/library/sqlite3.html" rel="nofollow">sqlite3 module</a>:</p>
  <div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="import sqlite3
  con = sqlite3.connect(":memory:")
  con.enable_load_extension(True)
  con.load_extension("./regex0")
  print(con.execute("select regex_version()").fetchone())
  # ('v0.1.0',)"><pre><span class="pl-k">import</span> <span class="pl-s1">sqlite3</span>
  <span class="pl-s1">con</span> <span class="pl-c1">=</span> <span class="pl-s1">sqlite3</span>.<span class="pl-c1">connect</span>(<span class="pl-s">":memory:"</span>)
  <span class="pl-s1">con</span>.<span class="pl-c1">enable_load_extension</span>(<span class="pl-c1">True</span>)
  <span class="pl-s1">con</span>.<span class="pl-c1">load_extension</span>(<span class="pl-s">"./regex0"</span>)
  <span class="pl-en">print</span>(<span class="pl-s1">con</span>.<span class="pl-c1">execute</span>(<span class="pl-s">"select regex_version()"</span>).<span class="pl-c1">fetchone</span>())
  <span class="pl-c"># ('v0.1.0',)</span></pre></div>
  <p dir="auto">Or in Node.js using <a href="https://github.com/WiseLibs/better-sqlite3">better-sqlite3</a>:</p>
  <div class="highlight highlight-source-js notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="const Database = require("better-sqlite3");
  const db = new Database(":memory:");
  db.loadExtension("./regex0");
  console.log(db.prepare("select regex_version()").get());
  // { 'regex_version()': 'v0.1.0' }"><pre><span class="pl-k">const</span> <span class="pl-v">Database</span> <span class="pl-c1">=</span> <span class="pl-en">require</span><span class="pl-kos">(</span><span class="pl-s">"better-sqlite3"</span><span class="pl-kos">)</span><span class="pl-kos">;</span>
  <span class="pl-k">const</span> <span class="pl-s1">db</span> <span class="pl-c1">=</span> <span class="pl-k">new</span> <span class="pl-v">Database</span><span class="pl-kos">(</span><span class="pl-s">":memory:"</span><span class="pl-kos">)</span><span class="pl-kos">;</span>
  <span class="pl-s1">db</span><span class="pl-kos">.</span><span class="pl-en">loadExtension</span><span class="pl-kos">(</span><span class="pl-s">"./regex0"</span><span class="pl-kos">)</span><span class="pl-kos">;</span>
  <span class="pl-smi">console</span><span class="pl-kos">.</span><span class="pl-en">log</span><span class="pl-kos">(</span><span class="pl-s1">db</span><span class="pl-kos">.</span><span class="pl-en">prepare</span><span class="pl-kos">(</span><span class="pl-s">"select regex_version()"</span><span class="pl-kos">)</span><span class="pl-kos">.</span><span class="pl-en">get</span><span class="pl-kos">(</span><span class="pl-kos">)</span><span class="pl-kos">)</span><span class="pl-kos">;</span>
  <span class="pl-c">// { 'regex_version()': 'v0.1.0' }</span></pre></div>
  <p dir="auto">Or with <a href="https://datasette.io/" rel="nofollow">Datasette</a>:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="datasette data.db --load-extension ./regex0"><pre class="notranslate"><code>datasette data.db --load-extension ./regex0
  </code></pre></div>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Supporting</h2><a id="user-content-supporting" class="anchor" aria-label="Permalink: Supporting" href="#supporting"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">I (Alex 👋🏼) spent a lot of time and energy on this project and <a href="https://github.com/asg017?tab=repositories&q=&type=&language=&sort=stargazers">many other open source projects</a>. If your company or organization uses this library (or you're feeling generous), then please <a href="https://alexgarcia.regex/work.html" rel="nofollow">consider supporting my work</a>, or share this project with a friend!</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">See also</h2><a id="user-content-see-also" class="anchor" aria-label="Permalink: See also" href="#see-also"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <ul dir="auto">
  <li><a href="https://github.com/asg017/sqlite-xsv">sqlite-xsv</a>, A SQLite extension for working with CSVs</li>
  <li><a href="https://github.com/asg017/sqlite-loadable-rs">sqlite-loadable</a>, A framework for writing SQLite extensions in Rust</li>
  <li><a href="https://github.com/asg017/sqlite-http">sqlite-http</a>, A SQLite extension for making HTTP requests</li>
  </ul>
  </article></div>


node_id: R_kgDOIX3aMg
name: sqlite-jsonschema
full_name: asg017/sqlite-jsonschema
html_url: https://github.com/asg017/sqlite-jsonschema
description: A SQLite extension for validating JSON objects with JSON Schema
created_at: 2022-11-04T18:34:59Z
updated_at: 2025-04-25T18:14:40Z
pushed_at: 2023-08-06T00:29:45Z
homepage: https://alexgarcia.xyz/sqlite-jsonschema
language: TypeScript
license: apache-2.0
topics: ["sqlite", "sqlite-extension"]
visibility: public
default_branch: main
permissions: {"admin": false, "maintain": false, "push": false, "triage": false, "pull": true}
readme:
  # sqlite-jsonschema

  A SQLite extension for validating JSON objects with [JSON Schema](https://json-schema.org/). Based on [`sqlite-loadable-rs`](https://github.com/asg017/sqlite-loadable-rs) and the [`jsonschema` crate](https://crates.io/crates/jsonschema).

  If your company or organization finds this library useful, consider [supporting my work](#supporting)!

  ## Usage

  ```sql
  .load ./jsonschema0
  select jsonschema_matches('{"maxLength": 5}', json_quote('alex'));
  ```

  Use with SQLite's [`CHECK` constraints](https://www.sqlite.org/lang_createtable.html#check_constraints) to validate JSON columns before inserting into a table.

  ```sql
  create table students(
    -- ensure that JSON objects stored in the data column have "firstName" strings,
    -- "lastName" strings, and "age" integers that are greater than 0.
    data json check (
      jsonschema_matches(
        json('
          {
            "type": "object",
            "properties": {
              "firstName": {
                "type": "string"
              },
              "lastName": {
                "type": "string"
              },
              "age": {
                "type": "integer",
                "minimum": 0
              }
            }
          }
          '),
        data
      )
    )
  );

  insert into students(data)
    values ('{"firstName": "Alex", "lastName": "Garcia", "age": 100}');
  -- ✓


  insert into students(data)
    values ('{"firstName": "Alex", "lastName": "Garcia", "age": -1}');
  -- Runtime error: CHECK constraint failed: jsonschema_matches

  ```

  Find all the values in a column that don't match a JSON Schema.

  ```sql
  select
    rowid,
    jsonschema_matches(
      '{
        "type": "array",
        "items": {
          "type": "number"
        }
      }',
      foo
    ) as valid
  from bar
  where not valid;
  ```

  ## Installing

  | Language       | Install                                                                  |                                                                                                                                                                                                         |
  | -------------- | ------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
  | Python         | `pip install sqlite-jsonschema`                                          | [![PyPI](https://img.shields.io/pypi/v/sqlite-jsonschema.svg?color=blue&logo=python&logoColor=white)](https://pypi.org/project/sqlite-jsonschema/)                                                      |
  | Datasette      | `datasette install datasette-sqlite-jsonschema`                          | [![Datasette](https://img.shields.io/pypi/v/datasette-sqlite-jsonschema.svg?color=B6B6D9&label=Datasette+plugin&logoColor=white&logo=python)](https://datasette.io/plugins/datasette-sqlite-jsonschema) |
  | Node.js        | `npm install sqlite-jsonschema`                                          | [![npm](https://img.shields.io/npm/v/sqlite-jsonschema.svg?color=green&logo=nodedotjs&logoColor=white)](https://www.npmjs.com/package/sqlite-jsonschema)                                                |
  | Deno           | [`deno.land/x/sqlite_jsonschema`](https://deno.land/x/sqlite_jsonschema) | [![deno.land/x release](https://img.shields.io/github/v/release/asg017/sqlite-jsonschema?color=fef8d2&include_prereleases&label=deno.land%2Fx&logo=deno)](https://deno.land/x/sqlite_jsonschema)        |
  | Ruby           | `gem install sqlite-jsonschema`                                          | ![Gem](https://img.shields.io/gem/v/sqlite-jsonschema?color=red&logo=rubygems&logoColor=white)                                                                                                          |
  | Github Release |                                                                          | ![GitHub tag (latest SemVer pre-release)](https://img.shields.io/github/v/tag/asg017/sqlite-jsonschema?color=lightgrey&include_prereleases&label=Github+release&logo=github)                            |
  | Rust           | `cargo add sqlite-jsonschema`                                            | [![Crates.io](https://img.shields.io/crates/v/sqlite-jsonschema?logo=rust)](https://crates.io/crates/sqlite-jsonschema)                                                                                 |

  <!--
  | Elixir         | [`hex.pm/packages/sqlite_jsonschema`](https://hex.pm/packages/sqlite_jsonschema) | [![Hex.pm](https://img.shields.io/hexpm/v/sqlite_jsonschema?color=purple&logo=elixir)](https://hex.pm/packages/sqlite_jsonschema)                                                                       |
  | Go             | `go get -u github.com/asg017/sqlite-jsonschema/bindings/go`               | [![Go Reference](https://pkg.go.dev/badge/github.com/asg017/sqlite-jsonschema/bindings/go.svg)](https://pkg.go.dev/github.com/asg017/sqlite-jsonschema/bindings/go)                                     |
  -->

  `sqlite-jsonschema` is distributed on pip, npm, and https://deno.land/x for Python, Node.js, and Deno programmers. There are also pre-built extensions available for use in other environments.

  ### Python

  For Python developers, use the [`sqlite-jsonschema` Python package](https://pypi.org/package/sqlite-jsonschema/):

  ```
  pip install sqlite-jsonschema
  ```

  The `sqlite-jsonschema` extension can then be loaded into a [`sqlite3` Connection object](https://docs.python.org/3/library/sqlite3.html#connection-objects).

  ```python
  import sqlite3
  import sqlite_jsonschema

  db = sqlite3.connect(':memory:')
  sqlite_jsonschema.load(db)
  db.execute('select jsonschema_version(), jsonschema()').fetchone()
  ```

  See [_Using `sqlite-jsonschema` with Python_](https://alexgarcia.jsonschema/sqlite-jsonschema/usage/python.html) for details.

  ### Node.js

  For Node.js developers, use the [`sqlite-jsonschema` NPM package](https://www.npmjs.com/package/sqlite-jsonschema):

  ```
  npm install sqlite-jsonschema
  ```

  The `sqlite-jsonschema` extension can then be loaded into a [`better-sqlite3`](https://github.com/WiseLibs/better-sqlite3) or [`node-sqlite3`](https://github.com/TryGhost/node-sqlite3) connection.

  ```javascript
  import Database from "better-sqlite3";
  import * as sqlite_jsonschema from "sqlite-jsonschema";

  const db = new Database(":memory:");

  db.loadExtension(sqlite_jsonschema.getLoadablePath());

  const version = db.prepare("select jsonschema_version()").pluck().get();
  console.log(version); // "v0.2.0"
  ```

  See [_Using `sqlite-jsonschema` with Node.js_](https://alexgarcia.jsonschema/sqlite-jsonschema/usage/node.html) for details.

  ### Deno

  For [Deno](https://deno.land/) developers, use the [x/sqlite_jsonschema](https://deno.land/x/sqlite_jsonschema@v0.2.2) Deno module with [`x/sqlite3`](https://deno.land/x/sqlite3@0.8.1).

  ```javascript
  import { Database } from "https://deno.land/x/sqlite3@0.8.0/mod.ts";
  import * as sqlite_jsonschema from "https://deno.land/x/sqlite_jsonschema/mod.ts";

  const db = new Database(":memory:");

  db.enableLoadExtension = true;
  db.loadExtension(sqlite_jsonschema.getLoadablePath());

  const [version] = db
    .prepare("select jsonschema_version()")
    .value<[string]>()!;

  console.log(version);
  ```

  See [_Using `sqlite-jsonschema` with Deno_](https://alexgarcia.jsonschema/sqlite-jsonschema/usage/deno.html) for details.

  ### Datasette

  For [Datasette](https://datasette.io/), use the [`datasette-sqlite-jsonschema` plugin](https://datasette.io/plugins/datasette-sqlite-jsonschema) to include `sqlite-jsonschema` functions to your Datasette instances.

  ```
  datasette install datasette-sqlite-jsonschema
  ```

  See [_Using `sqlite-jsonschema` with Datasette_](https://alexgarcia.jsonschema/sqlite-jsonschema/usage/datasette.html) for details.

  ### `sqlite3` CLI

  For [the `sqlite3` CLI](https://sqlite.org/cli.html), either [download a pre-compiled extension from the Releases page](https://github.com/asg017/sqlite-jsonschema/releases) or [build it yourself](#building-from-source). Then use the [`.load` dot command](https://sqlite.org/cli.html#loading_extensions).

  ```sql
  .load ./jsonschema0
  select jsonschema_version();
  'v0.2.1'
  ```

  ### As a loadable extension

  If you're using `sqlite-jsonschema` in a different way from those listed above, then [download a pre-compiled extension from the Releases page](https://github.com/asg017/sqlite-jsonschema/releases) and load it into your environment. Download the `jsonschema0.dylib` (for MacOS), `jsonschema0.so` (Linux), or `jsonschema0.dll` (Windows) file from a release and load it into your SQLite environment.

  > **Note:**
  > The `0` in the filename (`jsonschema0.dylib`/ `jsonschema0.so`/`jsonschema0.dll`) denotes the major version of `sqlite-jsonschema`. Currently `sqlite-jsonschema` is pre v1, so expect breaking changes in future versions.

  Chances are there is some method called "loadExtension" or "load_extension" in the SQLite client library you are using. Alternatively, as a last resort, use [the `load_extension()` SQL function](https://www.sqlite.org/lang_corefunc.html#load_extension).

  ### Building from source

  Make sure you have [Rust](https://www.rust-lang.org/tools/install), make, and a C compiler installed. Then `git clone` this repository and run `make loadable-release`.

  ```
  git clone https://github.com/asg017/sqlite-jsonschema.git
  cd sqlite-jsonschema
  make loadable-release
  ```

  Once complete, your compiled extension will appear under `dist/release/`, either as `jsonschema0.so`, `jsonschema0.dylib`, or `jsonschema0.dll`, depending on your operating system.

  ## Documentation

  See [the full API Reference](https://alexgarcia.jsonschema/sqlite-jsonschema/reference.html) for every `sqlite-jsonschema` SQL function.

  ## Supporting

  I (Alex 👋🏼) spent a lot of time and energy on this project and [many other open source projects](https://github.com/asg017?tab=repositories&q=&type=&language=&sort=stargazers). If your company or organization uses this library (or you're feeling generous), then please [consider supporting my work](https://alexgarcia.jsonschema/work.html), or share this project with a friend!

  ## See also

  - [`sqlite-xsv`](https://github.com/asg017/sqlite-xsv), A SQLite extension for working with CSVs
  - [`sqlite-http`](https://github.com/asg017/sqlite-http), A SQLite extension for making HTTP requests
  - [`sqlite-loadable-rs`](https://github.com/asg017/sqlite-loadable-rs), A framework for writing SQLite extensions in Rust

readme_html:
  <div id="readme" class="md" data-path="README.md"><article class="markdown-body entry-content container-lg" itemprop="text"><div class="markdown-heading" dir="auto"><h1 class="heading-element" dir="auto">sqlite-jsonschema</h1><a id="user-content-sqlite-jsonschema" class="anchor" aria-label="Permalink: sqlite-jsonschema" href="#sqlite-jsonschema"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">A SQLite extension for validating JSON objects with <a href="https://json-schema.org/" rel="nofollow">JSON Schema</a>. Based on <a href="https://github.com/asg017/sqlite-loadable-rs"><code>sqlite-loadable-rs</code></a> and the <a href="https://crates.io/crates/jsonschema" rel="nofollow"><code>jsonschema</code> crate</a>.</p>
  <p dir="auto">If your company or organization finds this library useful, consider <a href="#supporting">supporting my work</a>!</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Usage</h2><a id="user-content-usage" class="anchor" aria-label="Permalink: Usage" href="#usage"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <div class="highlight highlight-source-sql notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content=".load ./jsonschema0
  select jsonschema_matches('{"maxLength": 5}', json_quote('alex'));"><pre>.load .<span class="pl-k">/</span>jsonschema0
  <span class="pl-k">select</span> jsonschema_matches(<span class="pl-s"><span class="pl-pds">'</span>{"maxLength": 5}<span class="pl-pds">'</span></span>, json_quote(<span class="pl-s"><span class="pl-pds">'</span>alex<span class="pl-pds">'</span></span>));</pre></div>
  <p dir="auto">Use with SQLite's <a href="https://www.sqlite.org/lang_createtable.html#check_constraints" rel="nofollow"><code>CHECK</code> constraints</a> to validate JSON columns before inserting into a table.</p>
  <div class="highlight highlight-source-sql notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="create table students(
    -- ensure that JSON objects stored in the data column have "firstName" strings,
    -- "lastName" strings, and "age" integers that are greater than 0.
    data json check (
      jsonschema_matches(
        json('
          {
            "type": "object",
            "properties": {
              "firstName": {
                "type": "string"
              },
              "lastName": {
                "type": "string"
              },
              "age": {
                "type": "integer",
                "minimum": 0
              }
            }
          }
          '),
        data
      )
    )
  );

  insert into students(data)
    values ('{"firstName": "Alex", "lastName": "Garcia", "age": 100}');
  -- ✓


  insert into students(data)
    values ('{"firstName": "Alex", "lastName": "Garcia", "age": -1}');
  -- Runtime error: CHECK constraint failed: jsonschema_matches
  "><pre><span class="pl-k">create</span> <span class="pl-k">table</span> <span class="pl-en">students</span>(
    <span class="pl-c"><span class="pl-c">--</span> ensure that JSON objects stored in the data column have "firstName" strings,</span>
    <span class="pl-c"><span class="pl-c">--</span> "lastName" strings, and "age" integers that are greater than 0.</span>
    data json <span class="pl-k">check</span> (
      jsonschema_matches(
        json(<span class="pl-s"><span class="pl-pds">'</span></span>
  <span class="pl-s">        {</span>
  <span class="pl-s">          "type": "object",</span>
  <span class="pl-s">          "properties": {</span>
  <span class="pl-s">            "firstName": {</span>
  <span class="pl-s">              "type": "string"</span>
  <span class="pl-s">            },</span>
  <span class="pl-s">            "lastName": {</span>
  <span class="pl-s">              "type": "string"</span>
  <span class="pl-s">            },</span>
  <span class="pl-s">            "age": {</span>
  <span class="pl-s">              "type": "integer",</span>
  <span class="pl-s">              "minimum": 0</span>
  <span class="pl-s">            }</span>
  <span class="pl-s">          }</span>
  <span class="pl-s">        }</span>
  <span class="pl-s">        <span class="pl-pds">'</span></span>),
        data
      )
    )
  );

  <span class="pl-k">insert into</span> students(data)
    <span class="pl-k">values</span> (<span class="pl-s"><span class="pl-pds">'</span>{"firstName": "Alex", "lastName": "Garcia", "age": 100}<span class="pl-pds">'</span></span>);
  <span class="pl-c"><span class="pl-c">--</span> ✓</span>


  <span class="pl-k">insert into</span> students(data)
    <span class="pl-k">values</span> (<span class="pl-s"><span class="pl-pds">'</span>{"firstName": "Alex", "lastName": "Garcia", "age": -1}<span class="pl-pds">'</span></span>);
  <span class="pl-c"><span class="pl-c">--</span> Runtime error: CHECK constraint failed: jsonschema_matches</span>
  </pre></div>
  <p dir="auto">Find all the values in a column that don't match a JSON Schema.</p>
  <div class="highlight highlight-source-sql notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="select
    rowid,
    jsonschema_matches(
      '{
        "type": "array",
        "items": {
          "type": "number"
        }
      }',
      foo
    ) as valid
  from bar
  where not valid;"><pre><span class="pl-k">select</span>
    rowid,
    jsonschema_matches(
      <span class="pl-s"><span class="pl-pds">'</span>{</span>
  <span class="pl-s">      "type": "array",</span>
  <span class="pl-s">      "items": {</span>
  <span class="pl-s">        "type": "number"</span>
  <span class="pl-s">      }</span>
  <span class="pl-s">    }<span class="pl-pds">'</span></span>,
      foo
    ) <span class="pl-k">as</span> valid
  <span class="pl-k">from</span> bar
  <span class="pl-k">where</span> not valid;</pre></div>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Installing</h2><a id="user-content-installing" class="anchor" aria-label="Permalink: Installing" href="#installing"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <markdown-accessiblity-table><table>
  <thead>
  <tr>
  <th>Language</th>
  <th>Install</th>
  <th></th>
  </tr>
  </thead>
  <tbody>
  <tr>
  <td>Python</td>
  <td><code>pip install sqlite-jsonschema</code></td>
  <td><a href="https://pypi.org/project/sqlite-jsonschema/" rel="nofollow"><img src="https://img.shields.io/pypi/v/sqlite-jsonschema.svg?color=blue&logo=python&logoColor=white" alt="PyPI" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/dd58c73a3ea7adb3194cf38f73871f5f612ac0c391e6579f9f3654ac5fad8d66/68747470733a2f2f696d672e736869656c64732e696f2f707970692f762f73716c6974652d6a736f6e736368656d612e7376673f636f6c6f723d626c7565266c6f676f3d707974686f6e266c6f676f436f6c6f723d7768697465"></a></td>
  </tr>
  <tr>
  <td>Datasette</td>
  <td><code>datasette install datasette-sqlite-jsonschema</code></td>
  <td><a href="https://datasette.io/plugins/datasette-sqlite-jsonschema" rel="nofollow"><img src="https://img.shields.io/pypi/v/datasette-sqlite-jsonschema.svg?color=B6B6D9&label=Datasette+plugin&logoColor=white&logo=python" alt="Datasette" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/57b85a79d780f51002ccc48d9bd049b9f98bb30fc7469200050b34f88ef94dd2/68747470733a2f2f696d672e736869656c64732e696f2f707970692f762f6461746173657474652d73716c6974652d6a736f6e736368656d612e7376673f636f6c6f723d423642364439266c6162656c3d4461746173657474652b706c7567696e266c6f676f436f6c6f723d7768697465266c6f676f3d707974686f6e"></a></td>
  </tr>
  <tr>
  <td>Node.js</td>
  <td><code>npm install sqlite-jsonschema</code></td>
  <td><a href="https://www.npmjs.com/package/sqlite-jsonschema" rel="nofollow"><img src="https://img.shields.io/npm/v/sqlite-jsonschema.svg?color=green&logo=nodedotjs&logoColor=white" alt="npm" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/20c99636d747622e0cfdc2f1b72f80604cd6367b8d948cb69f53822570ed6081/68747470733a2f2f696d672e736869656c64732e696f2f6e706d2f762f73716c6974652d6a736f6e736368656d612e7376673f636f6c6f723d677265656e266c6f676f3d6e6f6465646f746a73266c6f676f436f6c6f723d7768697465"></a></td>
  </tr>
  <tr>
  <td>Deno</td>
  <td><a href="https://deno.land/x/sqlite_jsonschema" rel="nofollow"><code>deno.land/x/sqlite_jsonschema</code></a></td>
  <td><a href="https://deno.land/x/sqlite_jsonschema" rel="nofollow"><img src="https://img.shields.io/github/v/release/asg017/sqlite-jsonschema?color=fef8d2&include_prereleases&label=deno.land%2Fx&logo=deno" alt="deno.land/x release" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/c7af2d54294c90a7cd663046215c433193bbd88e26b3adfe8a630f30882e94e4/68747470733a2f2f696d672e736869656c64732e696f2f6769746875622f762f72656c656173652f6173673031372f73716c6974652d6a736f6e736368656d613f636f6c6f723d66656638643226696e636c7564655f70726572656c6561736573266c6162656c3d64656e6f2e6c616e6425324678266c6f676f3d64656e6f"></a></td>
  </tr>
  <tr>
  <td>Ruby</td>
  <td><code>gem install sqlite-jsonschema</code></td>
  <td><a target="_blank" rel="noopener noreferrer nofollow" href="https://camo.githubusercontent.com/93a1ab6362443da4a0bc010c3c2de94bf00f9b7c67c13a4daee4677f1623047b/68747470733a2f2f696d672e736869656c64732e696f2f67656d2f762f73716c6974652d6a736f6e736368656d613f636f6c6f723d726564266c6f676f3d7275627967656d73266c6f676f436f6c6f723d7768697465"><img src="https://img.shields.io/gem/v/sqlite-jsonschema?color=red&logo=rubygems&logoColor=white" alt="Gem" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/93a1ab6362443da4a0bc010c3c2de94bf00f9b7c67c13a4daee4677f1623047b/68747470733a2f2f696d672e736869656c64732e696f2f67656d2f762f73716c6974652d6a736f6e736368656d613f636f6c6f723d726564266c6f676f3d7275627967656d73266c6f676f436f6c6f723d7768697465"></a></td>
  </tr>
  <tr>
  <td>Github Release</td>
  <td></td>
  <td><a target="_blank" rel="noopener noreferrer nofollow" href="https://camo.githubusercontent.com/696c949f352fe82715065cae6d28da452fe9d163310246e7720b573ae1c4d9bf/68747470733a2f2f696d672e736869656c64732e696f2f6769746875622f762f7461672f6173673031372f73716c6974652d6a736f6e736368656d613f636f6c6f723d6c696768746772657926696e636c7564655f70726572656c6561736573266c6162656c3d4769746875622b72656c65617365266c6f676f3d676974687562"><img src="https://img.shields.io/github/v/tag/asg017/sqlite-jsonschema?color=lightgrey&include_prereleases&label=Github+release&logo=github" alt="GitHub tag (latest SemVer pre-release)" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/696c949f352fe82715065cae6d28da452fe9d163310246e7720b573ae1c4d9bf/68747470733a2f2f696d672e736869656c64732e696f2f6769746875622f762f7461672f6173673031372f73716c6974652d6a736f6e736368656d613f636f6c6f723d6c696768746772657926696e636c7564655f70726572656c6561736573266c6162656c3d4769746875622b72656c65617365266c6f676f3d676974687562"></a></td>
  </tr>
  <tr>
  <td>Rust</td>
  <td><code>cargo add sqlite-jsonschema</code></td>
  <td><a href="https://crates.io/crates/sqlite-jsonschema" rel="nofollow"><img src="https://img.shields.io/crates/v/sqlite-jsonschema?logo=rust" alt="Crates.io" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/b6f373be229c814ffc58c596e6e121c4f61e3a740aa759e2eebd22aee940f194/68747470733a2f2f696d672e736869656c64732e696f2f6372617465732f762f73716c6974652d6a736f6e736368656d613f6c6f676f3d72757374"></a></td>
  </tr>
  </tbody>
  </table></markdown-accessiblity-table>

  <p dir="auto"><code>sqlite-jsonschema</code> is distributed on pip, npm, and <a href="https://deno.land/x" rel="nofollow">https://deno.land/x</a> for Python, Node.js, and Deno programmers. There are also pre-built extensions available for use in other environments.</p>
  <div class="markdown-heading" dir="auto"><h3 class="heading-element" dir="auto">Python</h3><a id="user-content-python" class="anchor" aria-label="Permalink: Python" href="#python"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">For Python developers, use the <a href="https://pypi.org/package/sqlite-jsonschema/" rel="nofollow"><code>sqlite-jsonschema</code> Python package</a>:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="pip install sqlite-jsonschema"><pre class="notranslate"><code>pip install sqlite-jsonschema
  </code></pre></div>
  <p dir="auto">The <code>sqlite-jsonschema</code> extension can then be loaded into a <a href="https://docs.python.org/3/library/sqlite3.html#connection-objects" rel="nofollow"><code>sqlite3</code> Connection object</a>.</p>
  <div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="import sqlite3
  import sqlite_jsonschema

  db = sqlite3.connect(':memory:')
  sqlite_jsonschema.load(db)
  db.execute('select jsonschema_version(), jsonschema()').fetchone()"><pre><span class="pl-k">import</span> <span class="pl-s1">sqlite3</span>
  <span class="pl-k">import</span> <span class="pl-s1">sqlite_jsonschema</span>

  <span class="pl-s1">db</span> <span class="pl-c1">=</span> <span class="pl-s1">sqlite3</span>.<span class="pl-c1">connect</span>(<span class="pl-s">':memory:'</span>)
  <span class="pl-s1">sqlite_jsonschema</span>.<span class="pl-c1">load</span>(<span class="pl-s1">db</span>)
  <span class="pl-s1">db</span>.<span class="pl-c1">execute</span>(<span class="pl-s">'select jsonschema_version(), jsonschema()'</span>).<span class="pl-c1">fetchone</span>()</pre></div>
  <p dir="auto">See <a href="https://alexgarcia.jsonschema/sqlite-jsonschema/usage/python.html" rel="nofollow"><em>Using <code>sqlite-jsonschema</code> with Python</em></a> for details.</p>
  <div class="markdown-heading" dir="auto"><h3 class="heading-element" dir="auto">Node.js</h3><a id="user-content-nodejs" class="anchor" aria-label="Permalink: Node.js" href="#nodejs"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">For Node.js developers, use the <a href="https://www.npmjs.com/package/sqlite-jsonschema" rel="nofollow"><code>sqlite-jsonschema</code> NPM package</a>:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="npm install sqlite-jsonschema"><pre class="notranslate"><code>npm install sqlite-jsonschema
  </code></pre></div>
  <p dir="auto">The <code>sqlite-jsonschema</code> extension can then be loaded into a <a href="https://github.com/WiseLibs/better-sqlite3"><code>better-sqlite3</code></a> or <a href="https://github.com/TryGhost/node-sqlite3"><code>node-sqlite3</code></a> connection.</p>
  <div class="highlight highlight-source-js notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="import Database from "better-sqlite3";
  import * as sqlite_jsonschema from "sqlite-jsonschema";

  const db = new Database(":memory:");

  db.loadExtension(sqlite_jsonschema.getLoadablePath());

  const version = db.prepare("select jsonschema_version()").pluck().get();
  console.log(version); // "v0.2.0""><pre><span class="pl-k">import</span> <span class="pl-v">Database</span> <span class="pl-k">from</span> <span class="pl-s">"better-sqlite3"</span><span class="pl-kos">;</span>
  <span class="pl-k">import</span> <span class="pl-c1">*</span> <span class="pl-k">as</span> <span class="pl-s1">sqlite_jsonschema</span> <span class="pl-k">from</span> <span class="pl-s">"sqlite-jsonschema"</span><span class="pl-kos">;</span>

  <span class="pl-k">const</span> <span class="pl-s1">db</span> <span class="pl-c1">=</span> <span class="pl-k">new</span> <span class="pl-v">Database</span><span class="pl-kos">(</span><span class="pl-s">":memory:"</span><span class="pl-kos">)</span><span class="pl-kos">;</span>

  <span class="pl-s1">db</span><span class="pl-kos">.</span><span class="pl-en">loadExtension</span><span class="pl-kos">(</span><span class="pl-s1">sqlite_jsonschema</span><span class="pl-kos">.</span><span class="pl-en">getLoadablePath</span><span class="pl-kos">(</span><span class="pl-kos">)</span><span class="pl-kos">)</span><span class="pl-kos">;</span>

  <span class="pl-k">const</span> <span class="pl-s1">version</span> <span class="pl-c1">=</span> <span class="pl-s1">db</span><span class="pl-kos">.</span><span class="pl-en">prepare</span><span class="pl-kos">(</span><span class="pl-s">"select jsonschema_version()"</span><span class="pl-kos">)</span><span class="pl-kos">.</span><span class="pl-en">pluck</span><span class="pl-kos">(</span><span class="pl-kos">)</span><span class="pl-kos">.</span><span class="pl-en">get</span><span class="pl-kos">(</span><span class="pl-kos">)</span><span class="pl-kos">;</span>
  <span class="pl-smi">console</span><span class="pl-kos">.</span><span class="pl-en">log</span><span class="pl-kos">(</span><span class="pl-s1">version</span><span class="pl-kos">)</span><span class="pl-kos">;</span> <span class="pl-c">// "v0.2.0"</span></pre></div>
  <p dir="auto">See <a href="https://alexgarcia.jsonschema/sqlite-jsonschema/usage/node.html" rel="nofollow"><em>Using <code>sqlite-jsonschema</code> with Node.js</em></a> for details.</p>
  <div class="markdown-heading" dir="auto"><h3 class="heading-element" dir="auto">Deno</h3><a id="user-content-deno" class="anchor" aria-label="Permalink: Deno" href="#deno"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">For <a href="https://deno.land/" rel="nofollow">Deno</a> developers, use the <a href="https://deno.land/x/sqlite_jsonschema@v0.2.2" rel="nofollow">x/sqlite_jsonschema</a> Deno module with <a href="https://deno.land/x/sqlite3@0.8.1" rel="nofollow"><code>x/sqlite3</code></a>.</p>
  <div class="highlight highlight-source-js notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="import { Database } from "https://deno.land/x/sqlite3@0.8.0/mod.ts";
  import * as sqlite_jsonschema from "https://deno.land/x/sqlite_jsonschema/mod.ts";

  const db = new Database(":memory:");

  db.enableLoadExtension = true;
  db.loadExtension(sqlite_jsonschema.getLoadablePath());

  const [version] = db
    .prepare("select jsonschema_version()")
    .value<[string]>()!;

  console.log(version);"><pre><span class="pl-k">import</span> <span class="pl-kos">{</span> <span class="pl-v">Database</span> <span class="pl-kos">}</span> <span class="pl-k">from</span> <span class="pl-s">"https://deno.land/x/sqlite3@0.8.0/mod.ts"</span><span class="pl-kos">;</span>
  <span class="pl-k">import</span> <span class="pl-c1">*</span> <span class="pl-k">as</span> <span class="pl-s1">sqlite_jsonschema</span> <span class="pl-k">from</span> <span class="pl-s">"https://deno.land/x/sqlite_jsonschema/mod.ts"</span><span class="pl-kos">;</span>

  <span class="pl-k">const</span> <span class="pl-s1">db</span> <span class="pl-c1">=</span> <span class="pl-k">new</span> <span class="pl-v">Database</span><span class="pl-kos">(</span><span class="pl-s">":memory:"</span><span class="pl-kos">)</span><span class="pl-kos">;</span>

  <span class="pl-s1">db</span><span class="pl-kos">.</span><span class="pl-c1">enableLoadExtension</span> <span class="pl-c1">=</span> <span class="pl-c1">true</span><span class="pl-kos">;</span>
  <span class="pl-s1">db</span><span class="pl-kos">.</span><span class="pl-en">loadExtension</span><span class="pl-kos">(</span><span class="pl-s1">sqlite_jsonschema</span><span class="pl-kos">.</span><span class="pl-en">getLoadablePath</span><span class="pl-kos">(</span><span class="pl-kos">)</span><span class="pl-kos">)</span><span class="pl-kos">;</span>

  <span class="pl-k">const</span> <span class="pl-kos">[</span><span class="pl-s1">version</span><span class="pl-kos">]</span> <span class="pl-c1">=</span> <span class="pl-s1">db</span>
    <span class="pl-kos">.</span><span class="pl-en">prepare</span><span class="pl-kos">(</span><span class="pl-s">"select jsonschema_version()"</span><span class="pl-kos">)</span>
    <span class="pl-kos">.</span><span class="pl-c1">value</span><span class="pl-c1"><</span><span class="pl-kos">[</span><span class="pl-s1">string</span><span class="pl-kos">]</span><span class="pl-c1">></span><span class="pl-kos">(</span><span class="pl-kos">)</span><span class="pl-c1">!</span><span class="pl-kos">;</span>

  <span class="pl-smi">console</span><span class="pl-kos">.</span><span class="pl-en">log</span><span class="pl-kos">(</span><span class="pl-s1">version</span><span class="pl-kos">)</span><span class="pl-kos">;</span></pre></div>
  <p dir="auto">See <a href="https://alexgarcia.jsonschema/sqlite-jsonschema/usage/deno.html" rel="nofollow"><em>Using <code>sqlite-jsonschema</code> with Deno</em></a> for details.</p>
  <div class="markdown-heading" dir="auto"><h3 class="heading-element" dir="auto">Datasette</h3><a id="user-content-datasette" class="anchor" aria-label="Permalink: Datasette" href="#datasette"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">For <a href="https://datasette.io/" rel="nofollow">Datasette</a>, use the <a href="https://datasette.io/plugins/datasette-sqlite-jsonschema" rel="nofollow"><code>datasette-sqlite-jsonschema</code> plugin</a> to include <code>sqlite-jsonschema</code> functions to your Datasette instances.</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="datasette install datasette-sqlite-jsonschema"><pre class="notranslate"><code>datasette install datasette-sqlite-jsonschema
  </code></pre></div>
  <p dir="auto">See <a href="https://alexgarcia.jsonschema/sqlite-jsonschema/usage/datasette.html" rel="nofollow"><em>Using <code>sqlite-jsonschema</code> with Datasette</em></a> for details.</p>
  <div class="markdown-heading" dir="auto"><h3 class="heading-element" dir="auto"><code>sqlite3</code> CLI</h3><a id="user-content-sqlite3-cli" class="anchor" aria-label="Permalink: sqlite3 CLI" href="#sqlite3-cli"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">For <a href="https://sqlite.org/cli.html" rel="nofollow">the <code>sqlite3</code> CLI</a>, either <a href="https://github.com/asg017/sqlite-jsonschema/releases">download a pre-compiled extension from the Releases page</a> or <a href="#building-from-source">build it yourself</a>. Then use the <a href="https://sqlite.org/cli.html#loading_extensions" rel="nofollow"><code>.load</code> dot command</a>.</p>
  <div class="highlight highlight-source-sql notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content=".load ./jsonschema0
  select jsonschema_version();
  'v0.2.1'"><pre>.load .<span class="pl-k">/</span>jsonschema0
  <span class="pl-k">select</span> jsonschema_version();
  <span class="pl-s"><span class="pl-pds">'</span>v0.2.1<span class="pl-pds">'</span></span></pre></div>
  <div class="markdown-heading" dir="auto"><h3 class="heading-element" dir="auto">As a loadable extension</h3><a id="user-content-as-a-loadable-extension" class="anchor" aria-label="Permalink: As a loadable extension" href="#as-a-loadable-extension"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">If you're using <code>sqlite-jsonschema</code> in a different way from those listed above, then <a href="https://github.com/asg017/sqlite-jsonschema/releases">download a pre-compiled extension from the Releases page</a> and load it into your environment. Download the <code>jsonschema0.dylib</code> (for MacOS), <code>jsonschema0.so</code> (Linux), or <code>jsonschema0.dll</code> (Windows) file from a release and load it into your SQLite environment.</p>
  <blockquote>
  <p dir="auto"><strong>Note:</strong>
  The <code>0</code> in the filename (<code>jsonschema0.dylib</code>/ <code>jsonschema0.so</code>/<code>jsonschema0.dll</code>) denotes the major version of <code>sqlite-jsonschema</code>. Currently <code>sqlite-jsonschema</code> is pre v1, so expect breaking changes in future versions.</p>
  </blockquote>
  <p dir="auto">Chances are there is some method called "loadExtension" or "load_extension" in the SQLite client library you are using. Alternatively, as a last resort, use <a href="https://www.sqlite.org/lang_corefunc.html#load_extension" rel="nofollow">the <code>load_extension()</code> SQL function</a>.</p>
  <div class="markdown-heading" dir="auto"><h3 class="heading-element" dir="auto">Building from source</h3><a id="user-content-building-from-source" class="anchor" aria-label="Permalink: Building from source" href="#building-from-source"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">Make sure you have <a href="https://www.rust-lang.org/tools/install" rel="nofollow">Rust</a>, make, and a C compiler installed. Then <code>git clone</code> this repository and run <code>make loadable-release</code>.</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="git clone https://github.com/asg017/sqlite-jsonschema.git
  cd sqlite-jsonschema
  make loadable-release"><pre class="notranslate"><code>git clone https://github.com/asg017/sqlite-jsonschema.git
  cd sqlite-jsonschema
  make loadable-release
  </code></pre></div>
  <p dir="auto">Once complete, your compiled extension will appear under <code>dist/release/</code>, either as <code>jsonschema0.so</code>, <code>jsonschema0.dylib</code>, or <code>jsonschema0.dll</code>, depending on your operating system.</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Documentation</h2><a id="user-content-documentation" class="anchor" aria-label="Permalink: Documentation" href="#documentation"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">See <a href="https://alexgarcia.jsonschema/sqlite-jsonschema/reference.html" rel="nofollow">the full API Reference</a> for every <code>sqlite-jsonschema</code> SQL function.</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Supporting</h2><a id="user-content-supporting" class="anchor" aria-label="Permalink: Supporting" href="#supporting"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">I (Alex 👋🏼) spent a lot of time and energy on this project and <a href="https://github.com/asg017?tab=repositories&q=&type=&language=&sort=stargazers">many other open source projects</a>. If your company or organization uses this library (or you're feeling generous), then please <a href="https://alexgarcia.jsonschema/work.html" rel="nofollow">consider supporting my work</a>, or share this project with a friend!</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">See also</h2><a id="user-content-see-also" class="anchor" aria-label="Permalink: See also" href="#see-also"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <ul dir="auto">
  <li><a href="https://github.com/asg017/sqlite-xsv"><code>sqlite-xsv</code></a>, A SQLite extension for working with CSVs</li>
  <li><a href="https://github.com/asg017/sqlite-http"><code>sqlite-http</code></a>, A SQLite extension for making HTTP requests</li>
  <li><a href="https://github.com/asg017/sqlite-loadable-rs"><code>sqlite-loadable-rs</code></a>, A framework for writing SQLite extensions in Rust</li>
  </ul>
  </article></div>


node_id: R_kgDOIgngcA
name: datasette-export
full_name: simonw/datasette-export
html_url: https://github.com/simonw/datasette-export
description: Export pages from Datasette to files on disk
created_at: 2022-11-27T04:03:08Z
updated_at: 2022-12-09T23:11:24Z
pushed_at: 2022-11-27T04:20:09Z
language: Python
license: apache-2.0
topics: []
visibility: public
default_branch: main
permissions: {"admin": false, "maintain": false, "push": false, "triage": false, "pull": true}
readme:
  # datasette-export

  [![PyPI](https://img.shields.io/pypi/v/datasette-export.svg)](https://pypi.org/project/datasette-export/)
  [![Changelog](https://img.shields.io/github/v/release/simonw/datasette-export?include_prereleases&label=changelog)](https://github.com/simonw/datasette-export/releases)
  [![Tests](https://github.com/simonw/datasette-export/workflows/Test/badge.svg)](https://github.com/simonw/datasette-export/actions?query=workflow%3ATest)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/datasette-export/blob/main/LICENSE)

  Export pages from Datasette to files on disk

  ## Installation

  Install this plugin in the same environment as Datasette.

      datasette install datasette-export

  ## Usage

  This plugin adds a new `export` command. You can use this to export one or more pages from Datasette to files on disk.

  Pass the `--path` option one or more times to specify pages:

      datasette export mydata.db --path / --path /mydata.json --path /mydata/table1.csv

  This will create an `export/` directory and save the following files to it:

  - `index.html`
  - `mydata.json`
  - `mydata/table1.csv`

  Use `--output` to specify an alternative directory for the export:

      datasette export mydata.db --path / --output mywebsite/

  In addition to specifying paths, you can pass one or more SQL queries that can return lists of pages to be exported:

      datasette export mydata.db --sql "select path from pages"

  For example, to export JSON for every row in the `fixtures/facetable` table:

      datasette export fixtures.db \
        --sql "select '/fixtures/facetable/' || pk || '.json' from facetable"

  ## Development

  To set up this plugin locally, first checkout the code. Then create a new virtual environment:

      cd datasette-export
      python3 -m venv venv
      source venv/bin/activate

  Now install the dependencies and test dependencies:

      pip install -e '.[test]'

  To run the tests:

      pytest

readme_html:
  <div id="readme" class="md" data-path="README.md"><article class="markdown-body entry-content container-lg" itemprop="text"><div class="markdown-heading" dir="auto"><h1 class="heading-element" dir="auto">datasette-export</h1><a id="user-content-datasette-export" class="anchor" aria-label="Permalink: datasette-export" href="#datasette-export"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto"><a href="https://pypi.org/project/datasette-export/" rel="nofollow"><img src="https://img.shields.io/pypi/v/datasette-export.svg" alt="PyPI" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/4134efa5091d49e026f2dae3be99e14432f9df6087e0c45a951a346099902e9b/68747470733a2f2f696d672e736869656c64732e696f2f707970692f762f6461746173657474652d6578706f72742e737667"></a>
  <a href="https://github.com/simonw/datasette-export/releases"><img src="https://img.shields.io/github/v/release/simonw/datasette-export?include_prereleases&label=changelog" alt="Changelog" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/f18a72f8c2dce3f8b4c9d177a9e093d97ce4a1f495274efbfcfe7612ad3b339a/68747470733a2f2f696d672e736869656c64732e696f2f6769746875622f762f72656c656173652f73696d6f6e772f6461746173657474652d6578706f72743f696e636c7564655f70726572656c6561736573266c6162656c3d6368616e67656c6f67"></a>
  <a href="https://github.com/simonw/datasette-export/actions?query=workflow%3ATest"><img src="https://github.com/simonw/datasette-export/workflows/Test/badge.svg" alt="Tests" style="max-width: 100%;"></a>
  <a href="https://github.com/simonw/datasette-export/blob/main/LICENSE"><img src="https://img.shields.io/badge/license-Apache%202.0-blue.svg" alt="License" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/c355f200ea90fddaa407b6eaab303663a669248ea3ca7b1fcf77dbe04ff5f48c/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6c6963656e73652d417061636865253230322e302d626c75652e737667"></a></p>
  <p dir="auto">Export pages from Datasette to files on disk</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Installation</h2><a id="user-content-installation" class="anchor" aria-label="Permalink: Installation" href="#installation"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">Install this plugin in the same environment as Datasette.</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="datasette install datasette-export"><pre class="notranslate"><code>datasette install datasette-export
  </code></pre></div>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Usage</h2><a id="user-content-usage" class="anchor" aria-label="Permalink: Usage" href="#usage"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">This plugin adds a new <code>export</code> command. You can use this to export one or more pages from Datasette to files on disk.</p>
  <p dir="auto">Pass the <code>--path</code> option one or more times to specify pages:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="datasette export mydata.db --path / --path /mydata.json --path /mydata/table1.csv"><pre class="notranslate"><code>datasette export mydata.db --path / --path /mydata.json --path /mydata/table1.csv
  </code></pre></div>
  <p dir="auto">This will create an <code>export/</code> directory and save the following files to it:</p>
  <ul dir="auto">
  <li><code>index.html</code></li>
  <li><code>mydata.json</code></li>
  <li><code>mydata/table1.csv</code></li>
  </ul>
  <p dir="auto">Use <code>--output</code> to specify an alternative directory for the export:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="datasette export mydata.db --path / --output mywebsite/"><pre class="notranslate"><code>datasette export mydata.db --path / --output mywebsite/
  </code></pre></div>
  <p dir="auto">In addition to specifying paths, you can pass one or more SQL queries that can return lists of pages to be exported:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="datasette export mydata.db --sql "select path from pages""><pre class="notranslate"><code>datasette export mydata.db --sql "select path from pages"
  </code></pre></div>
  <p dir="auto">For example, to export JSON for every row in the <code>fixtures/facetable</code> table:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="datasette export fixtures.db \
    --sql "select '/fixtures/facetable/' || pk || '.json' from facetable""><pre class="notranslate"><code>datasette export fixtures.db \
    --sql "select '/fixtures/facetable/' || pk || '.json' from facetable"
  </code></pre></div>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Development</h2><a id="user-content-development" class="anchor" aria-label="Permalink: Development" href="#development"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">To set up this plugin locally, first checkout the code. Then create a new virtual environment:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="cd datasette-export
  python3 -m venv venv
  source venv/bin/activate"><pre class="notranslate"><code>cd datasette-export
  python3 -m venv venv
  source venv/bin/activate
  </code></pre></div>
  <p dir="auto">Now install the dependencies and test dependencies:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="pip install -e '.[test]'"><pre class="notranslate"><code>pip install -e '.[test]'
  </code></pre></div>
  <p dir="auto">To run the tests:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="pytest"><pre class="notranslate"><code>pytest
  </code></pre></div>
  </article></div>


node_id: R_kgDOIhVCuw
name: datasette-scraper
full_name: cldellow/datasette-scraper
html_url: https://github.com/cldellow/datasette-scraper
description: Add website scraping abilities to Datasette
created_at: 2022-11-29T00:19:56Z
updated_at: 2025-01-16T21:21:59Z
pushed_at: 2023-03-04T22:08:22Z
language: Python
license: apache-2.0
topics: ["datasette", "datasette-plugin", "scraping"]
visibility: public
default_branch: master
permissions: {"admin": false, "maintain": false, "push": false, "triage": false, "pull": true}
readme:
  # datasette-scraper

  [![PyPI](https://img.shields.io/pypi/v/datasette-scraper.svg)](https://pypi.org/project/datasette-scraper/)
  [![Changelog](https://img.shields.io/github/v/release/cldellow/datasette-scraper?include_prereleases&label=changelog)](https://github.com/cldellow/datasette-scraper/releases)
  [![Tests](https://github.com/cldellow/datasette-scraper/workflows/Test/badge.svg)](https://github.com/cldellow/datasette-scraper/actions?query=workflow%3ATest)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/cldellow/datasette-scraper/blob/main/LICENSE)

  `datasette-scraper` is a Datasette plugin to manage small-ish (~100K pages) crawl and extract jobs.

  - Opinionated yet extensible
    - Some useful tasks are possible out-of-the-box, or write your own pluggy hooks to go further
  - Leans heavily into SQLite
    - Introspect your crawls via ops tables exposed in Datasette
  - Built on robust libraries
    - [Datasette](https://datasette.io/) as a host
    - [selectolax](https://github.com/rushter/selectolax) for HTML parsing
    - [httpx](https://www.python-httpx.org/) for HTTP requests
    - [pluggy](https://pluggy.readthedocs.io/en/stable/) for extensibility
    - [zstandard](https://github.com/indygreg/python-zstandard) for efficiently compressing HTTP responses

  **Not for adversarial crawling**. Want to crawl a site that blocks bots? You're on your own.

  ## Installation

  Install this plugin in the same environment as Datasette.

      datasette install datasette-scraper

  ## Usage

  Configure `datasette-scraper` via `metadata.json`. You need to enable the plugin
  on a per-database level.

  To enable it in the `my-database` database, write something like this:

  ```
  {
    "databases": {
      "my-database": {
        "plugins": {
          "datasette-scraper": {
          }
        }
      }
    }
  }
  ```

  The next time you start datasette, the plugin will create several tables in
  the specified database. Go to the `dss_crawl` table to define a crawl.

  A 10-minute end-to-end walkthrough video is available:

  <div align="left">
        <a href="https://www.youtube.com/watch?v=zrSGnz7ErNI">
           <img src="https://img.youtube.com/vi/zrSGnz7ErNI/maxresdefault.jpg" style="width:100%;">
        </a>
  </div>

  ## Usage notes

  `datasette-scraper` requires a database in which to track its operational data,
  and a database in which to store scraped data. They can be the same database.

  Both databases will be put into WAL mode.

  The ops database's `user_version` pragma will be used to track schema versions.

  ## Architecture

  `datasette-scraper` handles the core bookkeeping for scraping--keeping track of
  URLs to be scraped, rate-limiting requests to origins, persisting data into the DB.
  It relies on plugins to do almost all the interesting work. For example, fetching
  the actual pages, following redirects, navigating sitemaps, extracting data.

  The tool comes with plugins for common use cases. Some users may want to author
  their own `after_fetch_url` or `extract_from_response` implementations to do custom
  processing.

  ### Overview

  ```mermaid
  flowchart LR
  direction TB

  subgraph init
    A(user starts crawl) --> B[get_seed_urls]
  end

  subgraph crawl [for each URL to crawl]
    before_fetch_url --> fetch_cached_url --> fetch_url --> after_fetch_url
    fetch_cached_url --> after_fetch_url
  end

  subgraph discover [for each URL crawled]
    discover_urls --> canonicalize_url --> canonicalize_url
    canonicalize_url --> x[queue URL to crawl]
    extract_from_response
  end

  init --> crawl --> discover
  ```

  ### Plugin hooks

  Most plugins will only implement a few of these hooks.

  - `conn` is a read/write `sqlite3.Connection` to the database
  - `config` is the crawl's config

  #### `get_seed_urls(config)`

  Returns a list of strings representing seed URLs to be fetched.

  They will be considered to have depth of 0, i.e. seeds.

  #### `before_fetch_url(conn, config, job_id, url, depth, request_headers)`

  `request_headers` is a dict, you can modify it to control what gets sent in the request.

  Returns:
    - truthy to indicate this URL should not be crawled (for example, crawl max page limit)
    - falsy to express no opinion

  > **Note** `before_fetch_url` vs `canonicalize_url`
  >
  > You can also use the `canonicalize_url` hook to reject URLs prior to them entering
  > the crawl queue.
  >
  > A URL rejected by `canonicalize_url` will not result in an entry in the
  > `dss_crawl_queue` and `dss_crawl_queue_history` tables.
  >
  > Which one you use is a matter of taste, in general, if you _never_ want the URL,
  > reject it at canonicalization time.

  #### `fetch_cached_url(conn, config, url, depth, request_headers)`

  Fetch a previously-cached HTTP response. The system will not have checked that
  there was rate limit available before calling this.

  Returns:
    - `None`, to indicate not handled
    - a response object, which is a dict with:
      - `fetched_at` - an ISO 8601 time like `2022-12-26 01:23:45.00`
      - `headers` - the response headers, eg `[['content-type', 'text/html']]`
      - `status_code` - the respones code, eg `200`
      - `text` - the response body

  Once any plugin has returned a truthy value, no other plugin's `fetch_url`
  hook will be invoked.


  #### `fetch_url(conn, config, url, request_headers)`

  Fetch an HTTP response from the live server. The system will have checked that there
  was rate limit available before calling this.

  Same return type and behaviour as `fetch_cached_url`.

  #### `after_fetch_url(conn, config, url, request_headers, response, fresh, fetch_duration)`

  Do something with a fetched URL.

  #### `discover_urls(config, url, response)`

  Returns a list of URLs to crawl.

  The URLs can be either strings, in which case they'll get enqueued as depth + 1, or tuple of URL and depth. This can be useful for paginated index pages, where you'd like to crawl to a max depth of, say, 2, but treat all the index pages as being at depth 1.

  #### `canonicalize_url(config, from_url, to_url, to_url_depth)`

  Returns:
    - `False` to filter URL
    - an URL to be crawled instead
    - `None` or `True` to no-op

  The URL to be crawled can be a string, or a tuple of string and depth.

  This hook is useful for:
    - blocking URLs that we never want
    - canonicalizing URLs, for example, by omitting query parameters
    - restricting crawls to same origin
    - resetting depth for pagination

  #### `extract_from_response(config, url, response)`

  Returns an object of rows-to-be-inserted-or-upserted:

  ```jsonc
  {
    "dbname": {  // can be omitted, in which case, current DB will be used
      "users": [
        {
          "id!": "cldellow@gmail.com",  // ! indicates pkey, compound OK
          "name": "Colin",
        },
        {
          "id!": "santa@northpole.com",
          "name": "Santa Claus",
        }
      ],
      "places": [
        {
          "id@": "santa@northpole.com",
          "__delete": true
        },
        {
          "id@": "cldellow@gmail.com",
          "city": "Kitchener",
        },
        {
          "id@": "cldellow@gmail.com",
          "city": "Dawson Creek"
        }
      ]
    }
  }
  ```

  Column names can have sigils at the end:
  - `!` says the column is part of the pkey; there can be at most 1 row with this value
  - `@` says the column should be indexed; there can be multiple rows with this value

  Columns with sigils must be known at table creation time. Although you can have
  multiple columns with sigils, you cannot mix `!` and `@` sigils in the same table.

  Any missing tables or columns will be created. Columns will have `ANY` data type.
  Columns will be nullable unless they have the `!` sigil.

  You can indicate that a row should be deleted by emitting `__delete` key in your object.

  `datasette-scraper` may commit your changes to the database in batches in order to
  reduce write transactions and improve throughput. It may also elide
  DELETE/INSERT statements entirely if it determines that the state of the database
  would be unchanged.

  If you'd like to control the schema more carefully, please create the table manually.

  #### Metadata hooks

  These hooks don't affect operation of the scrapes. They provide metadata to
  help validate a user's configuration and show UI to configure a crawl.

  ##### config_schema()

  Returns a `ConfigSchema` option that defines how this plugin is configured.

  Configuration is done via [JSON schema](https://json-schema.org/understanding-json-schema/). UI is done via [JSON Forms](https://jsonforms.io/).

  Look at the existing plugins to learn how to use this hook.

  The schema is optional; if omitted, you will need to configure the plug in
  out of band.

  ##### config_default_value()

  Returns `None` to indicate that new crawls should not use this plugin by default.

  Otherwise, returns a reasonable default value that conforms to the schema in `config_schema()`

  ## Development

  To set up this plugin locally, first checkout the code. Then create a new virtual environment:

      cd datasette-scraper
      python3 -m venv venv
      source venv/bin/activate

  Now install the dependencies and test dependencies:

      pip install -e '.[test]'

  To run the tests:

      pytest

readme_html:
  <div id="readme" class="md" data-path="README.md"><article class="markdown-body entry-content container-lg" itemprop="text"><div class="markdown-heading" dir="auto"><h1 class="heading-element" dir="auto">datasette-scraper</h1><a id="user-content-datasette-scraper" class="anchor" aria-label="Permalink: datasette-scraper" href="#datasette-scraper"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto"><a href="https://pypi.org/project/datasette-scraper/" rel="nofollow"><img src="https://img.shields.io/pypi/v/datasette-scraper.svg" alt="PyPI" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/0cf6ff091ff05efd7f44e66042b4dad7c6e913ef6729d8754d5e86c9acf36e87/68747470733a2f2f696d672e736869656c64732e696f2f707970692f762f6461746173657474652d736372617065722e737667"></a>
  <a href="https://github.com/cldellow/datasette-scraper/releases"><img src="https://img.shields.io/github/v/release/cldellow/datasette-scraper?include_prereleases&label=changelog" alt="Changelog" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/712f287f5e6a3309dfea03e72cfd5e9c846336958453f7dc074e2d50e358ad66/68747470733a2f2f696d672e736869656c64732e696f2f6769746875622f762f72656c656173652f636c64656c6c6f772f6461746173657474652d736372617065723f696e636c7564655f70726572656c6561736573266c6162656c3d6368616e67656c6f67"></a>
  <a href="https://github.com/cldellow/datasette-scraper/actions?query=workflow%3ATest"><img src="https://github.com/cldellow/datasette-scraper/workflows/Test/badge.svg" alt="Tests" style="max-width: 100%;"></a>
  <a href="https://github.com/cldellow/datasette-scraper/blob/main/LICENSE"><img src="https://img.shields.io/badge/license-Apache%202.0-blue.svg" alt="License" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/c355f200ea90fddaa407b6eaab303663a669248ea3ca7b1fcf77dbe04ff5f48c/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6c6963656e73652d417061636865253230322e302d626c75652e737667"></a></p>
  <p dir="auto"><code>datasette-scraper</code> is a Datasette plugin to manage small-ish (~100K pages) crawl and extract jobs.</p>
  <ul dir="auto">
  <li>Opinionated yet extensible
  <ul dir="auto">
  <li>Some useful tasks are possible out-of-the-box, or write your own pluggy hooks to go further</li>
  </ul>
  </li>
  <li>Leans heavily into SQLite
  <ul dir="auto">
  <li>Introspect your crawls via ops tables exposed in Datasette</li>
  </ul>
  </li>
  <li>Built on robust libraries
  <ul dir="auto">
  <li><a href="https://datasette.io/" rel="nofollow">Datasette</a> as a host</li>
  <li><a href="https://github.com/rushter/selectolax">selectolax</a> for HTML parsing</li>
  <li><a href="https://www.python-httpx.org/" rel="nofollow">httpx</a> for HTTP requests</li>
  <li><a href="https://pluggy.readthedocs.io/en/stable/" rel="nofollow">pluggy</a> for extensibility</li>
  <li><a href="https://github.com/indygreg/python-zstandard">zstandard</a> for efficiently compressing HTTP responses</li>
  </ul>
  </li>
  </ul>
  <p dir="auto"><strong>Not for adversarial crawling</strong>. Want to crawl a site that blocks bots? You're on your own.</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Installation</h2><a id="user-content-installation" class="anchor" aria-label="Permalink: Installation" href="#installation"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">Install this plugin in the same environment as Datasette.</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="datasette install datasette-scraper"><pre class="notranslate"><code>datasette install datasette-scraper
  </code></pre></div>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Usage</h2><a id="user-content-usage" class="anchor" aria-label="Permalink: Usage" href="#usage"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">Configure <code>datasette-scraper</code> via <code>metadata.json</code>. You need to enable the plugin
  on a per-database level.</p>
  <p dir="auto">To enable it in the <code>my-database</code> database, write something like this:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="{
    "databases": {
      "my-database": {
        "plugins": {
          "datasette-scraper": {
          }
        }
      }
    }
  }"><pre class="notranslate"><code>{
    "databases": {
      "my-database": {
        "plugins": {
          "datasette-scraper": {
          }
        }
      }
    }
  }
  </code></pre></div>
  <p dir="auto">The next time you start datasette, the plugin will create several tables in
  the specified database. Go to the <code>dss_crawl</code> table to define a crawl.</p>
  <p dir="auto">A 10-minute end-to-end walkthrough video is available:</p>
  <div align="left" dir="auto">
        <a href="https://www.youtube.com/watch?v=zrSGnz7ErNI" rel="nofollow">
           <img src="https://img.youtube.com/vi/zrSGnz7ErNI/maxresdefault.jpg" style="width: 100%; max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/e218c61e66d0d8e905573dcaa0f672c277633a7333414f34745ad065e3443b14/68747470733a2f2f696d672e796f75747562652e636f6d2f76692f7a7253476e7a3745724e492f6d617872657364656661756c742e6a7067">
        </a>
  </div>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Usage notes</h2><a id="user-content-usage-notes" class="anchor" aria-label="Permalink: Usage notes" href="#usage-notes"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto"><code>datasette-scraper</code> requires a database in which to track its operational data,
  and a database in which to store scraped data. They can be the same database.</p>
  <p dir="auto">Both databases will be put into WAL mode.</p>
  <p dir="auto">The ops database's <code>user_version</code> pragma will be used to track schema versions.</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Architecture</h2><a id="user-content-architecture" class="anchor" aria-label="Permalink: Architecture" href="#architecture"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto"><code>datasette-scraper</code> handles the core bookkeeping for scraping--keeping track of
  URLs to be scraped, rate-limiting requests to origins, persisting data into the DB.
  It relies on plugins to do almost all the interesting work. For example, fetching
  the actual pages, following redirects, navigating sitemaps, extracting data.</p>
  <p dir="auto">The tool comes with plugins for common use cases. Some users may want to author
  their own <code>after_fetch_url</code> or <code>extract_from_response</code> implementations to do custom
  processing.</p>
  <div class="markdown-heading" dir="auto"><h3 class="heading-element" dir="auto">Overview</h3><a id="user-content-overview" class="anchor" aria-label="Permalink: Overview" href="#overview"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <section class="js-render-needs-enrichment render-needs-enrichment position-relative" data-identity="802aebb7-2a4d-45b5-94be-46016edb84d6" data-host="https://viewscreen.githubusercontent.com" data-src="https://viewscreen.githubusercontent.com/markdown/mermaid?docs_host=https%3A%2F%2Fdocs.github.com" data-type="mermaid" aria-label="mermaid rendered output container">
    <div class="js-render-enrichment-target" data-json="{"data":"flowchart LR\ndirection TB\n\nsubgraph init\n  A(user starts crawl) --&gt; B[get_seed_urls]\nend\n\nsubgraph crawl [for each URL to crawl]\n  before_fetch_url --&gt; fetch_cached_url --&gt; fetch_url --&gt; after_fetch_url\n  fetch_cached_url --&gt; after_fetch_url\nend\n\nsubgraph discover [for each URL crawled]\n  discover_urls --&gt; canonicalize_url --&gt; canonicalize_url\n  canonicalize_url --&gt; x[queue URL to crawl]\n  extract_from_response\nend\n\ninit --&gt; crawl --&gt; discover\n"}" data-plain="flowchart LR
  direction TB

  subgraph init
    A(user starts crawl) --> B[get_seed_urls]
  end

  subgraph crawl [for each URL to crawl]
    before_fetch_url --> fetch_cached_url --> fetch_url --> after_fetch_url
    fetch_cached_url --> after_fetch_url
  end

  subgraph discover [for each URL crawled]
    discover_urls --> canonicalize_url --> canonicalize_url
    canonicalize_url --> x[queue URL to crawl]
    extract_from_response
  end

  init --> crawl --> discover
  " dir="auto">
      <div class="render-plaintext-hidden" dir="auto">
        <pre lang="mermaid" aria-label="Raw mermaid code">flowchart LR
  direction TB

  subgraph init
    A(user starts crawl) --> B[get_seed_urls]
  end

  subgraph crawl [for each URL to crawl]
    before_fetch_url --> fetch_cached_url --> fetch_url --> after_fetch_url
    fetch_cached_url --> after_fetch_url
  end

  subgraph discover [for each URL crawled]
    discover_urls --> canonicalize_url --> canonicalize_url
    canonicalize_url --> x[queue URL to crawl]
    extract_from_response
  end

  init --> crawl --> discover
  </pre>
      </div>
    </div>
    <span class="js-render-enrichment-loader d-flex flex-justify-center flex-items-center width-full" style="min-height:100px" role="presentation">
      <span data-view-component="true">
    <svg style="box-sizing: content-box; color: var(--color-icon-primary);" width="16" height="16" viewbox="0 0 16 16" fill="none" aria-hidden="true" data-view-component="true" class="octospinner mx-auto anim-rotate">
      <circle cx="8" cy="8" r="7" stroke="currentColor" stroke-opacity="0.25" stroke-width="2" vector-effect="non-scaling-stroke" fill="none"></circle>
      <path d="M15 8a7.002 7.002 0 00-7-7" stroke="currentColor" stroke-width="2" stroke-linecap="round" vector-effect="non-scaling-stroke"></path>
  </svg>    <span class="sr-only">Loading</span>
  </span>
    </span>
  </section>

  <div class="markdown-heading" dir="auto"><h3 class="heading-element" dir="auto">Plugin hooks</h3><a id="user-content-plugin-hooks" class="anchor" aria-label="Permalink: Plugin hooks" href="#plugin-hooks"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">Most plugins will only implement a few of these hooks.</p>
  <ul dir="auto">
  <li><code>conn</code> is a read/write <code>sqlite3.Connection</code> to the database</li>
  <li><code>config</code> is the crawl's config</li>
  </ul>
  <div class="markdown-heading" dir="auto"><h4 class="heading-element" dir="auto"><code>get_seed_urls(config)</code></h4><a id="user-content-get_seed_urlsconfig" class="anchor" aria-label="Permalink: get_seed_urls(config)" href="#get_seed_urlsconfig"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">Returns a list of strings representing seed URLs to be fetched.</p>
  <p dir="auto">They will be considered to have depth of 0, i.e. seeds.</p>
  <div class="markdown-heading" dir="auto"><h4 class="heading-element" dir="auto"><code>before_fetch_url(conn, config, job_id, url, depth, request_headers)</code></h4><a id="user-content-before_fetch_urlconn-config-job_id-url-depth-request_headers" class="anchor" aria-label="Permalink: before_fetch_url(conn, config, job_id, url, depth, request_headers)" href="#before_fetch_urlconn-config-job_id-url-depth-request_headers"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto"><code>request_headers</code> is a dict, you can modify it to control what gets sent in the request.</p>
  <p dir="auto">Returns:</p>
  <ul dir="auto">
  <li>truthy to indicate this URL should not be crawled (for example, crawl max page limit)</li>
  <li>falsy to express no opinion</li>
  </ul>
  <blockquote>
  <p dir="auto"><strong>Note</strong> <code>before_fetch_url</code> vs <code>canonicalize_url</code></p>
  <p dir="auto">You can also use the <code>canonicalize_url</code> hook to reject URLs prior to them entering
  the crawl queue.</p>
  <p dir="auto">A URL rejected by <code>canonicalize_url</code> will not result in an entry in the
  <code>dss_crawl_queue</code> and <code>dss_crawl_queue_history</code> tables.</p>
  <p dir="auto">Which one you use is a matter of taste, in general, if you <em>never</em> want the URL,
  reject it at canonicalization time.</p>
  </blockquote>
  <div class="markdown-heading" dir="auto"><h4 class="heading-element" dir="auto"><code>fetch_cached_url(conn, config, url, depth, request_headers)</code></h4><a id="user-content-fetch_cached_urlconn-config-url-depth-request_headers" class="anchor" aria-label="Permalink: fetch_cached_url(conn, config, url, depth, request_headers)" href="#fetch_cached_urlconn-config-url-depth-request_headers"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">Fetch a previously-cached HTTP response. The system will not have checked that
  there was rate limit available before calling this.</p>
  <p dir="auto">Returns:</p>
  <ul dir="auto">
  <li><code>None</code>, to indicate not handled</li>
  <li>a response object, which is a dict with:
  <ul dir="auto">
  <li><code>fetched_at</code> - an ISO 8601 time like <code>2022-12-26 01:23:45.00</code></li>
  <li><code>headers</code> - the response headers, eg <code>[['content-type', 'text/html']]</code></li>
  <li><code>status_code</code> - the respones code, eg <code>200</code></li>
  <li><code>text</code> - the response body</li>
  </ul>
  </li>
  </ul>
  <p dir="auto">Once any plugin has returned a truthy value, no other plugin's <code>fetch_url</code>
  hook will be invoked.</p>
  <div class="markdown-heading" dir="auto"><h4 class="heading-element" dir="auto"><code>fetch_url(conn, config, url, request_headers)</code></h4><a id="user-content-fetch_urlconn-config-url-request_headers" class="anchor" aria-label="Permalink: fetch_url(conn, config, url, request_headers)" href="#fetch_urlconn-config-url-request_headers"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">Fetch an HTTP response from the live server. The system will have checked that there
  was rate limit available before calling this.</p>
  <p dir="auto">Same return type and behaviour as <code>fetch_cached_url</code>.</p>
  <div class="markdown-heading" dir="auto"><h4 class="heading-element" dir="auto"><code>after_fetch_url(conn, config, url, request_headers, response, fresh, fetch_duration)</code></h4><a id="user-content-after_fetch_urlconn-config-url-request_headers-response-fresh-fetch_duration" class="anchor" aria-label="Permalink: after_fetch_url(conn, config, url, request_headers, response, fresh, fetch_duration)" href="#after_fetch_urlconn-config-url-request_headers-response-fresh-fetch_duration"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">Do something with a fetched URL.</p>
  <div class="markdown-heading" dir="auto"><h4 class="heading-element" dir="auto"><code>discover_urls(config, url, response)</code></h4><a id="user-content-discover_urlsconfig-url-response" class="anchor" aria-label="Permalink: discover_urls(config, url, response)" href="#discover_urlsconfig-url-response"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">Returns a list of URLs to crawl.</p>
  <p dir="auto">The URLs can be either strings, in which case they'll get enqueued as depth + 1, or tuple of URL and depth. This can be useful for paginated index pages, where you'd like to crawl to a max depth of, say, 2, but treat all the index pages as being at depth 1.</p>
  <div class="markdown-heading" dir="auto"><h4 class="heading-element" dir="auto"><code>canonicalize_url(config, from_url, to_url, to_url_depth)</code></h4><a id="user-content-canonicalize_urlconfig-from_url-to_url-to_url_depth" class="anchor" aria-label="Permalink: canonicalize_url(config, from_url, to_url, to_url_depth)" href="#canonicalize_urlconfig-from_url-to_url-to_url_depth"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">Returns:</p>
  <ul dir="auto">
  <li><code>False</code> to filter URL</li>
  <li>an URL to be crawled instead</li>
  <li><code>None</code> or <code>True</code> to no-op</li>
  </ul>
  <p dir="auto">The URL to be crawled can be a string, or a tuple of string and depth.</p>
  <p dir="auto">This hook is useful for:</p>
  <ul dir="auto">
  <li>blocking URLs that we never want</li>
  <li>canonicalizing URLs, for example, by omitting query parameters</li>
  <li>restricting crawls to same origin</li>
  <li>resetting depth for pagination</li>
  </ul>
  <div class="markdown-heading" dir="auto"><h4 class="heading-element" dir="auto"><code>extract_from_response(config, url, response)</code></h4><a id="user-content-extract_from_responseconfig-url-response" class="anchor" aria-label="Permalink: extract_from_response(config, url, response)" href="#extract_from_responseconfig-url-response"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">Returns an object of rows-to-be-inserted-or-upserted:</p>
  <div class="highlight highlight-source-json-comments notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="{
    "dbname": {  // can be omitted, in which case, current DB will be used
      "users": [
        {
          "id!": "cldellow@gmail.com",  // ! indicates pkey, compound OK
          "name": "Colin",
        },
        {
          "id!": "santa@northpole.com",
          "name": "Santa Claus",
        }
      ],
      "places": [
        {
          "id@": "santa@northpole.com",
          "__delete": true
        },
        {
          "id@": "cldellow@gmail.com",
          "city": "Kitchener",
        },
        {
          "id@": "cldellow@gmail.com",
          "city": "Dawson Creek"
        }
      ]
    }
  }"><pre>{
    <span class="pl-s">"dbname"</span>: {  <span class="pl-c"><span class="pl-c">//</span> can be omitted, in which case, current DB will be used</span>
      <span class="pl-s">"users"</span>: [
        {
          <span class="pl-s">"id!"</span>: <span class="pl-s"><span class="pl-pds">"</span>cldellow@gmail.com<span class="pl-pds">"</span></span>,  <span class="pl-c"><span class="pl-c">//</span> ! indicates pkey, compound OK</span>
          <span class="pl-s">"name"</span>: <span class="pl-s"><span class="pl-pds">"</span>Colin<span class="pl-pds">"</span></span>,
        },
        {
          <span class="pl-s">"id!"</span>: <span class="pl-s"><span class="pl-pds">"</span>santa@northpole.com<span class="pl-pds">"</span></span>,
          <span class="pl-s">"name"</span>: <span class="pl-s"><span class="pl-pds">"</span>Santa Claus<span class="pl-pds">"</span></span>,
        }
      ],
      <span class="pl-s">"places"</span>: [
        {
          <span class="pl-s">"id@"</span>: <span class="pl-s"><span class="pl-pds">"</span>santa@northpole.com<span class="pl-pds">"</span></span>,
          <span class="pl-s">"__delete"</span>: <span class="pl-c1">true</span>
        },
        {
          <span class="pl-s">"id@"</span>: <span class="pl-s"><span class="pl-pds">"</span>cldellow@gmail.com<span class="pl-pds">"</span></span>,
          <span class="pl-s">"city"</span>: <span class="pl-s"><span class="pl-pds">"</span>Kitchener<span class="pl-pds">"</span></span>,
        },
        {
          <span class="pl-s">"id@"</span>: <span class="pl-s"><span class="pl-pds">"</span>cldellow@gmail.com<span class="pl-pds">"</span></span>,
          <span class="pl-s">"city"</span>: <span class="pl-s"><span class="pl-pds">"</span>Dawson Creek<span class="pl-pds">"</span></span>
        }
      ]
    }
  }</pre></div>
  <p dir="auto">Column names can have sigils at the end:</p>
  <ul dir="auto">
  <li><code>!</code> says the column is part of the pkey; there can be at most 1 row with this value</li>
  <li><code>@</code> says the column should be indexed; there can be multiple rows with this value</li>
  </ul>
  <p dir="auto">Columns with sigils must be known at table creation time. Although you can have
  multiple columns with sigils, you cannot mix <code>!</code> and <code>@</code> sigils in the same table.</p>
  <p dir="auto">Any missing tables or columns will be created. Columns will have <code>ANY</code> data type.
  Columns will be nullable unless they have the <code>!</code> sigil.</p>
  <p dir="auto">You can indicate that a row should be deleted by emitting <code>__delete</code> key in your object.</p>
  <p dir="auto"><code>datasette-scraper</code> may commit your changes to the database in batches in order to
  reduce write transactions and improve throughput. It may also elide
  DELETE/INSERT statements entirely if it determines that the state of the database
  would be unchanged.</p>
  <p dir="auto">If you'd like to control the schema more carefully, please create the table manually.</p>
  <div class="markdown-heading" dir="auto"><h4 class="heading-element" dir="auto">Metadata hooks</h4><a id="user-content-metadata-hooks" class="anchor" aria-label="Permalink: Metadata hooks" href="#metadata-hooks"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">These hooks don't affect operation of the scrapes. They provide metadata to
  help validate a user's configuration and show UI to configure a crawl.</p>
  <div class="markdown-heading" dir="auto"><h5 class="heading-element" dir="auto">config_schema()</h5><a id="user-content-config_schema" class="anchor" aria-label="Permalink: config_schema()" href="#config_schema"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">Returns a <code>ConfigSchema</code> option that defines how this plugin is configured.</p>
  <p dir="auto">Configuration is done via <a href="https://json-schema.org/understanding-json-schema/" rel="nofollow">JSON schema</a>. UI is done via <a href="https://jsonforms.io/" rel="nofollow">JSON Forms</a>.</p>
  <p dir="auto">Look at the existing plugins to learn how to use this hook.</p>
  <p dir="auto">The schema is optional; if omitted, you will need to configure the plug in
  out of band.</p>
  <div class="markdown-heading" dir="auto"><h5 class="heading-element" dir="auto">config_default_value()</h5><a id="user-content-config_default_value" class="anchor" aria-label="Permalink: config_default_value()" href="#config_default_value"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">Returns <code>None</code> to indicate that new crawls should not use this plugin by default.</p>
  <p dir="auto">Otherwise, returns a reasonable default value that conforms to the schema in <code>config_schema()</code></p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Development</h2><a id="user-content-development" class="anchor" aria-label="Permalink: Development" href="#development"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">To set up this plugin locally, first checkout the code. Then create a new virtual environment:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="cd datasette-scraper
  python3 -m venv venv
  source venv/bin/activate"><pre class="notranslate"><code>cd datasette-scraper
  python3 -m venv venv
  source venv/bin/activate
  </code></pre></div>
  <p dir="auto">Now install the dependencies and test dependencies:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="pip install -e '.[test]'"><pre class="notranslate"><code>pip install -e '.[test]'
  </code></pre></div>
  <p dir="auto">To run the tests:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="pytest"><pre class="notranslate"><code>pytest
  </code></pre></div>
  </article></div>


node_id: R_kgDOIhvnWQ
name: datasette-ephemeral-tables
full_name: simonw/datasette-ephemeral-tables
html_url: https://github.com/simonw/datasette-ephemeral-tables
description: Provide tables that expire after a time limit
created_at: 2022-11-29T21:54:16Z
updated_at: 2025-11-09T23:54:11Z
pushed_at: 2025-11-09T23:54:08Z
language: Python
license: apache-2.0
topics: []
visibility: public
default_branch: main
permissions: {"admin": false, "maintain": false, "push": false, "triage": false, "pull": false}
pull_request_creation_policy: all
template_repository:
  {"id": 400878073, "node_id": "MDEwOlJlcG9zaXRvcnk0MDA4NzgwNzM=", "name": "datasette-plugin-template-repository", "full_name": "simonw/datasette-plugin-template-repository", "private": false, "owner": {"login": "simonw", "id": 9599, "node_id": "MDQ6VXNlcjk1OTk=", "avatar_url": "https://avatars.githubusercontent.com/u/9599?v=4", "gravatar_id": "", "url": "https://api.github.com/users/simonw", "html_url": "https://github.com/simonw", "followers_url": "https://api.github.com/users/simonw/followers", "following_url": "https://api.github.com/users/simonw/following{/other_user}", "gists_url": "https://api.github.com/users/simonw/gists{/gist_id}", "starred_url": "https://api.github.com/users/simonw/starred{/owner}{/repo}", "subscriptions_url": "https://api.github.com/users/simonw/subscriptions", "organizations_url": "https://api.github.com/users/simonw/orgs", "repos_url": "https://api.github.com/users/simonw/repos", "events_url": "https://api.github.com/users/simonw/events{/privacy}", "received_events_url": "https://api.github.com/users/simonw/received_events", "type": "User", "user_view_type": "public", "site_admin": false}, "html_url": "https://github.com/simonw/datasette-plugin-template-repository", "description": "GitHub template repository for creating new Datasette plugins, using the simonw/datasette-plugin cookiecutter template", "fork": false, "url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository", "forks_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/forks", "keys_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/keys{/key_id}", "collaborators_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/collaborators{/collaborator}", "teams_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/teams", "hooks_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/hooks", "issue_events_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/issues/events{/number}", "events_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/events", "assignees_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/assignees{/user}", "branches_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/branches{/branch}", "tags_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/tags", "blobs_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/git/blobs{/sha}", "git_tags_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/git/tags{/sha}", "git_refs_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/git/refs{/sha}", "trees_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/git/trees{/sha}", "statuses_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/statuses/{sha}", "languages_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/languages", "stargazers_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/stargazers", "contributors_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/contributors", "subscribers_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/subscribers", "subscription_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/subscription", "commits_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/commits{/sha}", "git_commits_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/git/commits{/sha}", "comments_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/comments{/number}", "issue_comment_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/issues/comments{/number}", "contents_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/contents/{+path}", "compare_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/compare/{base}...{head}", "merges_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/merges", "archive_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/{archive_format}{/ref}", "downloads_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/downloads", "issues_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/issues{/number}", "pulls_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/pulls{/number}", "milestones_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/milestones{/number}", "notifications_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/notifications{?since,all,participating}", "labels_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/labels{/name}", "releases_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/releases{/id}", "deployments_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/deployments", "created_at": "2021-08-28T19:50:28Z", "updated_at": "2026-03-03T05:49:18Z", "pushed_at": "2026-01-03T22:42:16Z", "git_url": "git://github.com/simonw/datasette-plugin-template-repository.git", "ssh_url": "git@github.com:simonw/datasette-plugin-template-repository.git", "clone_url": "https://github.com/simonw/datasette-plugin-template-repository.git", "svn_url": "https://github.com/simonw/datasette-plugin-template-repository", "homepage": "", "size": 27, "stargazers_count": 26, "watchers_count": 26, "language": null, "has_issues": true, "has_projects": true, "has_downloads": true, "has_wiki": true, "has_pages": false, "has_discussions": false, "forks_count": 0, "mirror_url": null, "archived": false, "disabled": false, "open_issues_count": 0, "license": null, "allow_forking": true, "is_template": true, "web_commit_signoff_required": false, "has_pull_requests": true, "pull_request_creation_policy": "all", "topics": [], "visibility": "public", "forks": 0, "open_issues": 0, "watchers": 26, "default_branch": "main", "permissions": {"admin": false, "maintain": false, "push": false, "triage": false, "pull": false}, "temp_clone_token": ""}

readme:
  # datasette-ephemeral-tables

  [![PyPI](https://img.shields.io/pypi/v/datasette-ephemeral-tables.svg)](https://pypi.org/project/datasette-ephemeral-tables/)
  [![Changelog](https://img.shields.io/github/v/release/simonw/datasette-ephemeral-tables?include_prereleases&label=changelog)](https://github.com/simonw/datasette-ephemeral-tables/releases)
  [![Tests](https://github.com/simonw/datasette-ephemeral-tables/workflows/Test/badge.svg)](https://github.com/simonw/datasette-ephemeral-tables/actions?query=workflow%3ATest)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/datasette-ephemeral-tables/blob/main/LICENSE)

  Provide tables that expire after a time limit

  ## Installation

  Install this plugin in the same environment as Datasette.

      datasette install datasette-ephemeral-tables

  ## Usage

  Once installed, this plugin will ensure Datasette starts running with a new named in-memory database called `ephemeral`.

  Any tables created in this database will be automatically dropped five minutes after creation.

  ## Configuration

  All plugin settings are optional. The full set of settings in `metadata.yml` looks like this:

  ```yaml
  plugins:
      datasette-ephemeral-tables:
          # The name of the in-memory database created by the plugin:
          database: ephemeral
          # After how many seconds should tables be dropped?
          table_ttl: 300
          # How often should the plugin check for expired tables (in seconds)?
          check_interval: 2
  ```
  The figures shown here are the default values.

  ## Development

  To set up this plugin locally, first checkout the code. Then create a new virtual environment:

      cd datasette-ephemeral-tables
      python3 -m venv venv
      source venv/bin/activate

  Now install the dependencies and test dependencies:

      pip install -e '.[test]'

  To run the tests:

      pytest

readme_html:
  <div id="readme" class="md" data-path="README.md"><article class="markdown-body entry-content container-lg" itemprop="text"><div class="markdown-heading" dir="auto"><h1 class="heading-element" dir="auto">datasette-ephemeral-tables</h1><a id="user-content-datasette-ephemeral-tables" class="anchor" aria-label="Permalink: datasette-ephemeral-tables" href="#datasette-ephemeral-tables"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto"><a href="https://pypi.org/project/datasette-ephemeral-tables/" rel="nofollow"><img src="https://img.shields.io/pypi/v/datasette-ephemeral-tables.svg" alt="PyPI" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/37c6a3ba24559691bafa31051c9a2263109f5466b0e3b1debbbd2e9497aa4185/68747470733a2f2f696d672e736869656c64732e696f2f707970692f762f6461746173657474652d657068656d6572616c2d7461626c65732e737667"></a>
  <a href="https://github.com/simonw/datasette-ephemeral-tables/releases"><img src="https://img.shields.io/github/v/release/simonw/datasette-ephemeral-tables?include_prereleases&label=changelog" alt="Changelog" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/ba4045ae716c7b8a5678f987b749fe918b5bfbf051d5dc37a7c47bdca4e26aee/68747470733a2f2f696d672e736869656c64732e696f2f6769746875622f762f72656c656173652f73696d6f6e772f6461746173657474652d657068656d6572616c2d7461626c65733f696e636c7564655f70726572656c6561736573266c6162656c3d6368616e67656c6f67"></a>
  <a href="https://github.com/simonw/datasette-ephemeral-tables/actions?query=workflow%3ATest"><img src="https://github.com/simonw/datasette-ephemeral-tables/workflows/Test/badge.svg" alt="Tests" style="max-width: 100%;"></a>
  <a href="https://github.com/simonw/datasette-ephemeral-tables/blob/main/LICENSE"><img src="https://img.shields.io/badge/license-Apache%202.0-blue.svg" alt="License" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/c355f200ea90fddaa407b6eaab303663a669248ea3ca7b1fcf77dbe04ff5f48c/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6c6963656e73652d417061636865253230322e302d626c75652e737667"></a></p>
  <p dir="auto">Provide tables that expire after a time limit</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Installation</h2><a id="user-content-installation" class="anchor" aria-label="Permalink: Installation" href="#installation"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">Install this plugin in the same environment as Datasette.</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="datasette install datasette-ephemeral-tables"><pre class="notranslate"><code>datasette install datasette-ephemeral-tables
  </code></pre></div>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Usage</h2><a id="user-content-usage" class="anchor" aria-label="Permalink: Usage" href="#usage"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">Once installed, this plugin will ensure Datasette starts running with a new named in-memory database called <code>ephemeral</code>.</p>
  <p dir="auto">Any tables created in this database will be automatically dropped five minutes after creation.</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Configuration</h2><a id="user-content-configuration" class="anchor" aria-label="Permalink: Configuration" href="#configuration"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">All plugin settings are optional. The full set of settings in <code>metadata.yml</code> looks like this:</p>
  <div class="highlight highlight-source-yaml notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="plugins:
      datasette-ephemeral-tables:
          # The name of the in-memory database created by the plugin:
          database: ephemeral
          # After how many seconds should tables be dropped?
          table_ttl: 300
          # How often should the plugin check for expired tables (in seconds)?
          check_interval: 2"><pre><span class="pl-ent">plugins</span>:
      <span class="pl-ent">datasette-ephemeral-tables</span>:
          <span class="pl-c"><span class="pl-c">#</span> The name of the in-memory database created by the plugin:</span>
          <span class="pl-ent">database</span>: <span class="pl-s">ephemeral</span>
          <span class="pl-c"><span class="pl-c">#</span> After how many seconds should tables be dropped?</span>
          <span class="pl-ent">table_ttl</span>: <span class="pl-c1">300</span>
          <span class="pl-c"><span class="pl-c">#</span> How often should the plugin check for expired tables (in seconds)?</span>
          <span class="pl-ent">check_interval</span>: <span class="pl-c1">2</span></pre></div>
  <p dir="auto">The figures shown here are the default values.</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Development</h2><a id="user-content-development" class="anchor" aria-label="Permalink: Development" href="#development"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">To set up this plugin locally, first checkout the code. Then create a new virtual environment:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="cd datasette-ephemeral-tables
  python3 -m venv venv
  source venv/bin/activate"><pre class="notranslate"><code>cd datasette-ephemeral-tables
  python3 -m venv venv
  source venv/bin/activate
  </code></pre></div>
  <p dir="auto">Now install the dependencies and test dependencies:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="pip install -e '.[test]'"><pre class="notranslate"><code>pip install -e '.[test]'
  </code></pre></div>
  <p dir="auto">To run the tests:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="pytest"><pre class="notranslate"><code>pytest
  </code></pre></div>
  </article></div>


node_id: R_kgDOIl_N4w
name: datasette-secret-santa
full_name: simonw/datasette-secret-santa
html_url: https://github.com/simonw/datasette-secret-santa
description: Run secret santa gift circles using Datasette
created_at: 2022-12-10T17:54:39Z
updated_at: 2025-03-04T12:45:03Z
pushed_at: 2022-12-11T01:41:10Z
language: Python
license: apache-2.0
topics: []
visibility: public
default_branch: main
permissions: {"admin": false, "maintain": false, "push": false, "triage": false, "pull": true}
readme:
  # datasette-secret-santa

  [![PyPI](https://img.shields.io/pypi/v/datasette-secret-santa.svg)](https://pypi.org/project/datasette-secret-santa/)
  [![Changelog](https://img.shields.io/github/v/release/simonw/datasette-secret-santa?include_prereleases&label=changelog)](https://github.com/simonw/datasette-secret-santa/releases)
  [![Tests](https://github.com/simonw/datasette-secret-santa/workflows/Test/badge.svg)](https://github.com/simonw/datasette-secret-santa/actions?query=workflow%3ATest)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/datasette-secret-santa/blob/main/LICENSE)

  Run a secret santa using Datasette

  ## Installation

  Install this plugin in the same environment as Datasette.

      datasette install datasette-secret-santa

  ## Running this on Glitch

  https://glitch.com/ is a great way to host this application.

  You can [remix this project on Glitch](https://glitch.com/~datasette-secret-santa) to get your own copy.

  Or you can create a new application and paste the following into your `glitch.json` file:

  ```json
  {
    "install": "pip3 install --user datasette datasette-secret-santa -U",
    "start": "datasette --create .data/santa.db -p 3000"
  }
  ```
  Then visit your new app's homepage and click the link to create a new secret santa group.

  ## Usage

  ![Animated GIF showing the plugin in action - the user adds three names, then gets the password for their account - then hits the assign button and uses their password to find out who they have been assigned.](https://raw.githubusercontent.com/simonw/datasette-secret-santa/main/secret-santa.gif)

  This plugin requires a database called `santa.db`. You can run it and create such a database like this:

      datasette santa.db --create

  It expects to be the only plugin installed, and will take over the `/` homepage.

  To create a new Secret Santa, visit `/santa/create_secret_santa` (linked from the homepage).

  ## Development

  To set up this plugin locally, first checkout the code. Then create a new virtual environment:

      cd datasette-secret-santa
      python3 -m venv venv
      source venv/bin/activate

  Now install the dependencies and test dependencies:

      pip install -e '.[test]'

  To run the tests:

      pytest

readme_html:
  <div id="readme" class="md" data-path="README.md"><article class="markdown-body entry-content container-lg" itemprop="text"><div class="markdown-heading" dir="auto"><h1 class="heading-element" dir="auto">datasette-secret-santa</h1><a id="user-content-datasette-secret-santa" class="anchor" aria-label="Permalink: datasette-secret-santa" href="#datasette-secret-santa"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto"><a href="https://pypi.org/project/datasette-secret-santa/" rel="nofollow"><img src="https://img.shields.io/pypi/v/datasette-secret-santa.svg" alt="PyPI" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/96c881bb8a11ddb3df3a7dac0b983503c795fa20a5658a46948f23830fec33f5/68747470733a2f2f696d672e736869656c64732e696f2f707970692f762f6461746173657474652d7365637265742d73616e74612e737667"></a>
  <a href="https://github.com/simonw/datasette-secret-santa/releases"><img src="https://img.shields.io/github/v/release/simonw/datasette-secret-santa?include_prereleases&label=changelog" alt="Changelog" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/15a899c398af58286d0ba3c2f77e2cd69b9b39c0006a7d7277ad119de5111dbe/68747470733a2f2f696d672e736869656c64732e696f2f6769746875622f762f72656c656173652f73696d6f6e772f6461746173657474652d7365637265742d73616e74613f696e636c7564655f70726572656c6561736573266c6162656c3d6368616e67656c6f67"></a>
  <a href="https://github.com/simonw/datasette-secret-santa/actions?query=workflow%3ATest"><img src="https://github.com/simonw/datasette-secret-santa/workflows/Test/badge.svg" alt="Tests" style="max-width: 100%;"></a>
  <a href="https://github.com/simonw/datasette-secret-santa/blob/main/LICENSE"><img src="https://img.shields.io/badge/license-Apache%202.0-blue.svg" alt="License" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/c355f200ea90fddaa407b6eaab303663a669248ea3ca7b1fcf77dbe04ff5f48c/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6c6963656e73652d417061636865253230322e302d626c75652e737667"></a></p>
  <p dir="auto">Run a secret santa using Datasette</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Installation</h2><a id="user-content-installation" class="anchor" aria-label="Permalink: Installation" href="#installation"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">Install this plugin in the same environment as Datasette.</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="datasette install datasette-secret-santa"><pre class="notranslate"><code>datasette install datasette-secret-santa
  </code></pre></div>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Running this on Glitch</h2><a id="user-content-running-this-on-glitch" class="anchor" aria-label="Permalink: Running this on Glitch" href="#running-this-on-glitch"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto"><a href="https://glitch.com/" rel="nofollow">https://glitch.com/</a> is a great way to host this application.</p>
  <p dir="auto">You can <a href="https://glitch.com/~datasette-secret-santa" rel="nofollow">remix this project on Glitch</a> to get your own copy.</p>
  <p dir="auto">Or you can create a new application and paste the following into your <code>glitch.json</code> file:</p>
  <div class="highlight highlight-source-json notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="{
    "install": "pip3 install --user datasette datasette-secret-santa -U",
    "start": "datasette --create .data/santa.db -p 3000"
  }"><pre>{
    <span class="pl-ent">"install"</span>: <span class="pl-s"><span class="pl-pds">"</span>pip3 install --user datasette datasette-secret-santa -U<span class="pl-pds">"</span></span>,
    <span class="pl-ent">"start"</span>: <span class="pl-s"><span class="pl-pds">"</span>datasette --create .data/santa.db -p 3000<span class="pl-pds">"</span></span>
  }</pre></div>
  <p dir="auto">Then visit your new app's homepage and click the link to create a new secret santa group.</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Usage</h2><a id="user-content-usage" class="anchor" aria-label="Permalink: Usage" href="#usage"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto"><a target="_blank" rel="noopener noreferrer nofollow" href="https://raw.githubusercontent.com/simonw/datasette-secret-santa/main/secret-santa.gif"><img src="https://raw.githubusercontent.com/simonw/datasette-secret-santa/main/secret-santa.gif" alt="Animated GIF showing the plugin in action - the user adds three names, then gets the password for their account - then hits the assign button and uses their password to find out who they have been assigned." data-animated-image="" style="max-width: 100%;"></a></p>
  <p dir="auto">This plugin requires a database called <code>santa.db</code>. You can run it and create such a database like this:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="datasette santa.db --create"><pre class="notranslate"><code>datasette santa.db --create
  </code></pre></div>
  <p dir="auto">It expects to be the only plugin installed, and will take over the <code>/</code> homepage.</p>
  <p dir="auto">To create a new Secret Santa, visit <code>/santa/create_secret_santa</code> (linked from the homepage).</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Development</h2><a id="user-content-development" class="anchor" aria-label="Permalink: Development" href="#development"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">To set up this plugin locally, first checkout the code. Then create a new virtual environment:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="cd datasette-secret-santa
  python3 -m venv venv
  source venv/bin/activate"><pre class="notranslate"><code>cd datasette-secret-santa
  python3 -m venv venv
  source venv/bin/activate
  </code></pre></div>
  <p dir="auto">Now install the dependencies and test dependencies:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="pip install -e '.[test]'"><pre class="notranslate"><code>pip install -e '.[test]'
  </code></pre></div>
  <p dir="auto">To run the tests:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="pytest"><pre class="notranslate"><code>pytest
  </code></pre></div>
  </article></div>


node_id: R_kgDOIpo-2g
name: sqlite-ulid
full_name: asg017/sqlite-ulid
html_url: https://github.com/asg017/sqlite-ulid
description: A SQLite extension for generating and working with ULIDs
created_at: 2022-12-20T19:58:44Z
updated_at: 2025-05-04T12:11:51Z
pushed_at: 2023-10-06T22:57:31Z
language: Python
license: apache-2.0
topics: ["sqlite", "sqlite-extension"]
visibility: public
default_branch: main
permissions: {"admin": false, "maintain": false, "push": false, "triage": false, "pull": true}
readme:
  # sqlite-ulid

  A SQLite extension for generating and working with [ULIDs](https://github.com/ulid/spec). Built on top of [sqlite-loadable-rs](https://github.com/asg017/sqlite-loadable-rs) and [ulid-rs](https://github.com/dylanhart/ulid-rs).

  If your company or organization finds this library useful, consider [supporting my work](#supporting)!

  ## Usage

  ```sql
  .load ./ulid0

  select ulid(); -- '01gqr4j69cc7w1xdbarkcbpq17'
  select ulid_bytes(); -- X'0185310899dd7662b8f1e5adf9a5e7c0'
  select ulid_with_prefix('invoice'); -- 'invoice_01gqr4jmhxhc92x1kqkpxb8j16'
  select ulid_with_datetime('2023-01-26 22:53:20.556); -- '01gqr4j69cc7w1xdbarkcbpq17'
  select ulid_datetime('01gqr4j69cc7w1xdbarkcbpq17') -- '2023-01-26 22:53:20.556'
  ```

  Use as a `PRIMARY KEY` for a table.

  ```sql
  create table log_events(
    id ulid primary key,
    data any
  );


  insert into log_events(id, data) values (ulid(), 1);
  insert into log_events(id, data) values (ulid(), 2);
  insert into log_events(id, data) values (ulid(), 3);

  select * from log_events;
  /*
  ┌────────────────────────────┬──────┐
  │             id             │ data │
  ├────────────────────────────┼──────┤
  │ 01gqr4vr487bytsf10ktfmheg4 │ 1    │
  │ 01gqr4vr4dfcfk80m2yp6j866z │ 2    │
  │ 01gqr4vrjxg0yex9jr0f100v1c │ 3    │
  └────────────────────────────┴──────┘
  */
  ```

  Consider using [`ulid_bytes()`](./docs.md#ulid_bytes) for speed and smaller IDs. They generate about 1.6x faster than `ulid()`, and take up 16 bytes instead of 26 bytes. You can use `ulid()` to create a text representation of a BLOB ULID.

  ```sql

  create table log_events(
    id ulid primary key,
    data any
  );


  insert into log_events(id, data) values (ulid_bytes(), 1);
  insert into log_events(id, data) values (ulid_bytes(), 2);
  insert into log_events(id, data) values (ulid_bytes(), 3);

  select hex(id), ulid(id), data from log_events;
  /*
  ┌──────────────────────────────────┬────────────────────────────┬──────┐
  │             hex(id)              │          ulid(id)          │ data │
  ├──────────────────────────────────┼────────────────────────────┼──────┤
  │ 0185F0539EBF286DA9F56BA4D9981783 │ 01gqr577nz51ptkxbbmkcsg5w3 │ 1    │
  │ 0185F0539EC54F85745C1ECB64DF3A97 │ 01gqr577p59y2q8q0ysdjdyemq │ 2    │
  │ 0185F0539ED48113F6F67BF3F6A4BFF7 │ 01gqr577pmg49zdxkvyfva9fzq │ 3    │
  └──────────────────────────────────┴────────────────────────────┴──────┘
  */
  ```

  Extract the timestamp component of a ULID with [`ulid_datetime()`](./docs.md#ulid_datetime).

  ```sql
  select ulid_datetime(ulid()); -- '2023-01-26 23:07:36.508'
  select unixepoch(ulid_datetime(ulid())); -- 1674774499
  select strftime('%Y-%m-%d', ulid_datetime(ulid())); -- '2023-01-26''
  ```

  Consider using [`ulid_with_prefix()`](./docs.md#ulid_with_prefix) to generate a text ULID with a given prefix, to differentiate between different ID types.

  ```sql
  select ulid_with_prefix('customer'); -- 'customer_01gqr5j1ebk31wv30wgp8ebehj'
  select ulid_with_prefix('product'); -- 'product_01gqr5prjgsa77dhrxf2dt1dgv'
  select ulid_with_prefix('order'); -- 'order_01gqr5q35n68jk0sycy1ntr083'


  ```

  ## Quick benchmarks

  Not definitive, hastily ran on a Macbook, not representative of real-life usecases. The `uuid()` SQL function comes from the [official `uuid.c` extension](https://sqlite.org/src/file/ext/misc/uuid.c).

  | Test case                                      | Time                                            |
  | ---------------------------------------------- | ----------------------------------------------- |
  | `generate_series()` to generate 1 million rows | `28.5 ms ±   0.8 ms` (`1x`)                     |
  | Calling `ulid_bytes()` 1 million times         | `88.4 ms ±   2.8 ms`,, `3.10 ± 0.13` slower     |
  | Calling `uuid()` 1 million times               | `141.6 ms ±   1.5 ms`, or `4.97 ± 0.15` slower  |
  | Calling `ulid()` 1 million times               | `344.3 ms ±  11.9 ms`, or `12.07 ± 0.53` slower |

  So `ulid_bytes()` is pretty fast, but returns an unreadable blob instead of a nicely formatted text ID. The `ulid()` function does that, but is more than twice as slow than `uuid()`.

  However, generating 1 million `ulid()` IDs in ~350ms is most likely "good enough" for most SQLite usecases.

  ## Using with...

  | Language       | Install                                                      |                                                                                                                                                                                             |
  | -------------- | ------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
  | Python         | `pip install sqlite-ulid`                                    | [![PyPI](https://img.shields.io/pypi/v/sqlite-ulid.svg?color=blue&logo=python&logoColor=white)](https://pypi.org/project/sqlite-ulid/)                                                      |
  | Datasette      | `datasette install datasette-sqlite-ulid`                    | [![Datasette](https://img.shields.io/pypi/v/datasette-sqlite-ulid.svg?color=B6B6D9&label=Datasette+plugin&logoColor=white&logo=python)](https://datasette.io/plugins/datasette-sqlite-ulid) |
  | Node.js        | `npm install sqlite-ulid`                                    | [![npm](https://img.shields.io/npm/v/sqlite-ulid.svg?color=green&logo=nodedotjs&logoColor=white)](https://www.npmjs.com/package/sqlite-ulid)                                                |
  | Deno           | [`deno.land/x/sqlite_ulid`](https://deno.land/x/sqlite_ulid) | [![deno.land/x release](https://img.shields.io/github/v/release/asg017/sqlite-ulid?color=fef8d2&include_prereleases&label=deno.land%2Fx&logo=deno)](https://deno.land/x/sqlite_ulid)        |
  | Ruby           | `gem install sqlite-ulid`                                    | ![Gem](https://img.shields.io/gem/v/sqlite-ulid?color=red&logo=rubygems&logoColor=white)                                                                                                    |
  | Rust           | `cargo add sqlite-ulid`                                      | [![Crates.io](https://img.shields.io/crates/v/sqlite-ulid?logo=rust)](https://crates.io/crates/sqlite-ulid)                                                                                 |
  | Github Release |                                                              | ![GitHub tag (latest SemVer pre-release)](https://img.shields.io/github/v/tag/asg017/sqlite-ulid?color=lightgrey&include_prereleases&label=Github+release&logo=github)                      |

  <!--
  | Elixir         | [`hex.pm/packages/sqlite_ulid`](https://hex.pm/packages/sqlite_ulid) | [![Hex.pm](https://img.shields.io/hexpm/v/sqlite_ulid?color=purple&logo=elixir)](https://hex.pm/packages/sqlite_ulid)                                                                       |
  | Go             | `go get -u github.com/asg017/sqlite-ulid/bindings/go`               | [![Go Reference](https://pkg.go.dev/badge/github.com/asg017/sqlite-ulid/bindings/go.svg)](https://pkg.go.dev/github.com/asg017/sqlite-ulid/bindings/go)                                     |
  -->

  The [Releases page](https://github.com/asg017/sqlite-ulid/releases) contains pre-built binaries for Linux x86_64, MacOS, and Windows.

  ### Python

  For Python developers, install the [`sqlite-ulid` package](https://pypi.org/package/sqlite-ulid/) with:

  ```
  pip install sqlite-ulid
  ```

  ```python
  import sqlite3
  import sqlite_ulid
  db = sqlite3.connect(':memory:')
  db.enable_load_extension(True)
  sqlite_ulid.load(db)
  db.execute('select ulid()').fetchone()
  # ('01gr7gwc5aq22ycea6j8kxq4s9',)
  ```

  See [`python/sqlite_ulid`](./python/sqlite_ulid/README.md) for more details.

  ### Node.js

  For Node.js developers, install the [`sqlite-ulid` npm package](https://www.npmjs.com/package/sqlite-ulid) with:

  ```
  npm install sqlite-ulid
  ```

  ```js
  import Database from "better-sqlite3";
  import * as sqlite_ulid from "sqlite-ulid";

  const db = new Database(":memory:");
  db.loadExtension(sqlite_ulid.getLoadablePath());
  ```

  See [`npm/sqlite-ulid/README.md`](./npm/sqlite-ulid/README.md) for more details.

  ### Deno

  For [Deno](https://deno.land/) developers, use the [deno.land/x/sqlite_ulid](https://deno.land/x/sqlite_ulid) module:

  ```ts
  import { Database } from "https://deno.land/x/sqlite3@0.8.0/mod.ts";
  import * as sqlite_ulid from "https://deno.land/x/sqlite_ulid@v${VERSION}/mod.ts";

  const db = new Database(":memory:");

  db.enableLoadExtension = true;
  sqlite_ulid.load(db);

  const [version] = db.prepare("select ulid_version()").value<[string]>()!;

  console.log(version);
  ```

  ### Datasette

  And for [Datasette](https://datasette.io/), install the [`datasette-sqlite-ulid` plugin](https://datasette.io/plugins/datasette-sqlite-ulid) with:

  ```
  datasette install datasette-sqlite-ulid
  ```

  See [`python/datasette_sqlite_ulid`](./python/datasette_sqlite_ulid/README.md) for more details.

  ### As a loadable extension

  If you want to use `sqlite-ulid` as a [Runtime-loadable extension](https://www.sqlite.org/loadext.html), Download the `ulid0.dylib` (for MacOS), `ulid0.so` (Linux), or `ulid0.dll` (Windows) file [from a release](https://github.com/asg017/sqlite-ulid/releases) and load it into your SQLite environment.

  > **Note:**
  > The `0` in the filename (`ulid0.dylib`/ `ulid0.so`/`ulid0.dll`) denotes the major version of `sqlite-ulid`. Currently `sqlite-ulid` is pre v1, so expect breaking changes in future versions.

  For example, if you are using the [SQLite CLI](https://www.sqlite.org/cli.html), you can load the library like so:

  ```sql
  .load ./ulid0
  select ulid_version();
  -- v0.1.0
  ```

  In Python, you should prefer the [`sqlite-ulid` Python package](./python/sqlite_ulid/README.md). However, you can manually load a pre-compiled extension with the builtin [sqlite3 module](https://docs.python.org/3/library/sqlite3.html):

  ```python
  import sqlite3
  con = sqlite3.connect(":memory:")
  con.enable_load_extension(True)
  con.load_extension("./ulid0")
  print(con.execute("select ulid_version()").fetchone())
  # ('v0.1.0',)
  ```

  Or in Node.js using [better-sqlite3](https://github.com/WiseLibs/better-sqlite3):

  ```javascript
  const Database = require("better-sqlite3");
  const db = new Database(":memory:");
  db.loadExtension("./ulid0");
  console.log(db.prepare("select ulid_version()").get());
  // { 'ulid_version()': 'v0.1.0' }
  ```

  With [Datasette](https://datasette.io/), you should prefer the [`datasette-sqlite-ulid` Datasette plugin](./python/datasette_sqlite_ulid/README.md). However, you can manually load a pre-compiled extension into a Datasette instance like so:

  ```
  datasette data.db --load-extension ./ulid0
  ```

  ## Supporting

  I (Alex 👋🏼) spent a lot of time and energy on this project and [many other open source projects](https://github.com/asg017?tab=repositories&q=&type=&language=&sort=stargazers). If your company or organization uses this library (or you're feeling generous), then please [consider supporting my work](https://alexgarcia.ulid/work.html), or share this project with a friend!

  ## See also

  - [sqlite-xsv](https://github.com/asg017/sqlite-xsv), A SQLite extension for working with CSVs
  - [sqlite-loadable](https://github.com/asg017/sqlite-loadable-rs), A framework for writing SQLite extensions in Rust
  - [sqlite-http](https://github.com/asg017/sqlite-http), A SQLite extension for making HTTP requests

readme_html:
  <div id="readme" class="md" data-path="README.md"><article class="markdown-body entry-content container-lg" itemprop="text"><div class="markdown-heading" dir="auto"><h1 class="heading-element" dir="auto">sqlite-ulid</h1><a id="user-content-sqlite-ulid" class="anchor" aria-label="Permalink: sqlite-ulid" href="#sqlite-ulid"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">A SQLite extension for generating and working with <a href="https://github.com/ulid/spec">ULIDs</a>. Built on top of <a href="https://github.com/asg017/sqlite-loadable-rs">sqlite-loadable-rs</a> and <a href="https://github.com/dylanhart/ulid-rs">ulid-rs</a>.</p>
  <p dir="auto">If your company or organization finds this library useful, consider <a href="#supporting">supporting my work</a>!</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Usage</h2><a id="user-content-usage" class="anchor" aria-label="Permalink: Usage" href="#usage"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <div class="highlight highlight-source-sql notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content=".load ./ulid0

  select ulid(); -- '01gqr4j69cc7w1xdbarkcbpq17'
  select ulid_bytes(); -- X'0185310899dd7662b8f1e5adf9a5e7c0'
  select ulid_with_prefix('invoice'); -- 'invoice_01gqr4jmhxhc92x1kqkpxb8j16'
  select ulid_with_datetime('2023-01-26 22:53:20.556); -- '01gqr4j69cc7w1xdbarkcbpq17'
  select ulid_datetime('01gqr4j69cc7w1xdbarkcbpq17') -- '2023-01-26 22:53:20.556'"><pre>.load .<span class="pl-k">/</span>ulid0

  <span class="pl-k">select</span> ulid(); <span class="pl-c"><span class="pl-c">--</span> '01gqr4j69cc7w1xdbarkcbpq17'</span>
  <span class="pl-k">select</span> ulid_bytes(); <span class="pl-c"><span class="pl-c">--</span> X'0185310899dd7662b8f1e5adf9a5e7c0'</span>
  <span class="pl-k">select</span> ulid_with_prefix(<span class="pl-s"><span class="pl-pds">'</span>invoice<span class="pl-pds">'</span></span>); <span class="pl-c"><span class="pl-c">--</span> 'invoice_01gqr4jmhxhc92x1kqkpxb8j16'</span>
  <span class="pl-k">select</span> ulid_with_datetime(<span class="pl-s"><span class="pl-pds">'</span>2023-01-26 22:53:20.556); -- <span class="pl-pds">'</span></span>01gqr4j69cc7w1xdbarkcbpq17<span class="pl-s"><span class="pl-pds">'</span></span>
  <span class="pl-s">select ulid_datetime(<span class="pl-pds">'</span></span>01gqr4j69cc7w1xdbarkcbpq17<span class="pl-s"><span class="pl-pds">'</span>) -- <span class="pl-pds">'</span></span><span class="pl-c1">2023</span><span class="pl-k">-</span><span class="pl-c1">01</span><span class="pl-k">-</span><span class="pl-c1">26</span> <span class="pl-c1">22</span>:<span class="pl-c1">53</span>:<span class="pl-c1">20</span>.<span class="pl-c1">556</span><span class="pl-s"><span class="pl-pds">'</span></span></pre></div>
  <p dir="auto">Use as a <code>PRIMARY KEY</code> for a table.</p>
  <div class="highlight highlight-source-sql notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="create table log_events(
    id ulid primary key,
    data any
  );


  insert into log_events(id, data) values (ulid(), 1);
  insert into log_events(id, data) values (ulid(), 2);
  insert into log_events(id, data) values (ulid(), 3);

  select * from log_events;
  /*
  ┌────────────────────────────┬──────┐
  │             id             │ data │
  ├────────────────────────────┼──────┤
  │ 01gqr4vr487bytsf10ktfmheg4 │ 1    │
  │ 01gqr4vr4dfcfk80m2yp6j866z │ 2    │
  │ 01gqr4vrjxg0yex9jr0f100v1c │ 3    │
  └────────────────────────────┴──────┘
  */"><pre><span class="pl-k">create</span> <span class="pl-k">table</span> <span class="pl-en">log_events</span>(
    id ulid <span class="pl-k">primary key</span>,
    data any
  );


  <span class="pl-k">insert into</span> log_events(id, data) <span class="pl-k">values</span> (ulid(), <span class="pl-c1">1</span>);
  <span class="pl-k">insert into</span> log_events(id, data) <span class="pl-k">values</span> (ulid(), <span class="pl-c1">2</span>);
  <span class="pl-k">insert into</span> log_events(id, data) <span class="pl-k">values</span> (ulid(), <span class="pl-c1">3</span>);

  <span class="pl-k">select</span> <span class="pl-k">*</span> <span class="pl-k">from</span> log_events;
  <span class="pl-c"><span class="pl-c">/*</span></span>
  <span class="pl-c">┌────────────────────────────┬──────┐</span>
  <span class="pl-c">│             id             │ data │</span>
  <span class="pl-c">├────────────────────────────┼──────┤</span>
  <span class="pl-c">│ 01gqr4vr487bytsf10ktfmheg4 │ 1    │</span>
  <span class="pl-c">│ 01gqr4vr4dfcfk80m2yp6j866z │ 2    │</span>
  <span class="pl-c">│ 01gqr4vrjxg0yex9jr0f100v1c │ 3    │</span>
  <span class="pl-c">└────────────────────────────┴──────┘</span>
  <span class="pl-c"><span class="pl-c">*/</span></span></pre></div>
  <p dir="auto">Consider using <a href="./docs.md#ulid_bytes"><code>ulid_bytes()</code></a> for speed and smaller IDs. They generate about 1.6x faster than <code>ulid()</code>, and take up 16 bytes instead of 26 bytes. You can use <code>ulid()</code> to create a text representation of a BLOB ULID.</p>
  <div class="highlight highlight-source-sql notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="
  create table log_events(
    id ulid primary key,
    data any
  );


  insert into log_events(id, data) values (ulid_bytes(), 1);
  insert into log_events(id, data) values (ulid_bytes(), 2);
  insert into log_events(id, data) values (ulid_bytes(), 3);

  select hex(id), ulid(id), data from log_events;
  /*
  ┌──────────────────────────────────┬────────────────────────────┬──────┐
  │             hex(id)              │          ulid(id)          │ data │
  ├──────────────────────────────────┼────────────────────────────┼──────┤
  │ 0185F0539EBF286DA9F56BA4D9981783 │ 01gqr577nz51ptkxbbmkcsg5w3 │ 1    │
  │ 0185F0539EC54F85745C1ECB64DF3A97 │ 01gqr577p59y2q8q0ysdjdyemq │ 2    │
  │ 0185F0539ED48113F6F67BF3F6A4BFF7 │ 01gqr577pmg49zdxkvyfva9fzq │ 3    │
  └──────────────────────────────────┴────────────────────────────┴──────┘
  */"><pre><span class="pl-k">create</span> <span class="pl-k">table</span> <span class="pl-en">log_events</span>(
    id ulid <span class="pl-k">primary key</span>,
    data any
  );


  <span class="pl-k">insert into</span> log_events(id, data) <span class="pl-k">values</span> (ulid_bytes(), <span class="pl-c1">1</span>);
  <span class="pl-k">insert into</span> log_events(id, data) <span class="pl-k">values</span> (ulid_bytes(), <span class="pl-c1">2</span>);
  <span class="pl-k">insert into</span> log_events(id, data) <span class="pl-k">values</span> (ulid_bytes(), <span class="pl-c1">3</span>);

  <span class="pl-k">select</span> hex(id), ulid(id), data <span class="pl-k">from</span> log_events;
  <span class="pl-c"><span class="pl-c">/*</span></span>
  <span class="pl-c">┌──────────────────────────────────┬────────────────────────────┬──────┐</span>
  <span class="pl-c">│             hex(id)              │          ulid(id)          │ data │</span>
  <span class="pl-c">├──────────────────────────────────┼────────────────────────────┼──────┤</span>
  <span class="pl-c">│ 0185F0539EBF286DA9F56BA4D9981783 │ 01gqr577nz51ptkxbbmkcsg5w3 │ 1    │</span>
  <span class="pl-c">│ 0185F0539EC54F85745C1ECB64DF3A97 │ 01gqr577p59y2q8q0ysdjdyemq │ 2    │</span>
  <span class="pl-c">│ 0185F0539ED48113F6F67BF3F6A4BFF7 │ 01gqr577pmg49zdxkvyfva9fzq │ 3    │</span>
  <span class="pl-c">└──────────────────────────────────┴────────────────────────────┴──────┘</span>
  <span class="pl-c"><span class="pl-c">*/</span></span></pre></div>
  <p dir="auto">Extract the timestamp component of a ULID with <a href="./docs.md#ulid_datetime"><code>ulid_datetime()</code></a>.</p>
  <div class="highlight highlight-source-sql notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="select ulid_datetime(ulid()); -- '2023-01-26 23:07:36.508'
  select unixepoch(ulid_datetime(ulid())); -- 1674774499
  select strftime('%Y-%m-%d', ulid_datetime(ulid())); -- '2023-01-26''"><pre><span class="pl-k">select</span> ulid_datetime(ulid()); <span class="pl-c"><span class="pl-c">--</span> '2023-01-26 23:07:36.508'</span>
  <span class="pl-k">select</span> unixepoch(ulid_datetime(ulid())); <span class="pl-c"><span class="pl-c">--</span> 1674774499</span>
  <span class="pl-k">select</span> strftime(<span class="pl-s"><span class="pl-pds">'</span>%Y-%m-%d<span class="pl-pds">'</span></span>, ulid_datetime(ulid())); <span class="pl-c"><span class="pl-c">--</span> '2023-01-26''</span></pre></div>
  <p dir="auto">Consider using <a href="./docs.md#ulid_with_prefix"><code>ulid_with_prefix()</code></a> to generate a text ULID with a given prefix, to differentiate between different ID types.</p>
  <div class="highlight highlight-source-sql notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="select ulid_with_prefix('customer'); -- 'customer_01gqr5j1ebk31wv30wgp8ebehj'
  select ulid_with_prefix('product'); -- 'product_01gqr5prjgsa77dhrxf2dt1dgv'
  select ulid_with_prefix('order'); -- 'order_01gqr5q35n68jk0sycy1ntr083'

  "><pre><span class="pl-k">select</span> ulid_with_prefix(<span class="pl-s"><span class="pl-pds">'</span>customer<span class="pl-pds">'</span></span>); <span class="pl-c"><span class="pl-c">--</span> 'customer_01gqr5j1ebk31wv30wgp8ebehj'</span>
  <span class="pl-k">select</span> ulid_with_prefix(<span class="pl-s"><span class="pl-pds">'</span>product<span class="pl-pds">'</span></span>); <span class="pl-c"><span class="pl-c">--</span> 'product_01gqr5prjgsa77dhrxf2dt1dgv'</span>
  <span class="pl-k">select</span> ulid_with_prefix(<span class="pl-s"><span class="pl-pds">'</span>order<span class="pl-pds">'</span></span>); <span class="pl-c"><span class="pl-c">--</span> 'order_01gqr5q35n68jk0sycy1ntr083'</span>

  </pre></div>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Quick benchmarks</h2><a id="user-content-quick-benchmarks" class="anchor" aria-label="Permalink: Quick benchmarks" href="#quick-benchmarks"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">Not definitive, hastily ran on a Macbook, not representative of real-life usecases. The <code>uuid()</code> SQL function comes from the <a href="https://sqlite.org/src/file/ext/misc/uuid.c" rel="nofollow">official <code>uuid.c</code> extension</a>.</p>
  <markdown-accessiblity-table><table>
  <thead>
  <tr>
  <th>Test case</th>
  <th>Time</th>
  </tr>
  </thead>
  <tbody>
  <tr>
  <td><code>generate_series()</code> to generate 1 million rows</td>
  <td><code>28.5 ms ±   0.8 ms</code> (<code>1x</code>)</td>
  </tr>
  <tr>
  <td>Calling <code>ulid_bytes()</code> 1 million times</td>
  <td><code>88.4 ms ±   2.8 ms</code>,, <code>3.10 ± 0.13</code> slower</td>
  </tr>
  <tr>
  <td>Calling <code>uuid()</code> 1 million times</td>
  <td><code>141.6 ms ±   1.5 ms</code>, or <code>4.97 ± 0.15</code> slower</td>
  </tr>
  <tr>
  <td>Calling <code>ulid()</code> 1 million times</td>
  <td><code>344.3 ms ±  11.9 ms</code>, or <code>12.07 ± 0.53</code> slower</td>
  </tr>
  </tbody>
  </table></markdown-accessiblity-table>
  <p dir="auto">So <code>ulid_bytes()</code> is pretty fast, but returns an unreadable blob instead of a nicely formatted text ID. The <code>ulid()</code> function does that, but is more than twice as slow than <code>uuid()</code>.</p>
  <p dir="auto">However, generating 1 million <code>ulid()</code> IDs in ~350ms is most likely "good enough" for most SQLite usecases.</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Using with...</h2><a id="user-content-using-with" class="anchor" aria-label="Permalink: Using with..." href="#using-with"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <markdown-accessiblity-table><table>
  <thead>
  <tr>
  <th>Language</th>
  <th>Install</th>
  <th></th>
  </tr>
  </thead>
  <tbody>
  <tr>
  <td>Python</td>
  <td><code>pip install sqlite-ulid</code></td>
  <td><a href="https://pypi.org/project/sqlite-ulid/" rel="nofollow"><img src="https://img.shields.io/pypi/v/sqlite-ulid.svg?color=blue&logo=python&logoColor=white" alt="PyPI" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/13ca429cb958e5fdc245b36b8c866224c30f4e80c5c076a6e8c9a7f258bb7775/68747470733a2f2f696d672e736869656c64732e696f2f707970692f762f73716c6974652d756c69642e7376673f636f6c6f723d626c7565266c6f676f3d707974686f6e266c6f676f436f6c6f723d7768697465"></a></td>
  </tr>
  <tr>
  <td>Datasette</td>
  <td><code>datasette install datasette-sqlite-ulid</code></td>
  <td><a href="https://datasette.io/plugins/datasette-sqlite-ulid" rel="nofollow"><img src="https://img.shields.io/pypi/v/datasette-sqlite-ulid.svg?color=B6B6D9&label=Datasette+plugin&logoColor=white&logo=python" alt="Datasette" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/2de9dbc2098b465ee365bb2675a5ea9ca919fd89f64fdec02e68c1bb9021c411/68747470733a2f2f696d672e736869656c64732e696f2f707970692f762f6461746173657474652d73716c6974652d756c69642e7376673f636f6c6f723d423642364439266c6162656c3d4461746173657474652b706c7567696e266c6f676f436f6c6f723d7768697465266c6f676f3d707974686f6e"></a></td>
  </tr>
  <tr>
  <td>Node.js</td>
  <td><code>npm install sqlite-ulid</code></td>
  <td><a href="https://www.npmjs.com/package/sqlite-ulid" rel="nofollow"><img src="https://img.shields.io/npm/v/sqlite-ulid.svg?color=green&logo=nodedotjs&logoColor=white" alt="npm" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/8b9a39c95d1883765b135fac562838c0a3ec6282280fb0d283f4ce5209b1cfea/68747470733a2f2f696d672e736869656c64732e696f2f6e706d2f762f73716c6974652d756c69642e7376673f636f6c6f723d677265656e266c6f676f3d6e6f6465646f746a73266c6f676f436f6c6f723d7768697465"></a></td>
  </tr>
  <tr>
  <td>Deno</td>
  <td><a href="https://deno.land/x/sqlite_ulid" rel="nofollow"><code>deno.land/x/sqlite_ulid</code></a></td>
  <td><a href="https://deno.land/x/sqlite_ulid" rel="nofollow"><img src="https://img.shields.io/github/v/release/asg017/sqlite-ulid?color=fef8d2&include_prereleases&label=deno.land%2Fx&logo=deno" alt="deno.land/x release" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/70a6f09813ea66d6e5c3378faee3ae7b3deb2facd53590f7039fbaba51a57e32/68747470733a2f2f696d672e736869656c64732e696f2f6769746875622f762f72656c656173652f6173673031372f73716c6974652d756c69643f636f6c6f723d66656638643226696e636c7564655f70726572656c6561736573266c6162656c3d64656e6f2e6c616e6425324678266c6f676f3d64656e6f"></a></td>
  </tr>
  <tr>
  <td>Ruby</td>
  <td><code>gem install sqlite-ulid</code></td>
  <td><a target="_blank" rel="noopener noreferrer nofollow" href="https://camo.githubusercontent.com/f0522ae9adb3ffe09d3f00b079a798e5970b3393e5db3ca0a7e935edb47227e4/68747470733a2f2f696d672e736869656c64732e696f2f67656d2f762f73716c6974652d756c69643f636f6c6f723d726564266c6f676f3d7275627967656d73266c6f676f436f6c6f723d7768697465"><img src="https://img.shields.io/gem/v/sqlite-ulid?color=red&logo=rubygems&logoColor=white" alt="Gem" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/f0522ae9adb3ffe09d3f00b079a798e5970b3393e5db3ca0a7e935edb47227e4/68747470733a2f2f696d672e736869656c64732e696f2f67656d2f762f73716c6974652d756c69643f636f6c6f723d726564266c6f676f3d7275627967656d73266c6f676f436f6c6f723d7768697465"></a></td>
  </tr>
  <tr>
  <td>Rust</td>
  <td><code>cargo add sqlite-ulid</code></td>
  <td><a href="https://crates.io/crates/sqlite-ulid" rel="nofollow"><img src="https://img.shields.io/crates/v/sqlite-ulid?logo=rust" alt="Crates.io" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/89300010405e6cf7625467121bb220c1185cd86a4a32b272671598a377a28d87/68747470733a2f2f696d672e736869656c64732e696f2f6372617465732f762f73716c6974652d756c69643f6c6f676f3d72757374"></a></td>
  </tr>
  <tr>
  <td>Github Release</td>
  <td></td>
  <td><a target="_blank" rel="noopener noreferrer nofollow" href="https://camo.githubusercontent.com/9d5fedf8cc1f03a66e8d9490d6aef41561c56cd8cb8b0068ec00c73e822bee2c/68747470733a2f2f696d672e736869656c64732e696f2f6769746875622f762f7461672f6173673031372f73716c6974652d756c69643f636f6c6f723d6c696768746772657926696e636c7564655f70726572656c6561736573266c6162656c3d4769746875622b72656c65617365266c6f676f3d676974687562"><img src="https://img.shields.io/github/v/tag/asg017/sqlite-ulid?color=lightgrey&include_prereleases&label=Github+release&logo=github" alt="GitHub tag (latest SemVer pre-release)" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/9d5fedf8cc1f03a66e8d9490d6aef41561c56cd8cb8b0068ec00c73e822bee2c/68747470733a2f2f696d672e736869656c64732e696f2f6769746875622f762f7461672f6173673031372f73716c6974652d756c69643f636f6c6f723d6c696768746772657926696e636c7564655f70726572656c6561736573266c6162656c3d4769746875622b72656c65617365266c6f676f3d676974687562"></a></td>
  </tr>
  </tbody>
  </table></markdown-accessiblity-table>

  <p dir="auto">The <a href="https://github.com/asg017/sqlite-ulid/releases">Releases page</a> contains pre-built binaries for Linux x86_64, MacOS, and Windows.</p>
  <div class="markdown-heading" dir="auto"><h3 class="heading-element" dir="auto">Python</h3><a id="user-content-python" class="anchor" aria-label="Permalink: Python" href="#python"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">For Python developers, install the <a href="https://pypi.org/package/sqlite-ulid/" rel="nofollow"><code>sqlite-ulid</code> package</a> with:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="pip install sqlite-ulid"><pre class="notranslate"><code>pip install sqlite-ulid
  </code></pre></div>
  <div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="import sqlite3
  import sqlite_ulid
  db = sqlite3.connect(':memory:')
  db.enable_load_extension(True)
  sqlite_ulid.load(db)
  db.execute('select ulid()').fetchone()
  # ('01gr7gwc5aq22ycea6j8kxq4s9',)"><pre><span class="pl-k">import</span> <span class="pl-s1">sqlite3</span>
  <span class="pl-k">import</span> <span class="pl-s1">sqlite_ulid</span>
  <span class="pl-s1">db</span> <span class="pl-c1">=</span> <span class="pl-s1">sqlite3</span>.<span class="pl-c1">connect</span>(<span class="pl-s">':memory:'</span>)
  <span class="pl-s1">db</span>.<span class="pl-c1">enable_load_extension</span>(<span class="pl-c1">True</span>)
  <span class="pl-s1">sqlite_ulid</span>.<span class="pl-c1">load</span>(<span class="pl-s1">db</span>)
  <span class="pl-s1">db</span>.<span class="pl-c1">execute</span>(<span class="pl-s">'select ulid()'</span>).<span class="pl-c1">fetchone</span>()
  <span class="pl-c"># ('01gr7gwc5aq22ycea6j8kxq4s9',)</span></pre></div>
  <p dir="auto">See <a href="./python/sqlite_ulid/README.md"><code>python/sqlite_ulid</code></a> for more details.</p>
  <div class="markdown-heading" dir="auto"><h3 class="heading-element" dir="auto">Node.js</h3><a id="user-content-nodejs" class="anchor" aria-label="Permalink: Node.js" href="#nodejs"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">For Node.js developers, install the <a href="https://www.npmjs.com/package/sqlite-ulid" rel="nofollow"><code>sqlite-ulid</code> npm package</a> with:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="npm install sqlite-ulid"><pre class="notranslate"><code>npm install sqlite-ulid
  </code></pre></div>
  <div class="highlight highlight-source-js notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="import Database from "better-sqlite3";
  import * as sqlite_ulid from "sqlite-ulid";

  const db = new Database(":memory:");
  db.loadExtension(sqlite_ulid.getLoadablePath());"><pre><span class="pl-k">import</span> <span class="pl-v">Database</span> <span class="pl-k">from</span> <span class="pl-s">"better-sqlite3"</span><span class="pl-kos">;</span>
  <span class="pl-k">import</span> <span class="pl-c1">*</span> <span class="pl-k">as</span> <span class="pl-s1">sqlite_ulid</span> <span class="pl-k">from</span> <span class="pl-s">"sqlite-ulid"</span><span class="pl-kos">;</span>

  <span class="pl-k">const</span> <span class="pl-s1">db</span> <span class="pl-c1">=</span> <span class="pl-k">new</span> <span class="pl-v">Database</span><span class="pl-kos">(</span><span class="pl-s">":memory:"</span><span class="pl-kos">)</span><span class="pl-kos">;</span>
  <span class="pl-s1">db</span><span class="pl-kos">.</span><span class="pl-en">loadExtension</span><span class="pl-kos">(</span><span class="pl-s1">sqlite_ulid</span><span class="pl-kos">.</span><span class="pl-en">getLoadablePath</span><span class="pl-kos">(</span><span class="pl-kos">)</span><span class="pl-kos">)</span><span class="pl-kos">;</span></pre></div>
  <p dir="auto">See <a href="./npm/sqlite-ulid/README.md"><code>npm/sqlite-ulid/README.md</code></a> for more details.</p>
  <div class="markdown-heading" dir="auto"><h3 class="heading-element" dir="auto">Deno</h3><a id="user-content-deno" class="anchor" aria-label="Permalink: Deno" href="#deno"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">For <a href="https://deno.land/" rel="nofollow">Deno</a> developers, use the <a href="https://deno.land/x/sqlite_ulid" rel="nofollow">deno.land/x/sqlite_ulid</a> module:</p>
  <div class="highlight highlight-source-ts notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="import { Database } from "https://deno.land/x/sqlite3@0.8.0/mod.ts";
  import * as sqlite_ulid from "https://deno.land/x/sqlite_ulid@v${VERSION}/mod.ts";

  const db = new Database(":memory:");

  db.enableLoadExtension = true;
  sqlite_ulid.load(db);

  const [version] = db.prepare("select ulid_version()").value<[string]>()!;

  console.log(version);"><pre><span class="pl-k">import</span> <span class="pl-kos">{</span> <span class="pl-v">Database</span> <span class="pl-kos">}</span> <span class="pl-k">from</span> <span class="pl-s">"https://deno.land/x/sqlite3@0.8.0/mod.ts"</span><span class="pl-kos">;</span>
  <span class="pl-k">import</span> <span class="pl-c1">*</span> <span class="pl-k">as</span> <span class="pl-s1">sqlite_ulid</span> <span class="pl-k">from</span> <span class="pl-s">"https://deno.land/x/sqlite_ulid@v${VERSION}/mod.ts"</span><span class="pl-kos">;</span>

  <span class="pl-k">const</span> <span class="pl-s1">db</span> <span class="pl-c1">=</span> <span class="pl-k">new</span> <span class="pl-v">Database</span><span class="pl-kos">(</span><span class="pl-s">":memory:"</span><span class="pl-kos">)</span><span class="pl-kos">;</span>

  <span class="pl-s1">db</span><span class="pl-kos">.</span><span class="pl-c1">enableLoadExtension</span> <span class="pl-c1">=</span> <span class="pl-c1">true</span><span class="pl-kos">;</span>
  <span class="pl-s1">sqlite_ulid</span><span class="pl-kos">.</span><span class="pl-en">load</span><span class="pl-kos">(</span><span class="pl-s1">db</span><span class="pl-kos">)</span><span class="pl-kos">;</span>

  <span class="pl-k">const</span> <span class="pl-kos">[</span><span class="pl-s1">version</span><span class="pl-kos">]</span> <span class="pl-c1">=</span> <span class="pl-s1">db</span><span class="pl-kos">.</span><span class="pl-en">prepare</span><span class="pl-kos">(</span><span class="pl-s">"select ulid_version()"</span><span class="pl-kos">)</span><span class="pl-kos">.</span><span class="pl-en">value</span><span class="pl-c1"><</span><span class="pl-kos">[</span><span class="pl-smi">string</span><span class="pl-kos">]</span><span class="pl-c1">></span><span class="pl-kos">(</span><span class="pl-kos">)</span><span class="pl-c1">!</span><span class="pl-kos">;</span>

  <span class="pl-smi">console</span><span class="pl-kos">.</span><span class="pl-en">log</span><span class="pl-kos">(</span><span class="pl-s1">version</span><span class="pl-kos">)</span><span class="pl-kos">;</span></pre></div>
  <div class="markdown-heading" dir="auto"><h3 class="heading-element" dir="auto">Datasette</h3><a id="user-content-datasette" class="anchor" aria-label="Permalink: Datasette" href="#datasette"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">And for <a href="https://datasette.io/" rel="nofollow">Datasette</a>, install the <a href="https://datasette.io/plugins/datasette-sqlite-ulid" rel="nofollow"><code>datasette-sqlite-ulid</code> plugin</a> with:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="datasette install datasette-sqlite-ulid"><pre class="notranslate"><code>datasette install datasette-sqlite-ulid
  </code></pre></div>
  <p dir="auto">See <a href="./python/datasette_sqlite_ulid/README.md"><code>python/datasette_sqlite_ulid</code></a> for more details.</p>
  <div class="markdown-heading" dir="auto"><h3 class="heading-element" dir="auto">As a loadable extension</h3><a id="user-content-as-a-loadable-extension" class="anchor" aria-label="Permalink: As a loadable extension" href="#as-a-loadable-extension"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">If you want to use <code>sqlite-ulid</code> as a <a href="https://www.sqlite.org/loadext.html" rel="nofollow">Runtime-loadable extension</a>, Download the <code>ulid0.dylib</code> (for MacOS), <code>ulid0.so</code> (Linux), or <code>ulid0.dll</code> (Windows) file <a href="https://github.com/asg017/sqlite-ulid/releases">from a release</a> and load it into your SQLite environment.</p>
  <blockquote>
  <p dir="auto"><strong>Note:</strong>
  The <code>0</code> in the filename (<code>ulid0.dylib</code>/ <code>ulid0.so</code>/<code>ulid0.dll</code>) denotes the major version of <code>sqlite-ulid</code>. Currently <code>sqlite-ulid</code> is pre v1, so expect breaking changes in future versions.</p>
  </blockquote>
  <p dir="auto">For example, if you are using the <a href="https://www.sqlite.org/cli.html" rel="nofollow">SQLite CLI</a>, you can load the library like so:</p>
  <div class="highlight highlight-source-sql notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content=".load ./ulid0
  select ulid_version();
  -- v0.1.0"><pre>.load .<span class="pl-k">/</span>ulid0
  <span class="pl-k">select</span> ulid_version();
  <span class="pl-c"><span class="pl-c">--</span> v0.1.0</span></pre></div>
  <p dir="auto">In Python, you should prefer the <a href="./python/sqlite_ulid/README.md"><code>sqlite-ulid</code> Python package</a>. However, you can manually load a pre-compiled extension with the builtin <a href="https://docs.python.org/3/library/sqlite3.html" rel="nofollow">sqlite3 module</a>:</p>
  <div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="import sqlite3
  con = sqlite3.connect(":memory:")
  con.enable_load_extension(True)
  con.load_extension("./ulid0")
  print(con.execute("select ulid_version()").fetchone())
  # ('v0.1.0',)"><pre><span class="pl-k">import</span> <span class="pl-s1">sqlite3</span>
  <span class="pl-s1">con</span> <span class="pl-c1">=</span> <span class="pl-s1">sqlite3</span>.<span class="pl-c1">connect</span>(<span class="pl-s">":memory:"</span>)
  <span class="pl-s1">con</span>.<span class="pl-c1">enable_load_extension</span>(<span class="pl-c1">True</span>)
  <span class="pl-s1">con</span>.<span class="pl-c1">load_extension</span>(<span class="pl-s">"./ulid0"</span>)
  <span class="pl-en">print</span>(<span class="pl-s1">con</span>.<span class="pl-c1">execute</span>(<span class="pl-s">"select ulid_version()"</span>).<span class="pl-c1">fetchone</span>())
  <span class="pl-c"># ('v0.1.0',)</span></pre></div>
  <p dir="auto">Or in Node.js using <a href="https://github.com/WiseLibs/better-sqlite3">better-sqlite3</a>:</p>
  <div class="highlight highlight-source-js notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="const Database = require("better-sqlite3");
  const db = new Database(":memory:");
  db.loadExtension("./ulid0");
  console.log(db.prepare("select ulid_version()").get());
  // { 'ulid_version()': 'v0.1.0' }"><pre><span class="pl-k">const</span> <span class="pl-v">Database</span> <span class="pl-c1">=</span> <span class="pl-en">require</span><span class="pl-kos">(</span><span class="pl-s">"better-sqlite3"</span><span class="pl-kos">)</span><span class="pl-kos">;</span>
  <span class="pl-k">const</span> <span class="pl-s1">db</span> <span class="pl-c1">=</span> <span class="pl-k">new</span> <span class="pl-v">Database</span><span class="pl-kos">(</span><span class="pl-s">":memory:"</span><span class="pl-kos">)</span><span class="pl-kos">;</span>
  <span class="pl-s1">db</span><span class="pl-kos">.</span><span class="pl-en">loadExtension</span><span class="pl-kos">(</span><span class="pl-s">"./ulid0"</span><span class="pl-kos">)</span><span class="pl-kos">;</span>
  <span class="pl-smi">console</span><span class="pl-kos">.</span><span class="pl-en">log</span><span class="pl-kos">(</span><span class="pl-s1">db</span><span class="pl-kos">.</span><span class="pl-en">prepare</span><span class="pl-kos">(</span><span class="pl-s">"select ulid_version()"</span><span class="pl-kos">)</span><span class="pl-kos">.</span><span class="pl-en">get</span><span class="pl-kos">(</span><span class="pl-kos">)</span><span class="pl-kos">)</span><span class="pl-kos">;</span>
  <span class="pl-c">// { 'ulid_version()': 'v0.1.0' }</span></pre></div>
  <p dir="auto">With <a href="https://datasette.io/" rel="nofollow">Datasette</a>, you should prefer the <a href="./python/datasette_sqlite_ulid/README.md"><code>datasette-sqlite-ulid</code> Datasette plugin</a>. However, you can manually load a pre-compiled extension into a Datasette instance like so:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="datasette data.db --load-extension ./ulid0"><pre class="notranslate"><code>datasette data.db --load-extension ./ulid0
  </code></pre></div>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Supporting</h2><a id="user-content-supporting" class="anchor" aria-label="Permalink: Supporting" href="#supporting"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">I (Alex 👋🏼) spent a lot of time and energy on this project and <a href="https://github.com/asg017?tab=repositories&q=&type=&language=&sort=stargazers">many other open source projects</a>. If your company or organization uses this library (or you're feeling generous), then please <a href="https://alexgarcia.ulid/work.html" rel="nofollow">consider supporting my work</a>, or share this project with a friend!</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">See also</h2><a id="user-content-see-also" class="anchor" aria-label="Permalink: See also" href="#see-also"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <ul dir="auto">
  <li><a href="https://github.com/asg017/sqlite-xsv">sqlite-xsv</a>, A SQLite extension for working with CSVs</li>
  <li><a href="https://github.com/asg017/sqlite-loadable-rs">sqlite-loadable</a>, A framework for writing SQLite extensions in Rust</li>
  <li><a href="https://github.com/asg017/sqlite-http">sqlite-http</a>, A SQLite extension for making HTTP requests</li>
  </ul>
  </article></div>


node_id: R_kgDOItf8Rg
name: openai-to-sqlite
full_name: simonw/openai-to-sqlite
html_url: https://github.com/simonw/openai-to-sqlite
description: Save OpenAI API results to a SQLite database
created_at: 2023-01-03T01:14:22Z
updated_at: 2025-04-08T03:20:37Z
pushed_at: 2024-04-17T02:58:05Z
language: Python
license: apache-2.0
topics: []
visibility: public
default_branch: main
permissions: {"admin": false, "maintain": false, "push": false, "triage": false, "pull": true}
readme:
  # openai-to-sqlite

  [![PyPI](https://img.shields.io/pypi/v/openai-to-sqlite.svg)](https://pypi.org/project/openai-to-sqlite/)
  [![Changelog](https://img.shields.io/github/v/release/simonw/openai-to-sqlite?include_prereleases&label=changelog)](https://github.com/simonw/openai-to-sqlite/releases)
  [![Tests](https://github.com/simonw/openai-to-sqlite/workflows/Test/badge.svg)](https://github.com/simonw/openai-to-sqlite/actions?query=workflow%3ATest)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/openai-to-sqlite/blob/main/LICENSE)

  This tool provides utilities for interacting with OpenAI APIs and storing the results in a SQLite database.

  See [Semantic search answers: Q&A against documentation with GPT3 + OpenAI embeddings](https://simonwillison.net/2023/Jan/13/semantic-search-answers/) for background on this project.

  For a tutorial on using this for related content, see [Storing and serving related documents with openai-to-sqlite and embeddings](https://til.simonwillison.net/llms/openai-embeddings-related-content).

  ## Installation

  Install this tool using `pip`:
  ```bash
  pip install openai-to-sqlite
  ```
  ## Configuration

  You will need an OpenAI API key to use this tool.

  You can create one at https://beta.openai.com/account/api-keys

  You can then either set the API key as an environment variable:
  ```bash
  export OPENAI_API_KEY=sk-...
  ```
  Or pass it to each command using the `--token sk-...` option.

  ## Calling OpenAI APIs with SQL functions

  The `openai-to-sqlite query` command can be used to execute SQL queries that call OpenAI APIs.

  Functions available are:

  - `chatgpt(prompt)` - call the OpenAI Chat API using model `gpt-3.5-turbo` with the specified prompt.
  - `chatgpt(prompt, system)` - call that API with the prompt and the specified system prompt.

  More functions are planned in the future.

  Here's how to use this command to run basic sentiment analysis against content in a table:
  ```bash
  openai-to-sqlite query database.db "
    update messages set sentiment = chatgpt(
      'Sentiment analysis for this message: ' || message ||
      ' - ONLY return a lowercase string from: positive, negative, neutral, unknown'
    )
    where sentiment not in ('positive', 'negative', 'neutral', 'unknown')
      or sentiment is null
  "
  ```
  This updates the `sentiment` column in a table called `messages`. It populates it with the response from the specified prompt.

  The command will display a progress bar indicating how many rows are being processed.

  You can add an empty `sentiment` column to a table using [sqlite-utils](https://sqlite-utils.datasette.io/) like this:

  ```bash
  sqlite-utils add-column database.db messages sentiment
  ```

  ## Embeddings

  The `embeddings` command can be used to calculate and store [OpenAI embeddings](https://beta.openai.com/docs/guides/embeddings) for strings of text.

  Each embedding has a cost, so be sure to familiarize yourself with [the pricing](https://openai.com/api/pricing/) for the embedding model.

  The command can accept data in four different ways:

  - As a JSON file containing a list of objects
  - As a CSV file
  - As a TSV file
  - By running queries against a SQLite database

  For all of these formats there should be an `id` column, followed by one or more text columns.

  The ID will be stored as the content ID. Any other columns will be concatenated together and used as the text to be embedded.

  The embeddings from the API will then be saved as binary blobs in the `embeddings` table of the specified SQLite database - or another table, if you pass the `-t/--table` option.

  ### JSON, CSV and TSV

  Given a CSV file like this:
  ```csv
  id,content
  1,This is a test
  2,This is another test
  ```
  Embeddings can be stored like so:
  ```bash
  openai-to-sqlite embeddings embeddings.db data.csv
  ```
  The resulting schema looks like this:
  ```sql
  CREATE TABLE [embeddings] (
     [id] TEXT PRIMARY KEY,
     [embedding] BLOB
  );
  ```
  The same data can be provided as TSV data:
  ```
  id    content
  1     This is a test
  2     This is another test
  ```
  Then imported like this:
  ```bash
  openai-to-sqlite embeddings embeddings.db data.tsv
  ```
  Or as JSON data:
  ```json
  [
    {"id": 1, "content": "This is a test"},
    {"id": 2, "content": "This is another test"}
  ]
  ```
  Imported like this:
  ```bash
  openai-to-sqlite embeddings embeddings.db data.json
  ```
  In each of these cases the tool automatically detects the format of the data. It does this by inspecting the data itself - it does not consider the file extension.

  If the automatic detection is not working, you can pass `--format json`, `csv` or `tsv` to explicitly specify a format:

  ```bash
  openai-to-sqlite embeddings embeddings.db data.tsv --format tsv
  ```
  ### Importing data from standard input

  You can use a filename of `-` to pipe data in to standard input:

  ```bash
  cat data.tsv | openai-to-sqlite embeddings embeddings.db -
  ```

  ### Data from a SQL query

  The `--sql` option can be used to read data to be embedded from the attached SQLite database. The query must return an `id` column and one or more text columns to be embedded.

  ```bash
  openai-to-sqlite embeddings content.db \
    --sql "select id, title from documents"
  ```
  This will create a `embeddings` table in the `content.db` database and populate it with embeddings calculated from the `title` column in that query.

  You can also store embeddings in one database while reading data from another database, using the `--attach alias filename.db` option:

  ```bash
  openai-to-sqlite embeddings embeddings.db \
    --attach documents documents.db \
    --sql "select id, title from documents.documents"
  ```
  A progress bar will be displayed when using `--sql` that indicates how long the embeddings are likely to take to calculate.

  The CSV/TSV/JSON options do not correctly display the progress bar. You can work around this by importing your data into SQLite first (e.g. [using sqlite-utils](https://sqlite-utils.datasette.io/en/stable/cli.html#inserting-json-data)) and then running the embeddings using `--sql`.

  ### Batching

  Embeddings will be sent to the OpenAI embeddings API in batches of 100. If you know that your data is short strings you can increase the batch size, up to 2048, using the `--batch-size` option:

  ```bash
  openai-to-sqlite embeddings embeddings.db data.csv --batch-size 2048
  ```

  ### Working with the stored embeddings

  The `embedding` column is a SQLite blob containing 1536 floating point numbers encoded as a sequence of 4 byte values.

  You can extract them back to an array of floating point values in Python like this:
  ```python
  import struct

  vector = struct.unpack(
      "f" * 1536, binary_embedding
  )
  ```

  ### Searching embeddings with the search command

  Having saved the embeddings for content, you can run searches using the `search` command:
  ```bash
  openai-to-sqlite search embeddings.db 'this is my search term'
  ```
  The output will be a list of cosine similarity scores and content IDs:
  ```bash
  openai-to-sqlite search blog.db 'cool datasette demo'
  ```
  ```
  0.843 7849
  0.830 8036
  0.828 8195
  0.826 8098
  0.818 8086
  0.817 8171
  0.816 8121
  0.815 7860
  0.815 7872
  0.814 8169
  ```

  Add the `-t/--table` option if your embeddings are stored in a different table:
  ```bash
  openai-to-sqlite search content.db 'this is my search term' -t documents

  Add `--count 20` to retrieve 20 results (the default is 10).
  ```

  ### Search for similar content with the similar command

  Having saved the embeddings for content, you can search for similar content with the `similar` command:
  ```bash
  oopenai-to-sqlite similar embeddings.db '<content identifier>'
  ```
  The output will be a list of cosine similarity scores and content IDs:
  ```bash
  openai-to-sqlite similar embeddings-bjcp-2021.db '23G Gose'
  ```
  ```
  23G Gose
    1.000 23G Gose
    0.929 24A Witbier
    0.921 23A Berliner Weisse
    0.909 05B Kölsch
    0.907 01D American Wheat Beer
    0.906 27 Historical Beer: Lichtenhainer
    0.905 23D Lambic
    0.905 10A Weissbier
    0.904 04B Festbier
    0.904 01B American Lager
  ```
  You can pass more than one IDs to see similarities calculated for each one:
  ```bash
  openai-to-sqlite similar embeddings-bjcp-2021.db \
    '23G Gose' '01A American Light Lager'
  ```
  Or pass `--all` to run similarity for every item in the database. This runs similarity calculations for the number of items squared so it can be quite a long running operation:
  ```bash
  openai-to-sqlite similar embeddings-bjcp-2021.db --all
  ```

  ### Saving similarity calculations to the database

  To save these calculations to a `similarities` table in the database, use the `--save` option:
  ```bash
  openai-to-sqlite similar embeddings-bjcp-2021.db --all --save
  ```
  The `--save` option disables output. You can re-enable output with `--print`:
  ```bash
  openai-to-sqlite similar embeddings-bjcp-2021.db --all --save --print
  ```
  To save to a database table with a name other than `similarities`, use `--table`:
  ```bash
  openai-to-sqlite similar embeddings-bjcp-2021.db \
    --all --save --table my_similarities
  ```

  ### --recalculate-for-matches

  Re-calculating similarities for every row in the database can be quite a lengthy operation.

  If you know which rows have just been added, you can speed things up using `--recalculate-for-matches`.

  This tells `openai-to-sqlite similar` to only re-calculate similarities for rows that are close matches to the specified rows.

  This means you can add one or two additional records and then trigger an update of the saved similarity scores for just those new records plus for the twenty closest matches to those new records like this:

  ```bash
  openai-to-sqlite similar embeddings-bjcp-2021.db \
    --save '23G Gose' '01A American Light Lager' \
    --recalculate-for-matches \
    --count 20 \
    --print
  ```

  ## Development

  To contribute to this tool, first checkout the code. Then create a new virtual environment:
  ```bash
  cd openai-to-sqlite
  python -m venv venv
  source venv/bin/activate
  ```
  Now install the dependencies and test dependencies:
  ```bash
  pip install -e '.[test]'
  ```
  To run the tests:
  ```bash
  pytest
  ```

readme_html:
  <div id="readme" class="md" data-path="README.md"><article class="markdown-body entry-content container-lg" itemprop="text"><div class="markdown-heading" dir="auto"><h1 class="heading-element" dir="auto">openai-to-sqlite</h1><a id="user-content-openai-to-sqlite" class="anchor" aria-label="Permalink: openai-to-sqlite" href="#openai-to-sqlite"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto"><a href="https://pypi.org/project/openai-to-sqlite/" rel="nofollow"><img src="https://img.shields.io/pypi/v/openai-to-sqlite.svg" alt="PyPI" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/faad565aa877eae5c2d6bff6e06a6b49ee28a8ab14e5ecfc6af25233acc9729e/68747470733a2f2f696d672e736869656c64732e696f2f707970692f762f6f70656e61692d746f2d73716c6974652e737667"></a>
  <a href="https://github.com/simonw/openai-to-sqlite/releases"><img src="https://img.shields.io/github/v/release/simonw/openai-to-sqlite?include_prereleases&label=changelog" alt="Changelog" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/a295e994bca6ab4af595103f71975c03354cd8e2e1b94aaf83ef9cb9108cedad/68747470733a2f2f696d672e736869656c64732e696f2f6769746875622f762f72656c656173652f73696d6f6e772f6f70656e61692d746f2d73716c6974653f696e636c7564655f70726572656c6561736573266c6162656c3d6368616e67656c6f67"></a>
  <a href="https://github.com/simonw/openai-to-sqlite/actions?query=workflow%3ATest"><img src="https://github.com/simonw/openai-to-sqlite/workflows/Test/badge.svg" alt="Tests" style="max-width: 100%;"></a>
  <a href="https://github.com/simonw/openai-to-sqlite/blob/main/LICENSE"><img src="https://img.shields.io/badge/license-Apache%202.0-blue.svg" alt="License" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/c355f200ea90fddaa407b6eaab303663a669248ea3ca7b1fcf77dbe04ff5f48c/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6c6963656e73652d417061636865253230322e302d626c75652e737667"></a></p>
  <p dir="auto">This tool provides utilities for interacting with OpenAI APIs and storing the results in a SQLite database.</p>
  <p dir="auto">See <a href="https://simonwillison.net/2023/Jan/13/semantic-search-answers/" rel="nofollow">Semantic search answers: Q&A against documentation with GPT3 + OpenAI embeddings</a> for background on this project.</p>
  <p dir="auto">For a tutorial on using this for related content, see <a href="https://til.simonwillison.net/llms/openai-embeddings-related-content" rel="nofollow">Storing and serving related documents with openai-to-sqlite and embeddings</a>.</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Installation</h2><a id="user-content-installation" class="anchor" aria-label="Permalink: Installation" href="#installation"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">Install this tool using <code>pip</code>:</p>
  <div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="pip install openai-to-sqlite"><pre>pip install openai-to-sqlite</pre></div>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Configuration</h2><a id="user-content-configuration" class="anchor" aria-label="Permalink: Configuration" href="#configuration"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">You will need an OpenAI API key to use this tool.</p>
  <p dir="auto">You can create one at <a href="https://beta.openai.com/account/api-keys" rel="nofollow">https://beta.openai.com/account/api-keys</a></p>
  <p dir="auto">You can then either set the API key as an environment variable:</p>
  <div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="export OPENAI_API_KEY=sk-..."><pre><span class="pl-k">export</span> OPENAI_API_KEY=sk-...</pre></div>
  <p dir="auto">Or pass it to each command using the <code>--token sk-...</code> option.</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Calling OpenAI APIs with SQL functions</h2><a id="user-content-calling-openai-apis-with-sql-functions" class="anchor" aria-label="Permalink: Calling OpenAI APIs with SQL functions" href="#calling-openai-apis-with-sql-functions"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">The <code>openai-to-sqlite query</code> command can be used to execute SQL queries that call OpenAI APIs.</p>
  <p dir="auto">Functions available are:</p>
  <ul dir="auto">
  <li><code>chatgpt(prompt)</code> - call the OpenAI Chat API using model <code>gpt-3.5-turbo</code> with the specified prompt.</li>
  <li><code>chatgpt(prompt, system)</code> - call that API with the prompt and the specified system prompt.</li>
  </ul>
  <p dir="auto">More functions are planned in the future.</p>
  <p dir="auto">Here's how to use this command to run basic sentiment analysis against content in a table:</p>
  <div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="openai-to-sqlite query database.db "
    update messages set sentiment = chatgpt(
      'Sentiment analysis for this message: ' || message ||
      ' - ONLY return a lowercase string from: positive, negative, neutral, unknown'
    )
    where sentiment not in ('positive', 'negative', 'neutral', 'unknown')
      or sentiment is null
  ""><pre>openai-to-sqlite query database.db <span class="pl-s"><span class="pl-pds">"</span></span>
  <span class="pl-s">  update messages set sentiment = chatgpt(</span>
  <span class="pl-s">    'Sentiment analysis for this message: ' || message ||</span>
  <span class="pl-s">    ' - ONLY return a lowercase string from: positive, negative, neutral, unknown'</span>
  <span class="pl-s">  )</span>
  <span class="pl-s">  where sentiment not in ('positive', 'negative', 'neutral', 'unknown')</span>
  <span class="pl-s">    or sentiment is null</span>
  <span class="pl-s"><span class="pl-pds">"</span></span></pre></div>
  <p dir="auto">This updates the <code>sentiment</code> column in a table called <code>messages</code>. It populates it with the response from the specified prompt.</p>
  <p dir="auto">The command will display a progress bar indicating how many rows are being processed.</p>
  <p dir="auto">You can add an empty <code>sentiment</code> column to a table using <a href="https://sqlite-utils.datasette.io/" rel="nofollow">sqlite-utils</a> like this:</p>
  <div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="sqlite-utils add-column database.db messages sentiment"><pre>sqlite-utils add-column database.db messages sentiment</pre></div>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Embeddings</h2><a id="user-content-embeddings" class="anchor" aria-label="Permalink: Embeddings" href="#embeddings"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">The <code>embeddings</code> command can be used to calculate and store <a href="https://beta.openai.com/docs/guides/embeddings" rel="nofollow">OpenAI embeddings</a> for strings of text.</p>
  <p dir="auto">Each embedding has a cost, so be sure to familiarize yourself with <a href="https://openai.com/api/pricing/" rel="nofollow">the pricing</a> for the embedding model.</p>
  <p dir="auto">The command can accept data in four different ways:</p>
  <ul dir="auto">
  <li>As a JSON file containing a list of objects</li>
  <li>As a CSV file</li>
  <li>As a TSV file</li>
  <li>By running queries against a SQLite database</li>
  </ul>
  <p dir="auto">For all of these formats there should be an <code>id</code> column, followed by one or more text columns.</p>
  <p dir="auto">The ID will be stored as the content ID. Any other columns will be concatenated together and used as the text to be embedded.</p>
  <p dir="auto">The embeddings from the API will then be saved as binary blobs in the <code>embeddings</code> table of the specified SQLite database - or another table, if you pass the <code>-t/--table</code> option.</p>
  <div class="markdown-heading" dir="auto"><h3 class="heading-element" dir="auto">JSON, CSV and TSV</h3><a id="user-content-json-csv-and-tsv" class="anchor" aria-label="Permalink: JSON, CSV and TSV" href="#json-csv-and-tsv"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">Given a CSV file like this:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="id,content
  1,This is a test
  2,This is another test"><pre lang="csv" class="notranslate"><code>id,content
  1,This is a test
  2,This is another test
  </code></pre></div>
  <p dir="auto">Embeddings can be stored like so:</p>
  <div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="openai-to-sqlite embeddings embeddings.db data.csv"><pre>openai-to-sqlite embeddings embeddings.db data.csv</pre></div>
  <p dir="auto">The resulting schema looks like this:</p>
  <div class="highlight highlight-source-sql notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="CREATE TABLE [embeddings] (
     [id] TEXT PRIMARY KEY,
     [embedding] BLOB
  );"><pre>CREATE TABLE [embeddings] (
     [id] <span class="pl-k">TEXT</span> <span class="pl-k">PRIMARY KEY</span>,
     [embedding] BLOB
  );</pre></div>
  <p dir="auto">The same data can be provided as TSV data:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="id    content
  1     This is a test
  2     This is another test"><pre class="notranslate"><code>id    content
  1     This is a test
  2     This is another test
  </code></pre></div>
  <p dir="auto">Then imported like this:</p>
  <div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="openai-to-sqlite embeddings embeddings.db data.tsv"><pre>openai-to-sqlite embeddings embeddings.db data.tsv</pre></div>
  <p dir="auto">Or as JSON data:</p>
  <div class="highlight highlight-source-json notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="[
    {"id": 1, "content": "This is a test"},
    {"id": 2, "content": "This is another test"}
  ]"><pre>[
    {<span class="pl-ent">"id"</span>: <span class="pl-c1">1</span>, <span class="pl-ent">"content"</span>: <span class="pl-s"><span class="pl-pds">"</span>This is a test<span class="pl-pds">"</span></span>},
    {<span class="pl-ent">"id"</span>: <span class="pl-c1">2</span>, <span class="pl-ent">"content"</span>: <span class="pl-s"><span class="pl-pds">"</span>This is another test<span class="pl-pds">"</span></span>}
  ]</pre></div>
  <p dir="auto">Imported like this:</p>
  <div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="openai-to-sqlite embeddings embeddings.db data.json"><pre>openai-to-sqlite embeddings embeddings.db data.json</pre></div>
  <p dir="auto">In each of these cases the tool automatically detects the format of the data. It does this by inspecting the data itself - it does not consider the file extension.</p>
  <p dir="auto">If the automatic detection is not working, you can pass <code>--format json</code>, <code>csv</code> or <code>tsv</code> to explicitly specify a format:</p>
  <div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="openai-to-sqlite embeddings embeddings.db data.tsv --format tsv"><pre>openai-to-sqlite embeddings embeddings.db data.tsv --format tsv</pre></div>
  <div class="markdown-heading" dir="auto"><h3 class="heading-element" dir="auto">Importing data from standard input</h3><a id="user-content-importing-data-from-standard-input" class="anchor" aria-label="Permalink: Importing data from standard input" href="#importing-data-from-standard-input"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">You can use a filename of <code>-</code> to pipe data in to standard input:</p>
  <div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="cat data.tsv | openai-to-sqlite embeddings embeddings.db -"><pre>cat data.tsv <span class="pl-k">|</span> openai-to-sqlite embeddings embeddings.db -</pre></div>
  <div class="markdown-heading" dir="auto"><h3 class="heading-element" dir="auto">Data from a SQL query</h3><a id="user-content-data-from-a-sql-query" class="anchor" aria-label="Permalink: Data from a SQL query" href="#data-from-a-sql-query"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">The <code>--sql</code> option can be used to read data to be embedded from the attached SQLite database. The query must return an <code>id</code> column and one or more text columns to be embedded.</p>
  <div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="openai-to-sqlite embeddings content.db \
    --sql "select id, title from documents""><pre>openai-to-sqlite embeddings content.db \
    --sql <span class="pl-s"><span class="pl-pds">"</span>select id, title from documents<span class="pl-pds">"</span></span></pre></div>
  <p dir="auto">This will create a <code>embeddings</code> table in the <code>content.db</code> database and populate it with embeddings calculated from the <code>title</code> column in that query.</p>
  <p dir="auto">You can also store embeddings in one database while reading data from another database, using the <code>--attach alias filename.db</code> option:</p>
  <div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="openai-to-sqlite embeddings embeddings.db \
    --attach documents documents.db \
    --sql "select id, title from documents.documents""><pre>openai-to-sqlite embeddings embeddings.db \
    --attach documents documents.db \
    --sql <span class="pl-s"><span class="pl-pds">"</span>select id, title from documents.documents<span class="pl-pds">"</span></span></pre></div>
  <p dir="auto">A progress bar will be displayed when using <code>--sql</code> that indicates how long the embeddings are likely to take to calculate.</p>
  <p dir="auto">The CSV/TSV/JSON options do not correctly display the progress bar. You can work around this by importing your data into SQLite first (e.g. <a href="https://sqlite-utils.datasette.io/en/stable/cli.html#inserting-json-data" rel="nofollow">using sqlite-utils</a>) and then running the embeddings using <code>--sql</code>.</p>
  <div class="markdown-heading" dir="auto"><h3 class="heading-element" dir="auto">Batching</h3><a id="user-content-batching" class="anchor" aria-label="Permalink: Batching" href="#batching"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">Embeddings will be sent to the OpenAI embeddings API in batches of 100. If you know that your data is short strings you can increase the batch size, up to 2048, using the <code>--batch-size</code> option:</p>
  <div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="openai-to-sqlite embeddings embeddings.db data.csv --batch-size 2048"><pre>openai-to-sqlite embeddings embeddings.db data.csv --batch-size 2048</pre></div>
  <div class="markdown-heading" dir="auto"><h3 class="heading-element" dir="auto">Working with the stored embeddings</h3><a id="user-content-working-with-the-stored-embeddings" class="anchor" aria-label="Permalink: Working with the stored embeddings" href="#working-with-the-stored-embeddings"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">The <code>embedding</code> column is a SQLite blob containing 1536 floating point numbers encoded as a sequence of 4 byte values.</p>
  <p dir="auto">You can extract them back to an array of floating point values in Python like this:</p>
  <div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="import struct

  vector = struct.unpack(
      "f" * 1536, binary_embedding
  )"><pre><span class="pl-k">import</span> <span class="pl-s1">struct</span>

  <span class="pl-s1">vector</span> <span class="pl-c1">=</span> <span class="pl-s1">struct</span>.<span class="pl-c1">unpack</span>(
      <span class="pl-s">"f"</span> <span class="pl-c1">*</span> <span class="pl-c1">1536</span>, <span class="pl-s1">binary_embedding</span>
  )</pre></div>
  <div class="markdown-heading" dir="auto"><h3 class="heading-element" dir="auto">Searching embeddings with the search command</h3><a id="user-content-searching-embeddings-with-the-search-command" class="anchor" aria-label="Permalink: Searching embeddings with the search command" href="#searching-embeddings-with-the-search-command"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">Having saved the embeddings for content, you can run searches using the <code>search</code> command:</p>
  <div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="openai-to-sqlite search embeddings.db 'this is my search term'"><pre>openai-to-sqlite search embeddings.db <span class="pl-s"><span class="pl-pds">'</span>this is my search term<span class="pl-pds">'</span></span></pre></div>
  <p dir="auto">The output will be a list of cosine similarity scores and content IDs:</p>
  <div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="openai-to-sqlite search blog.db 'cool datasette demo'"><pre>openai-to-sqlite search blog.db <span class="pl-s"><span class="pl-pds">'</span>cool datasette demo<span class="pl-pds">'</span></span></pre></div>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="0.843 7849
  0.830 8036
  0.828 8195
  0.826 8098
  0.818 8086
  0.817 8171
  0.816 8121
  0.815 7860
  0.815 7872
  0.814 8169"><pre class="notranslate"><code>0.843 7849
  0.830 8036
  0.828 8195
  0.826 8098
  0.818 8086
  0.817 8171
  0.816 8121
  0.815 7860
  0.815 7872
  0.814 8169
  </code></pre></div>
  <p dir="auto">Add the <code>-t/--table</code> option if your embeddings are stored in a different table:</p>
  <div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="openai-to-sqlite search content.db 'this is my search term' -t documents

  Add `--count 20` to retrieve 20 results (the default is 10)."><pre>openai-to-sqlite search content.db <span class="pl-s"><span class="pl-pds">'</span>this is my search term<span class="pl-pds">'</span></span> -t documents

  Add <span class="pl-s"><span class="pl-pds">`</span>--count 20<span class="pl-pds">`</span></span> to retrieve 20 results (the default is 10).</pre></div>
  <div class="markdown-heading" dir="auto"><h3 class="heading-element" dir="auto">Search for similar content with the similar command</h3><a id="user-content-search-for-similar-content-with-the-similar-command" class="anchor" aria-label="Permalink: Search for similar content with the similar command" href="#search-for-similar-content-with-the-similar-command"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">Having saved the embeddings for content, you can search for similar content with the <code>similar</code> command:</p>
  <div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="oopenai-to-sqlite similar embeddings.db '<content identifier>'"><pre>oopenai-to-sqlite similar embeddings.db <span class="pl-s"><span class="pl-pds">'</span><content identifier><span class="pl-pds">'</span></span></pre></div>
  <p dir="auto">The output will be a list of cosine similarity scores and content IDs:</p>
  <div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="openai-to-sqlite similar embeddings-bjcp-2021.db '23G Gose'"><pre>openai-to-sqlite similar embeddings-bjcp-2021.db <span class="pl-s"><span class="pl-pds">'</span>23G Gose<span class="pl-pds">'</span></span></pre></div>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="23G Gose
    1.000 23G Gose
    0.929 24A Witbier
    0.921 23A Berliner Weisse
    0.909 05B Kölsch
    0.907 01D American Wheat Beer
    0.906 27 Historical Beer: Lichtenhainer
    0.905 23D Lambic
    0.905 10A Weissbier
    0.904 04B Festbier
    0.904 01B American Lager"><pre class="notranslate"><code>23G Gose
    1.000 23G Gose
    0.929 24A Witbier
    0.921 23A Berliner Weisse
    0.909 05B Kölsch
    0.907 01D American Wheat Beer
    0.906 27 Historical Beer: Lichtenhainer
    0.905 23D Lambic
    0.905 10A Weissbier
    0.904 04B Festbier
    0.904 01B American Lager
  </code></pre></div>
  <p dir="auto">You can pass more than one IDs to see similarities calculated for each one:</p>
  <div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="openai-to-sqlite similar embeddings-bjcp-2021.db \
    '23G Gose' '01A American Light Lager'"><pre>openai-to-sqlite similar embeddings-bjcp-2021.db \
    <span class="pl-s"><span class="pl-pds">'</span>23G Gose<span class="pl-pds">'</span></span> <span class="pl-s"><span class="pl-pds">'</span>01A American Light Lager<span class="pl-pds">'</span></span></pre></div>
  <p dir="auto">Or pass <code>--all</code> to run similarity for every item in the database. This runs similarity calculations for the number of items squared so it can be quite a long running operation:</p>
  <div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="openai-to-sqlite similar embeddings-bjcp-2021.db --all"><pre>openai-to-sqlite similar embeddings-bjcp-2021.db --all</pre></div>
  <div class="markdown-heading" dir="auto"><h3 class="heading-element" dir="auto">Saving similarity calculations to the database</h3><a id="user-content-saving-similarity-calculations-to-the-database" class="anchor" aria-label="Permalink: Saving similarity calculations to the database" href="#saving-similarity-calculations-to-the-database"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">To save these calculations to a <code>similarities</code> table in the database, use the <code>--save</code> option:</p>
  <div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="openai-to-sqlite similar embeddings-bjcp-2021.db --all --save"><pre>openai-to-sqlite similar embeddings-bjcp-2021.db --all --save</pre></div>
  <p dir="auto">The <code>--save</code> option disables output. You can re-enable output with <code>--print</code>:</p>
  <div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="openai-to-sqlite similar embeddings-bjcp-2021.db --all --save --print"><pre>openai-to-sqlite similar embeddings-bjcp-2021.db --all --save --print</pre></div>
  <p dir="auto">To save to a database table with a name other than <code>similarities</code>, use <code>--table</code>:</p>
  <div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="openai-to-sqlite similar embeddings-bjcp-2021.db \
    --all --save --table my_similarities"><pre>openai-to-sqlite similar embeddings-bjcp-2021.db \
    --all --save --table my_similarities</pre></div>
  <div class="markdown-heading" dir="auto"><h3 class="heading-element" dir="auto">--recalculate-for-matches</h3><a id="user-content---recalculate-for-matches" class="anchor" aria-label="Permalink: --recalculate-for-matches" href="#--recalculate-for-matches"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">Re-calculating similarities for every row in the database can be quite a lengthy operation.</p>
  <p dir="auto">If you know which rows have just been added, you can speed things up using <code>--recalculate-for-matches</code>.</p>
  <p dir="auto">This tells <code>openai-to-sqlite similar</code> to only re-calculate similarities for rows that are close matches to the specified rows.</p>
  <p dir="auto">This means you can add one or two additional records and then trigger an update of the saved similarity scores for just those new records plus for the twenty closest matches to those new records like this:</p>
  <div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="openai-to-sqlite similar embeddings-bjcp-2021.db \
    --save '23G Gose' '01A American Light Lager' \
    --recalculate-for-matches \
    --count 20 \
    --print"><pre>openai-to-sqlite similar embeddings-bjcp-2021.db \
    --save <span class="pl-s"><span class="pl-pds">'</span>23G Gose<span class="pl-pds">'</span></span> <span class="pl-s"><span class="pl-pds">'</span>01A American Light Lager<span class="pl-pds">'</span></span> \
    --recalculate-for-matches \
    --count 20 \
    --print</pre></div>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Development</h2><a id="user-content-development" class="anchor" aria-label="Permalink: Development" href="#development"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">To contribute to this tool, first checkout the code. Then create a new virtual environment:</p>
  <div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="cd openai-to-sqlite
  python -m venv venv
  source venv/bin/activate"><pre><span class="pl-c1">cd</span> openai-to-sqlite
  python -m venv venv
  <span class="pl-c1">source</span> venv/bin/activate</pre></div>
  <p dir="auto">Now install the dependencies and test dependencies:</p>
  <div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="pip install -e '.[test]'"><pre>pip install -e <span class="pl-s"><span class="pl-pds">'</span>.[test]<span class="pl-pds">'</span></span></pre></div>
  <p dir="auto">To run the tests:</p>
  <div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="pytest"><pre>pytest</pre></div>
  </article></div>


node_id: R_kgDOItymQA
name: datasette-openai
full_name: simonw/datasette-openai
html_url: https://github.com/simonw/datasette-openai
description: SQL functions for calling OpenAI APIs
created_at: 2023-01-03T19:12:52Z
updated_at: 2024-09-04T07:37:30Z
pushed_at: 2023-01-14T02:29:43Z
homepage: https://datasette.io/plugins/datasette-openai
language: Python
license: apache-2.0
topics: ["datasette", "datasette-plugin", "gpt-3", "openai"]
visibility: public
default_branch: main
permissions: {"admin": false, "maintain": false, "push": false, "triage": false, "pull": true}
readme:
  # datasette-openai

  [![PyPI](https://img.shields.io/pypi/v/datasette-openai.svg)](https://pypi.org/project/datasette-openai/)
  [![Changelog](https://img.shields.io/github/v/release/simonw/datasette-openai?include_prereleases&label=changelog)](https://github.com/simonw/datasette-openai/releases)
  [![Tests](https://github.com/simonw/datasette-openai/workflows/Test/badge.svg)](https://github.com/simonw/datasette-openai/actions?query=workflow%3ATest)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/datasette-openai/blob/main/LICENSE)

  SQL functions for calling OpenAI APIs

  See [Semantic search answers: Q&A against documentation with GPT3 + OpenAI embeddings](https://simonwillison.net/2023/Jan/13/semantic-search-answers/) for background on this project.

  ## Installation

  Install this plugin in the same environment as Datasette.

      datasette install datasette-openai

  ## ⚠️ Warning ⚠️

  This plugin allows you to call a commercial, priced API using SQL queries.

  Use this with care! You could accidentally spend a lot of money.

  For example, the following query:

  ```sql
  select
    openai_davinci(
      'Summarize this text: ' || content, 200, 1, :api_key
  ) as summary
  from documents
  ```
  Would execute one paid API call for every item in the `documents` database. This could become very expensive.

  Be sure to familiarize yourself with [OpenAI pricing](https://openai.com/api/pricing/). You will need to obtain an [API key](https://beta.openai.com/account/api-keys).

  ## Usage

  This extension provides three new SQL functions:

  ### openai_davinci(prompt, max_tokens, temperature, api_key)

  This function runs a `text-davinci-003` completion against the provided prompt, with the specified values for max tokens and temperature.

  Da Vinci is currently 2 cents per thousand tokens.

  ### openai_embedding(text, api_key)

  This calls the OpenAI embedding endpoint and returns a binary object representing the floating point embedding for the provided text.

  ```sql
  select openai_embedding(:query, :api_key)
  ```
  An embedding is an array of 1536 floating point values. The returned value from this is a `blob` encoding of those values.

  It's mainly useful for using with the `openai_embedding_similarity()` function.

  The embedding API is very inexpensive: at time of writing, $0.0004 cents per thousand tokens, where a token is more-or-less a single word.

  ### openai_embedding_similarity(a, b)

  This function does not make any API calls. It takes two embedding blobs and returns the cosine similarity between the two.

  This function is particularly useful if you have stored embeddings of documents in a database table, and you want to find the most similar documents to a query or to another document.

  A simple search query could look like this:
  ```sql
  with query as (
    select
      openai_embedding(:query, :token) as q
  )
  select
    id,
    title,
    openai_embedding_similarity(query.q, embedding) as score
  from
    content, query
  order by
    score desc
  limit 10
  ```

  ### openai_build_prompt(text, prefix, suffix, completion_tokens, token_limit=4000)

  This aggregate function helps build a prompt from a number of inputs in a way that fits the GPT-3 prompt size limit.

  It takes the following argument:

  - `text` - this is the column that is being aggregated, so the function expects to have multiple values for this. All other arguments will only be read the first time they are passed, so should be consistent across all calls to the function.
  - `prefix` - text to use for the prefix of the prompt
  - `suffix` - text to use for the suffix of the prompt
  - `completion_tokens` - the number of tokens to reserve for the prompt response - this will be subtracted from the token limit
  - `token_limit` - this value is optional (there are 4-argument and 5-argument versions of the function registered). It defaults to the GPT-3 Da Vinci size limit of 4,000 tokens but can be changed if the model the prompt is being used with has a different size limit.

  Here's an example usage of this function, adapted from [this article](https://simonwillison.net/2023/Jan/13/semantic-search-answers/):

  ```sql
  with top_n as (
    select body from blog_entry order by id desc limit 3
  )
  select openai_build_prompt(body, 'Context:
  ------------
  ', '
  ------------
  Given the above context, answer the following question: ' || :question,
    500,
    2000
    ) from top_n
  ```
  [Try that here](https://datasette.simonwillison.net/simonwillisonblog?sql=with+top_n+as+%28%0D%0A++select+body+from+blog_entry+order+by+id+desc+limit+5%0D%0A%29%0D%0Aselect+openai_build_prompt%28body%2C+%27Context%3A%0D%0A------------%0D%0A%27%2C+%27%0D%0A------------%0D%0AGiven+the+above+context%2C+answer+the+following+question%3A+%27+%7C%7C+%3Aquestion%2C%0D%0A++500%2C%0D%0A++2000%0D%0A++%29+from+top_n&question=Examples+of+a+language+model%3F).

  This query first retrieves the three most recent blog entries, then constructs a prompt that with the provided prefix and suffix designed to fit 1500 tokens (2000 total, minus 500 reserved for the response).

  The output looks something like this (truncated for space):

  ```
  Context:
  ------------
  < p > If you 've spent any time with GPT - 3 or ChatGPT , you 've likely thought about how ...
  I release Datasette 0 . 64 this morning . This release is mainly a response to the realization that it 's not safe to run Datasette with the SpatiaLite extension loaded if that Datasette instance is configured to enable arbitrary SQL queries from untrusted users ...
  In lieu of my regular weeknotes ( I took two weeks off for the holidays ) here 's a look back at 2022 , mainly in terms of projects and things I 've written about ...
  ------------
  Given the above context, answer the following question: Examples of a language model?
  ```
  The body of each entry has been truncated to the number of tokens that will allow examples from all three entries to be included in the generated prompt.

  ### openai_strip_tags(text)

  Sometimes it can be useful to strip HTML tags from text in order to reduce the number of tokens used. This function does a very simple version of tag stripping - just removing anything that matches `<...>`.

  ### openai_tokenize(text)

  Returns a JSON array of tokens for the provided text.

  This uses a regular expression [extracted from OpenAI's GPT-2](https://github.com/openai/gpt-2/blob/a74da5d99abaaba920de8131d64da2862a8f213b/src/encoder.py#L53).

  ### openai_count_tokens(text)

  Returns a count of the number of tokens in the provided text.

  ## Development

  To set up this plugin locally, first checkout the code. Then create a new virtual environment:

      cd datasette-openai
      python3 -m venv venv
      source venv/bin/activate

  Now install the dependencies and test dependencies:

      pip install -e '.[test]'

  To run the tests:

      pytest

readme_html:
  <div id="readme" class="md" data-path="README.md"><article class="markdown-body entry-content container-lg" itemprop="text"><div class="markdown-heading" dir="auto"><h1 class="heading-element" dir="auto">datasette-openai</h1><a id="user-content-datasette-openai" class="anchor" aria-label="Permalink: datasette-openai" href="#datasette-openai"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto"><a href="https://pypi.org/project/datasette-openai/" rel="nofollow"><img src="https://img.shields.io/pypi/v/datasette-openai.svg" alt="PyPI" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/6176f84ae16e7ebc3b10248d7015a340443c7c8268e3e7b05632b7e2397ae5b5/68747470733a2f2f696d672e736869656c64732e696f2f707970692f762f6461746173657474652d6f70656e61692e737667"></a>
  <a href="https://github.com/simonw/datasette-openai/releases"><img src="https://img.shields.io/github/v/release/simonw/datasette-openai?include_prereleases&label=changelog" alt="Changelog" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/2dfc04e2b76872783c5855899ee2cd0a1224f5d49bed4d422dc552f8e68286ba/68747470733a2f2f696d672e736869656c64732e696f2f6769746875622f762f72656c656173652f73696d6f6e772f6461746173657474652d6f70656e61693f696e636c7564655f70726572656c6561736573266c6162656c3d6368616e67656c6f67"></a>
  <a href="https://github.com/simonw/datasette-openai/actions?query=workflow%3ATest"><img src="https://github.com/simonw/datasette-openai/workflows/Test/badge.svg" alt="Tests" style="max-width: 100%;"></a>
  <a href="https://github.com/simonw/datasette-openai/blob/main/LICENSE"><img src="https://img.shields.io/badge/license-Apache%202.0-blue.svg" alt="License" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/c355f200ea90fddaa407b6eaab303663a669248ea3ca7b1fcf77dbe04ff5f48c/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6c6963656e73652d417061636865253230322e302d626c75652e737667"></a></p>
  <p dir="auto">SQL functions for calling OpenAI APIs</p>
  <p dir="auto">See <a href="https://simonwillison.net/2023/Jan/13/semantic-search-answers/" rel="nofollow">Semantic search answers: Q&A against documentation with GPT3 + OpenAI embeddings</a> for background on this project.</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Installation</h2><a id="user-content-installation" class="anchor" aria-label="Permalink: Installation" href="#installation"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">Install this plugin in the same environment as Datasette.</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="datasette install datasette-openai"><pre class="notranslate"><code>datasette install datasette-openai
  </code></pre></div>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto"><g-emoji class="g-emoji" alias="warning">⚠️</g-emoji> Warning <g-emoji class="g-emoji" alias="warning">⚠️</g-emoji></h2><a id="user-content-️-warning-️" class="anchor" aria-label="Permalink: ⚠️ Warning ⚠️" href="#️-warning-️"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">This plugin allows you to call a commercial, priced API using SQL queries.</p>
  <p dir="auto">Use this with care! You could accidentally spend a lot of money.</p>
  <p dir="auto">For example, the following query:</p>
  <div class="highlight highlight-source-sql notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="select
    openai_davinci(
      'Summarize this text: ' || content, 200, 1, :api_key
  ) as summary
  from documents"><pre><span class="pl-k">select</span>
    openai_davinci(
      <span class="pl-s"><span class="pl-pds">'</span>Summarize this text: <span class="pl-pds">'</span></span> <span class="pl-k">||</span> content, <span class="pl-c1">200</span>, <span class="pl-c1">1</span>, :api_key
  ) <span class="pl-k">as</span> summary
  <span class="pl-k">from</span> documents</pre></div>
  <p dir="auto">Would execute one paid API call for every item in the <code>documents</code> database. This could become very expensive.</p>
  <p dir="auto">Be sure to familiarize yourself with <a href="https://openai.com/api/pricing/" rel="nofollow">OpenAI pricing</a>. You will need to obtain an <a href="https://beta.openai.com/account/api-keys" rel="nofollow">API key</a>.</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Usage</h2><a id="user-content-usage" class="anchor" aria-label="Permalink: Usage" href="#usage"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">This extension provides three new SQL functions:</p>
  <div class="markdown-heading" dir="auto"><h3 class="heading-element" dir="auto">openai_davinci(prompt, max_tokens, temperature, api_key)</h3><a id="user-content-openai_davinciprompt-max_tokens-temperature-api_key" class="anchor" aria-label="Permalink: openai_davinci(prompt, max_tokens, temperature, api_key)" href="#openai_davinciprompt-max_tokens-temperature-api_key"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">This function runs a <code>text-davinci-003</code> completion against the provided prompt, with the specified values for max tokens and temperature.</p>
  <p dir="auto">Da Vinci is currently 2 cents per thousand tokens.</p>
  <div class="markdown-heading" dir="auto"><h3 class="heading-element" dir="auto">openai_embedding(text, api_key)</h3><a id="user-content-openai_embeddingtext-api_key" class="anchor" aria-label="Permalink: openai_embedding(text, api_key)" href="#openai_embeddingtext-api_key"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">This calls the OpenAI embedding endpoint and returns a binary object representing the floating point embedding for the provided text.</p>
  <div class="highlight highlight-source-sql notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="select openai_embedding(:query, :api_key)"><pre><span class="pl-k">select</span> openai_embedding(:query, :api_key)</pre></div>
  <p dir="auto">An embedding is an array of 1536 floating point values. The returned value from this is a <code>blob</code> encoding of those values.</p>
  <p dir="auto">It's mainly useful for using with the <code>openai_embedding_similarity()</code> function.</p>
  <p dir="auto">The embedding API is very inexpensive: at time of writing, $0.0004 cents per thousand tokens, where a token is more-or-less a single word.</p>
  <div class="markdown-heading" dir="auto"><h3 class="heading-element" dir="auto">openai_embedding_similarity(a, b)</h3><a id="user-content-openai_embedding_similaritya-b" class="anchor" aria-label="Permalink: openai_embedding_similarity(a, b)" href="#openai_embedding_similaritya-b"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">This function does not make any API calls. It takes two embedding blobs and returns the cosine similarity between the two.</p>
  <p dir="auto">This function is particularly useful if you have stored embeddings of documents in a database table, and you want to find the most similar documents to a query or to another document.</p>
  <p dir="auto">A simple search query could look like this:</p>
  <div class="highlight highlight-source-sql notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="with query as (
    select
      openai_embedding(:query, :token) as q
  )
  select
    id,
    title,
    openai_embedding_similarity(query.q, embedding) as score
  from
    content, query
  order by
    score desc
  limit 10"><pre>with query <span class="pl-k">as</span> (
    <span class="pl-k">select</span>
      openai_embedding(:query, :token) <span class="pl-k">as</span> q
  )
  <span class="pl-k">select</span>
    id,
    title,
    openai_embedding_similarity(<span class="pl-c1">query</span>.<span class="pl-c1">q</span>, embedding) <span class="pl-k">as</span> score
  <span class="pl-k">from</span>
    content, query
  <span class="pl-k">order by</span>
    score <span class="pl-k">desc</span>
  <span class="pl-k">limit</span> <span class="pl-c1">10</span></pre></div>
  <div class="markdown-heading" dir="auto"><h3 class="heading-element" dir="auto">openai_build_prompt(text, prefix, suffix, completion_tokens, token_limit=4000)</h3><a id="user-content-openai_build_prompttext-prefix-suffix-completion_tokens-token_limit4000" class="anchor" aria-label="Permalink: openai_build_prompt(text, prefix, suffix, completion_tokens, token_limit=4000)" href="#openai_build_prompttext-prefix-suffix-completion_tokens-token_limit4000"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">This aggregate function helps build a prompt from a number of inputs in a way that fits the GPT-3 prompt size limit.</p>
  <p dir="auto">It takes the following argument:</p>
  <ul dir="auto">
  <li><code>text</code> - this is the column that is being aggregated, so the function expects to have multiple values for this. All other arguments will only be read the first time they are passed, so should be consistent across all calls to the function.</li>
  <li><code>prefix</code> - text to use for the prefix of the prompt</li>
  <li><code>suffix</code> - text to use for the suffix of the prompt</li>
  <li><code>completion_tokens</code> - the number of tokens to reserve for the prompt response - this will be subtracted from the token limit</li>
  <li><code>token_limit</code> - this value is optional (there are 4-argument and 5-argument versions of the function registered). It defaults to the GPT-3 Da Vinci size limit of 4,000 tokens but can be changed if the model the prompt is being used with has a different size limit.</li>
  </ul>
  <p dir="auto">Here's an example usage of this function, adapted from <a href="https://simonwillison.net/2023/Jan/13/semantic-search-answers/" rel="nofollow">this article</a>:</p>
  <div class="highlight highlight-source-sql notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="with top_n as (
    select body from blog_entry order by id desc limit 3
  )
  select openai_build_prompt(body, 'Context:
  ------------
  ', '
  ------------
  Given the above context, answer the following question: ' || :question,
    500,
    2000
    ) from top_n"><pre>with top_n <span class="pl-k">as</span> (
    <span class="pl-k">select</span> body <span class="pl-k">from</span> blog_entry <span class="pl-k">order by</span> id <span class="pl-k">desc</span> <span class="pl-k">limit</span> <span class="pl-c1">3</span>
  )
  <span class="pl-k">select</span> openai_build_prompt(body, <span class="pl-s"><span class="pl-pds">'</span>Context:</span>
  <span class="pl-s">------------</span>
  <span class="pl-s"><span class="pl-pds">'</span></span>, <span class="pl-s"><span class="pl-pds">'</span></span>
  <span class="pl-s">------------</span>
  <span class="pl-s">Given the above context, answer the following question: <span class="pl-pds">'</span></span> <span class="pl-k">||</span> :question,
    <span class="pl-c1">500</span>,
    <span class="pl-c1">2000</span>
    ) <span class="pl-k">from</span> top_n</pre></div>
  <p dir="auto"><a href="https://datasette.simonwillison.net/simonwillisonblog?sql=with+top_n+as+%28%0D%0A++select+body+from+blog_entry+order+by+id+desc+limit+5%0D%0A%29%0D%0Aselect+openai_build_prompt%28body%2C+%27Context%3A%0D%0A------------%0D%0A%27%2C+%27%0D%0A------------%0D%0AGiven+the+above+context%2C+answer+the+following+question%3A+%27+%7C%7C+%3Aquestion%2C%0D%0A++500%2C%0D%0A++2000%0D%0A++%29+from+top_n&question=Examples+of+a+language+model%3F" rel="nofollow">Try that here</a>.</p>
  <p dir="auto">This query first retrieves the three most recent blog entries, then constructs a prompt that with the provided prefix and suffix designed to fit 1500 tokens (2000 total, minus 500 reserved for the response).</p>
  <p dir="auto">The output looks something like this (truncated for space):</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="Context:
  ------------
  < p > If you 've spent any time with GPT - 3 or ChatGPT , you 've likely thought about how ...
  I release Datasette 0 . 64 this morning . This release is mainly a response to the realization that it 's not safe to run Datasette with the SpatiaLite extension loaded if that Datasette instance is configured to enable arbitrary SQL queries from untrusted users ...
  In lieu of my regular weeknotes ( I took two weeks off for the holidays ) here 's a look back at 2022 , mainly in terms of projects and things I 've written about ...
  ------------
  Given the above context, answer the following question: Examples of a language model?"><pre class="notranslate"><code>Context:
  ------------
  < p > If you 've spent any time with GPT - 3 or ChatGPT , you 've likely thought about how ...
  I release Datasette 0 . 64 this morning . This release is mainly a response to the realization that it 's not safe to run Datasette with the SpatiaLite extension loaded if that Datasette instance is configured to enable arbitrary SQL queries from untrusted users ...
  In lieu of my regular weeknotes ( I took two weeks off for the holidays ) here 's a look back at 2022 , mainly in terms of projects and things I 've written about ...
  ------------
  Given the above context, answer the following question: Examples of a language model?
  </code></pre></div>
  <p dir="auto">The body of each entry has been truncated to the number of tokens that will allow examples from all three entries to be included in the generated prompt.</p>
  <div class="markdown-heading" dir="auto"><h3 class="heading-element" dir="auto">openai_strip_tags(text)</h3><a id="user-content-openai_strip_tagstext" class="anchor" aria-label="Permalink: openai_strip_tags(text)" href="#openai_strip_tagstext"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">Sometimes it can be useful to strip HTML tags from text in order to reduce the number of tokens used. This function does a very simple version of tag stripping - just removing anything that matches <code><...></code>.</p>
  <div class="markdown-heading" dir="auto"><h3 class="heading-element" dir="auto">openai_tokenize(text)</h3><a id="user-content-openai_tokenizetext" class="anchor" aria-label="Permalink: openai_tokenize(text)" href="#openai_tokenizetext"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">Returns a JSON array of tokens for the provided text.</p>
  <p dir="auto">This uses a regular expression <a href="https://github.com/openai/gpt-2/blob/a74da5d99abaaba920de8131d64da2862a8f213b/src/encoder.py#L53">extracted from OpenAI's GPT-2</a>.</p>
  <div class="markdown-heading" dir="auto"><h3 class="heading-element" dir="auto">openai_count_tokens(text)</h3><a id="user-content-openai_count_tokenstext" class="anchor" aria-label="Permalink: openai_count_tokens(text)" href="#openai_count_tokenstext"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">Returns a count of the number of tokens in the provided text.</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Development</h2><a id="user-content-development" class="anchor" aria-label="Permalink: Development" href="#development"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">To set up this plugin locally, first checkout the code. Then create a new virtual environment:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="cd datasette-openai
  python3 -m venv venv
  source venv/bin/activate"><pre class="notranslate"><code>cd datasette-openai
  python3 -m venv venv
  source venv/bin/activate
  </code></pre></div>
  <p dir="auto">Now install the dependencies and test dependencies:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="pip install -e '.[test]'"><pre class="notranslate"><code>pip install -e '.[test]'
  </code></pre></div>
  <p dir="auto">To run the tests:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="pytest"><pre class="notranslate"><code>pytest
  </code></pre></div>
  </article></div>


node_id: R_kgDOIt2TLQ
name: sqlite-vss
full_name: asg017/sqlite-vss
html_url: https://github.com/asg017/sqlite-vss
description: A SQLite extension for efficient vector search, based on Faiss!
created_at: 2023-01-03T23:27:31Z
updated_at: 2025-05-12T18:41:33Z
pushed_at: 2024-05-05T20:00:16Z
language: C++
license: mit
topics: ["faiss", "sqlite", "sqlite-extension", "vector-search"]
visibility: public
default_branch: main
permissions: {"admin": false, "maintain": false, "push": false, "triage": false, "pull": true}
readme:
  # sqlite-vss


  > [!WARNING]  
  > `sqlite-vss` is not in active development. Instead, my effort is now going towards [`sqlite-vec`](https://github.com/asg017/sqlite-vec), which is a similar vector search SQLite extension, but should be much easier to install and use than `sqlite-vss`. See [this blog post](https://alexgarcia.xyz/blog/2024/building-new-vector-search-sqlite/index.html) for more info.
  >

  `sqlite-vss` (SQLite <b><u>V</u></b>ector <b><u>S</u></b>imilarity <b><u>S</u></b>earch) is a SQLite extension that brings vector search capabilities to SQLite, based on [Faiss](https://faiss.ai/). It can be used to build semantic search engines, recommendations, or questions-and-answering tools.

  See [_Introducing sqlite-vss: A SQLite Extension for Vector Search_](https://observablehq.com/@asg017/introducing-sqlite-vss) (February 2023) for more details and a live example!

  If your company or organization finds this library useful, consider [supporting my work](#supporting)!

  ## Usage

  ```sql
  .load ./vector0
  .load ./vss0

  select vss_version(); -- 'v0.0.1'

  ```

  `sqlite-vss` has a similar API to the [`fts5` Full-Text Search Extension](https://www.sqlite.org/fts5.html). Use the `vss0` module to create virtual tables that can efficiently store and query your vectors.

  ```sql
  -- 384 == number of dimensions for this example
  create virtual table vss_articles using vss0(
    headline_embedding(384),
    description_embedding(384),
  );
  ```

  `sqlite-vss` is a **Bring-your-own-vectors** database, it is compatable with any embedding or vector data you have. Consider using [OpenAI's Embeddings API](https://platform.openai.com/docs/guides/embeddings), [HuggingFace's Inference API](https://huggingface.co/blog/getting-started-with-embeddings#1-embedding-a-dataset), [`sentence-transformers`](https://www.sbert.net/), or [any of these open source model](https://github.com/embeddings-benchmark/mteb#leaderboard). In this example, we are using [sentence-transformers/all-MiniLM-L6-v2](https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2) to generate embeddings from our text, which have 384 dimensions.

  You can insert vectors into `vss0` tables as JSON or raw bytes.

  ```sql
  insert into vss_articles(rowid, headline_embedding)
    select rowid, headline_embedding from articles;
  ```

  To query for similar vectors ("k nearest neighbors"), use the `vss_search` function in the `WHERE` clause. Here we are searching for the 100 nearest neighbors to the embedding in row #123 in the `articles` table.

  ```sql
  select rowid, distance
  from vss_articles
  where vss_search(
    headline_embedding,
    (select headline_embedding from articles where rowid = 123)
  )
  limit 100;
  ```

  You can `INSERT` and `DELETE` into these tables as necessary, but [`UPDATE` operations aren't supported yet](https://github.com/asg017/sqlite-vss/issues/7). This can be used with triggers for automatically updated indexes. Also note that "small" `INSERT`/`DELETE` operations that only insert a few rows can be slow, so batch where necessary.

  ```sql
  begin;

  delete from vss_articles
    where rowid between 100 and 200;

  insert into vss_articles(rowid, headline_embedding, description_embedding)
    values (:rowid, :headline_embedding, :description_embedding)

  commit;
  ```

  You can pass in custom [Faiss factory strings](https://github.com/facebookresearch/faiss/wiki/The-index-factory) for specific columns to control how the Faiss index is stored and queried. By default the factory string is `"Flat,IDMap2"`, which can be slow to query as your database grows. Here, we add an [inverted file index](https://github.com/facebookresearch/faiss/wiki/The-index-factory#inverted-file-indexes) with 4096 centroids, a non-exhaustive option that makes large database queries much faster.

  ```sql
  create virtual table vss_ivf_articles using vss0(
    headline_embedding(384) factory="IVF4096,Flat,IDMap2",
    description_embedding(384) factory="IVF4096,Flat,IDMap2"
  );
  ```

  This IVF will require training! You can define training data with a `INSERT` command in a single transaction, with the special `operation="training"` constraint.

  ```sql
  insert into vss_ivf_articles(operation, headline_embedding, description_embedding)
    select
      'training',
      headline_embedding,
      description_embedding
    from articles;
  ```

  Beware! Indexes that require training can take a long time. With the [News Category Dataset](./examples/headlines/) (386 dimension over 210k vectors) that this example is based on, the default index would take 8 seconds to build. But with the custom `"IVF4096,Flat,IDMap2"` factory, it took 45 minutes to train and 4.5 minutes to insert data! This likely can be reduced with a smaller training set, but the faster queries can be helpful.

  ## Documentation

  See [`docs.md`](./docs.md) for a instructions to compile `sqlite-vss` yourself, as well as a full SQL API reference.

  ## Installing

  The [Releases page](https://github.com/asg017/sqlite-vss/releases) contains pre-built binaries for Linux x86_64 and MacOS x86_64 (MacOS Big Sur 11 or higher). More pre-compiled targets will be available in the future. Additionally, `sqlite-vss` is distributed on common package managers like `pip` for Python and `npm` for Node.js, see below for details.

  Do note that on Linux machines, you'll have to install some packages to make these options work:

  ```
  sudo apt-get update
  sudo apt-get install -y libgomp1 libatlas-base-dev liblapack-dev
  ```

  > **Note:**
  > The `0` in the filename (`vss0.dylib`/ `vss0.so`) denotes the major version of `sqlite-vss`. Currently `sqlite-vss` is pre v1, so expect breaking changes in future versions.

  | Language       | Install                                                            | More Info                                                                       |                                                                                                                                                                                           |
  | -------------- | ------------------------------------------------------------------ | ------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
  | Python         | `pip install sqlite-vss`                                           | [`sqlite-vss` with Python](https://alexgarcia.xyz/sqlite-vss/python.html)       | [![PyPI](https://img.shields.io/pypi/v/sqlite-vss.svg?color=blue&logo=python&logoColor=white)](https://pypi.org/project/sqlite-vss/)                                                      |
  | Datasette      | `datasette install datasette-sqlite-vss`                           | [`sqlite-vss` with Datasette](https://alexgarcia.xyz/sqlite-vss/datasette.html) | [![Datasette](https://img.shields.io/pypi/v/datasette-sqlite-vss.svg?color=B6B6D9&label=Datasette+plugin&logoColor=white&logo=python)](https://datasette.io/plugins/datasette-sqlite-vss) |
  | Node.js        | `npm install sqlite-vss`                                           | [`sqlite-vss` with Node.js](https://alexgarcia.xyz/sqlite-vss/nodejs.html)        | [![npm](https://img.shields.io/npm/v/sqlite-vss.svg?color=green&logo=nodedotjs&logoColor=white)](https://www.npmjs.com/package/sqlite-vss)                                                |
  | Deno           | [`deno.land/x/sqlite_vss`](https://deno.land/x/sqlite_vss)         | [`sqlite-vss` with Deno](https://alexgarcia.xyz/sqlite-vss/deno.html)           | [![deno version](https://deno.land/badge/sqlite_vss/version?color=fef8d2)](https://deno.land/x/sqlite_vss)                                                                                |
  | Ruby           | `gem install sqlite-vss`                                           | [`sqlite-vss` with Ruby](https://alexgarcia.xyz/sqlite-vss/ruby.html)           | ![Gem](https://img.shields.io/gem/v/sqlite-vss?color=red&logo=rubygems&logoColor=white)                                                                                                   |
  | Elixir         | [`hex.pm/packages/sqlite_vss`](https://hex.pm/packages/sqlite_vss) | [`sqlite-vss` with Elixir](https://alexgarcia.xyz/sqlite-vss/elixir.html)       | [![Hex.pm](https://img.shields.io/hexpm/v/sqlite_vss?color=purple&logo=elixir)](https://hex.pm/packages/sqlite_vss)                                                                       |
  | Go             | `go get -u github.com/asg017/sqlite-vss/bindings/go`               | [`sqlite-vss` with Go](https://alexgarcia.xyz/sqlite-vss/go.html)               | [![Go Reference](https://pkg.go.dev/badge/github.com/asg017/sqlite-vss/bindings/go.svg)](https://pkg.go.dev/github.com/asg017/sqlite-vss/bindings/go)                                     |
  | Rust           | `cargo add sqlite-vss`                                             | [`sqlite-vss` with Rust](https://alexgarcia.xyz/sqlite-vss/rust.html)           | [![Crates.io](https://img.shields.io/crates/v/sqlite-vss?logo=rust)](https://crates.io/crates/sqlite-vss)                                                                                 |
  | Github Release |                                                                    |                                                                                 | ![GitHub tag (latest SemVer pre-release)](https://img.shields.io/github/v/tag/asg017/sqlite-vss?color=lightgrey&include_prereleases&label=Github+release&logo=github)                     |

  ### With the `sqlite3` CLI

  For using `sqlite-vss` with [the official SQLite command line shell](https://www.sqlite.org/cli.html), download the `vector0.dylib`/`vss0.dylib` (for MacOS Big Sur 11 or higher) or `vector0.so`/`vss0.so` (Linux) files from a release and load it into your SQLite environment.

  The `vector0` extension is a required dependency, so make sure to load that before `vss0`.

  ```sql
  .load ./vector0
  .load ./vss0
  select vss_version();
  -- v0.0.1
  ```

  ### Python

  For Python developers, install the [`sqlite-vss` package](https://pypi.org/package/sqlite-vss/) with:

  ```
  pip install sqlite-vss
  ```

  ```python
  import sqlite3
  import sqlite_vss

  db = sqlite3.connect(':memory:')
  db.enable_load_extension(True)
  sqlite_vss.load(db)

  version, = db.execute('select vss_version()').fetchone()
  print(version)
  ```

  See [`bindings/python`](./bindings/python/README.md) for more details.

  ### Node.js

  For Node.js developers, install the [`sqlite-vss` npm package](https://www.npmjs.com/package/sqlite-vss) with:

  ```
  npm install sqlite-vss
  ```

  ```js
  import Database from "better-sqlite3"; // also compatible with node-sqlite3
  import * as sqlite_vss from "sqlite-vss";

  const db = new Database(":memory:");
  sqlite_vss.load(db);

  const version = db.prepare("select vss_version()").pluck().get();
  console.log(version);
  ```

  See [`npm/sqlite-vss/README.md`](./npm/sqlite-vss/README.md) for more details.

  ### Deno

  For [Deno](https://deno.land/) developers, use the [deno.land/x/sqlite_vss](https://deno.land/x/sqlite_vss) module:

  ```ts
  // Requires all permissions (-A) and the --unstable flag

  import { Database } from "https://deno.land/x/sqlite3@0.8.0/mod.ts";
  import * as sqlite_vss from "https://deno.land/x/sqlite_vss/mod.ts";

  const db = new Database(":memory:");

  db.enableLoadExtension = true;
  sqlite_vss.load(db);

  const [version] = db.prepare("select vss_version()").value<[string]>()!;

  console.log(version);
  ```

  See [`deno/sqlite-vss/README.md`](./deno/README.md) for more details.

  ### Datasette

  And for [Datasette](https://datasette.io/), install the [`datasette-sqlite-vss` plugin](https://datasette.io/plugins/datasette-sqlite-vss) with:

  ```
  datasette install datasette-sqlite-vss
  ```

  See [`bindings/datasette`](./bindings/datasette/README.md) for more details.

  ## Disadvantages

  - The underlying Faiss indicies are capped at 1GB. Follow [#1](https://github.com/asg017/sqlite-vss/issues/1) for updates.
  - Additional filtering on top of KNN searches aren't supported yet. Follow [#2](https://github.com/asg017/sqlite-vss/issues/2) for updates.
  - Only CPU Faiss indicies are supported, not GPU yet. Follow [#3](https://github.com/asg017/sqlite-vss/issues/3) for updates.
  - mmap'ed indices aren't supported yet, so indicies have to fit in RAM. Follow [#4](https://github.com/asg017/sqlite-vss/issues/4) for updates.
  - This extension is written in C++ and doesn't have fuzzy testing yet. Follow [#5](https://github.com/asg017/sqlite-vss/issues/5) for updates.
  - `UPDATE` statements on vss0 virtual tables are not supported, though `INSERT` and `DELETE` statements are. Follow [#7](https://github.com/asg017/sqlite-vss/issues/7) for updates.

  ## Supporting

  I (Alex 👋🏼) spent a lot of time and energy on this project and [many other open source projects](https://github.com/asg017?tab=repositories&q=&type=&language=&sort=stargazers). If your company or organization uses this library (or you're feeling generous), then please consider [sponsoring my work](https://github.com/sponsors/asg017/), sharing this project with a friend, or [hiring me for contract/consulting work](https://alexgarcia.xyz/work.html)!

  ## See Also

  - [`sqlite-http`](https://github.com/asg017/sqlite-http), a SQLite extension for making HTTP requests
  - [`sqlite-xsv`](https://github.com/asg017/sqlite-xsv), a fast SQLite extension for querying CSVs
  - [`sqlite-loadable-rs`](https://github.com/asg017/sqlite-loadable-rs), a framework for building SQLite extensions in Rust

readme_html:
  <div id="readme" class="md" data-path="README.md"><article class="markdown-body entry-content container-lg" itemprop="text"><div class="markdown-heading" dir="auto"><h1 class="heading-element" dir="auto">sqlite-vss</h1><a id="user-content-sqlite-vss" class="anchor" aria-label="Permalink: sqlite-vss" href="#sqlite-vss"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <div class="markdown-alert markdown-alert-warning" dir="auto"><p class="markdown-alert-title" dir="auto"><svg class="octicon octicon-alert mr-2" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="M6.457 1.047c.659-1.234 2.427-1.234 3.086 0l6.082 11.378A1.75 1.75 0 0 1 14.082 15H1.918a1.75 1.75 0 0 1-1.543-2.575Zm1.763.707a.25.25 0 0 0-.44 0L1.698 13.132a.25.25 0 0 0 .22.368h12.164a.25.25 0 0 0 .22-.368Zm.53 3.996v2.5a.75.75 0 0 1-1.5 0v-2.5a.75.75 0 0 1 1.5 0ZM9 11a1 1 0 1 1-2 0 1 1 0 0 1 2 0Z"></path></svg>Warning</p><p dir="auto"><code>sqlite-vss</code> is not in active development. Instead, my effort is now going towards <a href="https://github.com/asg017/sqlite-vec"><code>sqlite-vec</code></a>, which is a similar vector search SQLite extension, but should be much easier to install and use than <code>sqlite-vss</code>. See <a href="https://alexgarcia.xyz/blog/2024/building-new-vector-search-sqlite/index.html" rel="nofollow">this blog post</a> for more info.</p>
  </div>
  <p dir="auto"><code>sqlite-vss</code> (SQLite <b>V</b>ector <b>S</b>imilarity <b>S</b>earch) is a SQLite extension that brings vector search capabilities to SQLite, based on <a href="https://faiss.ai/" rel="nofollow">Faiss</a>. It can be used to build semantic search engines, recommendations, or questions-and-answering tools.</p>
  <p dir="auto">See <a href="https://observablehq.com/@asg017/introducing-sqlite-vss" rel="nofollow"><em>Introducing sqlite-vss: A SQLite Extension for Vector Search</em></a> (February 2023) for more details and a live example!</p>
  <p dir="auto">If your company or organization finds this library useful, consider <a href="#supporting">supporting my work</a>!</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Usage</h2><a id="user-content-usage" class="anchor" aria-label="Permalink: Usage" href="#usage"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <div class="highlight highlight-source-sql notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content=".load ./vector0
  .load ./vss0

  select vss_version(); -- 'v0.0.1'
  "><pre>.load .<span class="pl-k">/</span>vector0
  .load .<span class="pl-k">/</span>vss0

  <span class="pl-k">select</span> vss_version(); <span class="pl-c"><span class="pl-c">--</span> 'v0.0.1'</span>
  </pre></div>
  <p dir="auto"><code>sqlite-vss</code> has a similar API to the <a href="https://www.sqlite.org/fts5.html" rel="nofollow"><code>fts5</code> Full-Text Search Extension</a>. Use the <code>vss0</code> module to create virtual tables that can efficiently store and query your vectors.</p>
  <div class="highlight highlight-source-sql notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="-- 384 == number of dimensions for this example
  create virtual table vss_articles using vss0(
    headline_embedding(384),
    description_embedding(384),
  );"><pre><span class="pl-c"><span class="pl-c">--</span> 384 == number of dimensions for this example</span>
  create virtual table vss_articles using vss0(
    headline_embedding(<span class="pl-c1">384</span>),
    description_embedding(<span class="pl-c1">384</span>),
  );</pre></div>
  <p dir="auto"><code>sqlite-vss</code> is a <strong>Bring-your-own-vectors</strong> database, it is compatable with any embedding or vector data you have. Consider using <a href="https://platform.openai.com/docs/guides/embeddings" rel="nofollow">OpenAI's Embeddings API</a>, <a href="https://huggingface.co/blog/getting-started-with-embeddings#1-embedding-a-dataset" rel="nofollow">HuggingFace's Inference API</a>, <a href="https://www.sbert.net/" rel="nofollow"><code>sentence-transformers</code></a>, or <a href="https://github.com/embeddings-benchmark/mteb#leaderboard">any of these open source model</a>. In this example, we are using <a href="https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2" rel="nofollow">sentence-transformers/all-MiniLM-L6-v2</a> to generate embeddings from our text, which have 384 dimensions.</p>
  <p dir="auto">You can insert vectors into <code>vss0</code> tables as JSON or raw bytes.</p>
  <div class="highlight highlight-source-sql notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="insert into vss_articles(rowid, headline_embedding)
    select rowid, headline_embedding from articles;"><pre><span class="pl-k">insert into</span> vss_articles(rowid, headline_embedding)
    <span class="pl-k">select</span> rowid, headline_embedding <span class="pl-k">from</span> articles;</pre></div>
  <p dir="auto">To query for similar vectors ("k nearest neighbors"), use the <code>vss_search</code> function in the <code>WHERE</code> clause. Here we are searching for the 100 nearest neighbors to the embedding in row #123 in the <code>articles</code> table.</p>
  <div class="highlight highlight-source-sql notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="select rowid, distance
  from vss_articles
  where vss_search(
    headline_embedding,
    (select headline_embedding from articles where rowid = 123)
  )
  limit 100;"><pre><span class="pl-k">select</span> rowid, distance
  <span class="pl-k">from</span> vss_articles
  <span class="pl-k">where</span> vss_search(
    headline_embedding,
    (<span class="pl-k">select</span> headline_embedding <span class="pl-k">from</span> articles <span class="pl-k">where</span> rowid <span class="pl-k">=</span> <span class="pl-c1">123</span>)
  )
  <span class="pl-k">limit</span> <span class="pl-c1">100</span>;</pre></div>
  <p dir="auto">You can <code>INSERT</code> and <code>DELETE</code> into these tables as necessary, but <a href="https://github.com/asg017/sqlite-vss/issues/7" data-hovercard-type="issue" data-hovercard-url="/asg017/sqlite-vss/issues/7/hovercard"><code>UPDATE</code> operations aren't supported yet</a>. This can be used with triggers for automatically updated indexes. Also note that "small" <code>INSERT</code>/<code>DELETE</code> operations that only insert a few rows can be slow, so batch where necessary.</p>
  <div class="highlight highlight-source-sql notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="begin;

  delete from vss_articles
    where rowid between 100 and 200;

  insert into vss_articles(rowid, headline_embedding, description_embedding)
    values (:rowid, :headline_embedding, :description_embedding)

  commit;"><pre><span class="pl-k">begin</span>;

  <span class="pl-k">delete</span> <span class="pl-k">from</span> vss_articles
    <span class="pl-k">where</span> rowid between <span class="pl-c1">100</span> <span class="pl-k">and</span> <span class="pl-c1">200</span>;

  <span class="pl-k">insert into</span> vss_articles(rowid, headline_embedding, description_embedding)
    <span class="pl-k">values</span> (:rowid, :headline_embedding, :description_embedding)

  <span class="pl-k">commit</span>;</pre></div>
  <p dir="auto">You can pass in custom <a href="https://github.com/facebookresearch/faiss/wiki/The-index-factory">Faiss factory strings</a> for specific columns to control how the Faiss index is stored and queried. By default the factory string is <code>"Flat,IDMap2"</code>, which can be slow to query as your database grows. Here, we add an <a href="https://github.com/facebookresearch/faiss/wiki/The-index-factory#inverted-file-indexes">inverted file index</a> with 4096 centroids, a non-exhaustive option that makes large database queries much faster.</p>
  <div class="highlight highlight-source-sql notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="create virtual table vss_ivf_articles using vss0(
    headline_embedding(384) factory="IVF4096,Flat,IDMap2",
    description_embedding(384) factory="IVF4096,Flat,IDMap2"
  );"><pre>create virtual table vss_ivf_articles using vss0(
    headline_embedding(<span class="pl-c1">384</span>) factory<span class="pl-k">=</span><span class="pl-s"><span class="pl-pds">"</span>IVF4096,Flat,IDMap2<span class="pl-pds">"</span></span>,
    description_embedding(<span class="pl-c1">384</span>) factory<span class="pl-k">=</span><span class="pl-s"><span class="pl-pds">"</span>IVF4096,Flat,IDMap2<span class="pl-pds">"</span></span>
  );</pre></div>
  <p dir="auto">This IVF will require training! You can define training data with a <code>INSERT</code> command in a single transaction, with the special <code>operation="training"</code> constraint.</p>
  <div class="highlight highlight-source-sql notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="insert into vss_ivf_articles(operation, headline_embedding, description_embedding)
    select
      'training',
      headline_embedding,
      description_embedding
    from articles;"><pre><span class="pl-k">insert into</span> vss_ivf_articles(operation, headline_embedding, description_embedding)
    <span class="pl-k">select</span>
      <span class="pl-s"><span class="pl-pds">'</span>training<span class="pl-pds">'</span></span>,
      headline_embedding,
      description_embedding
    <span class="pl-k">from</span> articles;</pre></div>
  <p dir="auto">Beware! Indexes that require training can take a long time. With the <a href="./examples/headlines/">News Category Dataset</a> (386 dimension over 210k vectors) that this example is based on, the default index would take 8 seconds to build. But with the custom <code>"IVF4096,Flat,IDMap2"</code> factory, it took 45 minutes to train and 4.5 minutes to insert data! This likely can be reduced with a smaller training set, but the faster queries can be helpful.</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Documentation</h2><a id="user-content-documentation" class="anchor" aria-label="Permalink: Documentation" href="#documentation"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">See <a href="./docs.md"><code>docs.md</code></a> for a instructions to compile <code>sqlite-vss</code> yourself, as well as a full SQL API reference.</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Installing</h2><a id="user-content-installing" class="anchor" aria-label="Permalink: Installing" href="#installing"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">The <a href="https://github.com/asg017/sqlite-vss/releases">Releases page</a> contains pre-built binaries for Linux x86_64 and MacOS x86_64 (MacOS Big Sur 11 or higher). More pre-compiled targets will be available in the future. Additionally, <code>sqlite-vss</code> is distributed on common package managers like <code>pip</code> for Python and <code>npm</code> for Node.js, see below for details.</p>
  <p dir="auto">Do note that on Linux machines, you'll have to install some packages to make these options work:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="sudo apt-get update
  sudo apt-get install -y libgomp1 libatlas-base-dev liblapack-dev"><pre class="notranslate"><code>sudo apt-get update
  sudo apt-get install -y libgomp1 libatlas-base-dev liblapack-dev
  </code></pre></div>
  <blockquote>
  <p dir="auto"><strong>Note:</strong>
  The <code>0</code> in the filename (<code>vss0.dylib</code>/ <code>vss0.so</code>) denotes the major version of <code>sqlite-vss</code>. Currently <code>sqlite-vss</code> is pre v1, so expect breaking changes in future versions.</p>
  </blockquote>
  <markdown-accessiblity-table><table>
  <thead>
  <tr>
  <th>Language</th>
  <th>Install</th>
  <th>More Info</th>
  <th></th>
  </tr>
  </thead>
  <tbody>
  <tr>
  <td>Python</td>
  <td><code>pip install sqlite-vss</code></td>
  <td><a href="https://alexgarcia.xyz/sqlite-vss/python.html" rel="nofollow"><code>sqlite-vss</code> with Python</a></td>
  <td><a href="https://pypi.org/project/sqlite-vss/" rel="nofollow"><img src="https://img.shields.io/pypi/v/sqlite-vss.svg?color=blue&logo=python&logoColor=white" alt="PyPI" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/5a02a682880aca8c16a9dcd6adc76cc20c157ae8adaaef8bfd06fe1bd0907174/68747470733a2f2f696d672e736869656c64732e696f2f707970692f762f73716c6974652d7673732e7376673f636f6c6f723d626c7565266c6f676f3d707974686f6e266c6f676f436f6c6f723d7768697465"></a></td>
  </tr>
  <tr>
  <td>Datasette</td>
  <td><code>datasette install datasette-sqlite-vss</code></td>
  <td><a href="https://alexgarcia.xyz/sqlite-vss/datasette.html" rel="nofollow"><code>sqlite-vss</code> with Datasette</a></td>
  <td><a href="https://datasette.io/plugins/datasette-sqlite-vss" rel="nofollow"><img src="https://img.shields.io/pypi/v/datasette-sqlite-vss.svg?color=B6B6D9&label=Datasette+plugin&logoColor=white&logo=python" alt="Datasette" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/42e7f55e6f20d4c0676b88a14fbc4ef9a994b3c35524def6acd9443e83d72726/68747470733a2f2f696d672e736869656c64732e696f2f707970692f762f6461746173657474652d73716c6974652d7673732e7376673f636f6c6f723d423642364439266c6162656c3d4461746173657474652b706c7567696e266c6f676f436f6c6f723d7768697465266c6f676f3d707974686f6e"></a></td>
  </tr>
  <tr>
  <td>Node.js</td>
  <td><code>npm install sqlite-vss</code></td>
  <td><a href="https://alexgarcia.xyz/sqlite-vss/nodejs.html" rel="nofollow"><code>sqlite-vss</code> with Node.js</a></td>
  <td><a href="https://www.npmjs.com/package/sqlite-vss" rel="nofollow"><img src="https://img.shields.io/npm/v/sqlite-vss.svg?color=green&logo=nodedotjs&logoColor=white" alt="npm" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/ba65d4a7f9a288d71136422eab5f27b188c6f6ee1a4985cb4c6bd1c4638f10c8/68747470733a2f2f696d672e736869656c64732e696f2f6e706d2f762f73716c6974652d7673732e7376673f636f6c6f723d677265656e266c6f676f3d6e6f6465646f746a73266c6f676f436f6c6f723d7768697465"></a></td>
  </tr>
  <tr>
  <td>Deno</td>
  <td><a href="https://deno.land/x/sqlite_vss" rel="nofollow"><code>deno.land/x/sqlite_vss</code></a></td>
  <td><a href="https://alexgarcia.xyz/sqlite-vss/deno.html" rel="nofollow"><code>sqlite-vss</code> with Deno</a></td>
  <td><a href="https://deno.land/x/sqlite_vss" rel="nofollow"><img src="https://deno.land/badge/sqlite_vss/version?color=fef8d2" alt="deno version" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/5fc601d8173347586c499c8c145fed78fbeb7f7723d22d9b76a5cdeb693cad5c/68747470733a2f2f64656e6f2e6c616e642f62616467652f73716c6974655f7673732f76657273696f6e3f636f6c6f723d666566386432"></a></td>
  </tr>
  <tr>
  <td>Ruby</td>
  <td><code>gem install sqlite-vss</code></td>
  <td><a href="https://alexgarcia.xyz/sqlite-vss/ruby.html" rel="nofollow"><code>sqlite-vss</code> with Ruby</a></td>
  <td><a target="_blank" rel="noopener noreferrer nofollow" href="https://camo.githubusercontent.com/1499d84e7da81bc6d496ed5691492d301110975a0bf4c97fed1a4337bc40e1c5/68747470733a2f2f696d672e736869656c64732e696f2f67656d2f762f73716c6974652d7673733f636f6c6f723d726564266c6f676f3d7275627967656d73266c6f676f436f6c6f723d7768697465"><img src="https://img.shields.io/gem/v/sqlite-vss?color=red&logo=rubygems&logoColor=white" alt="Gem" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/1499d84e7da81bc6d496ed5691492d301110975a0bf4c97fed1a4337bc40e1c5/68747470733a2f2f696d672e736869656c64732e696f2f67656d2f762f73716c6974652d7673733f636f6c6f723d726564266c6f676f3d7275627967656d73266c6f676f436f6c6f723d7768697465"></a></td>
  </tr>
  <tr>
  <td>Elixir</td>
  <td><a href="https://hex.pm/packages/sqlite_vss" rel="nofollow"><code>hex.pm/packages/sqlite_vss</code></a></td>
  <td><a href="https://alexgarcia.xyz/sqlite-vss/elixir.html" rel="nofollow"><code>sqlite-vss</code> with Elixir</a></td>
  <td><a href="https://hex.pm/packages/sqlite_vss" rel="nofollow"><img src="https://img.shields.io/hexpm/v/sqlite_vss?color=purple&logo=elixir" alt="Hex.pm" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/b05133ca57bc1c5c19ad9dfdb71c2d0b9c39c462e687755b386a9c3f269190aa/68747470733a2f2f696d672e736869656c64732e696f2f686578706d2f762f73716c6974655f7673733f636f6c6f723d707572706c65266c6f676f3d656c69786972"></a></td>
  </tr>
  <tr>
  <td>Go</td>
  <td><code>go get -u github.com/asg017/sqlite-vss/bindings/go</code></td>
  <td><a href="https://alexgarcia.xyz/sqlite-vss/go.html" rel="nofollow"><code>sqlite-vss</code> with Go</a></td>
  <td><a href="https://pkg.go.dev/github.com/asg017/sqlite-vss/bindings/go" rel="nofollow"><img src="https://pkg.go.dev/badge/github.com/asg017/sqlite-vss/bindings/go.svg" alt="Go Reference" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/3ac2d40b6c2449dc1a09b1b0f60a53fa87a953968e7914e5d1aaa3647841f704/68747470733a2f2f706b672e676f2e6465762f62616467652f6769746875622e636f6d2f6173673031372f73716c6974652d7673732f62696e64696e67732f676f2e737667"></a></td>
  </tr>
  <tr>
  <td>Rust</td>
  <td><code>cargo add sqlite-vss</code></td>
  <td><a href="https://alexgarcia.xyz/sqlite-vss/rust.html" rel="nofollow"><code>sqlite-vss</code> with Rust</a></td>
  <td><a href="https://crates.io/crates/sqlite-vss" rel="nofollow"><img src="https://img.shields.io/crates/v/sqlite-vss?logo=rust" alt="Crates.io" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/46095004a3fe32069b6f37aa1b3f65195c46b97b2059e287d371e32b490ba685/68747470733a2f2f696d672e736869656c64732e696f2f6372617465732f762f73716c6974652d7673733f6c6f676f3d72757374"></a></td>
  </tr>
  <tr>
  <td>Github Release</td>
  <td></td>
  <td></td>
  <td><a target="_blank" rel="noopener noreferrer nofollow" href="https://camo.githubusercontent.com/3ed9cd239bfdcb37f1dd5e2d214cded8731f2f1d6ffb780819766fe2448df4e1/68747470733a2f2f696d672e736869656c64732e696f2f6769746875622f762f7461672f6173673031372f73716c6974652d7673733f636f6c6f723d6c696768746772657926696e636c7564655f70726572656c6561736573266c6162656c3d4769746875622b72656c65617365266c6f676f3d676974687562"><img src="https://img.shields.io/github/v/tag/asg017/sqlite-vss?color=lightgrey&include_prereleases&label=Github+release&logo=github" alt="GitHub tag (latest SemVer pre-release)" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/3ed9cd239bfdcb37f1dd5e2d214cded8731f2f1d6ffb780819766fe2448df4e1/68747470733a2f2f696d672e736869656c64732e696f2f6769746875622f762f7461672f6173673031372f73716c6974652d7673733f636f6c6f723d6c696768746772657926696e636c7564655f70726572656c6561736573266c6162656c3d4769746875622b72656c65617365266c6f676f3d676974687562"></a></td>
  </tr>
  </tbody>
  </table></markdown-accessiblity-table>
  <div class="markdown-heading" dir="auto"><h3 class="heading-element" dir="auto">With the <code>sqlite3</code> CLI</h3><a id="user-content-with-the-sqlite3-cli" class="anchor" aria-label="Permalink: With the sqlite3 CLI" href="#with-the-sqlite3-cli"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">For using <code>sqlite-vss</code> with <a href="https://www.sqlite.org/cli.html" rel="nofollow">the official SQLite command line shell</a>, download the <code>vector0.dylib</code>/<code>vss0.dylib</code> (for MacOS Big Sur 11 or higher) or <code>vector0.so</code>/<code>vss0.so</code> (Linux) files from a release and load it into your SQLite environment.</p>
  <p dir="auto">The <code>vector0</code> extension is a required dependency, so make sure to load that before <code>vss0</code>.</p>
  <div class="highlight highlight-source-sql notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content=".load ./vector0
  .load ./vss0
  select vss_version();
  -- v0.0.1"><pre>.load .<span class="pl-k">/</span>vector0
  .load .<span class="pl-k">/</span>vss0
  <span class="pl-k">select</span> vss_version();
  <span class="pl-c"><span class="pl-c">--</span> v0.0.1</span></pre></div>
  <div class="markdown-heading" dir="auto"><h3 class="heading-element" dir="auto">Python</h3><a id="user-content-python" class="anchor" aria-label="Permalink: Python" href="#python"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">For Python developers, install the <a href="https://pypi.org/package/sqlite-vss/" rel="nofollow"><code>sqlite-vss</code> package</a> with:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="pip install sqlite-vss"><pre class="notranslate"><code>pip install sqlite-vss
  </code></pre></div>
  <div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="import sqlite3
  import sqlite_vss

  db = sqlite3.connect(':memory:')
  db.enable_load_extension(True)
  sqlite_vss.load(db)

  version, = db.execute('select vss_version()').fetchone()
  print(version)"><pre><span class="pl-k">import</span> <span class="pl-s1">sqlite3</span>
  <span class="pl-k">import</span> <span class="pl-s1">sqlite_vss</span>

  <span class="pl-s1">db</span> <span class="pl-c1">=</span> <span class="pl-s1">sqlite3</span>.<span class="pl-c1">connect</span>(<span class="pl-s">':memory:'</span>)
  <span class="pl-s1">db</span>.<span class="pl-c1">enable_load_extension</span>(<span class="pl-c1">True</span>)
  <span class="pl-s1">sqlite_vss</span>.<span class="pl-c1">load</span>(<span class="pl-s1">db</span>)

  <span class="pl-s1">version</span>, <span class="pl-c1">=</span> <span class="pl-s1">db</span>.<span class="pl-c1">execute</span>(<span class="pl-s">'select vss_version()'</span>).<span class="pl-c1">fetchone</span>()
  <span class="pl-en">print</span>(<span class="pl-s1">version</span>)</pre></div>
  <p dir="auto">See <a href="./bindings/python/README.md"><code>bindings/python</code></a> for more details.</p>
  <div class="markdown-heading" dir="auto"><h3 class="heading-element" dir="auto">Node.js</h3><a id="user-content-nodejs" class="anchor" aria-label="Permalink: Node.js" href="#nodejs"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">For Node.js developers, install the <a href="https://www.npmjs.com/package/sqlite-vss" rel="nofollow"><code>sqlite-vss</code> npm package</a> with:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="npm install sqlite-vss"><pre class="notranslate"><code>npm install sqlite-vss
  </code></pre></div>
  <div class="highlight highlight-source-js notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="import Database from "better-sqlite3"; // also compatible with node-sqlite3
  import * as sqlite_vss from "sqlite-vss";

  const db = new Database(":memory:");
  sqlite_vss.load(db);

  const version = db.prepare("select vss_version()").pluck().get();
  console.log(version);"><pre><span class="pl-k">import</span> <span class="pl-v">Database</span> <span class="pl-k">from</span> <span class="pl-s">"better-sqlite3"</span><span class="pl-kos">;</span> <span class="pl-c">// also compatible with node-sqlite3</span>
  <span class="pl-k">import</span> <span class="pl-c1">*</span> <span class="pl-k">as</span> <span class="pl-s1">sqlite_vss</span> <span class="pl-k">from</span> <span class="pl-s">"sqlite-vss"</span><span class="pl-kos">;</span>

  <span class="pl-k">const</span> <span class="pl-s1">db</span> <span class="pl-c1">=</span> <span class="pl-k">new</span> <span class="pl-v">Database</span><span class="pl-kos">(</span><span class="pl-s">":memory:"</span><span class="pl-kos">)</span><span class="pl-kos">;</span>
  <span class="pl-s1">sqlite_vss</span><span class="pl-kos">.</span><span class="pl-en">load</span><span class="pl-kos">(</span><span class="pl-s1">db</span><span class="pl-kos">)</span><span class="pl-kos">;</span>

  <span class="pl-k">const</span> <span class="pl-s1">version</span> <span class="pl-c1">=</span> <span class="pl-s1">db</span><span class="pl-kos">.</span><span class="pl-en">prepare</span><span class="pl-kos">(</span><span class="pl-s">"select vss_version()"</span><span class="pl-kos">)</span><span class="pl-kos">.</span><span class="pl-en">pluck</span><span class="pl-kos">(</span><span class="pl-kos">)</span><span class="pl-kos">.</span><span class="pl-en">get</span><span class="pl-kos">(</span><span class="pl-kos">)</span><span class="pl-kos">;</span>
  <span class="pl-smi">console</span><span class="pl-kos">.</span><span class="pl-en">log</span><span class="pl-kos">(</span><span class="pl-s1">version</span><span class="pl-kos">)</span><span class="pl-kos">;</span></pre></div>
  <p dir="auto">See <a href="./npm/sqlite-vss/README.md"><code>npm/sqlite-vss/README.md</code></a> for more details.</p>
  <div class="markdown-heading" dir="auto"><h3 class="heading-element" dir="auto">Deno</h3><a id="user-content-deno" class="anchor" aria-label="Permalink: Deno" href="#deno"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">For <a href="https://deno.land/" rel="nofollow">Deno</a> developers, use the <a href="https://deno.land/x/sqlite_vss" rel="nofollow">deno.land/x/sqlite_vss</a> module:</p>
  <div class="highlight highlight-source-ts notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="// Requires all permissions (-A) and the --unstable flag

  import { Database } from "https://deno.land/x/sqlite3@0.8.0/mod.ts";
  import * as sqlite_vss from "https://deno.land/x/sqlite_vss/mod.ts";

  const db = new Database(":memory:");

  db.enableLoadExtension = true;
  sqlite_vss.load(db);

  const [version] = db.prepare("select vss_version()").value<[string]>()!;

  console.log(version);"><pre><span class="pl-c">// Requires all permissions (-A) and the --unstable flag</span>

  <span class="pl-k">import</span> <span class="pl-kos">{</span> <span class="pl-v">Database</span> <span class="pl-kos">}</span> <span class="pl-k">from</span> <span class="pl-s">"https://deno.land/x/sqlite3@0.8.0/mod.ts"</span><span class="pl-kos">;</span>
  <span class="pl-k">import</span> <span class="pl-c1">*</span> <span class="pl-k">as</span> <span class="pl-s1">sqlite_vss</span> <span class="pl-k">from</span> <span class="pl-s">"https://deno.land/x/sqlite_vss/mod.ts"</span><span class="pl-kos">;</span>

  <span class="pl-k">const</span> <span class="pl-s1">db</span> <span class="pl-c1">=</span> <span class="pl-k">new</span> <span class="pl-v">Database</span><span class="pl-kos">(</span><span class="pl-s">":memory:"</span><span class="pl-kos">)</span><span class="pl-kos">;</span>

  <span class="pl-s1">db</span><span class="pl-kos">.</span><span class="pl-c1">enableLoadExtension</span> <span class="pl-c1">=</span> <span class="pl-c1">true</span><span class="pl-kos">;</span>
  <span class="pl-s1">sqlite_vss</span><span class="pl-kos">.</span><span class="pl-en">load</span><span class="pl-kos">(</span><span class="pl-s1">db</span><span class="pl-kos">)</span><span class="pl-kos">;</span>

  <span class="pl-k">const</span> <span class="pl-kos">[</span><span class="pl-s1">version</span><span class="pl-kos">]</span> <span class="pl-c1">=</span> <span class="pl-s1">db</span><span class="pl-kos">.</span><span class="pl-en">prepare</span><span class="pl-kos">(</span><span class="pl-s">"select vss_version()"</span><span class="pl-kos">)</span><span class="pl-kos">.</span><span class="pl-en">value</span><span class="pl-c1"><</span><span class="pl-kos">[</span><span class="pl-smi">string</span><span class="pl-kos">]</span><span class="pl-c1">></span><span class="pl-kos">(</span><span class="pl-kos">)</span><span class="pl-c1">!</span><span class="pl-kos">;</span>

  <span class="pl-smi">console</span><span class="pl-kos">.</span><span class="pl-en">log</span><span class="pl-kos">(</span><span class="pl-s1">version</span><span class="pl-kos">)</span><span class="pl-kos">;</span></pre></div>
  <p dir="auto">See <a href="./deno/README.md"><code>deno/sqlite-vss/README.md</code></a> for more details.</p>
  <div class="markdown-heading" dir="auto"><h3 class="heading-element" dir="auto">Datasette</h3><a id="user-content-datasette" class="anchor" aria-label="Permalink: Datasette" href="#datasette"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">And for <a href="https://datasette.io/" rel="nofollow">Datasette</a>, install the <a href="https://datasette.io/plugins/datasette-sqlite-vss" rel="nofollow"><code>datasette-sqlite-vss</code> plugin</a> with:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="datasette install datasette-sqlite-vss"><pre class="notranslate"><code>datasette install datasette-sqlite-vss
  </code></pre></div>
  <p dir="auto">See <a href="./bindings/datasette/README.md"><code>bindings/datasette</code></a> for more details.</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Disadvantages</h2><a id="user-content-disadvantages" class="anchor" aria-label="Permalink: Disadvantages" href="#disadvantages"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <ul dir="auto">
  <li>The underlying Faiss indicies are capped at 1GB. Follow <a href="https://github.com/asg017/sqlite-vss/issues/1" data-hovercard-type="issue" data-hovercard-url="/asg017/sqlite-vss/issues/1/hovercard">#1</a> for updates.</li>
  <li>Additional filtering on top of KNN searches aren't supported yet. Follow <a href="https://github.com/asg017/sqlite-vss/issues/2" data-hovercard-type="issue" data-hovercard-url="/asg017/sqlite-vss/issues/2/hovercard">#2</a> for updates.</li>
  <li>Only CPU Faiss indicies are supported, not GPU yet. Follow <a href="https://github.com/asg017/sqlite-vss/issues/3" data-hovercard-type="issue" data-hovercard-url="/asg017/sqlite-vss/issues/3/hovercard">#3</a> for updates.</li>
  <li>mmap'ed indices aren't supported yet, so indicies have to fit in RAM. Follow <a href="https://github.com/asg017/sqlite-vss/issues/4" data-hovercard-type="issue" data-hovercard-url="/asg017/sqlite-vss/issues/4/hovercard">#4</a> for updates.</li>
  <li>This extension is written in C++ and doesn't have fuzzy testing yet. Follow <a href="https://github.com/asg017/sqlite-vss/issues/5" data-hovercard-type="issue" data-hovercard-url="/asg017/sqlite-vss/issues/5/hovercard">#5</a> for updates.</li>
  <li><code>UPDATE</code> statements on vss0 virtual tables are not supported, though <code>INSERT</code> and <code>DELETE</code> statements are. Follow <a href="https://github.com/asg017/sqlite-vss/issues/7" data-hovercard-type="issue" data-hovercard-url="/asg017/sqlite-vss/issues/7/hovercard">#7</a> for updates.</li>
  </ul>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Supporting</h2><a id="user-content-supporting" class="anchor" aria-label="Permalink: Supporting" href="#supporting"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">I (Alex 👋🏼) spent a lot of time and energy on this project and <a href="https://github.com/asg017?tab=repositories&q=&type=&language=&sort=stargazers">many other open source projects</a>. If your company or organization uses this library (or you're feeling generous), then please consider <a href="https://github.com/sponsors/asg017/">sponsoring my work</a>, sharing this project with a friend, or <a href="https://alexgarcia.xyz/work.html" rel="nofollow">hiring me for contract/consulting work</a>!</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">See Also</h2><a id="user-content-see-also" class="anchor" aria-label="Permalink: See Also" href="#see-also"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <ul dir="auto">
  <li><a href="https://github.com/asg017/sqlite-http"><code>sqlite-http</code></a>, a SQLite extension for making HTTP requests</li>
  <li><a href="https://github.com/asg017/sqlite-xsv"><code>sqlite-xsv</code></a>, a fast SQLite extension for querying CSVs</li>
  <li><a href="https://github.com/asg017/sqlite-loadable-rs"><code>sqlite-loadable-rs</code></a>, a framework for building SQLite extensions in Rust</li>
  </ul>
  </article></div>


node_id: R_kgDOIu6Scg
name: sqlite-fastrand
full_name: asg017/sqlite-fastrand
html_url: https://github.com/asg017/sqlite-fastrand
description: A SQLite extension for quickly generating random numbers, booleans, characters, and blobs
created_at: 2023-01-06T20:46:06Z
updated_at: 2024-11-28T16:35:13Z
pushed_at: 2023-11-27T16:07:05Z
language: Rust
license: apache-2.0
topics: ["datasette-plugin", "sqlite", "sqlite-extension"]
visibility: public
default_branch: main
permissions: {"admin": false, "maintain": false, "push": false, "triage": false, "pull": true}
readme:
  # sqlite-fastrandom

  A SQLite extension for quickly generating random numbers, booleans, characters, and blobs. **Not cryptographically secure.** Based on [`sqlite-loadable-rs`](https://github.com/asg017/sqlite-loadable-rs) and the [fastrand crate](https://crates.io/crates/fastrand).`

  According to my local benchmarks, `fastrand_int64()` is about 2.6x faster than SQLite's `random()`, and `fastrand_blob()` is about 1.6x faster than `randomblob()`. `sqlite-fastrand` also offers a more ergonomic API with custom ranges, seeds, and boolean/character support. However, it yields psuedo-random results and isn't "truly" random.

  If your company or organization finds this library useful, consider [supporting my work](#supporting)!

  ## Usage

  ```sql
  .load ./fastrand0
  select fastrand_int(); -- 556823563
  select fastrand_int(); -- 363294620
  select fastrand_int(); -- -320463573
  ```

  Set a seed for the underlying random number generator, for deterministic values.

  ```sql
  select fastrand_seed_set(1234);
  select fastrand_int(); -- -2058591105
  select fastrand_int(); -- -211244717
  select fastrand_int(); -- -1772832958

  select fastrand_seed_set(1234);
  select fastrand_int(); -- -2058591105
  select fastrand_int(); -- -211244717
  select fastrand_int(); -- -1772832958
  ```

  Include `start` and `end` (exclusive) parameters to generate random numbers within a range.

  ```sql
  select fastrand_int(0, 10); -- 0
  select fastrand_int(0, 10); -- 9
  select fastrand_int(0, 10); -- 6
  ```

  Generate random digits, lowercase/uppercase/alphabetic/alphanumeric characters.

  ```sql
  select fastrand_alphabetic(); -- 's'
  select fastrand_alphanumeric(); -- '2'
  select fastrand_char(); -- '񠞼'
  select fastrand_lowercase(); -- 'g'
  select fastrand_uppercase();-- 'M'

  select fastrand_digit(16); -- 'c'
  ```

  Generate a random float between 0 and 1.

  ```sql
  select fastrand_double(); -- 0.740834390248454
  select fastrand_double(); -- 0.46936608707793
  ```

  ### Differences from `random()` and `randomblob()`

  The builtin [`random()`](https://www.sqlite.org/lang_corefunc.html#random) and [`randomblob()`](https://www.sqlite.org/lang_corefunc.html#randomblob) are powerful tools that already exist in SQLite standard library, but they can be confusing at times.

  For example, the `random()` function returns "_... a pseudo-random integer between -9223372036854775808 and +9223372036854775807_", which are the minimum and maximum values of a 64 bit signed integer.

  ```sql
  select random(); -- 8247412491507365610
  select random(); -- 8124278049726255864
  ```

  This may work fine in your use-case, but typically I want a more constrained random number, like any number between `0-100`. This can technically be done with `random()` if you use `abs()` and the modulus `%` operator, but it gets awkward:

  ```sql
  select abs(random()) % 100; -- 96
  select abs(random()) % 100; -- 41
  ```

  The `fastrand_int64()` function works the same as `random()` but offers an optional `start` and `end` parameters to specify a range in which the random number should be generated in.

  ```sql
  select fastrand_int64(); -- 5216671854996406003
  select fastrand_int64(0, 100); -- 19
  ```

  [`randomblob(N)`](https://www.sqlite.org/lang_corefunc.html#randomblob)

  > _The randomblob(N) function return an N-byte blob containing pseudo-random bytes. If N is less than 1 then a 1-byte random blob is returned._

  ```sql
  select hex(randomblob(16)); -- '4E7EFDB9E687EED4F376359986CB695E'
  select hex(randomblob(16)); -- 'F6CFF9249E3BD8755E10D6BB3CA81C66'
  ```

  The [`fastrand_blob(N)`](./docs.md#fastrand_blob) function acts in the same way.

  ```sql
  select hex(fastrand_blob(16)); -- 'D86FF5409D3FAD7DBE707580C7E7DE14'
  select hex(fastrand_blob(16)); -- 'AB72BFE9480197F487933E8071072D4A'
  ```

  ## Installing

  | Language       | Install                                                              |                                                                                                                                                                                                     |
  | -------------- | -------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
  | Python         | `pip install sqlite-fastrand`                                        | [![PyPI](https://img.shields.io/pypi/v/sqlite-fastrand.svg?color=blue&logo=python&logoColor=white)](https://pypi.org/project/sqlite-fastrand/)                                                      |
  | Datasette      | `datasette install datasette-sqlite-fastrand`                        | [![Datasette](https://img.shields.io/pypi/v/datasette-sqlite-fastrand.svg?color=B6B6D9&label=Datasette+plugin&logoColor=white&logo=python)](https://datasette.io/plugins/datasette-sqlite-fastrand) |
  | Node.js        | `npm install sqlite-fastrand`                                        | [![npm](https://img.shields.io/npm/v/sqlite-fastrand.svg?color=green&logo=nodedotjs&logoColor=white)](https://www.npmjs.com/package/sqlite-fastrand)                                                |
  | Deno           | [`deno.land/x/sqlite_fastrand`](https://deno.land/x/sqlite_fastrand) | [![deno.land/x release](https://img.shields.io/github/v/release/asg017/sqlite-fastrand?color=fef8d2&include_prereleases&label=deno.land%2Fx&logo=deno)](https://deno.land/x/sqlite_fastrand)        |
  | Ruby           | `gem install sqlite-fastrand`                                        | ![Gem](https://img.shields.io/gem/v/sqlite-fastrand?color=red&logo=rubygems&logoColor=white)                                                                                                        |
  | Github Release |                                                                      | ![GitHub tag (latest SemVer pre-release)](https://img.shields.io/github/v/tag/asg017/sqlite-fastrand?color=lightgrey&include_prereleases&label=Github+release&logo=github)                          |
  | Rust           | `cargo add sqlite-fastrand`                                          | [![Crates.io](https://img.shields.io/crates/v/sqlite-fastrand?logo=rust)](https://crates.io/crates/sqlite-fastrand)                                                                                 |

  <!--
  | Elixir         | [`hex.pm/packages/sqlite_fastrand`](https://hex.pm/packages/sqlite_fastrand) | [![Hex.pm](https://img.shields.io/hexpm/v/sqlite_fastrand?color=purple&logo=elixir)](https://hex.pm/packages/sqlite_fastrand)                                                                       |
  | Go             | `go get -u github.com/asg017/sqlite-fastrand/bindings/go`               | [![Go Reference](https://pkg.go.dev/badge/github.com/asg017/sqlite-fastrand/bindings/go.svg)](https://pkg.go.dev/github.com/asg017/sqlite-fastrand/bindings/go)                                     |
  -->

  The [Releases page](https://github.com/asg017/sqlite-fastrand/releases) contains pre-built binaries for Linux x86_64, MacOS, and Windows.

  ### As a loadable extension

  If you want to use `sqlite-fastrand` as a [Runtime-loadable extension](https://www.sqlite.org/loadext.html), Download the `fastrand0.dylib` (for MacOS), `fastrand0.so` (Linux), or `fastrand0.dll` (Windows) file from a release and load it into your SQLite environment.

  > **Note:**
  > The `0` in the filename (`fastrand0.dylib`/ `fastrand0.so`/`fastrand0.dll`) denotes the major version of `sqlite-fastrand`. Currently `sqlite-fastrand` is pre v1, so expect breaking changes in future versions.

  For example, if you are using the [SQLite CLI](https://www.sqlite.org/cli.html), you can load the library like so:

  ```sql
  .load ./fastrand0
  select fastrand_version();
  -- v0.1.0
  ```

  Or in Python, using the builtin [sqlite3 module](https://docs.python.org/3/library/sqlite3.html):

  ```python
  import sqlite3
  con = sqlite3.connect(":memory:")
  con.enable_load_extension(True)
  con.load_extension("./fastrand0")
  print(con.execute("select fastrand_version()").fetchone())
  # ('v0.1.0',)
  ```

  Or in Node.js using [better-sqlite3](https://github.com/WiseLibs/better-sqlite3):

  ```javascript
  const Database = require("better-sqlite3");
  const db = new Database(":memory:");
  db.loadExtension("./fastrand0");
  console.log(db.prepare("select fastrand_version()").get());
  // { 'fastrand_version()': 'v0.1.0' }
  ```

  Or with [Datasette](https://datasette.io/):

  ```
  datasette data.db --load-extension ./fastrand0
  ```

  ## Supporting

  I (Alex 👋🏼) spent a lot of time and energy on this project and [many other open source projects](https://github.com/asg017?tab=repositories&q=&type=&language=&sort=stargazers). If your company or organization uses this library (or you're feeling generous), then please [consider supporting my work](https://alexgarcia.xyz/work.html), or share this project with a friend!

  ## See also

  - [sqlite-xsv](https://github.com/asg017/sqlite-xsv), A SQLite extension for working with CSVs
  - [sqlite-loadable](https://github.com/asg017/sqlite-loadable-rs), A framework for writing SQLite extensions in Rust
  - [sqlite-http](https://github.com/asg017/sqlite-http), A SQLite extension for making HTTP requests

readme_html:
  <div id="readme" class="md" data-path="README.md"><article class="markdown-body entry-content container-lg" itemprop="text"><div class="markdown-heading" dir="auto"><h1 class="heading-element" dir="auto">sqlite-fastrandom</h1><a id="user-content-sqlite-fastrandom" class="anchor" aria-label="Permalink: sqlite-fastrandom" href="#sqlite-fastrandom"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">A SQLite extension for quickly generating random numbers, booleans, characters, and blobs. <strong>Not cryptographically secure.</strong> Based on <a href="https://github.com/asg017/sqlite-loadable-rs"><code>sqlite-loadable-rs</code></a> and the <a href="https://crates.io/crates/fastrand" rel="nofollow">fastrand crate</a>.`</p>
  <p dir="auto">According to my local benchmarks, <code>fastrand_int64()</code> is about 2.6x faster than SQLite's <code>random()</code>, and <code>fastrand_blob()</code> is about 1.6x faster than <code>randomblob()</code>. <code>sqlite-fastrand</code> also offers a more ergonomic API with custom ranges, seeds, and boolean/character support. However, it yields psuedo-random results and isn't "truly" random.</p>
  <p dir="auto">If your company or organization finds this library useful, consider <a href="#supporting">supporting my work</a>!</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Usage</h2><a id="user-content-usage" class="anchor" aria-label="Permalink: Usage" href="#usage"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <div class="highlight highlight-source-sql notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content=".load ./fastrand0
  select fastrand_int(); -- 556823563
  select fastrand_int(); -- 363294620
  select fastrand_int(); -- -320463573"><pre>.load .<span class="pl-k">/</span>fastrand0
  <span class="pl-k">select</span> fastrand_int(); <span class="pl-c"><span class="pl-c">--</span> 556823563</span>
  <span class="pl-k">select</span> fastrand_int(); <span class="pl-c"><span class="pl-c">--</span> 363294620</span>
  <span class="pl-k">select</span> fastrand_int(); <span class="pl-c"><span class="pl-c">--</span> -320463573</span></pre></div>
  <p dir="auto">Set a seed for the underlying random number generator, for deterministic values.</p>
  <div class="highlight highlight-source-sql notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="select fastrand_seed_set(1234);
  select fastrand_int(); -- -2058591105
  select fastrand_int(); -- -211244717
  select fastrand_int(); -- -1772832958

  select fastrand_seed_set(1234);
  select fastrand_int(); -- -2058591105
  select fastrand_int(); -- -211244717
  select fastrand_int(); -- -1772832958"><pre><span class="pl-k">select</span> fastrand_seed_set(<span class="pl-c1">1234</span>);
  <span class="pl-k">select</span> fastrand_int(); <span class="pl-c"><span class="pl-c">--</span> -2058591105</span>
  <span class="pl-k">select</span> fastrand_int(); <span class="pl-c"><span class="pl-c">--</span> -211244717</span>
  <span class="pl-k">select</span> fastrand_int(); <span class="pl-c"><span class="pl-c">--</span> -1772832958</span>

  <span class="pl-k">select</span> fastrand_seed_set(<span class="pl-c1">1234</span>);
  <span class="pl-k">select</span> fastrand_int(); <span class="pl-c"><span class="pl-c">--</span> -2058591105</span>
  <span class="pl-k">select</span> fastrand_int(); <span class="pl-c"><span class="pl-c">--</span> -211244717</span>
  <span class="pl-k">select</span> fastrand_int(); <span class="pl-c"><span class="pl-c">--</span> -1772832958</span></pre></div>
  <p dir="auto">Include <code>start</code> and <code>end</code> (exclusive) parameters to generate random numbers within a range.</p>
  <div class="highlight highlight-source-sql notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="select fastrand_int(0, 10); -- 0
  select fastrand_int(0, 10); -- 9
  select fastrand_int(0, 10); -- 6"><pre><span class="pl-k">select</span> fastrand_int(<span class="pl-c1">0</span>, <span class="pl-c1">10</span>); <span class="pl-c"><span class="pl-c">--</span> 0</span>
  <span class="pl-k">select</span> fastrand_int(<span class="pl-c1">0</span>, <span class="pl-c1">10</span>); <span class="pl-c"><span class="pl-c">--</span> 9</span>
  <span class="pl-k">select</span> fastrand_int(<span class="pl-c1">0</span>, <span class="pl-c1">10</span>); <span class="pl-c"><span class="pl-c">--</span> 6</span></pre></div>
  <p dir="auto">Generate random digits, lowercase/uppercase/alphabetic/alphanumeric characters.</p>
  <div class="highlight highlight-source-sql notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="select fastrand_alphabetic(); -- 's'
  select fastrand_alphanumeric(); -- '2'
  select fastrand_char(); -- '񠞼'
  select fastrand_lowercase(); -- 'g'
  select fastrand_uppercase();-- 'M'

  select fastrand_digit(16); -- 'c'"><pre><span class="pl-k">select</span> fastrand_alphabetic(); <span class="pl-c"><span class="pl-c">--</span> 's'</span>
  <span class="pl-k">select</span> fastrand_alphanumeric(); <span class="pl-c"><span class="pl-c">--</span> '2'</span>
  <span class="pl-k">select</span> fastrand_char(); <span class="pl-c"><span class="pl-c">--</span> '񠞼'</span>
  <span class="pl-k">select</span> fastrand_lowercase(); <span class="pl-c"><span class="pl-c">--</span> 'g'</span>
  <span class="pl-k">select</span> fastrand_uppercase();<span class="pl-c"><span class="pl-c">--</span> 'M'</span>

  <span class="pl-k">select</span> fastrand_digit(<span class="pl-c1">16</span>); <span class="pl-c"><span class="pl-c">--</span> 'c'</span></pre></div>
  <p dir="auto">Generate a random float between 0 and 1.</p>
  <div class="highlight highlight-source-sql notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="select fastrand_double(); -- 0.740834390248454
  select fastrand_double(); -- 0.46936608707793"><pre><span class="pl-k">select</span> fastrand_double(); <span class="pl-c"><span class="pl-c">--</span> 0.740834390248454</span>
  <span class="pl-k">select</span> fastrand_double(); <span class="pl-c"><span class="pl-c">--</span> 0.46936608707793</span></pre></div>
  <div class="markdown-heading" dir="auto"><h3 class="heading-element" dir="auto">Differences from <code>random()</code> and <code>randomblob()</code></h3><a id="user-content-differences-from-random-and-randomblob" class="anchor" aria-label="Permalink: Differences from random() and randomblob()" href="#differences-from-random-and-randomblob"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">The builtin <a href="https://www.sqlite.org/lang_corefunc.html#random" rel="nofollow"><code>random()</code></a> and <a href="https://www.sqlite.org/lang_corefunc.html#randomblob" rel="nofollow"><code>randomblob()</code></a> are powerful tools that already exist in SQLite standard library, but they can be confusing at times.</p>
  <p dir="auto">For example, the <code>random()</code> function returns "<em>... a pseudo-random integer between -9223372036854775808 and +9223372036854775807</em>", which are the minimum and maximum values of a 64 bit signed integer.</p>
  <div class="highlight highlight-source-sql notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="select random(); -- 8247412491507365610
  select random(); -- 8124278049726255864"><pre><span class="pl-k">select</span> random(); <span class="pl-c"><span class="pl-c">--</span> 8247412491507365610</span>
  <span class="pl-k">select</span> random(); <span class="pl-c"><span class="pl-c">--</span> 8124278049726255864</span></pre></div>
  <p dir="auto">This may work fine in your use-case, but typically I want a more constrained random number, like any number between <code>0-100</code>. This can technically be done with <code>random()</code> if you use <code>abs()</code> and the modulus <code>%</code> operator, but it gets awkward:</p>
  <div class="highlight highlight-source-sql notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="select abs(random()) % 100; -- 96
  select abs(random()) % 100; -- 41"><pre><span class="pl-k">select</span> abs(random()) % <span class="pl-c1">100</span>; <span class="pl-c"><span class="pl-c">--</span> 96</span>
  <span class="pl-k">select</span> abs(random()) % <span class="pl-c1">100</span>; <span class="pl-c"><span class="pl-c">--</span> 41</span></pre></div>
  <p dir="auto">The <code>fastrand_int64()</code> function works the same as <code>random()</code> but offers an optional <code>start</code> and <code>end</code> parameters to specify a range in which the random number should be generated in.</p>
  <div class="highlight highlight-source-sql notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="select fastrand_int64(); -- 5216671854996406003
  select fastrand_int64(0, 100); -- 19"><pre><span class="pl-k">select</span> fastrand_int64(); <span class="pl-c"><span class="pl-c">--</span> 5216671854996406003</span>
  <span class="pl-k">select</span> fastrand_int64(<span class="pl-c1">0</span>, <span class="pl-c1">100</span>); <span class="pl-c"><span class="pl-c">--</span> 19</span></pre></div>
  <p dir="auto"><a href="https://www.sqlite.org/lang_corefunc.html#randomblob" rel="nofollow"><code>randomblob(N)</code></a></p>
  <blockquote>
  <p dir="auto"><em>The randomblob(N) function return an N-byte blob containing pseudo-random bytes. If N is less than 1 then a 1-byte random blob is returned.</em></p>
  </blockquote>
  <div class="highlight highlight-source-sql notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="select hex(randomblob(16)); -- '4E7EFDB9E687EED4F376359986CB695E'
  select hex(randomblob(16)); -- 'F6CFF9249E3BD8755E10D6BB3CA81C66'"><pre><span class="pl-k">select</span> hex(randomblob(<span class="pl-c1">16</span>)); <span class="pl-c"><span class="pl-c">--</span> '4E7EFDB9E687EED4F376359986CB695E'</span>
  <span class="pl-k">select</span> hex(randomblob(<span class="pl-c1">16</span>)); <span class="pl-c"><span class="pl-c">--</span> 'F6CFF9249E3BD8755E10D6BB3CA81C66'</span></pre></div>
  <p dir="auto">The <a href="./docs.md#fastrand_blob"><code>fastrand_blob(N)</code></a> function acts in the same way.</p>
  <div class="highlight highlight-source-sql notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="select hex(fastrand_blob(16)); -- 'D86FF5409D3FAD7DBE707580C7E7DE14'
  select hex(fastrand_blob(16)); -- 'AB72BFE9480197F487933E8071072D4A'"><pre><span class="pl-k">select</span> hex(fastrand_blob(<span class="pl-c1">16</span>)); <span class="pl-c"><span class="pl-c">--</span> 'D86FF5409D3FAD7DBE707580C7E7DE14'</span>
  <span class="pl-k">select</span> hex(fastrand_blob(<span class="pl-c1">16</span>)); <span class="pl-c"><span class="pl-c">--</span> 'AB72BFE9480197F487933E8071072D4A'</span></pre></div>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Installing</h2><a id="user-content-installing" class="anchor" aria-label="Permalink: Installing" href="#installing"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <markdown-accessiblity-table><table>
  <thead>
  <tr>
  <th>Language</th>
  <th>Install</th>
  <th></th>
  </tr>
  </thead>
  <tbody>
  <tr>
  <td>Python</td>
  <td><code>pip install sqlite-fastrand</code></td>
  <td><a href="https://pypi.org/project/sqlite-fastrand/" rel="nofollow"><img src="https://img.shields.io/pypi/v/sqlite-fastrand.svg?color=blue&logo=python&logoColor=white" alt="PyPI" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/77a3eb5c47f6cc9eb996f7460336ac6b9d2fb4c1f5af982eb1c6876db91eed2d/68747470733a2f2f696d672e736869656c64732e696f2f707970692f762f73716c6974652d6661737472616e642e7376673f636f6c6f723d626c7565266c6f676f3d707974686f6e266c6f676f436f6c6f723d7768697465"></a></td>
  </tr>
  <tr>
  <td>Datasette</td>
  <td><code>datasette install datasette-sqlite-fastrand</code></td>
  <td><a href="https://datasette.io/plugins/datasette-sqlite-fastrand" rel="nofollow"><img src="https://img.shields.io/pypi/v/datasette-sqlite-fastrand.svg?color=B6B6D9&label=Datasette+plugin&logoColor=white&logo=python" alt="Datasette" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/7f805327da5b6bfb6238ee45e0ca36c37ce373971b217436e6858af222164b92/68747470733a2f2f696d672e736869656c64732e696f2f707970692f762f6461746173657474652d73716c6974652d6661737472616e642e7376673f636f6c6f723d423642364439266c6162656c3d4461746173657474652b706c7567696e266c6f676f436f6c6f723d7768697465266c6f676f3d707974686f6e"></a></td>
  </tr>
  <tr>
  <td>Node.js</td>
  <td><code>npm install sqlite-fastrand</code></td>
  <td><a href="https://www.npmjs.com/package/sqlite-fastrand" rel="nofollow"><img src="https://img.shields.io/npm/v/sqlite-fastrand.svg?color=green&logo=nodedotjs&logoColor=white" alt="npm" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/fe4894fb0d2b726d82065e44341ffb25058c1c0337a4a243edb9a3140b1b2b3b/68747470733a2f2f696d672e736869656c64732e696f2f6e706d2f762f73716c6974652d6661737472616e642e7376673f636f6c6f723d677265656e266c6f676f3d6e6f6465646f746a73266c6f676f436f6c6f723d7768697465"></a></td>
  </tr>
  <tr>
  <td>Deno</td>
  <td><a href="https://deno.land/x/sqlite_fastrand" rel="nofollow"><code>deno.land/x/sqlite_fastrand</code></a></td>
  <td><a href="https://deno.land/x/sqlite_fastrand" rel="nofollow"><img src="https://img.shields.io/github/v/release/asg017/sqlite-fastrand?color=fef8d2&include_prereleases&label=deno.land%2Fx&logo=deno" alt="deno.land/x release" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/626168930ce05855777e502e431cc2409b5d1b1863df799b3365876663a1825a/68747470733a2f2f696d672e736869656c64732e696f2f6769746875622f762f72656c656173652f6173673031372f73716c6974652d6661737472616e643f636f6c6f723d66656638643226696e636c7564655f70726572656c6561736573266c6162656c3d64656e6f2e6c616e6425324678266c6f676f3d64656e6f"></a></td>
  </tr>
  <tr>
  <td>Ruby</td>
  <td><code>gem install sqlite-fastrand</code></td>
  <td><a target="_blank" rel="noopener noreferrer nofollow" href="https://camo.githubusercontent.com/ef29465ac66e1a03c2c67e098288988819efd1ea87c7f71d1c45ce0141a3e2e0/68747470733a2f2f696d672e736869656c64732e696f2f67656d2f762f73716c6974652d6661737472616e643f636f6c6f723d726564266c6f676f3d7275627967656d73266c6f676f436f6c6f723d7768697465"><img src="https://img.shields.io/gem/v/sqlite-fastrand?color=red&logo=rubygems&logoColor=white" alt="Gem" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/ef29465ac66e1a03c2c67e098288988819efd1ea87c7f71d1c45ce0141a3e2e0/68747470733a2f2f696d672e736869656c64732e696f2f67656d2f762f73716c6974652d6661737472616e643f636f6c6f723d726564266c6f676f3d7275627967656d73266c6f676f436f6c6f723d7768697465"></a></td>
  </tr>
  <tr>
  <td>Github Release</td>
  <td></td>
  <td><a target="_blank" rel="noopener noreferrer nofollow" href="https://camo.githubusercontent.com/80b9101bd9bfc91fdb779dd9e310f73f71edf9dcda82cd87a86dfd0480903ec0/68747470733a2f2f696d672e736869656c64732e696f2f6769746875622f762f7461672f6173673031372f73716c6974652d6661737472616e643f636f6c6f723d6c696768746772657926696e636c7564655f70726572656c6561736573266c6162656c3d4769746875622b72656c65617365266c6f676f3d676974687562"><img src="https://img.shields.io/github/v/tag/asg017/sqlite-fastrand?color=lightgrey&include_prereleases&label=Github+release&logo=github" alt="GitHub tag (latest SemVer pre-release)" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/80b9101bd9bfc91fdb779dd9e310f73f71edf9dcda82cd87a86dfd0480903ec0/68747470733a2f2f696d672e736869656c64732e696f2f6769746875622f762f7461672f6173673031372f73716c6974652d6661737472616e643f636f6c6f723d6c696768746772657926696e636c7564655f70726572656c6561736573266c6162656c3d4769746875622b72656c65617365266c6f676f3d676974687562"></a></td>
  </tr>
  <tr>
  <td>Rust</td>
  <td><code>cargo add sqlite-fastrand</code></td>
  <td><a href="https://crates.io/crates/sqlite-fastrand" rel="nofollow"><img src="https://img.shields.io/crates/v/sqlite-fastrand?logo=rust" alt="Crates.io" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/e6a05a7e4feb9f89cc13c43be64d2493335a19445a59b7b1440802d6058621bb/68747470733a2f2f696d672e736869656c64732e696f2f6372617465732f762f73716c6974652d6661737472616e643f6c6f676f3d72757374"></a></td>
  </tr>
  </tbody>
  </table></markdown-accessiblity-table>

  <p dir="auto">The <a href="https://github.com/asg017/sqlite-fastrand/releases">Releases page</a> contains pre-built binaries for Linux x86_64, MacOS, and Windows.</p>
  <div class="markdown-heading" dir="auto"><h3 class="heading-element" dir="auto">As a loadable extension</h3><a id="user-content-as-a-loadable-extension" class="anchor" aria-label="Permalink: As a loadable extension" href="#as-a-loadable-extension"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">If you want to use <code>sqlite-fastrand</code> as a <a href="https://www.sqlite.org/loadext.html" rel="nofollow">Runtime-loadable extension</a>, Download the <code>fastrand0.dylib</code> (for MacOS), <code>fastrand0.so</code> (Linux), or <code>fastrand0.dll</code> (Windows) file from a release and load it into your SQLite environment.</p>
  <blockquote>
  <p dir="auto"><strong>Note:</strong>
  The <code>0</code> in the filename (<code>fastrand0.dylib</code>/ <code>fastrand0.so</code>/<code>fastrand0.dll</code>) denotes the major version of <code>sqlite-fastrand</code>. Currently <code>sqlite-fastrand</code> is pre v1, so expect breaking changes in future versions.</p>
  </blockquote>
  <p dir="auto">For example, if you are using the <a href="https://www.sqlite.org/cli.html" rel="nofollow">SQLite CLI</a>, you can load the library like so:</p>
  <div class="highlight highlight-source-sql notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content=".load ./fastrand0
  select fastrand_version();
  -- v0.1.0"><pre>.load .<span class="pl-k">/</span>fastrand0
  <span class="pl-k">select</span> fastrand_version();
  <span class="pl-c"><span class="pl-c">--</span> v0.1.0</span></pre></div>
  <p dir="auto">Or in Python, using the builtin <a href="https://docs.python.org/3/library/sqlite3.html" rel="nofollow">sqlite3 module</a>:</p>
  <div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="import sqlite3
  con = sqlite3.connect(":memory:")
  con.enable_load_extension(True)
  con.load_extension("./fastrand0")
  print(con.execute("select fastrand_version()").fetchone())
  # ('v0.1.0',)"><pre><span class="pl-k">import</span> <span class="pl-s1">sqlite3</span>
  <span class="pl-s1">con</span> <span class="pl-c1">=</span> <span class="pl-s1">sqlite3</span>.<span class="pl-c1">connect</span>(<span class="pl-s">":memory:"</span>)
  <span class="pl-s1">con</span>.<span class="pl-c1">enable_load_extension</span>(<span class="pl-c1">True</span>)
  <span class="pl-s1">con</span>.<span class="pl-c1">load_extension</span>(<span class="pl-s">"./fastrand0"</span>)
  <span class="pl-en">print</span>(<span class="pl-s1">con</span>.<span class="pl-c1">execute</span>(<span class="pl-s">"select fastrand_version()"</span>).<span class="pl-c1">fetchone</span>())
  <span class="pl-c"># ('v0.1.0',)</span></pre></div>
  <p dir="auto">Or in Node.js using <a href="https://github.com/WiseLibs/better-sqlite3">better-sqlite3</a>:</p>
  <div class="highlight highlight-source-js notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="const Database = require("better-sqlite3");
  const db = new Database(":memory:");
  db.loadExtension("./fastrand0");
  console.log(db.prepare("select fastrand_version()").get());
  // { 'fastrand_version()': 'v0.1.0' }"><pre><span class="pl-k">const</span> <span class="pl-v">Database</span> <span class="pl-c1">=</span> <span class="pl-en">require</span><span class="pl-kos">(</span><span class="pl-s">"better-sqlite3"</span><span class="pl-kos">)</span><span class="pl-kos">;</span>
  <span class="pl-k">const</span> <span class="pl-s1">db</span> <span class="pl-c1">=</span> <span class="pl-k">new</span> <span class="pl-v">Database</span><span class="pl-kos">(</span><span class="pl-s">":memory:"</span><span class="pl-kos">)</span><span class="pl-kos">;</span>
  <span class="pl-s1">db</span><span class="pl-kos">.</span><span class="pl-en">loadExtension</span><span class="pl-kos">(</span><span class="pl-s">"./fastrand0"</span><span class="pl-kos">)</span><span class="pl-kos">;</span>
  <span class="pl-smi">console</span><span class="pl-kos">.</span><span class="pl-en">log</span><span class="pl-kos">(</span><span class="pl-s1">db</span><span class="pl-kos">.</span><span class="pl-en">prepare</span><span class="pl-kos">(</span><span class="pl-s">"select fastrand_version()"</span><span class="pl-kos">)</span><span class="pl-kos">.</span><span class="pl-en">get</span><span class="pl-kos">(</span><span class="pl-kos">)</span><span class="pl-kos">)</span><span class="pl-kos">;</span>
  <span class="pl-c">// { 'fastrand_version()': 'v0.1.0' }</span></pre></div>
  <p dir="auto">Or with <a href="https://datasette.io/" rel="nofollow">Datasette</a>:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="datasette data.db --load-extension ./fastrand0"><pre class="notranslate"><code>datasette data.db --load-extension ./fastrand0
  </code></pre></div>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Supporting</h2><a id="user-content-supporting" class="anchor" aria-label="Permalink: Supporting" href="#supporting"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">I (Alex 👋🏼) spent a lot of time and energy on this project and <a href="https://github.com/asg017?tab=repositories&q=&type=&language=&sort=stargazers">many other open source projects</a>. If your company or organization uses this library (or you're feeling generous), then please <a href="https://alexgarcia.xyz/work.html" rel="nofollow">consider supporting my work</a>, or share this project with a friend!</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">See also</h2><a id="user-content-see-also" class="anchor" aria-label="Permalink: See also" href="#see-also"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <ul dir="auto">
  <li><a href="https://github.com/asg017/sqlite-xsv">sqlite-xsv</a>, A SQLite extension for working with CSVs</li>
  <li><a href="https://github.com/asg017/sqlite-loadable-rs">sqlite-loadable</a>, A framework for writing SQLite extensions in Rust</li>
  <li><a href="https://github.com/asg017/sqlite-http">sqlite-http</a>, A SQLite extension for making HTTP requests</li>
  </ul>
  </article></div>


