# repos

node_id: R_kgDOHI-jKw
name: pypi-to-sqlite
full_name: simonw/pypi-to-sqlite
html_url: https://github.com/simonw/pypi-to-sqlite
description: Load data about Python packages from PyPI into SQLite
created_at: 2022-04-07T23:09:02Z
updated_at: 2025-02-01T18:58:51Z
pushed_at: 2024-03-06T06:16:31Z
language: Python
license: apache-2.0
topics: []
visibility: public
default_branch: main
permissions: {"admin": false, "maintain": false, "push": false, "triage": false, "pull": true}
readme:
  # pypi-to-sqlite

  [![PyPI](https://img.shields.io/pypi/v/pypi-to-sqlite.svg)](https://pypi.org/project/pypi-to-sqlite/)
  [![Changelog](https://img.shields.io/github/v/release/simonw/pypi-to-sqlite?include_prereleases&label=changelog)](https://github.com/simonw/pypi-to-sqlite/releases)
  [![Tests](https://github.com/simonw/pypi-to-sqlite/workflows/Test/badge.svg)](https://github.com/simonw/pypi-to-sqlite/actions?query=workflow%3ATest)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/pypi-to-sqlite/blob/master/LICENSE)

  Load data about Python packages from PyPI into SQLite

  ## Installation

  Install this tool using `pip`:

      pip install pypi-to-sqlite

  ## Usage

  To create a SQLite database with details of one or more packages, run:

      pypi-to-sqlite pypi.db datasette sqlite-utils

  You can also process JSON that you have previously saved to disk like so:

      curl -o datasette.json https://pypi.org/pypi/datasette/json
      pypi-to-sqlite pypi.db -f datasette.json

  The tool will create three tables: `packages`, `versions` and `releases`. The full table schema is shown below.

  To create the tables with a prefix, use `--prefix prefix`. For example:

      pypi-to-sqlite pypi.db datasette --prefix pypi_

  This will create tables called `pypi_packages`, `pypi_versions` and `pypi_releases`.

  ## Demo

  You can see examples of tables created using this tool running in [Datasette](https://datasette.io/) here:

  - [packages](https://datasette.io/content/pypi_packages)
  - [versions](https://datasette.io/content/pypi_versions)
  - [releases](https://datasette.io/content/pypi_releases)

  ## Database schema

  <!-- [[[cog
  import cog, json
  from pypi_to_sqlite import cli
  from click.testing import CliRunner
  import sqlite_utils
  import tempfile, pathlib
  tmpdir = pathlib.Path(tempfile.mkdtemp())
  db_path = str(tmpdir / "pypi.db")
  runner = CliRunner()
  result = runner.invoke(cli.cli, [db_path, "-f", "tests/datasette-block.json"])
  cog.out("```sql\n")
  cog.out(sqlite_utils.Database(db_path).schema)
  cog.out("\n```")
  ]]] -->
  ```sql
  CREATE TABLE [packages] (
     [name] TEXT PRIMARY KEY,
     [summary] TEXT,
     [classifiers] TEXT,
     [description] TEXT,
     [author] TEXT,
     [author_email] TEXT,
     [description_content_type] TEXT,
     [home_page] TEXT,
     [keywords] TEXT,
     [license] TEXT,
     [maintainer] TEXT,
     [maintainer_email] TEXT,
     [package_url] TEXT,
     [platform] TEXT,
     [project_url] TEXT,
     [project_urls] TEXT,
     [release_url] TEXT,
     [requires_dist] TEXT,
     [requires_python] TEXT,
     [version] TEXT,
     [yanked] INTEGER,
     [yanked_reason] TEXT
  );
  CREATE TABLE [versions] (
     [id] TEXT PRIMARY KEY,
     [package] TEXT REFERENCES [packages]([name]),
     [name] TEXT
  );
  CREATE TABLE [releases] (
     [md5_digest] TEXT PRIMARY KEY,
     [package] TEXT REFERENCES [packages]([name]),
     [version] TEXT REFERENCES [versions]([id]),
     [packagetype] TEXT,
     [filename] TEXT,
     [comment_text] TEXT,
     [digests] TEXT,
     [has_sig] INTEGER,
     [python_version] TEXT,
     [requires_python] TEXT,
     [size] INTEGER,
     [upload_time] TEXT,
     [upload_time_iso_8601] TEXT,
     [url] TEXT,
     [yanked] INTEGER,
     [yanked_reason] TEXT
  );
  ```
  <!-- [[[end]]] -->

  ## pypi-to-sqlite --help

  <!-- [[[cog
  result = runner.invoke(cli.cli, ["--help"])
  cog.out("```\n")
  cog.out(result.output.replace("Usage: cli", "Usage: pypi-to-sqlite"))
  cog.out("\n```")
  ]]] -->
  ```
  Usage: pypi-to-sqlite [OPTIONS] DB_PATH [PACKAGE]...

    Load data about Python packages from PyPI into SQLite

    Usage example:

        pypi-to-sqlite pypy.db datasette sqlite-utils

    Use -f to load data from a JSON file instead:

        pypi-to-sqlite pypy.db -f datasette.json

    Created tables will be packages, versions and releases

    To create tables called pypi_packages, pypi_versions, pypi_releases use
    --prefix pypi_:

        pypi-to-sqlite pypy.db datasette sqlite-utils --prefix pypi_

  Options:
    --version            Show the version and exit.
    -f, --file FILENAME  Import JSON from this file
    -d, --delay FLOAT    Wait this many seconds between requests
    --prefix TEXT        Prefix to use for the created database tables
    --help               Show this message and exit.

  ```
  <!-- [[[end]]] -->

  ## Development

  To contribute to this tool, first checkout the code. Then create a new virtual environment:

      cd pypi-to-sqlite
      python -m venv venv
      source venv/bin/activate

  Now install the dependencies and test dependencies:

      pip install -e '.[test]'

  To run the tests:

      pytest

readme_html:
  <div id="readme" class="md" data-path="README.md"><article class="markdown-body entry-content container-lg" itemprop="text"><div class="markdown-heading" dir="auto"><h1 class="heading-element" dir="auto">pypi-to-sqlite</h1><a id="user-content-pypi-to-sqlite" class="anchor" aria-label="Permalink: pypi-to-sqlite" href="#pypi-to-sqlite"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto"><a href="https://pypi.org/project/pypi-to-sqlite/" rel="nofollow"><img src="https://img.shields.io/pypi/v/pypi-to-sqlite.svg" alt="PyPI" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/b67c04a6587d2100f4ad33b2ae70d70fc1fbd485a29c5cae4b1cf9b9039aee86/68747470733a2f2f696d672e736869656c64732e696f2f707970692f762f707970692d746f2d73716c6974652e737667"></a>
  <a href="https://github.com/simonw/pypi-to-sqlite/releases"><img src="https://img.shields.io/github/v/release/simonw/pypi-to-sqlite?include_prereleases&label=changelog" alt="Changelog" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/48aeacd23ee3322e373006485967f0f705a53c4bd8c0192956ed17dd143bb6db/68747470733a2f2f696d672e736869656c64732e696f2f6769746875622f762f72656c656173652f73696d6f6e772f707970692d746f2d73716c6974653f696e636c7564655f70726572656c6561736573266c6162656c3d6368616e67656c6f67"></a>
  <a href="https://github.com/simonw/pypi-to-sqlite/actions?query=workflow%3ATest"><img src="https://github.com/simonw/pypi-to-sqlite/workflows/Test/badge.svg" alt="Tests" style="max-width: 100%;"></a>
  <a href="https://github.com/simonw/pypi-to-sqlite/blob/master/LICENSE"><img src="https://img.shields.io/badge/license-Apache%202.0-blue.svg" alt="License" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/c355f200ea90fddaa407b6eaab303663a669248ea3ca7b1fcf77dbe04ff5f48c/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6c6963656e73652d417061636865253230322e302d626c75652e737667"></a></p>
  <p dir="auto">Load data about Python packages from PyPI into SQLite</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Installation</h2><a id="user-content-installation" class="anchor" aria-label="Permalink: Installation" href="#installation"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">Install this tool using <code>pip</code>:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="pip install pypi-to-sqlite"><pre class="notranslate"><code>pip install pypi-to-sqlite
  </code></pre></div>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Usage</h2><a id="user-content-usage" class="anchor" aria-label="Permalink: Usage" href="#usage"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">To create a SQLite database with details of one or more packages, run:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="pypi-to-sqlite pypi.db datasette sqlite-utils"><pre class="notranslate"><code>pypi-to-sqlite pypi.db datasette sqlite-utils
  </code></pre></div>
  <p dir="auto">You can also process JSON that you have previously saved to disk like so:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="curl -o datasette.json https://pypi.org/pypi/datasette/json
  pypi-to-sqlite pypi.db -f datasette.json"><pre class="notranslate"><code>curl -o datasette.json https://pypi.org/pypi/datasette/json
  pypi-to-sqlite pypi.db -f datasette.json
  </code></pre></div>
  <p dir="auto">The tool will create three tables: <code>packages</code>, <code>versions</code> and <code>releases</code>. The full table schema is shown below.</p>
  <p dir="auto">To create the tables with a prefix, use <code>--prefix prefix</code>. For example:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="pypi-to-sqlite pypi.db datasette --prefix pypi_"><pre class="notranslate"><code>pypi-to-sqlite pypi.db datasette --prefix pypi_
  </code></pre></div>
  <p dir="auto">This will create tables called <code>pypi_packages</code>, <code>pypi_versions</code> and <code>pypi_releases</code>.</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Demo</h2><a id="user-content-demo" class="anchor" aria-label="Permalink: Demo" href="#demo"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">You can see examples of tables created using this tool running in <a href="https://datasette.io/" rel="nofollow">Datasette</a> here:</p>
  <ul dir="auto">
  <li><a href="https://datasette.io/content/pypi_packages" rel="nofollow">packages</a></li>
  <li><a href="https://datasette.io/content/pypi_versions" rel="nofollow">versions</a></li>
  <li><a href="https://datasette.io/content/pypi_releases" rel="nofollow">releases</a></li>
  </ul>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Database schema</h2><a id="user-content-database-schema" class="anchor" aria-label="Permalink: Database schema" href="#database-schema"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>

  <div class="highlight highlight-source-sql notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="CREATE TABLE [packages] (
     [name] TEXT PRIMARY KEY,
     [summary] TEXT,
     [classifiers] TEXT,
     [description] TEXT,
     [author] TEXT,
     [author_email] TEXT,
     [description_content_type] TEXT,
     [home_page] TEXT,
     [keywords] TEXT,
     [license] TEXT,
     [maintainer] TEXT,
     [maintainer_email] TEXT,
     [package_url] TEXT,
     [platform] TEXT,
     [project_url] TEXT,
     [project_urls] TEXT,
     [release_url] TEXT,
     [requires_dist] TEXT,
     [requires_python] TEXT,
     [version] TEXT,
     [yanked] INTEGER,
     [yanked_reason] TEXT
  );
  CREATE TABLE [versions] (
     [id] TEXT PRIMARY KEY,
     [package] TEXT REFERENCES [packages]([name]),
     [name] TEXT
  );
  CREATE TABLE [releases] (
     [md5_digest] TEXT PRIMARY KEY,
     [package] TEXT REFERENCES [packages]([name]),
     [version] TEXT REFERENCES [versions]([id]),
     [packagetype] TEXT,
     [filename] TEXT,
     [comment_text] TEXT,
     [digests] TEXT,
     [has_sig] INTEGER,
     [python_version] TEXT,
     [requires_python] TEXT,
     [size] INTEGER,
     [upload_time] TEXT,
     [upload_time_iso_8601] TEXT,
     [url] TEXT,
     [yanked] INTEGER,
     [yanked_reason] TEXT
  );"><pre>CREATE TABLE [packages] (
     [name] <span class="pl-k">TEXT</span> <span class="pl-k">PRIMARY KEY</span>,
     [summary] <span class="pl-k">TEXT</span>,
     [classifiers] <span class="pl-k">TEXT</span>,
     [description] <span class="pl-k">TEXT</span>,
     [author] <span class="pl-k">TEXT</span>,
     [author_email] <span class="pl-k">TEXT</span>,
     [description_content_type] <span class="pl-k">TEXT</span>,
     [home_page] <span class="pl-k">TEXT</span>,
     [keywords] <span class="pl-k">TEXT</span>,
     [license] <span class="pl-k">TEXT</span>,
     [maintainer] <span class="pl-k">TEXT</span>,
     [maintainer_email] <span class="pl-k">TEXT</span>,
     [package_url] <span class="pl-k">TEXT</span>,
     [platform] <span class="pl-k">TEXT</span>,
     [project_url] <span class="pl-k">TEXT</span>,
     [project_urls] <span class="pl-k">TEXT</span>,
     [release_url] <span class="pl-k">TEXT</span>,
     [requires_dist] <span class="pl-k">TEXT</span>,
     [requires_python] <span class="pl-k">TEXT</span>,
     [version] <span class="pl-k">TEXT</span>,
     [yanked] <span class="pl-k">INTEGER</span>,
     [yanked_reason] <span class="pl-k">TEXT</span>
  );
  CREATE TABLE [versions] (
     [id] <span class="pl-k">TEXT</span> <span class="pl-k">PRIMARY KEY</span>,
     [package] <span class="pl-k">TEXT</span> <span class="pl-k">REFERENCES</span> [packages]([name]),
     [name] <span class="pl-k">TEXT</span>
  );
  CREATE TABLE [releases] (
     [md5_digest] <span class="pl-k">TEXT</span> <span class="pl-k">PRIMARY KEY</span>,
     [package] <span class="pl-k">TEXT</span> <span class="pl-k">REFERENCES</span> [packages]([name]),
     [version] <span class="pl-k">TEXT</span> <span class="pl-k">REFERENCES</span> [versions]([id]),
     [packagetype] <span class="pl-k">TEXT</span>,
     [filename] <span class="pl-k">TEXT</span>,
     [comment_text] <span class="pl-k">TEXT</span>,
     [digests] <span class="pl-k">TEXT</span>,
     [has_sig] <span class="pl-k">INTEGER</span>,
     [python_version] <span class="pl-k">TEXT</span>,
     [requires_python] <span class="pl-k">TEXT</span>,
     [size] <span class="pl-k">INTEGER</span>,
     [upload_time] <span class="pl-k">TEXT</span>,
     [upload_time_iso_8601] <span class="pl-k">TEXT</span>,
     [url] <span class="pl-k">TEXT</span>,
     [yanked] <span class="pl-k">INTEGER</span>,
     [yanked_reason] <span class="pl-k">TEXT</span>
  );</pre></div>

  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">pypi-to-sqlite --help</h2><a id="user-content-pypi-to-sqlite---help" class="anchor" aria-label="Permalink: pypi-to-sqlite --help" href="#pypi-to-sqlite---help"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>

  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="Usage: pypi-to-sqlite [OPTIONS] DB_PATH [PACKAGE]...

    Load data about Python packages from PyPI into SQLite

    Usage example:

        pypi-to-sqlite pypy.db datasette sqlite-utils

    Use -f to load data from a JSON file instead:

        pypi-to-sqlite pypy.db -f datasette.json

    Created tables will be packages, versions and releases

    To create tables called pypi_packages, pypi_versions, pypi_releases use
    --prefix pypi_:

        pypi-to-sqlite pypy.db datasette sqlite-utils --prefix pypi_

  Options:
    --version            Show the version and exit.
    -f, --file FILENAME  Import JSON from this file
    -d, --delay FLOAT    Wait this many seconds between requests
    --prefix TEXT        Prefix to use for the created database tables
    --help               Show this message and exit.
  "><pre class="notranslate"><code>Usage: pypi-to-sqlite [OPTIONS] DB_PATH [PACKAGE]...

    Load data about Python packages from PyPI into SQLite

    Usage example:

        pypi-to-sqlite pypy.db datasette sqlite-utils

    Use -f to load data from a JSON file instead:

        pypi-to-sqlite pypy.db -f datasette.json

    Created tables will be packages, versions and releases

    To create tables called pypi_packages, pypi_versions, pypi_releases use
    --prefix pypi_:

        pypi-to-sqlite pypy.db datasette sqlite-utils --prefix pypi_

  Options:
    --version            Show the version and exit.
    -f, --file FILENAME  Import JSON from this file
    -d, --delay FLOAT    Wait this many seconds between requests
    --prefix TEXT        Prefix to use for the created database tables
    --help               Show this message and exit.

  </code></pre></div>

  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Development</h2><a id="user-content-development" class="anchor" aria-label="Permalink: Development" href="#development"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">To contribute to this tool, first checkout the code. Then create a new virtual environment:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="cd pypi-to-sqlite
  python -m venv venv
  source venv/bin/activate"><pre class="notranslate"><code>cd pypi-to-sqlite
  python -m venv venv
  source venv/bin/activate
  </code></pre></div>
  <p dir="auto">Now install the dependencies and test dependencies:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="pip install -e '.[test]'"><pre class="notranslate"><code>pip install -e '.[test]'
  </code></pre></div>
  <p dir="auto">To run the tests:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="pytest"><pre class="notranslate"><code>pytest
  </code></pre></div>
  </article></div>


node_id: R_kgDOHPAsug
name: sqlite-lines
full_name: asg017/sqlite-lines
html_url: https://github.com/asg017/sqlite-lines
description: A SQLite extension for reading large files line-by-line (NDJSON, logs, txt, etc.)
created_at: 2022-04-25T19:11:46Z
updated_at: 2025-03-28T07:54:17Z
pushed_at: 2023-10-07T06:04:44Z
language: C
license: mit
topics: ["sqlite", "sqlite-extension"]
visibility: public
default_branch: main
permissions: {"admin": false, "maintain": false, "push": false, "triage": false, "pull": true}
readme:
  # sqlite-lines

  `sqlite-lines` is a SQLite extension for reading lines from a file or blob.

  <img src="./benchmarks/calc.png" alt="Benchmark between sqlite-lines and various other data processing tools" width="600"/>

  <small>See [Benchmarks](./benchmarks) for more info.</small>

  ## Usage

  ```sql
  .load ./lines0
  select line from lines_read('logs.txt');
  ```

  `sqlite-lines` is great for line-oriented datasets, like [ndjson](https://ndjson.org/) or [JSON Lines](https://jsonlines.org/), when paired with SQLite's [JSON support](https://www.sqlite.org/json1.html). Here, we calculate the top 5 country participants in Google's [Quick, Draw!](https://quickdraw.withgoogle.com/data) dataset for [`calendars.ndjson`](https://storage.googleapis.com/quickdraw_dataset/full/simplified/calendar.ndjson):

  ```sql
  select
    line ->> '$.countrycode' as countrycode,
    count(*)
  from lines_read('./calendar.ndjson')
  group by 1
  order by 2 desc
  limit 5;
  /*
  ┌─────────────┬──────────┐
  │ countrycode │ count(*) │
  ├─────────────┼──────────┤
  │ US          │ 141001   │
  │ GB          │ 22560    │
  │ CA          │ 11759    │
  │ RU          │ 9250     │
  │ DE          │ 8748     │
  └─────────────┴──────────┘
  */
  ```

  Use the SQLite CLI's [`fsdir()`](https://sqlite.org/cli.html#file_i_o_functions) table functions with `lines_read()` to read lines from every file in a directory.

  ```sql
  select
    name as file,
    lines.rowid as line_number,
    line
  from fsdir('logs')
  join lines_read(name) as lines
  where name like '%.txt';
  /*
  ┌─────────────────────┬──────┐
  │ file  │ line_number | line │
  ├───────┼─────────────┤──────┤
  | a.txt | 1           | x    |
  | a.txt | 2           | y    |
  | a.txt | 3           | z    |
  | b.txt | 1           | xx   |
  | b.txt | 2           | yy   |
  | c.txt | 1           | xxx  |
  └───────┴─────────────┴──────┘
  */
  ```

  ## Documentation

  See [`docs.md`](./docs.md) for a full API Reference and detailed documentation.

  ## Installing

  | Language       | Install                                                    |                                                                                                                                                                                           |
  | -------------- | ---------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
  | Python         | `pip install sqlite-lines`                                   | [![PyPI](https://img.shields.io/pypi/v/sqlite-lines.svg?color=blue&logo=python&logoColor=white)](https://pypi.org/project/sqlite-lines/)                                                      |
  | Datasette      | `datasette install datasette-sqlite-lines`                   | [![Datasette](https://img.shields.io/pypi/v/datasette-sqlite-lines.svg?color=B6B6D9&label=Datasette+plugin&logoColor=white&logo=python)](https://datasette.io/plugins/datasette-sqlite-lines) |
  | Node.js        | `npm install sqlite-lines`                                   | [![npm](https://img.shields.io/npm/v/sqlite-lines.svg?color=green&logo=nodedotjs&logoColor=white)](https://www.npmjs.com/package/sqlite-lines)                                                |
  | Deno           | [`deno.land/x/sqlite_lines`](https://deno.land/x/sqlite_lines) | [![deno.land/x release](https://img.shields.io/github/v/release/asg017/sqlite-lines?color=fef8d2&include_prereleases&label=deno.land%2Fx&logo=deno)](https://deno.land/x/sqlite_lines)        |
  | Ruby           | `gem install sqlite-lines`                                   | ![Gem](https://img.shields.io/gem/v/sqlite-lines?color=red&logo=rubygems&logoColor=white)                                                                                                   |
  | Github Release |                                                            | ![GitHub tag (latest SemVer pre-release)](https://img.shields.io/github/v/tag/asg017/sqlite-lines?color=lightgrey&include_prereleases&label=Github+release&logo=github)                     |

  <!--
  | Elixir         | [`hex.pm/packages/sqlite_lines`](https://hex.pm/packages/sqlite_lines) | [![Hex.pm](https://img.shields.io/hexpm/v/sqlite_lines?color=purple&logo=elixir)](https://hex.pm/packages/sqlite_lines)                                                                       |
  | Go             | `go get -u github.com/asg017/sqlite-lines/bindings/go`               | [![Go Reference](https://pkg.go.dev/badge/github.com/asg017/sqlite-lines/bindings/go.svg)](https://pkg.go.dev/github.com/asg017/sqlite-lines/bindings/go)                                     |
  | Rust           | `cargo add sqlite-lines`                                             | [![Crates.io](https://img.shields.io/crates/v/sqlite-lines?logo=rust)](https://crates.io/crates/sqlite-lines)                                                                                 |
  -->

  The [Releases page](https://github.com/asg017/sqlite-lines/releases) contains pre-built binaries for Linux amd64 and MacOS (amd64, no arm).

  ### As a loadable extension

  If you want to use `sqlite-lines` as a [Runtime-loadable extension](https://www.sqlite.org/loadext.html), Download the `lines0.dylib` (for MacOS) or `lines0.so` file from a release and load it into your SQLite environment.

  > **Note:**
  > The `0` in the filename (`lines0.dylib` or `lines0.so`) denotes the major version of `sqlite-lines`. Currently `sqlite-lines` is pre v1, so expect breaking changes in future versions.

  For example, if you are using the [SQLite CLI](https://www.sqlite.org/cli.html), you can load the library like so:

  ```sql
  .load ./lines0
  select lines_version();
  -- v0.0.1
  ```

  Or in Python, using the builtin [sqlite3 module](https://docs.python.org/3/library/sqlite3.html):

  ```python
  import sqlite3

  con = sqlite3.connect(":memory:")

  con.enable_load_extension(True)
  con.load_extension("./lines0")

  print(con.execute("select lines_version()").fetchone())
  # ('v0.0.1',)
  ```

  Or in Node.js using [better-sqlite3](https://github.com/WiseLibs/better-sqlite3):

  ```javascript
  const Database = require("better-sqlite3");
  const db = new Database(":memory:");

  db.loadExtension("./lines0");

  console.log(db.prepare("select lines_version()").get());
  // { 'lines_version()': 'v0.0.1' }
  ```

  Or with [Datasette](https://datasette.io/) (using the "no filesystem" version to limit security vulnerabilities):

  ```
  datasette data.db --load-extension ./lines_nofs0
  ```

  Windows is not supported - [yet](https://github.com/asg017/sqlite-lines/issues/4)!

  ### From the browser with WASM/JavaScript

  `sqlite-lines` is also distributed as a standalone [SQL.js](https://github.com/sql-js/sql.js) library. It's essentially a fork of the original SQL.js library, with the addition of `sqlite-lines` functions like `lines_version()` and `lines()`.

  Check out [this Observable notebook](https://observablehq.com/@asg017/introducing-sqlite-lines) for the full demonstration. The [Releases page](https://github.com/asg017/sqlite-lines/releases) contains the JavaScript and WASM files.

  ### The sqlite-lines CLI

  `sqlite-lines` comes with an example CLI modeled after [ndjson-cli](https://github.com/mbostock/ndjson-cli) that demos the speed and versatility of `sqlite-lines`. Download a pre-compiled version from the [Releases page](https://github.com/asg017/sqlite-lines/releases), or build yourself with:

  ```
  make cli
  ./dist/sqlite-lines
  ```

  The `sqlite-lines` CLI reads data from stdin and applies transformations with SQL code through its arguments.

  The first argument should be a SQL expression that is used transform a single line from stdlin. The available columns are `rowid`, which is the "line number" that is being processed, and `d`, an alias for `line`, which is the text content of the current line (inspired by ndjson-cli). For example, to uppercase every line from a file with [`upper()`](https://www.sqlite.org/lang_corefunc.html#upper):

  ```bash
  $ cat names.txt | sqlite-lines 'rowid || upper(d)'
  1ALEX
  2BRIAN
  3CRAIG
  ```

  This includes SQLite's new JSON `->` and `->>` operators for NDJSON/JSONL files:

  ```bash
  $ cat data.ndjson | sqlite-lines 'd ->> "$.id"'
  $ cat data.ndjson | sqlite-lines 'json_object("name", d ->> "$.name", "age": d ->> "$.stats.age")'
  ```

  The second argument is another SQL expression that's used in the WHERE statement of the underlying SQL query to filter out lines.

  ```bash
  # get the names of all people older than 40
  cat data.ndjson | sqlite-lines 'd ->> "$.name"' 'd ->> "$.age" > 40'
  ```

  The third argument is another SQL expression that's used in the GROUP BY statement of the underlying SQL query to aggregate lines.

  ### A Note on CSV Parsing

  `sqlite-lines` isn't a great option for CSVs. Technically you can, but the moment your data has a `\n` character in a field or header, then you'll get corrupted results.

  Instead, you should use the "official" [CSV Virtual Table](https://www.sqlite.org/csv.html), or use the [`.import`](https://www.sqlite.org/cli.html#csv) command in the SQLite CLI.

readme_html:
  <div id="readme" class="md" data-path="README.md"><article class="markdown-body entry-content container-lg" itemprop="text"><div class="markdown-heading" dir="auto"><h1 class="heading-element" dir="auto">sqlite-lines</h1><a id="user-content-sqlite-lines" class="anchor" aria-label="Permalink: sqlite-lines" href="#sqlite-lines"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto"><code>sqlite-lines</code> is a SQLite extension for reading lines from a file or blob.</p>
  <p dir="auto"><a target="_blank" rel="noopener noreferrer" href="./benchmarks/calc.png"><img src="./benchmarks/calc.png" alt="Benchmark between sqlite-lines and various other data processing tools" width="600" style="max-width: 100%;"></a></p>
  <p dir="auto">See <a href="./benchmarks">Benchmarks</a> for more info.</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Usage</h2><a id="user-content-usage" class="anchor" aria-label="Permalink: Usage" href="#usage"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <div class="highlight highlight-source-sql notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content=".load ./lines0
  select line from lines_read('logs.txt');"><pre>.load .<span class="pl-k">/</span>lines0
  <span class="pl-k">select</span> <span class="pl-k">line</span> <span class="pl-k">from</span> lines_read(<span class="pl-s"><span class="pl-pds">'</span>logs.txt<span class="pl-pds">'</span></span>);</pre></div>
  <p dir="auto"><code>sqlite-lines</code> is great for line-oriented datasets, like <a href="https://ndjson.org/" rel="nofollow">ndjson</a> or <a href="https://jsonlines.org/" rel="nofollow">JSON Lines</a>, when paired with SQLite's <a href="https://www.sqlite.org/json1.html" rel="nofollow">JSON support</a>. Here, we calculate the top 5 country participants in Google's <a href="https://quickdraw.withgoogle.com/data" rel="nofollow">Quick, Draw!</a> dataset for <a href="https://storage.googleapis.com/quickdraw_dataset/full/simplified/calendar.ndjson" rel="nofollow"><code>calendars.ndjson</code></a>:</p>
  <div class="highlight highlight-source-sql notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="select
    line ->> '$.countrycode' as countrycode,
    count(*)
  from lines_read('./calendar.ndjson')
  group by 1
  order by 2 desc
  limit 5;
  /*
  ┌─────────────┬──────────┐
  │ countrycode │ count(*) │
  ├─────────────┼──────────┤
  │ US          │ 141001   │
  │ GB          │ 22560    │
  │ CA          │ 11759    │
  │ RU          │ 9250     │
  │ DE          │ 8748     │
  └─────────────┴──────────┘
  */"><pre><span class="pl-k">select</span>
    <span class="pl-k">line</span> <span class="pl-k">-</span><span class="pl-k">>></span> <span class="pl-s"><span class="pl-pds">'</span>$.countrycode<span class="pl-pds">'</span></span> <span class="pl-k">as</span> countrycode,
    <span class="pl-c1">count</span>(<span class="pl-k">*</span>)
  <span class="pl-k">from</span> lines_read(<span class="pl-s"><span class="pl-pds">'</span>./calendar.ndjson<span class="pl-pds">'</span></span>)
  <span class="pl-k">group by</span> <span class="pl-c1">1</span>
  <span class="pl-k">order by</span> <span class="pl-c1">2</span> <span class="pl-k">desc</span>
  <span class="pl-k">limit</span> <span class="pl-c1">5</span>;
  <span class="pl-c"><span class="pl-c">/*</span></span>
  <span class="pl-c">┌─────────────┬──────────┐</span>
  <span class="pl-c">│ countrycode │ count(*) │</span>
  <span class="pl-c">├─────────────┼──────────┤</span>
  <span class="pl-c">│ US          │ 141001   │</span>
  <span class="pl-c">│ GB          │ 22560    │</span>
  <span class="pl-c">│ CA          │ 11759    │</span>
  <span class="pl-c">│ RU          │ 9250     │</span>
  <span class="pl-c">│ DE          │ 8748     │</span>
  <span class="pl-c">└─────────────┴──────────┘</span>
  <span class="pl-c"><span class="pl-c">*/</span></span></pre></div>
  <p dir="auto">Use the SQLite CLI's <a href="https://sqlite.org/cli.html#file_i_o_functions" rel="nofollow"><code>fsdir()</code></a> table functions with <code>lines_read()</code> to read lines from every file in a directory.</p>
  <div class="highlight highlight-source-sql notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="select
    name as file,
    lines.rowid as line_number,
    line
  from fsdir('logs')
  join lines_read(name) as lines
  where name like '%.txt';
  /*
  ┌─────────────────────┬──────┐
  │ file  │ line_number | line │
  ├───────┼─────────────┤──────┤
  | a.txt | 1           | x    |
  | a.txt | 2           | y    |
  | a.txt | 3           | z    |
  | b.txt | 1           | xx   |
  | b.txt | 2           | yy   |
  | c.txt | 1           | xxx  |
  └───────┴─────────────┴──────┘
  */"><pre><span class="pl-k">select</span>
    name <span class="pl-k">as</span> file,
    <span class="pl-c1">lines</span>.<span class="pl-c1">rowid</span> <span class="pl-k">as</span> line_number,
    <span class="pl-k">line</span>
  <span class="pl-k">from</span> fsdir(<span class="pl-s"><span class="pl-pds">'</span>logs<span class="pl-pds">'</span></span>)
  <span class="pl-k">join</span> lines_read(name) <span class="pl-k">as</span> lines
  <span class="pl-k">where</span> name <span class="pl-k">like</span> <span class="pl-s"><span class="pl-pds">'</span>%.txt<span class="pl-pds">'</span></span>;
  <span class="pl-c"><span class="pl-c">/*</span></span>
  <span class="pl-c">┌─────────────────────┬──────┐</span>
  <span class="pl-c">│ file  │ line_number | line │</span>
  <span class="pl-c">├───────┼─────────────┤──────┤</span>
  <span class="pl-c">| a.txt | 1           | x    |</span>
  <span class="pl-c">| a.txt | 2           | y    |</span>
  <span class="pl-c">| a.txt | 3           | z    |</span>
  <span class="pl-c">| b.txt | 1           | xx   |</span>
  <span class="pl-c">| b.txt | 2           | yy   |</span>
  <span class="pl-c">| c.txt | 1           | xxx  |</span>
  <span class="pl-c">└───────┴─────────────┴──────┘</span>
  <span class="pl-c"><span class="pl-c">*/</span></span></pre></div>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Documentation</h2><a id="user-content-documentation" class="anchor" aria-label="Permalink: Documentation" href="#documentation"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">See <a href="./docs.md"><code>docs.md</code></a> for a full API Reference and detailed documentation.</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Installing</h2><a id="user-content-installing" class="anchor" aria-label="Permalink: Installing" href="#installing"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <markdown-accessiblity-table><table>
  <thead>
  <tr>
  <th>Language</th>
  <th>Install</th>
  <th></th>
  </tr>
  </thead>
  <tbody>
  <tr>
  <td>Python</td>
  <td><code>pip install sqlite-lines</code></td>
  <td><a href="https://pypi.org/project/sqlite-lines/" rel="nofollow"><img src="https://img.shields.io/pypi/v/sqlite-lines.svg?color=blue&logo=python&logoColor=white" alt="PyPI" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/d6af0247157eaff59b9bf9fab6a34ca30c47b7de976533350f703fa9d14a40a3/68747470733a2f2f696d672e736869656c64732e696f2f707970692f762f73716c6974652d6c696e65732e7376673f636f6c6f723d626c7565266c6f676f3d707974686f6e266c6f676f436f6c6f723d7768697465"></a></td>
  </tr>
  <tr>
  <td>Datasette</td>
  <td><code>datasette install datasette-sqlite-lines</code></td>
  <td><a href="https://datasette.io/plugins/datasette-sqlite-lines" rel="nofollow"><img src="https://img.shields.io/pypi/v/datasette-sqlite-lines.svg?color=B6B6D9&label=Datasette+plugin&logoColor=white&logo=python" alt="Datasette" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/9052e90c280ec7960fe6402fb2a5afb296146d5ecd2aacae71022d848738b3cb/68747470733a2f2f696d672e736869656c64732e696f2f707970692f762f6461746173657474652d73716c6974652d6c696e65732e7376673f636f6c6f723d423642364439266c6162656c3d4461746173657474652b706c7567696e266c6f676f436f6c6f723d7768697465266c6f676f3d707974686f6e"></a></td>
  </tr>
  <tr>
  <td>Node.js</td>
  <td><code>npm install sqlite-lines</code></td>
  <td><a href="https://www.npmjs.com/package/sqlite-lines" rel="nofollow"><img src="https://img.shields.io/npm/v/sqlite-lines.svg?color=green&logo=nodedotjs&logoColor=white" alt="npm" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/25db89155302e5621699b4a30248ff3f0e01739bb0255f80bcb69b4d36c587f9/68747470733a2f2f696d672e736869656c64732e696f2f6e706d2f762f73716c6974652d6c696e65732e7376673f636f6c6f723d677265656e266c6f676f3d6e6f6465646f746a73266c6f676f436f6c6f723d7768697465"></a></td>
  </tr>
  <tr>
  <td>Deno</td>
  <td><a href="https://deno.land/x/sqlite_lines" rel="nofollow"><code>deno.land/x/sqlite_lines</code></a></td>
  <td><a href="https://deno.land/x/sqlite_lines" rel="nofollow"><img src="https://img.shields.io/github/v/release/asg017/sqlite-lines?color=fef8d2&include_prereleases&label=deno.land%2Fx&logo=deno" alt="deno.land/x release" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/3640eb589fca101e76bc053c4fac1404ae75df1fde819ed73590bca0dac38426/68747470733a2f2f696d672e736869656c64732e696f2f6769746875622f762f72656c656173652f6173673031372f73716c6974652d6c696e65733f636f6c6f723d66656638643226696e636c7564655f70726572656c6561736573266c6162656c3d64656e6f2e6c616e6425324678266c6f676f3d64656e6f"></a></td>
  </tr>
  <tr>
  <td>Ruby</td>
  <td><code>gem install sqlite-lines</code></td>
  <td><a target="_blank" rel="noopener noreferrer nofollow" href="https://camo.githubusercontent.com/c4c97b2b3db78181ed66ef37026ee2a24d3d9121a672995d779d19b215e79b4e/68747470733a2f2f696d672e736869656c64732e696f2f67656d2f762f73716c6974652d6c696e65733f636f6c6f723d726564266c6f676f3d7275627967656d73266c6f676f436f6c6f723d7768697465"><img src="https://img.shields.io/gem/v/sqlite-lines?color=red&logo=rubygems&logoColor=white" alt="Gem" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/c4c97b2b3db78181ed66ef37026ee2a24d3d9121a672995d779d19b215e79b4e/68747470733a2f2f696d672e736869656c64732e696f2f67656d2f762f73716c6974652d6c696e65733f636f6c6f723d726564266c6f676f3d7275627967656d73266c6f676f436f6c6f723d7768697465"></a></td>
  </tr>
  <tr>
  <td>Github Release</td>
  <td></td>
  <td><a target="_blank" rel="noopener noreferrer nofollow" href="https://camo.githubusercontent.com/837c2a93a36e91e70c32470e23ee6ca1f9d1b0482bdfc360e3ec035ac6adcd9b/68747470733a2f2f696d672e736869656c64732e696f2f6769746875622f762f7461672f6173673031372f73716c6974652d6c696e65733f636f6c6f723d6c696768746772657926696e636c7564655f70726572656c6561736573266c6162656c3d4769746875622b72656c65617365266c6f676f3d676974687562"><img src="https://img.shields.io/github/v/tag/asg017/sqlite-lines?color=lightgrey&include_prereleases&label=Github+release&logo=github" alt="GitHub tag (latest SemVer pre-release)" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/837c2a93a36e91e70c32470e23ee6ca1f9d1b0482bdfc360e3ec035ac6adcd9b/68747470733a2f2f696d672e736869656c64732e696f2f6769746875622f762f7461672f6173673031372f73716c6974652d6c696e65733f636f6c6f723d6c696768746772657926696e636c7564655f70726572656c6561736573266c6162656c3d4769746875622b72656c65617365266c6f676f3d676974687562"></a></td>
  </tr>
  </tbody>
  </table></markdown-accessiblity-table>

  <p dir="auto">The <a href="https://github.com/asg017/sqlite-lines/releases">Releases page</a> contains pre-built binaries for Linux amd64 and MacOS (amd64, no arm).</p>
  <div class="markdown-heading" dir="auto"><h3 class="heading-element" dir="auto">As a loadable extension</h3><a id="user-content-as-a-loadable-extension" class="anchor" aria-label="Permalink: As a loadable extension" href="#as-a-loadable-extension"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">If you want to use <code>sqlite-lines</code> as a <a href="https://www.sqlite.org/loadext.html" rel="nofollow">Runtime-loadable extension</a>, Download the <code>lines0.dylib</code> (for MacOS) or <code>lines0.so</code> file from a release and load it into your SQLite environment.</p>
  <blockquote>
  <p dir="auto"><strong>Note:</strong>
  The <code>0</code> in the filename (<code>lines0.dylib</code> or <code>lines0.so</code>) denotes the major version of <code>sqlite-lines</code>. Currently <code>sqlite-lines</code> is pre v1, so expect breaking changes in future versions.</p>
  </blockquote>
  <p dir="auto">For example, if you are using the <a href="https://www.sqlite.org/cli.html" rel="nofollow">SQLite CLI</a>, you can load the library like so:</p>
  <div class="highlight highlight-source-sql notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content=".load ./lines0
  select lines_version();
  -- v0.0.1"><pre>.load .<span class="pl-k">/</span>lines0
  <span class="pl-k">select</span> lines_version();
  <span class="pl-c"><span class="pl-c">--</span> v0.0.1</span></pre></div>
  <p dir="auto">Or in Python, using the builtin <a href="https://docs.python.org/3/library/sqlite3.html" rel="nofollow">sqlite3 module</a>:</p>
  <div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="import sqlite3

  con = sqlite3.connect(":memory:")

  con.enable_load_extension(True)
  con.load_extension("./lines0")

  print(con.execute("select lines_version()").fetchone())
  # ('v0.0.1',)"><pre><span class="pl-k">import</span> <span class="pl-s1">sqlite3</span>

  <span class="pl-s1">con</span> <span class="pl-c1">=</span> <span class="pl-s1">sqlite3</span>.<span class="pl-c1">connect</span>(<span class="pl-s">":memory:"</span>)

  <span class="pl-s1">con</span>.<span class="pl-c1">enable_load_extension</span>(<span class="pl-c1">True</span>)
  <span class="pl-s1">con</span>.<span class="pl-c1">load_extension</span>(<span class="pl-s">"./lines0"</span>)

  <span class="pl-en">print</span>(<span class="pl-s1">con</span>.<span class="pl-c1">execute</span>(<span class="pl-s">"select lines_version()"</span>).<span class="pl-c1">fetchone</span>())
  <span class="pl-c"># ('v0.0.1',)</span></pre></div>
  <p dir="auto">Or in Node.js using <a href="https://github.com/WiseLibs/better-sqlite3">better-sqlite3</a>:</p>
  <div class="highlight highlight-source-js notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="const Database = require("better-sqlite3");
  const db = new Database(":memory:");

  db.loadExtension("./lines0");

  console.log(db.prepare("select lines_version()").get());
  // { 'lines_version()': 'v0.0.1' }"><pre><span class="pl-k">const</span> <span class="pl-v">Database</span> <span class="pl-c1">=</span> <span class="pl-en">require</span><span class="pl-kos">(</span><span class="pl-s">"better-sqlite3"</span><span class="pl-kos">)</span><span class="pl-kos">;</span>
  <span class="pl-k">const</span> <span class="pl-s1">db</span> <span class="pl-c1">=</span> <span class="pl-k">new</span> <span class="pl-v">Database</span><span class="pl-kos">(</span><span class="pl-s">":memory:"</span><span class="pl-kos">)</span><span class="pl-kos">;</span>

  <span class="pl-s1">db</span><span class="pl-kos">.</span><span class="pl-en">loadExtension</span><span class="pl-kos">(</span><span class="pl-s">"./lines0"</span><span class="pl-kos">)</span><span class="pl-kos">;</span>

  <span class="pl-smi">console</span><span class="pl-kos">.</span><span class="pl-en">log</span><span class="pl-kos">(</span><span class="pl-s1">db</span><span class="pl-kos">.</span><span class="pl-en">prepare</span><span class="pl-kos">(</span><span class="pl-s">"select lines_version()"</span><span class="pl-kos">)</span><span class="pl-kos">.</span><span class="pl-en">get</span><span class="pl-kos">(</span><span class="pl-kos">)</span><span class="pl-kos">)</span><span class="pl-kos">;</span>
  <span class="pl-c">// { 'lines_version()': 'v0.0.1' }</span></pre></div>
  <p dir="auto">Or with <a href="https://datasette.io/" rel="nofollow">Datasette</a> (using the "no filesystem" version to limit security vulnerabilities):</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="datasette data.db --load-extension ./lines_nofs0"><pre class="notranslate"><code>datasette data.db --load-extension ./lines_nofs0
  </code></pre></div>
  <p dir="auto">Windows is not supported - <a href="https://github.com/asg017/sqlite-lines/issues/4" data-hovercard-type="issue" data-hovercard-url="/asg017/sqlite-lines/issues/4/hovercard">yet</a>!</p>
  <div class="markdown-heading" dir="auto"><h3 class="heading-element" dir="auto">From the browser with WASM/JavaScript</h3><a id="user-content-from-the-browser-with-wasmjavascript" class="anchor" aria-label="Permalink: From the browser with WASM/JavaScript" href="#from-the-browser-with-wasmjavascript"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto"><code>sqlite-lines</code> is also distributed as a standalone <a href="https://github.com/sql-js/sql.js">SQL.js</a> library. It's essentially a fork of the original SQL.js library, with the addition of <code>sqlite-lines</code> functions like <code>lines_version()</code> and <code>lines()</code>.</p>
  <p dir="auto">Check out <a href="https://observablehq.com/@asg017/introducing-sqlite-lines" rel="nofollow">this Observable notebook</a> for the full demonstration. The <a href="https://github.com/asg017/sqlite-lines/releases">Releases page</a> contains the JavaScript and WASM files.</p>
  <div class="markdown-heading" dir="auto"><h3 class="heading-element" dir="auto">The sqlite-lines CLI</h3><a id="user-content-the-sqlite-lines-cli" class="anchor" aria-label="Permalink: The sqlite-lines CLI" href="#the-sqlite-lines-cli"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto"><code>sqlite-lines</code> comes with an example CLI modeled after <a href="https://github.com/mbostock/ndjson-cli">ndjson-cli</a> that demos the speed and versatility of <code>sqlite-lines</code>. Download a pre-compiled version from the <a href="https://github.com/asg017/sqlite-lines/releases">Releases page</a>, or build yourself with:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="make cli
  ./dist/sqlite-lines"><pre class="notranslate"><code>make cli
  ./dist/sqlite-lines
  </code></pre></div>
  <p dir="auto">The <code>sqlite-lines</code> CLI reads data from stdin and applies transformations with SQL code through its arguments.</p>
  <p dir="auto">The first argument should be a SQL expression that is used transform a single line from stdlin. The available columns are <code>rowid</code>, which is the "line number" that is being processed, and <code>d</code>, an alias for <code>line</code>, which is the text content of the current line (inspired by ndjson-cli). For example, to uppercase every line from a file with <a href="https://www.sqlite.org/lang_corefunc.html#upper" rel="nofollow"><code>upper()</code></a>:</p>
  <div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="$ cat names.txt | sqlite-lines 'rowid || upper(d)'
  1ALEX
  2BRIAN
  3CRAIG"><pre>$ cat names.txt <span class="pl-k">|</span> sqlite-lines <span class="pl-s"><span class="pl-pds">'</span>rowid || upper(d)<span class="pl-pds">'</span></span>
  1ALEX
  2BRIAN
  3CRAIG</pre></div>
  <p dir="auto">This includes SQLite's new JSON <code>-></code> and <code>->></code> operators for NDJSON/JSONL files:</p>
  <div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="$ cat data.ndjson | sqlite-lines 'd ->> "$.id"'
  $ cat data.ndjson | sqlite-lines 'json_object("name", d ->> "$.name", "age": d ->> "$.stats.age")'"><pre>$ cat data.ndjson <span class="pl-k">|</span> sqlite-lines <span class="pl-s"><span class="pl-pds">'</span>d ->> "$.id"<span class="pl-pds">'</span></span>
  $ cat data.ndjson <span class="pl-k">|</span> sqlite-lines <span class="pl-s"><span class="pl-pds">'</span>json_object("name", d ->> "$.name", "age": d ->> "$.stats.age")<span class="pl-pds">'</span></span></pre></div>
  <p dir="auto">The second argument is another SQL expression that's used in the WHERE statement of the underlying SQL query to filter out lines.</p>
  <div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="# get the names of all people older than 40
  cat data.ndjson | sqlite-lines 'd ->> "$.name"' 'd ->> "$.age" > 40'"><pre><span class="pl-c"><span class="pl-c">#</span> get the names of all people older than 40</span>
  cat data.ndjson <span class="pl-k">|</span> sqlite-lines <span class="pl-s"><span class="pl-pds">'</span>d ->> "$.name"<span class="pl-pds">'</span></span> <span class="pl-s"><span class="pl-pds">'</span>d ->> "$.age" > 40<span class="pl-pds">'</span></span></pre></div>
  <p dir="auto">The third argument is another SQL expression that's used in the GROUP BY statement of the underlying SQL query to aggregate lines.</p>
  <div class="markdown-heading" dir="auto"><h3 class="heading-element" dir="auto">A Note on CSV Parsing</h3><a id="user-content-a-note-on-csv-parsing" class="anchor" aria-label="Permalink: A Note on CSV Parsing" href="#a-note-on-csv-parsing"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto"><code>sqlite-lines</code> isn't a great option for CSVs. Technically you can, but the moment your data has a <code>\n</code> character in a field or header, then you'll get corrupted results.</p>
  <p dir="auto">Instead, you should use the "official" <a href="https://www.sqlite.org/csv.html" rel="nofollow">CSV Virtual Table</a>, or use the <a href="https://www.sqlite.org/cli.html#csv" rel="nofollow"><code>.import</code></a> command in the SQLite CLI.</p>
  </article></div>


node_id: R_kgDOHPc0Nw
name: datasette-total-page-time
full_name: simonw/datasette-total-page-time
html_url: https://github.com/simonw/datasette-total-page-time
description: Add a note to the Datasette footer measuring the total page load time
created_at: 2022-04-26T22:09:58Z
updated_at: 2022-04-27T15:06:00Z
pushed_at: 2022-04-26T22:11:48Z
language: Python
license: apache-2.0
topics: []
visibility: public
default_branch: main
permissions: {"admin": false, "maintain": false, "push": false, "triage": false, "pull": true}
readme:
  # datasette-total-page-time

  [![PyPI](https://img.shields.io/pypi/v/datasette-total-page-time.svg)](https://pypi.org/project/datasette-total-page-time/)
  [![Changelog](https://img.shields.io/github/v/release/simonw/datasette-total-page-time?include_prereleases&label=changelog)](https://github.com/simonw/datasette-total-page-time/releases)
  [![Tests](https://github.com/simonw/datasette-total-page-time/workflows/Test/badge.svg)](https://github.com/simonw/datasette-total-page-time/actions?query=workflow%3ATest)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/datasette-total-page-time/blob/main/LICENSE)

  Add a note to the Datasette footer measuring the total page load time

  ## Installation

  Install this plugin in the same environment as Datasette.

      datasette install datasette-total-page-time

  ## Usage

  Once this plugin is installed, a note will appear in the footer of every page showing how long the page took to generate.

  > Queries took 326.74ms · Page took 386.310ms

  ## How it works

  Measuring how long a page takes to load and then injecting that note into the page is tricky, because you need to finish generating the page before you know how long it took to load it!

  This plugin uses the [asgi_wrapper](https://docs.datasette.io/en/stable/plugin_hooks.html#asgi-wrapper-datasette) plugin hook to measure the time taken by Datasette and then inject the following JavaScript at the bottom of the response, after the closing `</html>` tag but with the correct measured value:

  ```html
  <script>
  let footer = document.querySelector("footer");
  if (footer) {
      let ms = 37.224;
      let s = ` &middot; Page took ${ms.toFixed(3)}ms`;
      footer.innerHTML += s;
  }
  </script>
  ```
  This script is injected only on pages with the `text/html` content type - so it should not affect JSON or CSV returned by Datasette.

  ## Development

  To set up this plugin locally, first checkout the code. Then create a new virtual environment:

      cd datasette-total-page-time
      python3 -mvenv venv
      source venv/bin/activate

  Now install the dependencies and test dependencies:

      pip install -e '.[test]'

  To run the tests:

      pytest

readme_html:
  <div id="readme" class="md" data-path="README.md"><article class="markdown-body entry-content container-lg" itemprop="text"><div class="markdown-heading" dir="auto"><h1 class="heading-element" dir="auto">datasette-total-page-time</h1><a id="user-content-datasette-total-page-time" class="anchor" aria-label="Permalink: datasette-total-page-time" href="#datasette-total-page-time"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto"><a href="https://pypi.org/project/datasette-total-page-time/" rel="nofollow"><img src="https://img.shields.io/pypi/v/datasette-total-page-time.svg" alt="PyPI" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/f37deb8b856d95a8e2ad86d54edd331e94965704e4760e05a995572028ecabf1/68747470733a2f2f696d672e736869656c64732e696f2f707970692f762f6461746173657474652d746f74616c2d706167652d74696d652e737667"></a>
  <a href="https://github.com/simonw/datasette-total-page-time/releases"><img src="https://img.shields.io/github/v/release/simonw/datasette-total-page-time?include_prereleases&label=changelog" alt="Changelog" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/80087ac405ef431a7d0c1ba01919867bea4c8b8b5693f92b0b93219af4155b7c/68747470733a2f2f696d672e736869656c64732e696f2f6769746875622f762f72656c656173652f73696d6f6e772f6461746173657474652d746f74616c2d706167652d74696d653f696e636c7564655f70726572656c6561736573266c6162656c3d6368616e67656c6f67"></a>
  <a href="https://github.com/simonw/datasette-total-page-time/actions?query=workflow%3ATest"><img src="https://github.com/simonw/datasette-total-page-time/workflows/Test/badge.svg" alt="Tests" style="max-width: 100%;"></a>
  <a href="https://github.com/simonw/datasette-total-page-time/blob/main/LICENSE"><img src="https://img.shields.io/badge/license-Apache%202.0-blue.svg" alt="License" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/c355f200ea90fddaa407b6eaab303663a669248ea3ca7b1fcf77dbe04ff5f48c/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6c6963656e73652d417061636865253230322e302d626c75652e737667"></a></p>
  <p dir="auto">Add a note to the Datasette footer measuring the total page load time</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Installation</h2><a id="user-content-installation" class="anchor" aria-label="Permalink: Installation" href="#installation"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">Install this plugin in the same environment as Datasette.</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="datasette install datasette-total-page-time"><pre class="notranslate"><code>datasette install datasette-total-page-time
  </code></pre></div>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Usage</h2><a id="user-content-usage" class="anchor" aria-label="Permalink: Usage" href="#usage"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">Once this plugin is installed, a note will appear in the footer of every page showing how long the page took to generate.</p>
  <blockquote>
  <p dir="auto">Queries took 326.74ms · Page took 386.310ms</p>
  </blockquote>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">How it works</h2><a id="user-content-how-it-works" class="anchor" aria-label="Permalink: How it works" href="#how-it-works"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">Measuring how long a page takes to load and then injecting that note into the page is tricky, because you need to finish generating the page before you know how long it took to load it!</p>
  <p dir="auto">This plugin uses the <a href="https://docs.datasette.io/en/stable/plugin_hooks.html#asgi-wrapper-datasette" rel="nofollow">asgi_wrapper</a> plugin hook to measure the time taken by Datasette and then inject the following JavaScript at the bottom of the response, after the closing <code></html></code> tag but with the correct measured value:</p>
  <div class="highlight highlight-text-html-basic notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="<script>
  let footer = document.querySelector("footer");
  if (footer) {
      let ms = 37.224;
      let s = ` &middot; Page took ${ms.toFixed(3)}ms`;
      footer.innerHTML += s;
  }
  </script>"><pre><span class="pl-kos"><</span><span class="pl-ent">script</span><span class="pl-kos">></span>
  <span class="pl-k">let</span> <span class="pl-s1">footer</span> <span class="pl-c1">=</span> <span class="pl-smi">document</span><span class="pl-kos">.</span><span class="pl-en">querySelector</span><span class="pl-kos">(</span><span class="pl-s">"footer"</span><span class="pl-kos">)</span><span class="pl-kos">;</span>
  <span class="pl-k">if</span> <span class="pl-kos">(</span><span class="pl-s1">footer</span><span class="pl-kos">)</span> <span class="pl-kos">{</span>
      <span class="pl-k">let</span> <span class="pl-s1">ms</span> <span class="pl-c1">=</span> <span class="pl-c1">37.224</span><span class="pl-kos">;</span>
      <span class="pl-k">let</span> <span class="pl-s1">s</span> <span class="pl-c1">=</span> <span class="pl-s">` &middot; Page took <span class="pl-s1"><span class="pl-kos">${</span><span class="pl-s1">ms</span><span class="pl-kos">.</span><span class="pl-en">toFixed</span><span class="pl-kos">(</span><span class="pl-c1">3</span><span class="pl-kos">)</span><span class="pl-kos">}</span></span>ms`</span><span class="pl-kos">;</span>
      <span class="pl-s1">footer</span><span class="pl-kos">.</span><span class="pl-c1">innerHTML</span> <span class="pl-c1">+=</span> <span class="pl-s1">s</span><span class="pl-kos">;</span>
  <span class="pl-kos">}</span>
  <span class="pl-kos"></</span><span class="pl-ent">script</span><span class="pl-kos">></span></pre></div>
  <p dir="auto">This script is injected only on pages with the <code>text/html</code> content type - so it should not affect JSON or CSV returned by Datasette.</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Development</h2><a id="user-content-development" class="anchor" aria-label="Permalink: Development" href="#development"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">To set up this plugin locally, first checkout the code. Then create a new virtual environment:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="cd datasette-total-page-time
  python3 -mvenv venv
  source venv/bin/activate"><pre class="notranslate"><code>cd datasette-total-page-time
  python3 -mvenv venv
  source venv/bin/activate
  </code></pre></div>
  <p dir="auto">Now install the dependencies and test dependencies:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="pip install -e '.[test]'"><pre class="notranslate"><code>pip install -e '.[test]'
  </code></pre></div>
  <p dir="auto">To run the tests:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="pytest"><pre class="notranslate"><code>pytest
  </code></pre></div>
  </article></div>


node_id: R_kgDOHPj_Zg
name: datasette-gzip
full_name: simonw/datasette-gzip
html_url: https://github.com/simonw/datasette-gzip
description: Add gzip compression to Datasette
created_at: 2022-04-27T06:54:52Z
updated_at: 2026-04-09T04:03:29Z
pushed_at: 2026-04-09T04:04:18Z
language: Python
license: apache-2.0
topics: []
visibility: public
default_branch: main
permissions: {"admin": false, "maintain": false, "push": false, "triage": false, "pull": false}
pull_request_creation_policy: all
readme:
  # datasette-gzip

  [![PyPI](https://img.shields.io/pypi/v/datasette-gzip.svg)](https://pypi.org/project/datasette-gzip/)
  [![Changelog](https://img.shields.io/github/v/release/simonw/datasette-gzip?include_prereleases&label=changelog)](https://github.com/simonw/datasette-gzip/releases)
  [![Tests](https://github.com/simonw/datasette-gzip/workflows/Test/badge.svg)](https://github.com/simonw/datasette-gzip/actions?query=workflow%3ATest)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/datasette-gzip/blob/main/LICENSE)

  Add gzip compression to Datasette

  ## Installation

  Install this plugin in the same environment as Datasette.

      datasette install datasette-gzip

  ## Usage

  Once installed, Datasette will obey the `Accept-Encoding:` header sent by browsers or other user agents and return content compressed in the most appropriate way.

  This plugin is a thin wrapper for the [asgi-gzip library](https://github.com/simonw/asgi-gzip), which extracts the [GzipMiddleware](https://www.starlette.io/middleware/#gzipmiddleware) from Starlette.

  ## Development

  To set up this plugin locally, first checkout the code. Then create a new virtual environment:

      cd datasette-gzip
      python3 -mvenv venv
      source venv/bin/activate

  Now install the dependencies and test dependencies:

      pip install -e '.[test]'

  To run the tests:

      pytest

readme_html:
  <div id="readme" class="md" data-path="README.md"><article class="markdown-body entry-content container-lg" itemprop="text"><div class="markdown-heading" dir="auto"><h1 class="heading-element" dir="auto">datasette-gzip</h1><a id="user-content-datasette-gzip" class="anchor" aria-label="Permalink: datasette-gzip" href="#datasette-gzip"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto"><a href="https://pypi.org/project/datasette-gzip/" rel="nofollow"><img src="https://img.shields.io/pypi/v/datasette-gzip.svg" alt="PyPI" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/cb6b52c635294603ac996754557ebf182932d9f656106a374117ba4904c111d7/68747470733a2f2f696d672e736869656c64732e696f2f707970692f762f6461746173657474652d677a69702e737667"></a>
  <a href="https://github.com/simonw/datasette-gzip/releases"><img src="https://img.shields.io/github/v/release/simonw/datasette-gzip?include_prereleases&label=changelog" alt="Changelog" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/9a1f8a4bcf3516fa6563bb48b42698ae6e5d3510033d6c5b3e9fbb3b790ab693/68747470733a2f2f696d672e736869656c64732e696f2f6769746875622f762f72656c656173652f73696d6f6e772f6461746173657474652d677a69703f696e636c7564655f70726572656c6561736573266c6162656c3d6368616e67656c6f67"></a>
  <a href="https://github.com/simonw/datasette-gzip/actions?query=workflow%3ATest"><img src="https://github.com/simonw/datasette-gzip/workflows/Test/badge.svg" alt="Tests" style="max-width: 100%;"></a>
  <a href="https://github.com/simonw/datasette-gzip/blob/main/LICENSE"><img src="https://img.shields.io/badge/license-Apache%202.0-blue.svg" alt="License" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/c355f200ea90fddaa407b6eaab303663a669248ea3ca7b1fcf77dbe04ff5f48c/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6c6963656e73652d417061636865253230322e302d626c75652e737667"></a></p>
  <p dir="auto">Add gzip compression to Datasette</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Installation</h2><a id="user-content-installation" class="anchor" aria-label="Permalink: Installation" href="#installation"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">Install this plugin in the same environment as Datasette.</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="datasette install datasette-gzip"><pre class="notranslate"><code>datasette install datasette-gzip
  </code></pre></div>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Usage</h2><a id="user-content-usage" class="anchor" aria-label="Permalink: Usage" href="#usage"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">Once installed, Datasette will obey the <code>Accept-Encoding:</code> header sent by browsers or other user agents and return content compressed in the most appropriate way.</p>
  <p dir="auto">This plugin is a thin wrapper for the <a href="https://github.com/simonw/asgi-gzip">asgi-gzip library</a>, which extracts the <a href="https://www.starlette.io/middleware/#gzipmiddleware" rel="nofollow">GzipMiddleware</a> from Starlette.</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Development</h2><a id="user-content-development" class="anchor" aria-label="Permalink: Development" href="#development"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">To set up this plugin locally, first checkout the code. Then create a new virtual environment:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="cd datasette-gzip
  python3 -mvenv venv
  source venv/bin/activate"><pre class="notranslate"><code>cd datasette-gzip
  python3 -mvenv venv
  source venv/bin/activate
  </code></pre></div>
  <p dir="auto">Now install the dependencies and test dependencies:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="pip install -e '.[test]'"><pre class="notranslate"><code>pip install -e '.[test]'
  </code></pre></div>
  <p dir="auto">To run the tests:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="pytest"><pre class="notranslate"><code>pytest
  </code></pre></div>
  </article></div>


node_id: R_kgDOHQLzfQ
name: datasette-copy-to-memory
full_name: simonw/datasette-copy-to-memory
html_url: https://github.com/simonw/datasette-copy-to-memory
description: Copy database files into an in-memory database on startup
created_at: 2022-04-28T20:02:21Z
updated_at: 2023-09-29T15:36:18Z
pushed_at: 2022-04-30T19:49:29Z
language: Python
license: apache-2.0
topics: ["datasette", "datasette-plugin"]
visibility: public
default_branch: main
permissions: {"admin": false, "maintain": false, "push": false, "triage": false, "pull": true}
readme:
  # datasette-copy-to-memory

  [![PyPI](https://img.shields.io/pypi/v/datasette-copy-to-memory.svg)](https://pypi.org/project/datasette-copy-to-memory/)
  [![Changelog](https://img.shields.io/github/v/release/simonw/datasette-copy-to-memory?include_prereleases&label=changelog)](https://github.com/simonw/datasette-copy-to-memory/releases)
  [![Tests](https://github.com/simonw/datasette-copy-to-memory/workflows/Test/badge.svg)](https://github.com/simonw/datasette-copy-to-memory/actions?query=workflow%3ATest)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/datasette-copy-to-memory/blob/main/LICENSE)

  Copy database files into an in-memory database on startup

  This plugin is **highly experimental**. It currently exists to support Datasette performance research, and is not designed for actual production usage.

  ## Installation

  Install this plugin in the same environment as Datasette.

      datasette install datasette-copy-to-memory

  ## Usage

  On startup, Datasette will create an in-memory named database for each attached database. This database will have the same name but with `_memory` at the end.

  So running this:

      datasette fixtures.db

  Will serve two databases: the original at `/fixtures` and the in-memory copy at `/fixtures_memory`.

  ## Demo

  A demo is running on [latest-with-plugins.datasette.io](https://latest-with-plugins.datasette.io/) - the [/fixtures_memory](https://latest-with-plugins.datasette.io/fixtures_memory) table there is provided by this plugin.

  ## Configuration

  By default every attached database file will be loaded into a `_memory` copy.

  You can use plugin configuration to specify just a subset of the database. For example, to create `github_memory` but not `fixtures_memory` you would use the following `metadata.yml` file:

  ```yaml
  plugins:
    datasette-copy-to-memory:
      databases:
      - github
  ```
  Then start Datasette like this:

      datasette github.db fixtures.db -m metadata.yml

  If you don't want to have a `fixtures` and `fixtures_memory` database, you can use `replace: true` to have the plugin replace the file-backed database with the new in-memory one, reusing the same database name:

  ```yaml
  plugins:
    datasette-copy-to-memory:
      replace: true
  ```
  Then:

      datasette github.db fixtures.db -m metadata.yml

  This will result in both `/github` and `/fixtures` but no `/github_memory` or `/fixtures_memory`.

  ## Development

  To set up this plugin locally, first checkout the code. Then create a new virtual environment:

      cd datasette-copy-to-memory
      python3 -m venv venv
      source venv/bin/activate

  Now install the dependencies and test dependencies:

      pip install -e '.[test]'

  To run the tests:

      pytest

readme_html:
  <div id="readme" class="md" data-path="README.md"><article class="markdown-body entry-content container-lg" itemprop="text"><div class="markdown-heading" dir="auto"><h1 class="heading-element" dir="auto">datasette-copy-to-memory</h1><a id="user-content-datasette-copy-to-memory" class="anchor" aria-label="Permalink: datasette-copy-to-memory" href="#datasette-copy-to-memory"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto"><a href="https://pypi.org/project/datasette-copy-to-memory/" rel="nofollow"><img src="https://img.shields.io/pypi/v/datasette-copy-to-memory.svg" alt="PyPI" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/6c8477ba1ce409c65ae9f34bf99ddd6ee4beaa34a5a05f7f3c0b0e482af1efbe/68747470733a2f2f696d672e736869656c64732e696f2f707970692f762f6461746173657474652d636f70792d746f2d6d656d6f72792e737667"></a>
  <a href="https://github.com/simonw/datasette-copy-to-memory/releases"><img src="https://img.shields.io/github/v/release/simonw/datasette-copy-to-memory?include_prereleases&label=changelog" alt="Changelog" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/8643d7ef64d28b6913a4e85ae32f18db7cf8325252f4cba406a89750ee57f578/68747470733a2f2f696d672e736869656c64732e696f2f6769746875622f762f72656c656173652f73696d6f6e772f6461746173657474652d636f70792d746f2d6d656d6f72793f696e636c7564655f70726572656c6561736573266c6162656c3d6368616e67656c6f67"></a>
  <a href="https://github.com/simonw/datasette-copy-to-memory/actions?query=workflow%3ATest"><img src="https://github.com/simonw/datasette-copy-to-memory/workflows/Test/badge.svg" alt="Tests" style="max-width: 100%;"></a>
  <a href="https://github.com/simonw/datasette-copy-to-memory/blob/main/LICENSE"><img src="https://img.shields.io/badge/license-Apache%202.0-blue.svg" alt="License" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/c355f200ea90fddaa407b6eaab303663a669248ea3ca7b1fcf77dbe04ff5f48c/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6c6963656e73652d417061636865253230322e302d626c75652e737667"></a></p>
  <p dir="auto">Copy database files into an in-memory database on startup</p>
  <p dir="auto">This plugin is <strong>highly experimental</strong>. It currently exists to support Datasette performance research, and is not designed for actual production usage.</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Installation</h2><a id="user-content-installation" class="anchor" aria-label="Permalink: Installation" href="#installation"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">Install this plugin in the same environment as Datasette.</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="datasette install datasette-copy-to-memory"><pre class="notranslate"><code>datasette install datasette-copy-to-memory
  </code></pre></div>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Usage</h2><a id="user-content-usage" class="anchor" aria-label="Permalink: Usage" href="#usage"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">On startup, Datasette will create an in-memory named database for each attached database. This database will have the same name but with <code>_memory</code> at the end.</p>
  <p dir="auto">So running this:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="datasette fixtures.db"><pre class="notranslate"><code>datasette fixtures.db
  </code></pre></div>
  <p dir="auto">Will serve two databases: the original at <code>/fixtures</code> and the in-memory copy at <code>/fixtures_memory</code>.</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Demo</h2><a id="user-content-demo" class="anchor" aria-label="Permalink: Demo" href="#demo"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">A demo is running on <a href="https://latest-with-plugins.datasette.io/" rel="nofollow">latest-with-plugins.datasette.io</a> - the <a href="https://latest-with-plugins.datasette.io/fixtures_memory" rel="nofollow">/fixtures_memory</a> table there is provided by this plugin.</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Configuration</h2><a id="user-content-configuration" class="anchor" aria-label="Permalink: Configuration" href="#configuration"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">By default every attached database file will be loaded into a <code>_memory</code> copy.</p>
  <p dir="auto">You can use plugin configuration to specify just a subset of the database. For example, to create <code>github_memory</code> but not <code>fixtures_memory</code> you would use the following <code>metadata.yml</code> file:</p>
  <div class="highlight highlight-source-yaml notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="plugins:
    datasette-copy-to-memory:
      databases:
      - github"><pre><span class="pl-ent">plugins</span>:
    <span class="pl-ent">datasette-copy-to-memory</span>:
      <span class="pl-ent">databases</span>:
      - <span class="pl-s">github</span></pre></div>
  <p dir="auto">Then start Datasette like this:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="datasette github.db fixtures.db -m metadata.yml"><pre class="notranslate"><code>datasette github.db fixtures.db -m metadata.yml
  </code></pre></div>
  <p dir="auto">If you don't want to have a <code>fixtures</code> and <code>fixtures_memory</code> database, you can use <code>replace: true</code> to have the plugin replace the file-backed database with the new in-memory one, reusing the same database name:</p>
  <div class="highlight highlight-source-yaml notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="plugins:
    datasette-copy-to-memory:
      replace: true"><pre><span class="pl-ent">plugins</span>:
    <span class="pl-ent">datasette-copy-to-memory</span>:
      <span class="pl-ent">replace</span>: <span class="pl-c1">true</span></pre></div>
  <p dir="auto">Then:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="datasette github.db fixtures.db -m metadata.yml"><pre class="notranslate"><code>datasette github.db fixtures.db -m metadata.yml
  </code></pre></div>
  <p dir="auto">This will result in both <code>/github</code> and <code>/fixtures</code> but no <code>/github_memory</code> or <code>/fixtures_memory</code>.</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Development</h2><a id="user-content-development" class="anchor" aria-label="Permalink: Development" href="#development"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">To set up this plugin locally, first checkout the code. Then create a new virtual environment:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="cd datasette-copy-to-memory
  python3 -m venv venv
  source venv/bin/activate"><pre class="notranslate"><code>cd datasette-copy-to-memory
  python3 -m venv venv
  source venv/bin/activate
  </code></pre></div>
  <p dir="auto">Now install the dependencies and test dependencies:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="pip install -e '.[test]'"><pre class="notranslate"><code>pip install -e '.[test]'
  </code></pre></div>
  <p dir="auto">To run the tests:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="pytest"><pre class="notranslate"><code>pytest
  </code></pre></div>
  </article></div>


node_id: R_kgDOHSfuMg
name: datasette-upload-dbs
full_name: simonw/datasette-upload-dbs
html_url: https://github.com/simonw/datasette-upload-dbs
description: Upload SQLite database files to Datasette
created_at: 2022-05-05T23:36:51Z
updated_at: 2025-11-10T21:11:31Z
pushed_at: 2025-11-10T21:11:58Z
language: Python
license: apache-2.0
topics: []
visibility: public
default_branch: main
permissions: {"admin": false, "maintain": false, "push": false, "triage": false, "pull": false}
pull_request_creation_policy: all
readme:
  # datasette-upload-dbs

  [![PyPI](https://img.shields.io/pypi/v/datasette-upload-dbs.svg)](https://pypi.org/project/datasette-upload-dbs/)
  [![Changelog](https://img.shields.io/github/v/release/simonw/datasette-upload-dbs?include_prereleases&label=changelog)](https://github.com/simonw/datasette-upload-dbs/releases)
  [![Tests](https://github.com/simonw/datasette-upload-dbs/workflows/Test/badge.svg)](https://github.com/simonw/datasette-upload-dbs/actions?query=workflow%3ATest)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/datasette-upload-dbs/blob/main/LICENSE)

  Upload SQLite database files to Datasette

  ## Installation

  Install this plugin in the same environment as Datasette.

      datasette install datasette-upload-dbs

  ## Configuration

  This plugin requires you to configure a directory in which uploaded files will be stored.

  On startup, Datasette will automatically load any SQLite files that it finds in that directory. This means it is safe to restart your server in between file uploads.

  To configure the directory as `/home/datasette/uploads`, add this to a `metadata.yml` configuration file:

  ```yaml
  plugins:
    datasette-upload-dbs:
      directory: /home/datasette/uploads
  ```

  Or if you are using `metadata.json`:

  ```json
  {
    "plugins": {
      "datasette-upload-dbs": {
        "directory": "/home/datasette/uploads"
      }
    }
  }
  ```
  You can use `"."` for the current folder when the server starts, or `"uploads"` for a folder relative to that folder. The folder will be created on startup if it does not already exist.

  Then start Datasette like this:

      datasette -m metadata.yml

  The plugin defaults to loading all databases in the configured directory.

  You can disable this by adding the following setting:
  ```
  "skip_startup_scan": true
  ```
  ## Usage

  Only users with the `upload-dbs` permission will be able to upload files. The `root` user has this permission by default - other users can be granted access using permission plugins, see the [Permissions](https://docs.datasette.io/en/stable/authentication.html#permissions) documentation for details.

  To start Datasette as the root user, run this:

      datasette -m metadata.yml --root

  And follow the link that is displayed on the console.

  If a user has that permission they will see an "Upload database" link in the navigation menu.

  This will take them to `/-/upload-dbs` where they will be able to upload database files, by selecting them or by dragging them onto the drop area.

  ![Animated demo showing a file being dropped onto a box, then uploading and redirecting to the database page](https://github.com/simonw/datasette-upload-dbs/raw/main/upload-demo.gif)

  ## Development

  To set up this plugin locally, first checkout the code. Then create a new virtual environment:

      cd datasette-upload-dbs
      python3 -m venv venv
      source venv/bin/activate

  Now install the dependencies and test dependencies:

      pip install -e '.[test]'

  To run the tests:

      pytest

readme_html:
  <div id="readme" class="md" data-path="README.md"><article class="markdown-body entry-content container-lg" itemprop="text"><div class="markdown-heading" dir="auto"><h1 class="heading-element" dir="auto">datasette-upload-dbs</h1><a id="user-content-datasette-upload-dbs" class="anchor" aria-label="Permalink: datasette-upload-dbs" href="#datasette-upload-dbs"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto"><a href="https://pypi.org/project/datasette-upload-dbs/" rel="nofollow"><img src="https://img.shields.io/pypi/v/datasette-upload-dbs.svg" alt="PyPI" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/cb5cdb4bc47f16d0479f8d6dd9ac50cbbdf64f01316b4f5090632ceb207df8ca/68747470733a2f2f696d672e736869656c64732e696f2f707970692f762f6461746173657474652d75706c6f61642d6462732e737667"></a>
  <a href="https://github.com/simonw/datasette-upload-dbs/releases"><img src="https://img.shields.io/github/v/release/simonw/datasette-upload-dbs?include_prereleases&label=changelog" alt="Changelog" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/e897c486b85bdfed0edecee247e979baf184068158943c55306d0e49298b67e4/68747470733a2f2f696d672e736869656c64732e696f2f6769746875622f762f72656c656173652f73696d6f6e772f6461746173657474652d75706c6f61642d6462733f696e636c7564655f70726572656c6561736573266c6162656c3d6368616e67656c6f67"></a>
  <a href="https://github.com/simonw/datasette-upload-dbs/actions?query=workflow%3ATest"><img src="https://github.com/simonw/datasette-upload-dbs/workflows/Test/badge.svg" alt="Tests" style="max-width: 100%;"></a>
  <a href="https://github.com/simonw/datasette-upload-dbs/blob/main/LICENSE"><img src="https://img.shields.io/badge/license-Apache%202.0-blue.svg" alt="License" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/c355f200ea90fddaa407b6eaab303663a669248ea3ca7b1fcf77dbe04ff5f48c/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6c6963656e73652d417061636865253230322e302d626c75652e737667"></a></p>
  <p dir="auto">Upload SQLite database files to Datasette</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Installation</h2><a id="user-content-installation" class="anchor" aria-label="Permalink: Installation" href="#installation"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">Install this plugin in the same environment as Datasette.</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="datasette install datasette-upload-dbs"><pre class="notranslate"><code>datasette install datasette-upload-dbs
  </code></pre></div>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Configuration</h2><a id="user-content-configuration" class="anchor" aria-label="Permalink: Configuration" href="#configuration"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">This plugin requires you to configure a directory in which uploaded files will be stored.</p>
  <p dir="auto">On startup, Datasette will automatically load any SQLite files that it finds in that directory. This means it is safe to restart your server in between file uploads.</p>
  <p dir="auto">To configure the directory as <code>/home/datasette/uploads</code>, add this to a <code>metadata.yml</code> configuration file:</p>
  <div class="highlight highlight-source-yaml notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="plugins:
    datasette-upload-dbs:
      directory: /home/datasette/uploads"><pre><span class="pl-ent">plugins</span>:
    <span class="pl-ent">datasette-upload-dbs</span>:
      <span class="pl-ent">directory</span>: <span class="pl-s">/home/datasette/uploads</span></pre></div>
  <p dir="auto">Or if you are using <code>metadata.json</code>:</p>
  <div class="highlight highlight-source-json notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="{
    "plugins": {
      "datasette-upload-dbs": {
        "directory": "/home/datasette/uploads"
      }
    }
  }"><pre>{
    <span class="pl-ent">"plugins"</span>: {
      <span class="pl-ent">"datasette-upload-dbs"</span>: {
        <span class="pl-ent">"directory"</span>: <span class="pl-s"><span class="pl-pds">"</span>/home/datasette/uploads<span class="pl-pds">"</span></span>
      }
    }
  }</pre></div>
  <p dir="auto">You can use <code>"."</code> for the current folder when the server starts, or <code>"uploads"</code> for a folder relative to that folder. The folder will be created on startup if it does not already exist.</p>
  <p dir="auto">Then start Datasette like this:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="datasette -m metadata.yml"><pre class="notranslate"><code>datasette -m metadata.yml
  </code></pre></div>
  <p dir="auto">The plugin defaults to loading all databases in the configured directory.</p>
  <p dir="auto">You can disable this by adding the following setting:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content=""skip_startup_scan": true"><pre class="notranslate"><code>"skip_startup_scan": true
  </code></pre></div>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Usage</h2><a id="user-content-usage" class="anchor" aria-label="Permalink: Usage" href="#usage"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">Only users with the <code>upload-dbs</code> permission will be able to upload files. The <code>root</code> user has this permission by default - other users can be granted access using permission plugins, see the <a href="https://docs.datasette.io/en/stable/authentication.html#permissions" rel="nofollow">Permissions</a> documentation for details.</p>
  <p dir="auto">To start Datasette as the root user, run this:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="datasette -m metadata.yml --root"><pre class="notranslate"><code>datasette -m metadata.yml --root
  </code></pre></div>
  <p dir="auto">And follow the link that is displayed on the console.</p>
  <p dir="auto">If a user has that permission they will see an "Upload database" link in the navigation menu.</p>
  <p dir="auto">This will take them to <code>/-/upload-dbs</code> where they will be able to upload database files, by selecting them or by dragging them onto the drop area.</p>
  <p dir="auto"><a target="_blank" rel="noopener noreferrer" href="https://github.com/simonw/datasette-upload-dbs/raw/main/upload-demo.gif"><img src="https://github.com/simonw/datasette-upload-dbs/raw/main/upload-demo.gif" alt="Animated demo showing a file being dropped onto a box, then uploading and redirecting to the database page" data-animated-image="" style="max-width: 100%;"></a></p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Development</h2><a id="user-content-development" class="anchor" aria-label="Permalink: Development" href="#development"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">To set up this plugin locally, first checkout the code. Then create a new virtual environment:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="cd datasette-upload-dbs
  python3 -m venv venv
  source venv/bin/activate"><pre class="notranslate"><code>cd datasette-upload-dbs
  python3 -m venv venv
  source venv/bin/activate
  </code></pre></div>
  <p dir="auto">Now install the dependencies and test dependencies:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="pip install -e '.[test]'"><pre class="notranslate"><code>pip install -e '.[test]'
  </code></pre></div>
  <p dir="auto">To run the tests:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="pytest"><pre class="notranslate"><code>pytest
  </code></pre></div>
  </article></div>


node_id: R_kgDOHcwLAg
name: datasette-query-files
full_name: eyeseast/datasette-query-files
html_url: https://github.com/eyeseast/datasette-query-files
description: Write Datasette canned queries as plain SQL files
created_at: 2022-06-04T18:52:07Z
updated_at: 2024-07-05T20:44:59Z
pushed_at: 2022-07-02T20:40:51Z
language: Python
license: apache-2.0
topics: ["datasette", "datasette-plugin", "python", "sql", "sqlite"]
visibility: public
default_branch: main
permissions: {"admin": false, "maintain": false, "push": false, "triage": false, "pull": true}
template_repository:
[cell repeats an earlier one in this file: 6377 characters]

readme:
  # datasette-query-files

  [![PyPI](https://img.shields.io/pypi/v/datasette-query-files.svg)](https://pypi.org/project/datasette-query-files/)
  [![Changelog](https://img.shields.io/github/v/release/eyeseast/datasette-query-files?include_prereleases&label=changelog)](https://github.com/eyeseast/datasette-query-files/releases)
  [![Tests](https://github.com/eyeseast/datasette-query-files/workflows/Test/badge.svg)](https://github.com/eyeseast/datasette-query-files/actions?query=workflow%3ATest)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/eyeseast/datasette-query-files/blob/main/LICENSE)

  Write Datasette canned queries as plain SQL files.

  ## Installation

  Install this plugin in the same environment as Datasette.

      datasette install datasette-query-files

  Or using `pip` or `pipenv`:

      pip install datasette-query-files
      pipenv install datasette-query-files

  ## Usage

  This plugin will look for [canned queries](https://docs.datasette.io/en/stable/sql_queries.html#canned-queries) in the filesystem, in addition any defined in metadata.

  Let's say you're working in a directory called `project-directory`, with a database file called `my-project.db`. Start by creating a `queries` directory with a `my-project` directory inside it. Any SQL file inside that `my-project` folder will become a canned query that can be run on the `my-project` database. If you have a `query-name.sql` file and a `query-name.json` (or `query-name.yml`) file in the same directory, the JSON file will be used as query metadata.

  ```
  project-directory/
    my-project.db
    queries/
      my-project/
        query-name.sql # a query
        query-name.yml # query metadata
  ```

  ## Development

  To set up this plugin locally, first checkout the code. Then create a new virtual environment:

      cd datasette-query-files
      python3 -m venv venv
      source venv/bin/activate

  Now install the dependencies and test dependencies:

      pip install -e '.[test]'

  To run the tests:

      pytest

readme_html:
  <div id="readme" class="md" data-path="README.md"><article class="markdown-body entry-content container-lg" itemprop="text"><div class="markdown-heading" dir="auto"><h1 class="heading-element" dir="auto">datasette-query-files</h1><a id="user-content-datasette-query-files" class="anchor" aria-label="Permalink: datasette-query-files" href="#datasette-query-files"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto"><a href="https://pypi.org/project/datasette-query-files/" rel="nofollow"><img src="https://img.shields.io/pypi/v/datasette-query-files.svg" alt="PyPI" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/488778e9d747e7d163e120174e98fd103ed89bb6667e4014ffe54cef35418116/68747470733a2f2f696d672e736869656c64732e696f2f707970692f762f6461746173657474652d71756572792d66696c65732e737667"></a>
  <a href="https://github.com/eyeseast/datasette-query-files/releases"><img src="https://img.shields.io/github/v/release/eyeseast/datasette-query-files?include_prereleases&label=changelog" alt="Changelog" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/8317a221121c138a3b466dcefb1c242d5c855dec120b767e7078e8796d42ed1e/68747470733a2f2f696d672e736869656c64732e696f2f6769746875622f762f72656c656173652f65796573656173742f6461746173657474652d71756572792d66696c65733f696e636c7564655f70726572656c6561736573266c6162656c3d6368616e67656c6f67"></a>
  <a href="https://github.com/eyeseast/datasette-query-files/actions?query=workflow%3ATest"><img src="https://github.com/eyeseast/datasette-query-files/workflows/Test/badge.svg" alt="Tests" style="max-width: 100%;"></a>
  <a href="https://github.com/eyeseast/datasette-query-files/blob/main/LICENSE"><img src="https://img.shields.io/badge/license-Apache%202.0-blue.svg" alt="License" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/c355f200ea90fddaa407b6eaab303663a669248ea3ca7b1fcf77dbe04ff5f48c/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6c6963656e73652d417061636865253230322e302d626c75652e737667"></a></p>
  <p dir="auto">Write Datasette canned queries as plain SQL files.</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Installation</h2><a id="user-content-installation" class="anchor" aria-label="Permalink: Installation" href="#installation"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">Install this plugin in the same environment as Datasette.</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="datasette install datasette-query-files"><pre class="notranslate"><code>datasette install datasette-query-files
  </code></pre></div>
  <p dir="auto">Or using <code>pip</code> or <code>pipenv</code>:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="pip install datasette-query-files
  pipenv install datasette-query-files"><pre class="notranslate"><code>pip install datasette-query-files
  pipenv install datasette-query-files
  </code></pre></div>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Usage</h2><a id="user-content-usage" class="anchor" aria-label="Permalink: Usage" href="#usage"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">This plugin will look for <a href="https://docs.datasette.io/en/stable/sql_queries.html#canned-queries" rel="nofollow">canned queries</a> in the filesystem, in addition any defined in metadata.</p>
  <p dir="auto">Let's say you're working in a directory called <code>project-directory</code>, with a database file called <code>my-project.db</code>. Start by creating a <code>queries</code> directory with a <code>my-project</code> directory inside it. Any SQL file inside that <code>my-project</code> folder will become a canned query that can be run on the <code>my-project</code> database. If you have a <code>query-name.sql</code> file and a <code>query-name.json</code> (or <code>query-name.yml</code>) file in the same directory, the JSON file will be used as query metadata.</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="project-directory/
    my-project.db
    queries/
      my-project/
        query-name.sql # a query
        query-name.yml # query metadata"><pre class="notranslate"><code>project-directory/
    my-project.db
    queries/
      my-project/
        query-name.sql # a query
        query-name.yml # query metadata
  </code></pre></div>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Development</h2><a id="user-content-development" class="anchor" aria-label="Permalink: Development" href="#development"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">To set up this plugin locally, first checkout the code. Then create a new virtual environment:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="cd datasette-query-files
  python3 -m venv venv
  source venv/bin/activate"><pre class="notranslate"><code>cd datasette-query-files
  python3 -m venv venv
  source venv/bin/activate
  </code></pre></div>
  <p dir="auto">Now install the dependencies and test dependencies:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="pip install -e '.[test]'"><pre class="notranslate"><code>pip install -e '.[test]'
  </code></pre></div>
  <p dir="auto">To run the tests:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="pytest"><pre class="notranslate"><code>pytest
  </code></pre></div>
  </article></div>


node_id: R_kgDOHeFHyg
name: datasette-socrata
full_name: simonw/datasette-socrata
html_url: https://github.com/simonw/datasette-socrata
description: Import data from Socrata into Datasette
created_at: 2022-06-08T15:13:44Z
updated_at: 2022-06-09T21:13:34Z
pushed_at: 2023-08-14T16:08:54Z
language: Python
license: apache-2.0
topics: []
visibility: public
default_branch: main
permissions: {"admin": false, "maintain": false, "push": false, "triage": false, "pull": true}
readme:
  # datasette-socrata

  [![PyPI](https://img.shields.io/pypi/v/datasette-socrata.svg)](https://pypi.org/project/datasette-socrata/)
  [![Changelog](https://img.shields.io/github/v/release/simonw/datasette-socrata?include_prereleases&label=changelog)](https://github.com/simonw/datasette-socrata/releases)
  [![Tests](https://github.com/simonw/datasette-socrata/workflows/Test/badge.svg)](https://github.com/simonw/datasette-socrata/actions?query=workflow%3ATest)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/datasette-socrata/blob/main/LICENSE)

  Import data from [Socrata](https://dev.socrata.com/) into [Datasette](https://datasette.io/)

  ## Installation

  Install this plugin in the same environment as Datasette.

      datasette install datasette-socrata

  ## Usage

  Make sure you have [enabled WAL mode](https://til.simonwillison.net/sqlite/enabling-wal-mode) on your database files before using this plugin.

  Once installed, an interface for importing data from Socrata will become available at this URL:

      /-/import-socrata

  Users will be able to paste in a URL to a dataset on Socrata in order to initialize an import.

  You can also pre-fill the form by passing a `?url=` parameter, for example:

      /-/import-socrata?url=https://data.sfgov.org/City-Infrastructure/Street-Tree-List/tkzw-k3nq

  Any database that is attached to Datasette, is NOT loaded as immutable (with the `-i` option) and that has WAL mode enabled will be available for users to import data into.

  The `import-socrata` permission governs access. By default the `root` actor (accessible using `datasette --root` to start Datasette) is granted that permission.

  You can use permission plugins such as [datasette-permissions-sql](https://github.com/simonw/datasette-permissions-sql) to grant additional access to other users.

  ## Configuration

  If you only want Socrata imports to be allowed to a specific database, you can configure that using plugin configration in `metadata.yml`:

  ```yaml
  plugins:
    datasette-socrata:
      database: socrata
  ```

  ## Development

  To set up this plugin locally, first checkout the code. Then create a new virtual environment:

      cd datasette-socrata
      python3 -m venv venv
      source venv/bin/activate

  Now install the dependencies and test dependencies:

      pip install -e '.[test]'

  To run the tests:

      pytest

readme_html:
  <div id="readme" class="md" data-path="README.md"><article class="markdown-body entry-content container-lg" itemprop="text"><div class="markdown-heading" dir="auto"><h1 class="heading-element" dir="auto">datasette-socrata</h1><a id="user-content-datasette-socrata" class="anchor" aria-label="Permalink: datasette-socrata" href="#datasette-socrata"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto"><a href="https://pypi.org/project/datasette-socrata/" rel="nofollow"><img src="https://img.shields.io/pypi/v/datasette-socrata.svg" alt="PyPI" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/5b191489c577757d8f6333b84c4043279253c596e36d9a79e57b59d3632778dc/68747470733a2f2f696d672e736869656c64732e696f2f707970692f762f6461746173657474652d736f63726174612e737667"></a>
  <a href="https://github.com/simonw/datasette-socrata/releases"><img src="https://img.shields.io/github/v/release/simonw/datasette-socrata?include_prereleases&label=changelog" alt="Changelog" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/f08e839336f1a52b5f4e4cc9d44e4372400012e805a4b1241b4a380f4cd83fba/68747470733a2f2f696d672e736869656c64732e696f2f6769746875622f762f72656c656173652f73696d6f6e772f6461746173657474652d736f63726174613f696e636c7564655f70726572656c6561736573266c6162656c3d6368616e67656c6f67"></a>
  <a href="https://github.com/simonw/datasette-socrata/actions?query=workflow%3ATest"><img src="https://github.com/simonw/datasette-socrata/workflows/Test/badge.svg" alt="Tests" style="max-width: 100%;"></a>
  <a href="https://github.com/simonw/datasette-socrata/blob/main/LICENSE"><img src="https://img.shields.io/badge/license-Apache%202.0-blue.svg" alt="License" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/c355f200ea90fddaa407b6eaab303663a669248ea3ca7b1fcf77dbe04ff5f48c/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6c6963656e73652d417061636865253230322e302d626c75652e737667"></a></p>
  <p dir="auto">Import data from <a href="https://dev.socrata.com/" rel="nofollow">Socrata</a> into <a href="https://datasette.io/" rel="nofollow">Datasette</a></p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Installation</h2><a id="user-content-installation" class="anchor" aria-label="Permalink: Installation" href="#installation"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">Install this plugin in the same environment as Datasette.</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="datasette install datasette-socrata"><pre class="notranslate"><code>datasette install datasette-socrata
  </code></pre></div>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Usage</h2><a id="user-content-usage" class="anchor" aria-label="Permalink: Usage" href="#usage"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">Make sure you have <a href="https://til.simonwillison.net/sqlite/enabling-wal-mode" rel="nofollow">enabled WAL mode</a> on your database files before using this plugin.</p>
  <p dir="auto">Once installed, an interface for importing data from Socrata will become available at this URL:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="/-/import-socrata"><pre class="notranslate"><code>/-/import-socrata
  </code></pre></div>
  <p dir="auto">Users will be able to paste in a URL to a dataset on Socrata in order to initialize an import.</p>
  <p dir="auto">You can also pre-fill the form by passing a <code>?url=</code> parameter, for example:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="/-/import-socrata?url=https://data.sfgov.org/City-Infrastructure/Street-Tree-List/tkzw-k3nq"><pre class="notranslate"><code>/-/import-socrata?url=https://data.sfgov.org/City-Infrastructure/Street-Tree-List/tkzw-k3nq
  </code></pre></div>
  <p dir="auto">Any database that is attached to Datasette, is NOT loaded as immutable (with the <code>-i</code> option) and that has WAL mode enabled will be available for users to import data into.</p>
  <p dir="auto">The <code>import-socrata</code> permission governs access. By default the <code>root</code> actor (accessible using <code>datasette --root</code> to start Datasette) is granted that permission.</p>
  <p dir="auto">You can use permission plugins such as <a href="https://github.com/simonw/datasette-permissions-sql">datasette-permissions-sql</a> to grant additional access to other users.</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Configuration</h2><a id="user-content-configuration" class="anchor" aria-label="Permalink: Configuration" href="#configuration"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">If you only want Socrata imports to be allowed to a specific database, you can configure that using plugin configration in <code>metadata.yml</code>:</p>
  <div class="highlight highlight-source-yaml notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="plugins:
    datasette-socrata:
      database: socrata"><pre><span class="pl-ent">plugins</span>:
    <span class="pl-ent">datasette-socrata</span>:
      <span class="pl-ent">database</span>: <span class="pl-s">socrata</span></pre></div>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Development</h2><a id="user-content-development" class="anchor" aria-label="Permalink: Development" href="#development"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">To set up this plugin locally, first checkout the code. Then create a new virtual environment:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="cd datasette-socrata
  python3 -m venv venv
  source venv/bin/activate"><pre class="notranslate"><code>cd datasette-socrata
  python3 -m venv venv
  source venv/bin/activate
  </code></pre></div>
  <p dir="auto">Now install the dependencies and test dependencies:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="pip install -e '.[test]'"><pre class="notranslate"><code>pip install -e '.[test]'
  </code></pre></div>
  <p dir="auto">To run the tests:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="pytest"><pre class="notranslate"><code>pytest
  </code></pre></div>
  </article></div>


node_id: R_kgDOHilbpw
name: datasette-scale-to-zero
full_name: simonw/datasette-scale-to-zero
html_url: https://github.com/simonw/datasette-scale-to-zero
description: Quit Datasette if it has not received traffic for a specified time period
created_at: 2022-06-21T22:49:04Z
updated_at: 2026-01-19T20:45:23Z
pushed_at: 2026-02-18T03:10:13Z
homepage: https://datasette.io/plugins/datasette-scale-to-zero
language: Python
license: apache-2.0
topics: ["datasette", "datasette-plugin"]
visibility: public
default_branch: main
permissions: {"admin": false, "maintain": false, "push": false, "triage": false, "pull": false}
pull_request_creation_policy: all
readme:
  # datasette-scale-to-zero

  [![PyPI](https://img.shields.io/pypi/v/datasette-scale-to-zero.svg)](https://pypi.org/project/datasette-scale-to-zero/)
  [![Changelog](https://img.shields.io/github/v/release/simonw/datasette-scale-to-zero?include_prereleases&label=changelog)](https://github.com/simonw/datasette-scale-to-zero/releases)
  [![Tests](https://github.com/simonw/datasette-scale-to-zero/workflows/Test/badge.svg)](https://github.com/simonw/datasette-scale-to-zero/actions?query=workflow%3ATest)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/datasette-scale-to-zero/blob/main/LICENSE)

  Quit Datasette if it has not received traffic for a specified time period

  Some hosting providers such as [Fly](https://fly.io/) offer a scale to zero mechanism, where servers can shut down and will be automatically started when new traffic arrives.

  This plugin can be used to configure Datasette to quit X minutes (or seconds, or hours) after the last request it received. It can also cause the Datasette server to exit after a configured maximum time whether or not it is receiving traffic.

  ## Installation

  Install this plugin in the same environment as Datasette.

      datasette install datasette-scale-to-zero

  ## Configuration

  This plugin will only take effect if it has been configured.

  Add the following to your ``metadata.json`` or ``metadata.yml`` configuration file:

  ```json
  {
      "plugins": {
          "datasette-scale-to-zero": {
              "duration": "10m"
          }
      }
  }
  ```
  This will cause Datasette to quit if it has not received any HTTP traffic for 10 minutes.

  You can set this value using a suffix of `m` for minutes, `h` for hours or `s` for seconds.

  To cause Datasette to exit if the server has been running for longer than a specific time, use `"max_age"`:

  ```json
  {
      "plugins": {
          "datasette-scale-to-zero": {
              "max_age": "10h"
          }
      }
  }
  ```
  This example will exit the Datasette server if it has been running for more than ten hours.

  You can use `"duration"` and `"max_age"` together in the same configuration file:

  ```json
  {
      "plugins": {
          "datasette-scale-to-zero": {
              "max_age": "10h",
              "duration": "5m"
          }
      }
  }
  ```
  This example will quit if no traffic has been received in five minutes, or if the server has been running for ten hours.

  ## Configuring a shutdown HTTP message

  You can also configure the plugin to send an HTTP request somewhere right before it quits, using the `"shutdown_url"` option:

  ```json
  {
      "plugins": {
          "datasette-scale-to-zero": {
              "duration": "10m",
              "shutdown_url": "https://example.com/shutdown"
          }
      }
  }
  ```
  You can add additional headers to the GET request - for example to send Authorization headers - using `"shutdown_headers"`:

  ```json
  {
      "plugins": {
          "datasette-scale-to-zero": {
              "duration": "10m",
              "shutdown_url": "https://example.com/shutdown",
              "shutdown_headers": {
                  "Authorization": "Bearer secret"
              }
          }
      }
  }
  ```
  Use `"shutdown_method"` to set a different HTTP method, e.g. for `POST`. You can also set `shutdown_body` to specify the body that should be sent with the request:

  ```json
  {
      "plugins": {
          "datasette-scale-to-zero": {
              "duration": "10m",
              "shutdown_url": "https://example.com/shutdown",
              "shutdown_method": "POST",
              "shutdown_headers": {
                  "Authorization": "Bearer secret",
                  "Content-Type": "application/json"
              },
              "shutdown_body": "{\"message\": \"shutting down\"}"
          }
      }
  }
  ```

  ## Development

  To set up this plugin locally, first checkout the code. Then create a new virtual environment:

      cd datasette-scale-to-zero
      python3 -m venv venv
      source venv/bin/activate

  Now install the dependencies and test dependencies:

      pip install -e '.[test]'

  To run the tests:

      pytest

readme_html:
  <div id="readme" class="md" data-path="README.md"><article class="markdown-body entry-content container-lg" itemprop="text"><div class="markdown-heading" dir="auto"><h1 class="heading-element" dir="auto">datasette-scale-to-zero</h1><a id="user-content-datasette-scale-to-zero" class="anchor" aria-label="Permalink: datasette-scale-to-zero" href="#datasette-scale-to-zero"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto"><a href="https://pypi.org/project/datasette-scale-to-zero/" rel="nofollow"><img src="https://img.shields.io/pypi/v/datasette-scale-to-zero.svg" alt="PyPI" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/29280dbf327bbb2a5aa156d9173f9e184eb1d094d8c8dd6faf1687f97a5d1bbd/68747470733a2f2f696d672e736869656c64732e696f2f707970692f762f6461746173657474652d7363616c652d746f2d7a65726f2e737667"></a>
  <a href="https://github.com/simonw/datasette-scale-to-zero/releases"><img src="https://img.shields.io/github/v/release/simonw/datasette-scale-to-zero?include_prereleases&label=changelog" alt="Changelog" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/6ff6f0df807b866684b7dc369371b8420cdf13aef06230785d7dce3b4d49087d/68747470733a2f2f696d672e736869656c64732e696f2f6769746875622f762f72656c656173652f73696d6f6e772f6461746173657474652d7363616c652d746f2d7a65726f3f696e636c7564655f70726572656c6561736573266c6162656c3d6368616e67656c6f67"></a>
  <a href="https://github.com/simonw/datasette-scale-to-zero/actions?query=workflow%3ATest"><img src="https://github.com/simonw/datasette-scale-to-zero/workflows/Test/badge.svg" alt="Tests" style="max-width: 100%;"></a>
  <a href="https://github.com/simonw/datasette-scale-to-zero/blob/main/LICENSE"><img src="https://img.shields.io/badge/license-Apache%202.0-blue.svg" alt="License" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/c355f200ea90fddaa407b6eaab303663a669248ea3ca7b1fcf77dbe04ff5f48c/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6c6963656e73652d417061636865253230322e302d626c75652e737667"></a></p>
  <p dir="auto">Quit Datasette if it has not received traffic for a specified time period</p>
  <p dir="auto">Some hosting providers such as <a href="https://fly.io/" rel="nofollow">Fly</a> offer a scale to zero mechanism, where servers can shut down and will be automatically started when new traffic arrives.</p>
  <p dir="auto">This plugin can be used to configure Datasette to quit X minutes (or seconds, or hours) after the last request it received. It can also cause the Datasette server to exit after a configured maximum time whether or not it is receiving traffic.</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Installation</h2><a id="user-content-installation" class="anchor" aria-label="Permalink: Installation" href="#installation"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">Install this plugin in the same environment as Datasette.</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="datasette install datasette-scale-to-zero"><pre class="notranslate"><code>datasette install datasette-scale-to-zero
  </code></pre></div>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Configuration</h2><a id="user-content-configuration" class="anchor" aria-label="Permalink: Configuration" href="#configuration"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">This plugin will only take effect if it has been configured.</p>
  <p dir="auto">Add the following to your <code>metadata.json</code> or <code>metadata.yml</code> configuration file:</p>
  <div class="highlight highlight-source-json notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="{
      "plugins": {
          "datasette-scale-to-zero": {
              "duration": "10m"
          }
      }
  }"><pre>{
      <span class="pl-ent">"plugins"</span>: {
          <span class="pl-ent">"datasette-scale-to-zero"</span>: {
              <span class="pl-ent">"duration"</span>: <span class="pl-s"><span class="pl-pds">"</span>10m<span class="pl-pds">"</span></span>
          }
      }
  }</pre></div>
  <p dir="auto">This will cause Datasette to quit if it has not received any HTTP traffic for 10 minutes.</p>
  <p dir="auto">You can set this value using a suffix of <code>m</code> for minutes, <code>h</code> for hours or <code>s</code> for seconds.</p>
  <p dir="auto">To cause Datasette to exit if the server has been running for longer than a specific time, use <code>"max_age"</code>:</p>
  <div class="highlight highlight-source-json notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="{
      "plugins": {
          "datasette-scale-to-zero": {
              "max_age": "10h"
          }
      }
  }"><pre>{
      <span class="pl-ent">"plugins"</span>: {
          <span class="pl-ent">"datasette-scale-to-zero"</span>: {
              <span class="pl-ent">"max_age"</span>: <span class="pl-s"><span class="pl-pds">"</span>10h<span class="pl-pds">"</span></span>
          }
      }
  }</pre></div>
  <p dir="auto">This example will exit the Datasette server if it has been running for more than ten hours.</p>
  <p dir="auto">You can use <code>"duration"</code> and <code>"max_age"</code> together in the same configuration file:</p>
  <div class="highlight highlight-source-json notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="{
      "plugins": {
          "datasette-scale-to-zero": {
              "max_age": "10h",
              "duration": "5m"
          }
      }
  }"><pre>{
      <span class="pl-ent">"plugins"</span>: {
          <span class="pl-ent">"datasette-scale-to-zero"</span>: {
              <span class="pl-ent">"max_age"</span>: <span class="pl-s"><span class="pl-pds">"</span>10h<span class="pl-pds">"</span></span>,
              <span class="pl-ent">"duration"</span>: <span class="pl-s"><span class="pl-pds">"</span>5m<span class="pl-pds">"</span></span>
          }
      }
  }</pre></div>
  <p dir="auto">This example will quit if no traffic has been received in five minutes, or if the server has been running for ten hours.</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Configuring a shutdown HTTP message</h2><a id="user-content-configuring-a-shutdown-http-message" class="anchor" aria-label="Permalink: Configuring a shutdown HTTP message" href="#configuring-a-shutdown-http-message"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">You can also configure the plugin to send an HTTP request somewhere right before it quits, using the <code>"shutdown_url"</code> option:</p>
  <div class="highlight highlight-source-json notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="{
      "plugins": {
          "datasette-scale-to-zero": {
              "duration": "10m",
              "shutdown_url": "https://example.com/shutdown"
          }
      }
  }"><pre>{
      <span class="pl-ent">"plugins"</span>: {
          <span class="pl-ent">"datasette-scale-to-zero"</span>: {
              <span class="pl-ent">"duration"</span>: <span class="pl-s"><span class="pl-pds">"</span>10m<span class="pl-pds">"</span></span>,
              <span class="pl-ent">"shutdown_url"</span>: <span class="pl-s"><span class="pl-pds">"</span>https://example.com/shutdown<span class="pl-pds">"</span></span>
          }
      }
  }</pre></div>
  <p dir="auto">You can add additional headers to the GET request - for example to send Authorization headers - using <code>"shutdown_headers"</code>:</p>
  <div class="highlight highlight-source-json notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="{
      "plugins": {
          "datasette-scale-to-zero": {
              "duration": "10m",
              "shutdown_url": "https://example.com/shutdown",
              "shutdown_headers": {
                  "Authorization": "Bearer secret"
              }
          }
      }
  }"><pre>{
      <span class="pl-ent">"plugins"</span>: {
          <span class="pl-ent">"datasette-scale-to-zero"</span>: {
              <span class="pl-ent">"duration"</span>: <span class="pl-s"><span class="pl-pds">"</span>10m<span class="pl-pds">"</span></span>,
              <span class="pl-ent">"shutdown_url"</span>: <span class="pl-s"><span class="pl-pds">"</span>https://example.com/shutdown<span class="pl-pds">"</span></span>,
              <span class="pl-ent">"shutdown_headers"</span>: {
                  <span class="pl-ent">"Authorization"</span>: <span class="pl-s"><span class="pl-pds">"</span>Bearer secret<span class="pl-pds">"</span></span>
              }
          }
      }
  }</pre></div>
  <p dir="auto">Use <code>"shutdown_method"</code> to set a different HTTP method, e.g. for <code>POST</code>. You can also set <code>shutdown_body</code> to specify the body that should be sent with the request:</p>
  <div class="highlight highlight-source-json notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="{
      "plugins": {
          "datasette-scale-to-zero": {
              "duration": "10m",
              "shutdown_url": "https://example.com/shutdown",
              "shutdown_method": "POST",
              "shutdown_headers": {
                  "Authorization": "Bearer secret",
                  "Content-Type": "application/json"
              },
              "shutdown_body": "{\"message\": \"shutting down\"}"
          }
      }
  }"><pre>{
      <span class="pl-ent">"plugins"</span>: {
          <span class="pl-ent">"datasette-scale-to-zero"</span>: {
              <span class="pl-ent">"duration"</span>: <span class="pl-s"><span class="pl-pds">"</span>10m<span class="pl-pds">"</span></span>,
              <span class="pl-ent">"shutdown_url"</span>: <span class="pl-s"><span class="pl-pds">"</span>https://example.com/shutdown<span class="pl-pds">"</span></span>,
              <span class="pl-ent">"shutdown_method"</span>: <span class="pl-s"><span class="pl-pds">"</span>POST<span class="pl-pds">"</span></span>,
              <span class="pl-ent">"shutdown_headers"</span>: {
                  <span class="pl-ent">"Authorization"</span>: <span class="pl-s"><span class="pl-pds">"</span>Bearer secret<span class="pl-pds">"</span></span>,
                  <span class="pl-ent">"Content-Type"</span>: <span class="pl-s"><span class="pl-pds">"</span>application/json<span class="pl-pds">"</span></span>
              },
              <span class="pl-ent">"shutdown_body"</span>: <span class="pl-s"><span class="pl-pds">"</span>{<span class="pl-cce">\"</span>message<span class="pl-cce">\"</span>: <span class="pl-cce">\"</span>shutting down<span class="pl-cce">\"</span>}<span class="pl-pds">"</span></span>
          }
      }
  }</pre></div>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Development</h2><a id="user-content-development" class="anchor" aria-label="Permalink: Development" href="#development"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">To set up this plugin locally, first checkout the code. Then create a new virtual environment:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="cd datasette-scale-to-zero
  python3 -m venv venv
  source venv/bin/activate"><pre class="notranslate"><code>cd datasette-scale-to-zero
  python3 -m venv venv
  source venv/bin/activate
  </code></pre></div>
  <p dir="auto">Now install the dependencies and test dependencies:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="pip install -e '.[test]'"><pre class="notranslate"><code>pip install -e '.[test]'
  </code></pre></div>
  <p dir="auto">To run the tests:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="pytest"><pre class="notranslate"><code>pytest
  </code></pre></div>
  </article></div>


node_id: R_kgDOHk6Aqw
name: s3-ocr
full_name: simonw/s3-ocr
html_url: https://github.com/simonw/s3-ocr
description: Tools for running OCR against files stored in S3
created_at: 2022-06-28T21:33:09Z
updated_at: 2025-02-06T02:38:33Z
pushed_at: 2022-08-10T04:43:17Z
language: Python
license: apache-2.0
topics: ["ocr", "s3", "textract"]
visibility: public
default_branch: main
permissions: {"admin": false, "maintain": false, "push": false, "triage": false, "pull": true}
readme:
  # s3-ocr

  [![PyPI](https://img.shields.io/pypi/v/s3-ocr.svg)](https://pypi.org/project/s3-ocr/)
  [![Changelog](https://img.shields.io/github/v/release/simonw/s3-ocr?include_prereleases&label=changelog)](https://github.com/simonw/s3-ocr/releases)
  [![Tests](https://github.com/simonw/s3-ocr/workflows/Test/badge.svg)](https://github.com/simonw/s3-ocr/actions?query=workflow%3ATest)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/s3-ocr/blob/master/LICENSE)

  Tools for running OCR against files stored in S3

  Background on this project: [s3-ocr: Extract text from PDF files stored in an S3 bucket](https://simonwillison.net/2022/Jun/30/s3-ocr/)

  ## Installation

  Install this tool using `pip`:

      pip install s3-ocr

  ## Demo

  You can see the results of running this tool against three PDFs from the Internet Archive ([one](https://archive.org/details/unmaskingrobert00houdgoog), [two](https://archive.org/details/practicalmagicia00harr), [three](https://archive.org/details/latestmagicbeing00hoff)) in [this example table](https://s3-ocr-demo.datasette.io/pages/pages?_facet=path#facet-path) hosted using [Datasette](https://datasette.io/).

  ## Starting OCR against PDFs in a bucket

  The `start` command takes a list of keys and submits them to [Textract](https://aws.amazon.com/textract/) for OCR processing.

  You need to have AWS configured using environment variables, credentials file in your home directory or a JSON or INI file generated using [s3-credentials](https://datasette.io/tools/s3-credentials).

  You can start the process running like this:

      s3-ocr start name-of-your-bucket my-pdf-file.pdf

  The paths you specify should be paths within the bucket. If you stored your PDF files in folders inside the bucket it should look like this:

      s3-ocr start name-of-your-bucket path/to/one.pdf path/to/two.pdf

  OCR can take some time. The results of the OCR will be stored in `textract-output` in your bucket.

  To process every file in the bucket with a `.pdf` extension use `--all`:

      s3-ocr start name-of-bucket --all

  To process every file with a `.pdf` extension within a specific folder, use `--prefix`:

      s3-ocr start name-of-bucket --prefix path/to/folder

  ### s3-ocr start --help

  <!-- [[[cog
  import cog
  from s3_ocr import cli
  from click.testing import CliRunner
  runner = CliRunner()
  result = runner.invoke(cli.cli, ["start", "--help"])
  help = result.output.replace("Usage: cli", "Usage: s3-ocr")
  cog.out(
      "```\n{}\n```".format(help)
  )
  ]]] -->
  ```
  Usage: s3-ocr start [OPTIONS] BUCKET [KEYS]...

    Start OCR tasks for PDF files in an S3 bucket

        s3-ocr start name-of-bucket path/to/one.pdf path/to/two.pdf

    To process every file with a .pdf extension:

        s3-ocr start name-of-bucket --all

    To process every .pdf in the PUBLIC/ folder:

        s3-ocr start name-of-bucket --prefix PUBLIC/

  Options:
    --all                 Process all PDF files in the bucket
    --prefix TEXT         Process all PDF files within this prefix
    --dry-run             Show what this would do, but don't actually do it
    --no-retry            Don't retry failed requests
    --access-key TEXT     AWS access key ID
    --secret-key TEXT     AWS secret access key
    --session-token TEXT  AWS session token
    --endpoint-url TEXT   Custom endpoint URL
    -a, --auth FILENAME   Path to JSON/INI file containing credentials
    --help                Show this message and exit.

  ```
  <!-- [[[end]]] -->

  ## Checking status

  The `s3-ocr status <bucket-name>` command shows a rough indication of progress through the tasks:

  ```
  % s3-ocr status sfms-history
  153 complete out of 532 jobs
  ```
  It compares the jobs that have been submitted, based on `.s3-ocr.json` files, to the jobs that have their results written to the `textract-output/` folder.

  ### s3-ocr status --help

  <!-- [[[cog
  result = runner.invoke(cli.cli, ["status", "--help"])
  help = result.output.replace("Usage: cli", "Usage: s3-ocr")
  cog.out(
      "```\n{}\n```".format(help.split("--access-key")[0] + "--access-key ...")
  )
  ]]] -->
  ```
  Usage: s3-ocr status [OPTIONS] BUCKET

    Show status of OCR jobs for a bucket

  Options:
    --access-key ...
  ```
  <!-- [[[end]]] -->

  ## Inspecting a job

  The `s3-ocr inspect-job <job_id>` command can be used to check the status of a specific job ID:
  ```
  % s3-ocr inspect-job b267282745685226339b7e0d4366c4ff6887b7e293ed4b304dc8bb8b991c7864
  {
    "DocumentMetadata": {
      "Pages": 583
    },
    "JobStatus": "SUCCEEDED",
    "DetectDocumentTextModelVersion": "1.0"
  }
  ```

  ### s3-ocr inspect-job --help

  <!-- [[[cog
  result = runner.invoke(cli.cli, ["inspect-job", "--help"])
  help = result.output.replace("Usage: cli", "Usage: s3-ocr")
  cog.out(
      "```\n{}\n```".format(help.split("--access-key")[0] + "--access-key ...")
  )
  ]]] -->
  ```
  Usage: s3-ocr inspect-job [OPTIONS] JOB_ID

    Show the current status of an OCR job

        s3-ocr inspect-job <job_id>

  Options:
    --access-key ...
  ```
  <!-- [[[end]]] -->

  ## Fetching the results

  Once an OCR job has completed you can download the resulting JSON using the `fetch` command:

      s3-ocr fetch name-of-bucket path/to/file.pdf

  This will save files in the current directory with names like this:

  - `4d9b5cf580e761fdb16fd24edce14737ebc562972526ef6617554adfa11d6038-1.json`
  - `4d9b5cf580e761fdb16fd24edce14737ebc562972526ef6617554adfa11d6038-2.json`

  The number of files will vary depending on the length of the document.

  If you don't want separate files you can combine them together using the `-c/--combine` option:

      s3-ocr fetch name-of-bucket path/to/file.pdf --combine output.json

  The `output.json` file will then contain data that looks something like this:

  ```
  {
    "Blocks": [
      {
        "BlockType": "PAGE",
        "Geometry": {...}
        "Page": 1,
        ...
      },
      {
        "BlockType": "LINE",
        "Page": 1,
        ...
        "Text": "Barry",
      },
  ```
  ### s3-ocr fetch --help

  <!-- [[[cog
  result = runner.invoke(cli.cli, ["fetch", "--help"])
  help = result.output.replace("Usage: cli", "Usage: s3-ocr")
  cog.out(
      "```\n{}\n```".format(help.split("--access-key")[0] + "--access-key ...")
  )
  ]]] -->
  ```
  Usage: s3-ocr fetch [OPTIONS] BUCKET KEY

    Fetch the OCR results for a specified file

        s3-ocr fetch name-of-bucket path/to/key.pdf

    This will save files in the current directory called things like

        a806e67e504fc15f...48314e-1.json     a806e67e504fc15f...48314e-2.json

    To combine these together into a single JSON file with a specified name, use:

        s3-ocr fetch name-of-bucket path/to/key.pdf --combine output.json

    Use "--output -" to print the combined JSON to standard output instead.

  Options:
    -c, --combine FILENAME  Write combined JSON to file
    --access-key ...
  ```
  <!-- [[[end]]] -->

  ## Fetching just the text of a page

  If you don't want to deal with the JSON directly, you can use the `text` command to retrieve just the text extracted from a PDF:

      s3-ocr text name-of-bucket path/to/file.pdf

  This will output plain text to standard output.

  To save that to a file, use this:

      s3-ocr text name-of-bucket path/to/file.pdf > text.txt

  Separate pages will be separated by three newlines. To separate them using a `----` horizontal divider instead add `--divider`:

      s3-ocr text name-of-bucket path/to/file.pdf --divider

  ### s3-ocr text --help

  <!-- [[[cog
  result = runner.invoke(cli.cli, ["text", "--help"])
  help = result.output.replace("Usage: cli", "Usage: s3-ocr")
  cog.out(
      "```\n{}\n```".format(help.split("--access-key")[0] + "--access-key ...")
  )
  ]]] -->
  ```
  Usage: s3-ocr text [OPTIONS] BUCKET KEY

    Retrieve the text from an OCRd PDF file

        s3-ocr text name-of-bucket path/to/key.pdf

  Options:
    --divider             Add ---- between pages
    --access-key ...
  ```
  <!-- [[[end]]] -->

  ## Avoiding processing duplicates

  If you move files around within your S3 bucket `s3-ocr` can lose track of which files have already been processed. This can lead to additional Textract charges for processing should you run `s3-ocr start` against those new files.

  The `s3-ocr dedupe` command addresses this by scanning your bucket for files that have a new name but have previously been processed. It does this by looking at the `ETag` for each file, which represents the MD5 hash of the file contents.

  The command will write out new `.s3ocr.json` files for each detected duplicate. This will avoid those duplicates being run those duplicates through OCR a second time should yo run `s3-ocr start`.

      s3-ocr dedupe name-of-bucket

  Add `--dry-run` for a preview of the changes that will be made to your bucket.

  ### s3-ocr dedupe --help

  <!-- [[[cog
  result = runner.invoke(cli.cli, ["dedupe", "--help"])
  help = result.output.replace("Usage: cli", "Usage: s3-ocr")
  cog.out(
      "```\n{}\n```".format(help.split("--access-key")[0] + "--access-key ...")
  )
  ]]] -->
  ```
  Usage: s3-ocr dedupe [OPTIONS] BUCKET

    Scan every file in the bucket checking for duplicates - files that have not
    yet been OCRd but that have the same contents (based on ETag) as a file that
    HAS been OCRd.

        s3-ocr dedupe name-of-bucket

  Options:
    --dry-run             Show output without writing anything to S3
    --access-key ...
  ```
  <!-- [[[end]]] -->

  ## Changes made to your bucket

  To keep track of which files have been submitted for processing, `s3-ocr` will create a JSON file for every file that it adds to the OCR queue.

  This file will be called:

      path-to-file/name-of-file.pdf.s3-ocr.json

  Each of these JSON files contains data that looks like this:

  ```json
  {
    "job_id": "a34eb4e8dc7e70aa9668f7272aa403e85997364199a654422340bc5ada43affe",
    "etag": "\"b0c77472e15500347ebf46032a454e8e\""
  }
  ```
  The recorded `job_id` can be used later to associate the file with the results of the OCR task in `textract-output/`.

  The `etag` is the ETag of the S3 object at the time it was submitted. This can be used later to determine if a file has changed since it last had OCR run against it.

  This design for the tool, with the `.s3-ocr.json` files tracking jobs that have been submitted, means that it is safe to run `s3-ocr start` against the same bucket multiple times without the risk of starting duplicate OCR jobs.

  ## Creating a SQLite index of your OCR results

  The `s3-ocr index <bucket> <database_file>` command creates a SQLite database containing the results of the OCR, and configures SQLite full-text search against the text:

  ```
  % s3-ocr index sfms-history index.db
  Fetching job details  [####################################]  100%
  Populating pages table  [####################----------------]   55%  00:03:18
  ```
  The schema of the resulting database looks like this (excluding the FTS tables):
  ```sql
  CREATE TABLE [pages] (
     [path] TEXT,
     [page] INTEGER,
     [folder] TEXT,
     [text] TEXT,
     PRIMARY KEY ([path], [page])
  );
  CREATE TABLE [ocr_jobs] (
     [key] TEXT PRIMARY KEY,
     [job_id] TEXT,
     [etag] TEXT,
     [s3_ocr_etag] TEXT
  );
  CREATE TABLE [fetched_jobs] (
     [job_id] TEXT PRIMARY KEY
  );
  ```
  The database is designed to be used with [Datasette](https://datasette.io).

  ### s3-ocr index --help

  <!-- [[[cog
  result = runner.invoke(cli.cli, ["index", "--help"])
  help = result.output.replace("Usage: cli", "Usage: s3-ocr")
  cog.out(
      "```\n{}\n```".format(help.split("--access-key")[0] + "--access-key ...")
  )
  ]]] -->
  ```
  Usage: s3-ocr index [OPTIONS] BUCKET DATABASE

    Create a SQLite database with OCR results for files in a bucket

  Options:
    --access-key ...
  ```
  <!-- [[[end]]] -->

  ## Development

  To contribute to this tool, first checkout the code. Then create a new virtual environment:

      cd s3-ocr
      python -m venv venv
      source venv/bin/activate

  Now install the dependencies and test dependencies:

      pip install -e '.[test]'

  To run the tests:

      pytest

  To regenerate the README file with the latest `--help`:

      cog -r README.md

readme_html:
  <div id="readme" class="md" data-path="README.md"><article class="markdown-body entry-content container-lg" itemprop="text"><div class="markdown-heading" dir="auto"><h1 class="heading-element" dir="auto">s3-ocr</h1><a id="user-content-s3-ocr" class="anchor" aria-label="Permalink: s3-ocr" href="#s3-ocr"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto"><a href="https://pypi.org/project/s3-ocr/" rel="nofollow"><img src="https://img.shields.io/pypi/v/s3-ocr.svg" alt="PyPI" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/13d8770d081d8fa819021c7d015de2d500adfcdedcaca37bfefa2afdcee536b1/68747470733a2f2f696d672e736869656c64732e696f2f707970692f762f73332d6f63722e737667"></a>
  <a href="https://github.com/simonw/s3-ocr/releases"><img src="https://img.shields.io/github/v/release/simonw/s3-ocr?include_prereleases&label=changelog" alt="Changelog" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/6a83c5391e6a5d9f7a4b5ce2f51a5ab4a5a1511d31b729742278b561bed357c9/68747470733a2f2f696d672e736869656c64732e696f2f6769746875622f762f72656c656173652f73696d6f6e772f73332d6f63723f696e636c7564655f70726572656c6561736573266c6162656c3d6368616e67656c6f67"></a>
  <a href="https://github.com/simonw/s3-ocr/actions?query=workflow%3ATest"><img src="https://github.com/simonw/s3-ocr/workflows/Test/badge.svg" alt="Tests" style="max-width: 100%;"></a>
  <a href="https://github.com/simonw/s3-ocr/blob/master/LICENSE"><img src="https://img.shields.io/badge/license-Apache%202.0-blue.svg" alt="License" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/c355f200ea90fddaa407b6eaab303663a669248ea3ca7b1fcf77dbe04ff5f48c/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6c6963656e73652d417061636865253230322e302d626c75652e737667"></a></p>
  <p dir="auto">Tools for running OCR against files stored in S3</p>
  <p dir="auto">Background on this project: <a href="https://simonwillison.net/2022/Jun/30/s3-ocr/" rel="nofollow">s3-ocr: Extract text from PDF files stored in an S3 bucket</a></p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Installation</h2><a id="user-content-installation" class="anchor" aria-label="Permalink: Installation" href="#installation"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">Install this tool using <code>pip</code>:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="pip install s3-ocr"><pre class="notranslate"><code>pip install s3-ocr
  </code></pre></div>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Demo</h2><a id="user-content-demo" class="anchor" aria-label="Permalink: Demo" href="#demo"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">You can see the results of running this tool against three PDFs from the Internet Archive (<a href="https://archive.org/details/unmaskingrobert00houdgoog" rel="nofollow">one</a>, <a href="https://archive.org/details/practicalmagicia00harr" rel="nofollow">two</a>, <a href="https://archive.org/details/latestmagicbeing00hoff" rel="nofollow">three</a>) in <a href="https://s3-ocr-demo.datasette.io/pages/pages?_facet=path#facet-path" rel="nofollow">this example table</a> hosted using <a href="https://datasette.io/" rel="nofollow">Datasette</a>.</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Starting OCR against PDFs in a bucket</h2><a id="user-content-starting-ocr-against-pdfs-in-a-bucket" class="anchor" aria-label="Permalink: Starting OCR against PDFs in a bucket" href="#starting-ocr-against-pdfs-in-a-bucket"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">The <code>start</code> command takes a list of keys and submits them to <a href="https://aws.amazon.com/textract/" rel="nofollow">Textract</a> for OCR processing.</p>
  <p dir="auto">You need to have AWS configured using environment variables, credentials file in your home directory or a JSON or INI file generated using <a href="https://datasette.io/tools/s3-credentials" rel="nofollow">s3-credentials</a>.</p>
  <p dir="auto">You can start the process running like this:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="s3-ocr start name-of-your-bucket my-pdf-file.pdf"><pre class="notranslate"><code>s3-ocr start name-of-your-bucket my-pdf-file.pdf
  </code></pre></div>
  <p dir="auto">The paths you specify should be paths within the bucket. If you stored your PDF files in folders inside the bucket it should look like this:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="s3-ocr start name-of-your-bucket path/to/one.pdf path/to/two.pdf"><pre class="notranslate"><code>s3-ocr start name-of-your-bucket path/to/one.pdf path/to/two.pdf
  </code></pre></div>
  <p dir="auto">OCR can take some time. The results of the OCR will be stored in <code>textract-output</code> in your bucket.</p>
  <p dir="auto">To process every file in the bucket with a <code>.pdf</code> extension use <code>--all</code>:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="s3-ocr start name-of-bucket --all"><pre class="notranslate"><code>s3-ocr start name-of-bucket --all
  </code></pre></div>
  <p dir="auto">To process every file with a <code>.pdf</code> extension within a specific folder, use <code>--prefix</code>:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="s3-ocr start name-of-bucket --prefix path/to/folder"><pre class="notranslate"><code>s3-ocr start name-of-bucket --prefix path/to/folder
  </code></pre></div>
  <div class="markdown-heading" dir="auto"><h3 class="heading-element" dir="auto">s3-ocr start --help</h3><a id="user-content-s3-ocr-start---help" class="anchor" aria-label="Permalink: s3-ocr start --help" href="#s3-ocr-start---help"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>

  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="Usage: s3-ocr start [OPTIONS] BUCKET [KEYS]...

    Start OCR tasks for PDF files in an S3 bucket

        s3-ocr start name-of-bucket path/to/one.pdf path/to/two.pdf

    To process every file with a .pdf extension:

        s3-ocr start name-of-bucket --all

    To process every .pdf in the PUBLIC/ folder:

        s3-ocr start name-of-bucket --prefix PUBLIC/

  Options:
    --all                 Process all PDF files in the bucket
    --prefix TEXT         Process all PDF files within this prefix
    --dry-run             Show what this would do, but don't actually do it
    --no-retry            Don't retry failed requests
    --access-key TEXT     AWS access key ID
    --secret-key TEXT     AWS secret access key
    --session-token TEXT  AWS session token
    --endpoint-url TEXT   Custom endpoint URL
    -a, --auth FILENAME   Path to JSON/INI file containing credentials
    --help                Show this message and exit.
  "><pre class="notranslate"><code>Usage: s3-ocr start [OPTIONS] BUCKET [KEYS]...

    Start OCR tasks for PDF files in an S3 bucket

        s3-ocr start name-of-bucket path/to/one.pdf path/to/two.pdf

    To process every file with a .pdf extension:

        s3-ocr start name-of-bucket --all

    To process every .pdf in the PUBLIC/ folder:

        s3-ocr start name-of-bucket --prefix PUBLIC/

  Options:
    --all                 Process all PDF files in the bucket
    --prefix TEXT         Process all PDF files within this prefix
    --dry-run             Show what this would do, but don't actually do it
    --no-retry            Don't retry failed requests
    --access-key TEXT     AWS access key ID
    --secret-key TEXT     AWS secret access key
    --session-token TEXT  AWS session token
    --endpoint-url TEXT   Custom endpoint URL
    -a, --auth FILENAME   Path to JSON/INI file containing credentials
    --help                Show this message and exit.

  </code></pre></div>

  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Checking status</h2><a id="user-content-checking-status" class="anchor" aria-label="Permalink: Checking status" href="#checking-status"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">The <code>s3-ocr status <bucket-name></code> command shows a rough indication of progress through the tasks:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="% s3-ocr status sfms-history
  153 complete out of 532 jobs"><pre class="notranslate"><code>% s3-ocr status sfms-history
  153 complete out of 532 jobs
  </code></pre></div>
  <p dir="auto">It compares the jobs that have been submitted, based on <code>.s3-ocr.json</code> files, to the jobs that have their results written to the <code>textract-output/</code> folder.</p>
  <div class="markdown-heading" dir="auto"><h3 class="heading-element" dir="auto">s3-ocr status --help</h3><a id="user-content-s3-ocr-status---help" class="anchor" aria-label="Permalink: s3-ocr status --help" href="#s3-ocr-status---help"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>

  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="Usage: s3-ocr status [OPTIONS] BUCKET

    Show status of OCR jobs for a bucket

  Options:
    --access-key ..."><pre class="notranslate"><code>Usage: s3-ocr status [OPTIONS] BUCKET

    Show status of OCR jobs for a bucket

  Options:
    --access-key ...
  </code></pre></div>

  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Inspecting a job</h2><a id="user-content-inspecting-a-job" class="anchor" aria-label="Permalink: Inspecting a job" href="#inspecting-a-job"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">The <code>s3-ocr inspect-job <job_id></code> command can be used to check the status of a specific job ID:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="% s3-ocr inspect-job b267282745685226339b7e0d4366c4ff6887b7e293ed4b304dc8bb8b991c7864
  {
    "DocumentMetadata": {
      "Pages": 583
    },
    "JobStatus": "SUCCEEDED",
    "DetectDocumentTextModelVersion": "1.0"
  }"><pre class="notranslate"><code>% s3-ocr inspect-job b267282745685226339b7e0d4366c4ff6887b7e293ed4b304dc8bb8b991c7864
  {
    "DocumentMetadata": {
      "Pages": 583
    },
    "JobStatus": "SUCCEEDED",
    "DetectDocumentTextModelVersion": "1.0"
  }
  </code></pre></div>
  <div class="markdown-heading" dir="auto"><h3 class="heading-element" dir="auto">s3-ocr inspect-job --help</h3><a id="user-content-s3-ocr-inspect-job---help" class="anchor" aria-label="Permalink: s3-ocr inspect-job --help" href="#s3-ocr-inspect-job---help"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>

  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="Usage: s3-ocr inspect-job [OPTIONS] JOB_ID

    Show the current status of an OCR job

        s3-ocr inspect-job <job_id>

  Options:
    --access-key ..."><pre class="notranslate"><code>Usage: s3-ocr inspect-job [OPTIONS] JOB_ID

    Show the current status of an OCR job

        s3-ocr inspect-job <job_id>

  Options:
    --access-key ...
  </code></pre></div>

  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Fetching the results</h2><a id="user-content-fetching-the-results" class="anchor" aria-label="Permalink: Fetching the results" href="#fetching-the-results"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">Once an OCR job has completed you can download the resulting JSON using the <code>fetch</code> command:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="s3-ocr fetch name-of-bucket path/to/file.pdf"><pre class="notranslate"><code>s3-ocr fetch name-of-bucket path/to/file.pdf
  </code></pre></div>
  <p dir="auto">This will save files in the current directory with names like this:</p>
  <ul dir="auto">
  <li><code>4d9b5cf580e761fdb16fd24edce14737ebc562972526ef6617554adfa11d6038-1.json</code></li>
  <li><code>4d9b5cf580e761fdb16fd24edce14737ebc562972526ef6617554adfa11d6038-2.json</code></li>
  </ul>
  <p dir="auto">The number of files will vary depending on the length of the document.</p>
  <p dir="auto">If you don't want separate files you can combine them together using the <code>-c/--combine</code> option:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="s3-ocr fetch name-of-bucket path/to/file.pdf --combine output.json"><pre class="notranslate"><code>s3-ocr fetch name-of-bucket path/to/file.pdf --combine output.json
  </code></pre></div>
  <p dir="auto">The <code>output.json</code> file will then contain data that looks something like this:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="{
    "Blocks": [
      {
        "BlockType": "PAGE",
        "Geometry": {...}
        "Page": 1,
        ...
      },
      {
        "BlockType": "LINE",
        "Page": 1,
        ...
        "Text": "Barry",
      },"><pre class="notranslate"><code>{
    "Blocks": [
      {
        "BlockType": "PAGE",
        "Geometry": {...}
        "Page": 1,
        ...
      },
      {
        "BlockType": "LINE",
        "Page": 1,
        ...
        "Text": "Barry",
      },
  </code></pre></div>
  <div class="markdown-heading" dir="auto"><h3 class="heading-element" dir="auto">s3-ocr fetch --help</h3><a id="user-content-s3-ocr-fetch---help" class="anchor" aria-label="Permalink: s3-ocr fetch --help" href="#s3-ocr-fetch---help"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>

  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="Usage: s3-ocr fetch [OPTIONS] BUCKET KEY

    Fetch the OCR results for a specified file

        s3-ocr fetch name-of-bucket path/to/key.pdf

    This will save files in the current directory called things like

        a806e67e504fc15f...48314e-1.json     a806e67e504fc15f...48314e-2.json

    To combine these together into a single JSON file with a specified name, use:

        s3-ocr fetch name-of-bucket path/to/key.pdf --combine output.json

    Use "--output -" to print the combined JSON to standard output instead.

  Options:
    -c, --combine FILENAME  Write combined JSON to file
    --access-key ..."><pre class="notranslate"><code>Usage: s3-ocr fetch [OPTIONS] BUCKET KEY

    Fetch the OCR results for a specified file

        s3-ocr fetch name-of-bucket path/to/key.pdf

    This will save files in the current directory called things like

        a806e67e504fc15f...48314e-1.json     a806e67e504fc15f...48314e-2.json

    To combine these together into a single JSON file with a specified name, use:

        s3-ocr fetch name-of-bucket path/to/key.pdf --combine output.json

    Use "--output -" to print the combined JSON to standard output instead.

  Options:
    -c, --combine FILENAME  Write combined JSON to file
    --access-key ...
  </code></pre></div>

  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Fetching just the text of a page</h2><a id="user-content-fetching-just-the-text-of-a-page" class="anchor" aria-label="Permalink: Fetching just the text of a page" href="#fetching-just-the-text-of-a-page"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">If you don't want to deal with the JSON directly, you can use the <code>text</code> command to retrieve just the text extracted from a PDF:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="s3-ocr text name-of-bucket path/to/file.pdf"><pre class="notranslate"><code>s3-ocr text name-of-bucket path/to/file.pdf
  </code></pre></div>
  <p dir="auto">This will output plain text to standard output.</p>
  <p dir="auto">To save that to a file, use this:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="s3-ocr text name-of-bucket path/to/file.pdf > text.txt"><pre class="notranslate"><code>s3-ocr text name-of-bucket path/to/file.pdf > text.txt
  </code></pre></div>
  <p dir="auto">Separate pages will be separated by three newlines. To separate them using a <code>----</code> horizontal divider instead add <code>--divider</code>:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="s3-ocr text name-of-bucket path/to/file.pdf --divider"><pre class="notranslate"><code>s3-ocr text name-of-bucket path/to/file.pdf --divider
  </code></pre></div>
  <div class="markdown-heading" dir="auto"><h3 class="heading-element" dir="auto">s3-ocr text --help</h3><a id="user-content-s3-ocr-text---help" class="anchor" aria-label="Permalink: s3-ocr text --help" href="#s3-ocr-text---help"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>

  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="Usage: s3-ocr text [OPTIONS] BUCKET KEY

    Retrieve the text from an OCRd PDF file

        s3-ocr text name-of-bucket path/to/key.pdf

  Options:
    --divider             Add ---- between pages
    --access-key ..."><pre class="notranslate"><code>Usage: s3-ocr text [OPTIONS] BUCKET KEY

    Retrieve the text from an OCRd PDF file

        s3-ocr text name-of-bucket path/to/key.pdf

  Options:
    --divider             Add ---- between pages
    --access-key ...
  </code></pre></div>

  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Avoiding processing duplicates</h2><a id="user-content-avoiding-processing-duplicates" class="anchor" aria-label="Permalink: Avoiding processing duplicates" href="#avoiding-processing-duplicates"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">If you move files around within your S3 bucket <code>s3-ocr</code> can lose track of which files have already been processed. This can lead to additional Textract charges for processing should you run <code>s3-ocr start</code> against those new files.</p>
  <p dir="auto">The <code>s3-ocr dedupe</code> command addresses this by scanning your bucket for files that have a new name but have previously been processed. It does this by looking at the <code>ETag</code> for each file, which represents the MD5 hash of the file contents.</p>
  <p dir="auto">The command will write out new <code>.s3ocr.json</code> files for each detected duplicate. This will avoid those duplicates being run those duplicates through OCR a second time should yo run <code>s3-ocr start</code>.</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="s3-ocr dedupe name-of-bucket"><pre class="notranslate"><code>s3-ocr dedupe name-of-bucket
  </code></pre></div>
  <p dir="auto">Add <code>--dry-run</code> for a preview of the changes that will be made to your bucket.</p>
  <div class="markdown-heading" dir="auto"><h3 class="heading-element" dir="auto">s3-ocr dedupe --help</h3><a id="user-content-s3-ocr-dedupe---help" class="anchor" aria-label="Permalink: s3-ocr dedupe --help" href="#s3-ocr-dedupe---help"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>

  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="Usage: s3-ocr dedupe [OPTIONS] BUCKET

    Scan every file in the bucket checking for duplicates - files that have not
    yet been OCRd but that have the same contents (based on ETag) as a file that
    HAS been OCRd.

        s3-ocr dedupe name-of-bucket

  Options:
    --dry-run             Show output without writing anything to S3
    --access-key ..."><pre class="notranslate"><code>Usage: s3-ocr dedupe [OPTIONS] BUCKET

    Scan every file in the bucket checking for duplicates - files that have not
    yet been OCRd but that have the same contents (based on ETag) as a file that
    HAS been OCRd.

        s3-ocr dedupe name-of-bucket

  Options:
    --dry-run             Show output without writing anything to S3
    --access-key ...
  </code></pre></div>

  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Changes made to your bucket</h2><a id="user-content-changes-made-to-your-bucket" class="anchor" aria-label="Permalink: Changes made to your bucket" href="#changes-made-to-your-bucket"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">To keep track of which files have been submitted for processing, <code>s3-ocr</code> will create a JSON file for every file that it adds to the OCR queue.</p>
  <p dir="auto">This file will be called:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="path-to-file/name-of-file.pdf.s3-ocr.json"><pre class="notranslate"><code>path-to-file/name-of-file.pdf.s3-ocr.json
  </code></pre></div>
  <p dir="auto">Each of these JSON files contains data that looks like this:</p>
  <div class="highlight highlight-source-json notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="{
    "job_id": "a34eb4e8dc7e70aa9668f7272aa403e85997364199a654422340bc5ada43affe",
    "etag": "\"b0c77472e15500347ebf46032a454e8e\""
  }"><pre>{
    <span class="pl-ent">"job_id"</span>: <span class="pl-s"><span class="pl-pds">"</span>a34eb4e8dc7e70aa9668f7272aa403e85997364199a654422340bc5ada43affe<span class="pl-pds">"</span></span>,
    <span class="pl-ent">"etag"</span>: <span class="pl-s"><span class="pl-pds">"</span><span class="pl-cce">\"</span>b0c77472e15500347ebf46032a454e8e<span class="pl-cce">\"</span><span class="pl-pds">"</span></span>
  }</pre></div>
  <p dir="auto">The recorded <code>job_id</code> can be used later to associate the file with the results of the OCR task in <code>textract-output/</code>.</p>
  <p dir="auto">The <code>etag</code> is the ETag of the S3 object at the time it was submitted. This can be used later to determine if a file has changed since it last had OCR run against it.</p>
  <p dir="auto">This design for the tool, with the <code>.s3-ocr.json</code> files tracking jobs that have been submitted, means that it is safe to run <code>s3-ocr start</code> against the same bucket multiple times without the risk of starting duplicate OCR jobs.</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Creating a SQLite index of your OCR results</h2><a id="user-content-creating-a-sqlite-index-of-your-ocr-results" class="anchor" aria-label="Permalink: Creating a SQLite index of your OCR results" href="#creating-a-sqlite-index-of-your-ocr-results"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">The <code>s3-ocr index <bucket> <database_file></code> command creates a SQLite database containing the results of the OCR, and configures SQLite full-text search against the text:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="% s3-ocr index sfms-history index.db
  Fetching job details  [####################################]  100%
  Populating pages table  [####################----------------]   55%  00:03:18"><pre class="notranslate"><code>% s3-ocr index sfms-history index.db
  Fetching job details  [####################################]  100%
  Populating pages table  [####################----------------]   55%  00:03:18
  </code></pre></div>
  <p dir="auto">The schema of the resulting database looks like this (excluding the FTS tables):</p>
  <div class="highlight highlight-source-sql notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="CREATE TABLE [pages] (
     [path] TEXT,
     [page] INTEGER,
     [folder] TEXT,
     [text] TEXT,
     PRIMARY KEY ([path], [page])
  );
  CREATE TABLE [ocr_jobs] (
     [key] TEXT PRIMARY KEY,
     [job_id] TEXT,
     [etag] TEXT,
     [s3_ocr_etag] TEXT
  );
  CREATE TABLE [fetched_jobs] (
     [job_id] TEXT PRIMARY KEY
  );"><pre>CREATE TABLE [pages] (
     [<span class="pl-k">path</span>] <span class="pl-k">TEXT</span>,
     [page] <span class="pl-k">INTEGER</span>,
     [folder] <span class="pl-k">TEXT</span>,
     [<span class="pl-k">text</span>] <span class="pl-k">TEXT</span>,
     <span class="pl-k">PRIMARY KEY</span> ([<span class="pl-k">path</span>], [page])
  );
  CREATE TABLE [ocr_jobs] (
     [key] <span class="pl-k">TEXT</span> <span class="pl-k">PRIMARY KEY</span>,
     [job_id] <span class="pl-k">TEXT</span>,
     [etag] <span class="pl-k">TEXT</span>,
     [s3_ocr_etag] <span class="pl-k">TEXT</span>
  );
  CREATE TABLE [fetched_jobs] (
     [job_id] <span class="pl-k">TEXT</span> <span class="pl-k">PRIMARY KEY</span>
  );</pre></div>
  <p dir="auto">The database is designed to be used with <a href="https://datasette.io" rel="nofollow">Datasette</a>.</p>
  <div class="markdown-heading" dir="auto"><h3 class="heading-element" dir="auto">s3-ocr index --help</h3><a id="user-content-s3-ocr-index---help" class="anchor" aria-label="Permalink: s3-ocr index --help" href="#s3-ocr-index---help"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>

  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="Usage: s3-ocr index [OPTIONS] BUCKET DATABASE

    Create a SQLite database with OCR results for files in a bucket

  Options:
    --access-key ..."><pre class="notranslate"><code>Usage: s3-ocr index [OPTIONS] BUCKET DATABASE

    Create a SQLite database with OCR results for files in a bucket

  Options:
    --access-key ...
  </code></pre></div>

  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Development</h2><a id="user-content-development" class="anchor" aria-label="Permalink: Development" href="#development"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">To contribute to this tool, first checkout the code. Then create a new virtual environment:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="cd s3-ocr
  python -m venv venv
  source venv/bin/activate"><pre class="notranslate"><code>cd s3-ocr
  python -m venv venv
  source venv/bin/activate
  </code></pre></div>
  <p dir="auto">Now install the dependencies and test dependencies:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="pip install -e '.[test]'"><pre class="notranslate"><code>pip install -e '.[test]'
  </code></pre></div>
  <p dir="auto">To run the tests:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="pytest"><pre class="notranslate"><code>pytest
  </code></pre></div>
  <p dir="auto">To regenerate the README file with the latest <code>--help</code>:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="cog -r README.md"><pre class="notranslate"><code>cog -r README.md
  </code></pre></div>
  </article></div>


node_id: R_kgDOHmgSAw
name: datasette-expose-env
full_name: simonw/datasette-expose-env
html_url: https://github.com/simonw/datasette-expose-env
description: Datasette plugin to expose selected environment variables at /-/env for debugging
created_at: 2022-07-03T21:14:29Z
updated_at: 2024-04-11T19:18:24Z
pushed_at: 2024-03-03T00:06:22Z
homepage: https://datasette.io/plugins/datasette-expose-env
language: Python
license: apache-2.0
topics: ["datasette", "datasette-plugin"]
visibility: public
default_branch: main
permissions: {"admin": false, "maintain": false, "push": false, "triage": false, "pull": true}
template_repository:
[cell repeats an earlier one in this file: 6377 characters]

readme:
  # datasette-expose-env

  [![PyPI](https://img.shields.io/pypi/v/datasette-expose-env.svg)](https://pypi.org/project/datasette-expose-env/)
  [![Changelog](https://img.shields.io/github/v/release/simonw/datasette-expose-env?include_prereleases&label=changelog)](https://github.com/simonw/datasette-expose-env/releases)
  [![Tests](https://github.com/simonw/datasette-expose-env/workflows/Test/badge.svg)](https://github.com/simonw/datasette-expose-env/actions?query=workflow%3ATest)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/datasette-expose-env/blob/main/LICENSE)

  Datasette plugin to expose selected environment variables at `/-/env` for debugging

  ## Installation

  Install this plugin in the same environment as Datasette.
  ```bash
  datasette install datasette-expose-env
  ```
  ## Configuration

  Decide on a list of environment variables you would like to expose, then add the following to your `metadata.yml` configuration:

  ```yaml
  plugins:
    datasette-expose-env:
      variables:
      - ENV_VAR_1
      - ENV_VAR_2
      - ENV_VAR_3
  ```

  If you are using JSON in a `metadata.json` file use the following:

  ```json
  {
    "plugins": {
      "datasette-expose-env": {
        "variables": [
          "ENV_VAR_1",
          "ENV_VAR_2",
          "ENV_VAR_3"
        ]
      }
    }
  }
  ```
  To show a full list of redacted environment variables use `"show_all_redacted": true`:

  ```yaml
  plugins:
    datasette-expose-env:
      show_all_redacted: true
      variables:
      - ENV_VAR_1
      - ENV_VAR_2
  ```
  This will show the values for `ENV_VAR_1` and `ENV_VAR_2` and then a full list of other variables with their value shown as `...`.


  Visit `/-/env` on your Datasette instance to see the values of the environment variables.

  ## Development

  To set up this plugin locally, first checkout the code. Then create a new virtual environment:
  ```bash
  cd datasette-expose-env
  python3 -m venv venv
  source venv/bin/activate
  ```
  Now install the dependencies and test dependencies:
  ```bash
  pip install -e '.[test]'
  ```
  To run the tests:
  ```bash
  pytest
  ```

readme_html:
  <div id="readme" class="md" data-path="README.md"><article class="markdown-body entry-content container-lg" itemprop="text"><div class="markdown-heading" dir="auto"><h1 class="heading-element" dir="auto">datasette-expose-env</h1><a id="user-content-datasette-expose-env" class="anchor" aria-label="Permalink: datasette-expose-env" href="#datasette-expose-env"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto"><a href="https://pypi.org/project/datasette-expose-env/" rel="nofollow"><img src="https://img.shields.io/pypi/v/datasette-expose-env.svg" alt="PyPI" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/de96aad99f3c544bd9312ebbe97c14a52826e3cd1d128535823c0b18a3552e51/68747470733a2f2f696d672e736869656c64732e696f2f707970692f762f6461746173657474652d6578706f73652d656e762e737667"></a>
  <a href="https://github.com/simonw/datasette-expose-env/releases"><img src="https://img.shields.io/github/v/release/simonw/datasette-expose-env?include_prereleases&label=changelog" alt="Changelog" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/f59004ab9999db0870986636932d9abaa87ba48bd8b27a3d42e7110b9553b047/68747470733a2f2f696d672e736869656c64732e696f2f6769746875622f762f72656c656173652f73696d6f6e772f6461746173657474652d6578706f73652d656e763f696e636c7564655f70726572656c6561736573266c6162656c3d6368616e67656c6f67"></a>
  <a href="https://github.com/simonw/datasette-expose-env/actions?query=workflow%3ATest"><img src="https://github.com/simonw/datasette-expose-env/workflows/Test/badge.svg" alt="Tests" style="max-width: 100%;"></a>
  <a href="https://github.com/simonw/datasette-expose-env/blob/main/LICENSE"><img src="https://img.shields.io/badge/license-Apache%202.0-blue.svg" alt="License" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/c355f200ea90fddaa407b6eaab303663a669248ea3ca7b1fcf77dbe04ff5f48c/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6c6963656e73652d417061636865253230322e302d626c75652e737667"></a></p>
  <p dir="auto">Datasette plugin to expose selected environment variables at <code>/-/env</code> for debugging</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Installation</h2><a id="user-content-installation" class="anchor" aria-label="Permalink: Installation" href="#installation"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">Install this plugin in the same environment as Datasette.</p>
  <div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="datasette install datasette-expose-env"><pre>datasette install datasette-expose-env</pre></div>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Configuration</h2><a id="user-content-configuration" class="anchor" aria-label="Permalink: Configuration" href="#configuration"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">Decide on a list of environment variables you would like to expose, then add the following to your <code>metadata.yml</code> configuration:</p>
  <div class="highlight highlight-source-yaml notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="plugins:
    datasette-expose-env:
      variables:
      - ENV_VAR_1
      - ENV_VAR_2
      - ENV_VAR_3"><pre><span class="pl-ent">plugins</span>:
    <span class="pl-ent">datasette-expose-env</span>:
      <span class="pl-ent">variables</span>:
      - <span class="pl-s">ENV_VAR_1</span>
      - <span class="pl-s">ENV_VAR_2</span>
      - <span class="pl-s">ENV_VAR_3</span></pre></div>
  <p dir="auto">If you are using JSON in a <code>metadata.json</code> file use the following:</p>
  <div class="highlight highlight-source-json notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="{
    "plugins": {
      "datasette-expose-env": {
        "variables": [
          "ENV_VAR_1",
          "ENV_VAR_2",
          "ENV_VAR_3"
        ]
      }
    }
  }"><pre>{
    <span class="pl-ent">"plugins"</span>: {
      <span class="pl-ent">"datasette-expose-env"</span>: {
        <span class="pl-ent">"variables"</span>: [
          <span class="pl-s"><span class="pl-pds">"</span>ENV_VAR_1<span class="pl-pds">"</span></span>,
          <span class="pl-s"><span class="pl-pds">"</span>ENV_VAR_2<span class="pl-pds">"</span></span>,
          <span class="pl-s"><span class="pl-pds">"</span>ENV_VAR_3<span class="pl-pds">"</span></span>
        ]
      }
    }
  }</pre></div>
  <p dir="auto">To show a full list of redacted environment variables use <code>"show_all_redacted": true</code>:</p>
  <div class="highlight highlight-source-yaml notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="plugins:
    datasette-expose-env:
      show_all_redacted: true
      variables:
      - ENV_VAR_1
      - ENV_VAR_2"><pre><span class="pl-ent">plugins</span>:
    <span class="pl-ent">datasette-expose-env</span>:
      <span class="pl-ent">show_all_redacted</span>: <span class="pl-c1">true</span>
      <span class="pl-ent">variables</span>:
      - <span class="pl-s">ENV_VAR_1</span>
      - <span class="pl-s">ENV_VAR_2</span></pre></div>
  <p dir="auto">This will show the values for <code>ENV_VAR_1</code> and <code>ENV_VAR_2</code> and then a full list of other variables with their value shown as <code>...</code>.</p>
  <p dir="auto">Visit <code>/-/env</code> on your Datasette instance to see the values of the environment variables.</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Development</h2><a id="user-content-development" class="anchor" aria-label="Permalink: Development" href="#development"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">To set up this plugin locally, first checkout the code. Then create a new virtual environment:</p>
  <div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="cd datasette-expose-env
  python3 -m venv venv
  source venv/bin/activate"><pre><span class="pl-c1">cd</span> datasette-expose-env
  python3 -m venv venv
  <span class="pl-c1">source</span> venv/bin/activate</pre></div>
  <p dir="auto">Now install the dependencies and test dependencies:</p>
  <div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="pip install -e '.[test]'"><pre>pip install -e <span class="pl-s"><span class="pl-pds">'</span>.[test]<span class="pl-pds">'</span></span></pre></div>
  <p dir="auto">To run the tests:</p>
  <div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="pytest"><pre>pytest</pre></div>
  </article></div>


node_id: R_kgDOHoFAng
name: sqlite-comprehend
full_name: simonw/sqlite-comprehend
html_url: https://github.com/simonw/sqlite-comprehend
description: Tools for running data in a SQLite database through AWS Comprehend
created_at: 2022-07-08T06:26:15Z
updated_at: 2022-07-11T21:44:34Z
pushed_at: 2022-07-12T14:21:42Z
language: Python
license: apache-2.0
topics: []
visibility: public
default_branch: main
permissions: {"admin": false, "maintain": false, "push": false, "triage": false, "pull": true}
readme:
  # sqlite-comprehend

  [![PyPI](https://img.shields.io/pypi/v/sqlite-comprehend.svg)](https://pypi.org/project/sqlite-comprehend/)
  [![Changelog](https://img.shields.io/github/v/release/simonw/sqlite-comprehend?include_prereleases&label=changelog)](https://github.com/simonw/sqlite-comprehend/releases)
  [![Tests](https://github.com/simonw/sqlite-comprehend/workflows/Test/badge.svg)](https://github.com/simonw/sqlite-comprehend/actions?query=workflow%3ATest)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/sqlite-comprehend/blob/master/LICENSE)

  Tools for running data in a SQLite database through [AWS Comprehend](https://aws.amazon.com/comprehend/)

  See [sqlite-comprehend: run AWS entity extraction against content in a SQLite database](https://simonwillison.net/2022/Jul/11/sqlite-comprehend/) for background on this project.

  ## Installation

  Install this tool using `pip`:

      pip install sqlite-comprehend

  ## Demo

  You can see examples of tables generated using this command here:

  - [comprehend_entities](https://datasette.simonwillison.net/simonwillisonblog/comprehend_entities) - the extracted entities, classified by type
  - [blog_entry_comprehend_entities](https://datasette.simonwillison.net/simonwillisonblog/blog_entry_comprehend_entities) - a table relating entities to the entries that they appear in
  - [comprehend_entity_types](https://datasette.simonwillison.net/simonwillisonblog/comprehend_entity_types) - a small lookup table of entity types

  ## Configuration

  You will need AWS credentials with the `comprehend:BatchDetectEntities` [IAM permission](https://docs.aws.amazon.com/comprehend/latest/dg/access-control-managing-permissions.html).

  You can configure credentials [using these instructions](https://boto3.amazonaws.com/v1/documentation/api/latest/guide/credentials.html). You can also save them to a JSON or INI configuration file and pass them to the command using `-a credentials.ini`, or pass them using the `--access-key` and `--secret-key` options.

  ## Entity extraction

  The `sqlite-comprehend entities` command runs entity extraction against every row in the specified table and saves the results to your database.

  Specify the database, the table and one or more columns containing text in that table. The following runs against the `text` column in the `pages` table of the `sfms.db` SQLite database:

      sqlite-comprehend sfms.db pages text

  Results will be written into a `pages_comprehend_entities` table. Change the name of the output table by passing `-o other_table_name`.

  You can run against a subset of rows by adding a `--where` clause:

      sqlite-comprehend sfms.db pages text --where 'id < 10'

  You can also used named parameters in your `--where` clause:

      sqlite-comprehend sfms.db pages text --where 'id < :maxid' -p maxid 10

  Only the first 5,000 characters for each row will be considered. Be sure to review [Comprehend's pricing](https://aws.amazon.com/comprehend/pricing/) - which starts at $0.0001 per hundred characters.

  If your context includes HTML tags, you can strip them out before extracting entities by adding `--strip-tags`:

      sqlite-comprehend sfms.db pages text --strip-tags

  Rows that have been processed are recorded in the `pages_comprehend_entities_done` table. If you run the command more than once it will only process rows that have been newly added.

  You can delete records from that `_done` table to run them again.

  ### sqlite-comprehend entities --help

  <!-- [[[cog
  from click.testing import CliRunner
  from sqlite_comprehend import cli
  runner = CliRunner()
  result = runner.invoke(cli.cli, ["entities", "--help"])
  help = result.output.replace("Usage: cli", "Usage: sqlite-comprehend")
  cog.out(
      "```\n{}\n```".format(help)
  )
  ]]] -->
  ```
  Usage: sqlite-comprehend entities [OPTIONS] DATABASE TABLE COLUMNS...

    Detect entities in columns in a table

    To extract entities from columns text1 and text2 in mytable:

        sqlite-comprehend entities my.db mytable text1 text2

    To run against just a subset of the rows in the table, add:

        --where "id < :max_id" -p max_id 50

    Results will be written to a table called mytable_comprehend_entities

    To specify a different output table, use -o custom_table_name

  Options:
    --where TEXT                WHERE clause to filter table
    -p, --param <TEXT TEXT>...  Named :parameters for SQL query
    -o, --output TEXT           Custom output table
    -r, --reset                 Start from scratch, deleting previous results
    --strip-tags                Strip HTML tags before extracting entities
    --access-key TEXT           AWS access key ID
    --secret-key TEXT           AWS secret access key
    --session-token TEXT        AWS session token
    --endpoint-url TEXT         Custom endpoint URL
    -a, --auth FILENAME         Path to JSON/INI file containing credentials
    --help                      Show this message and exit.

  ```
  <!-- [[[end]]] -->

  ## Schema

  Assuming an input table called `pages` the tables created by this tool will have the following schema:

  <!-- [[[cog
  import cog, json
  from sqlite_comprehend import cli
  from unittest.mock import patch
  from click.testing import CliRunner
  import sqlite_utils
  import tempfile, pathlib
  tmpdir = pathlib.Path(tempfile.mkdtemp())
  db_path = str(tmpdir / "data.db")
  db = sqlite_utils.Database(db_path)
  db["pages"].insert_all(
      [
          {
              "id": 1,
              "text": "John Bob",
          },
          {
              "id": 2,
              "text": "Sandra X",
          },
      ],
      pk="id",
  )
  with patch('boto3.client') as client:
      client.return_value.batch_detect_entities.return_value = {
          "ResultList": [
              {
                  "Index": 0,
                  "Entities": [
                      {
                          "Score": 0.8,
                          "Type": "PERSON",
                          "Text": "John Bob",
                          "BeginOffset": 0,
                          "EndOffset": 5,
                      },
                  ],
              },
              {
                  "Index": 1,
                  "Entities": [
                      {
                          "Score": 0.8,
                          "Type": "PERSON",
                          "Text": "Sandra X",
                          "BeginOffset": 0,
                          "EndOffset": 5,
                      },
                  ],
              },
          ],
          "ErrorList": [],
      }
      runner = CliRunner()
      result = runner.invoke(cli.cli, [
          "entities", db_path, "pages", "text"
      ])
  cog.out("```sql\n")
  cog.out(db.schema)
  cog.out("\n```")
  ]]] -->
  ```sql
  CREATE TABLE [pages] (
     [id] INTEGER PRIMARY KEY,
     [text] TEXT
  );
  CREATE TABLE [comprehend_entity_types] (
     [id] INTEGER PRIMARY KEY,
     [value] TEXT
  );
  CREATE TABLE [comprehend_entities] (
     [id] INTEGER PRIMARY KEY,
     [name] TEXT,
     [type] INTEGER REFERENCES [comprehend_entity_types]([id])
  );
  CREATE TABLE [pages_comprehend_entities] (
     [id] INTEGER REFERENCES [pages]([id]),
     [score] FLOAT,
     [entity] INTEGER REFERENCES [comprehend_entities]([id]),
     [begin_offset] INTEGER,
     [end_offset] INTEGER
  );
  CREATE UNIQUE INDEX [idx_comprehend_entity_types_value]
      ON [comprehend_entity_types] ([value]);
  CREATE UNIQUE INDEX [idx_comprehend_entities_type_name]
      ON [comprehend_entities] ([type], [name]);
  CREATE TABLE [pages_comprehend_entities_done] (
     [id] INTEGER PRIMARY KEY REFERENCES [pages]([id])
  );
  ```
  <!-- [[[end]]] -->

  ## Development

  To contribute to this tool, first checkout the code. Then create a new virtual environment:

      cd sqlite-comprehend
      python -m venv venv
      source venv/bin/activate

  Now install the dependencies and test dependencies:

      pip install -e '.[test]'

  To run the tests:

      pytest

readme_html:
  <div id="readme" class="md" data-path="README.md"><article class="markdown-body entry-content container-lg" itemprop="text"><div class="markdown-heading" dir="auto"><h1 class="heading-element" dir="auto">sqlite-comprehend</h1><a id="user-content-sqlite-comprehend" class="anchor" aria-label="Permalink: sqlite-comprehend" href="#sqlite-comprehend"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto"><a href="https://pypi.org/project/sqlite-comprehend/" rel="nofollow"><img src="https://img.shields.io/pypi/v/sqlite-comprehend.svg" alt="PyPI" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/aee97e43d65987ab201259ae25ec910ca56cd15afc82ff511cae9038850c8e4a/68747470733a2f2f696d672e736869656c64732e696f2f707970692f762f73716c6974652d636f6d70726568656e642e737667"></a>
  <a href="https://github.com/simonw/sqlite-comprehend/releases"><img src="https://img.shields.io/github/v/release/simonw/sqlite-comprehend?include_prereleases&label=changelog" alt="Changelog" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/036dfd9683ce63a410cf705cdda466f6f3af9d75d253018d984b677360d35fa5/68747470733a2f2f696d672e736869656c64732e696f2f6769746875622f762f72656c656173652f73696d6f6e772f73716c6974652d636f6d70726568656e643f696e636c7564655f70726572656c6561736573266c6162656c3d6368616e67656c6f67"></a>
  <a href="https://github.com/simonw/sqlite-comprehend/actions?query=workflow%3ATest"><img src="https://github.com/simonw/sqlite-comprehend/workflows/Test/badge.svg" alt="Tests" style="max-width: 100%;"></a>
  <a href="https://github.com/simonw/sqlite-comprehend/blob/master/LICENSE"><img src="https://img.shields.io/badge/license-Apache%202.0-blue.svg" alt="License" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/c355f200ea90fddaa407b6eaab303663a669248ea3ca7b1fcf77dbe04ff5f48c/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6c6963656e73652d417061636865253230322e302d626c75652e737667"></a></p>
  <p dir="auto">Tools for running data in a SQLite database through <a href="https://aws.amazon.com/comprehend/" rel="nofollow">AWS Comprehend</a></p>
  <p dir="auto">See <a href="https://simonwillison.net/2022/Jul/11/sqlite-comprehend/" rel="nofollow">sqlite-comprehend: run AWS entity extraction against content in a SQLite database</a> for background on this project.</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Installation</h2><a id="user-content-installation" class="anchor" aria-label="Permalink: Installation" href="#installation"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">Install this tool using <code>pip</code>:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="pip install sqlite-comprehend"><pre class="notranslate"><code>pip install sqlite-comprehend
  </code></pre></div>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Demo</h2><a id="user-content-demo" class="anchor" aria-label="Permalink: Demo" href="#demo"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">You can see examples of tables generated using this command here:</p>
  <ul dir="auto">
  <li><a href="https://datasette.simonwillison.net/simonwillisonblog/comprehend_entities" rel="nofollow">comprehend_entities</a> - the extracted entities, classified by type</li>
  <li><a href="https://datasette.simonwillison.net/simonwillisonblog/blog_entry_comprehend_entities" rel="nofollow">blog_entry_comprehend_entities</a> - a table relating entities to the entries that they appear in</li>
  <li><a href="https://datasette.simonwillison.net/simonwillisonblog/comprehend_entity_types" rel="nofollow">comprehend_entity_types</a> - a small lookup table of entity types</li>
  </ul>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Configuration</h2><a id="user-content-configuration" class="anchor" aria-label="Permalink: Configuration" href="#configuration"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">You will need AWS credentials with the <code>comprehend:BatchDetectEntities</code> <a href="https://docs.aws.amazon.com/comprehend/latest/dg/access-control-managing-permissions.html" rel="nofollow">IAM permission</a>.</p>
  <p dir="auto">You can configure credentials <a href="https://boto3.amazonaws.com/v1/documentation/api/latest/guide/credentials.html" rel="nofollow">using these instructions</a>. You can also save them to a JSON or INI configuration file and pass them to the command using <code>-a credentials.ini</code>, or pass them using the <code>--access-key</code> and <code>--secret-key</code> options.</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Entity extraction</h2><a id="user-content-entity-extraction" class="anchor" aria-label="Permalink: Entity extraction" href="#entity-extraction"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">The <code>sqlite-comprehend entities</code> command runs entity extraction against every row in the specified table and saves the results to your database.</p>
  <p dir="auto">Specify the database, the table and one or more columns containing text in that table. The following runs against the <code>text</code> column in the <code>pages</code> table of the <code>sfms.db</code> SQLite database:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="sqlite-comprehend sfms.db pages text"><pre class="notranslate"><code>sqlite-comprehend sfms.db pages text
  </code></pre></div>
  <p dir="auto">Results will be written into a <code>pages_comprehend_entities</code> table. Change the name of the output table by passing <code>-o other_table_name</code>.</p>
  <p dir="auto">You can run against a subset of rows by adding a <code>--where</code> clause:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="sqlite-comprehend sfms.db pages text --where 'id < 10'"><pre class="notranslate"><code>sqlite-comprehend sfms.db pages text --where 'id < 10'
  </code></pre></div>
  <p dir="auto">You can also used named parameters in your <code>--where</code> clause:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="sqlite-comprehend sfms.db pages text --where 'id < :maxid' -p maxid 10"><pre class="notranslate"><code>sqlite-comprehend sfms.db pages text --where 'id < :maxid' -p maxid 10
  </code></pre></div>
  <p dir="auto">Only the first 5,000 characters for each row will be considered. Be sure to review <a href="https://aws.amazon.com/comprehend/pricing/" rel="nofollow">Comprehend's pricing</a> - which starts at $0.0001 per hundred characters.</p>
  <p dir="auto">If your context includes HTML tags, you can strip them out before extracting entities by adding <code>--strip-tags</code>:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="sqlite-comprehend sfms.db pages text --strip-tags"><pre class="notranslate"><code>sqlite-comprehend sfms.db pages text --strip-tags
  </code></pre></div>
  <p dir="auto">Rows that have been processed are recorded in the <code>pages_comprehend_entities_done</code> table. If you run the command more than once it will only process rows that have been newly added.</p>
  <p dir="auto">You can delete records from that <code>_done</code> table to run them again.</p>
  <div class="markdown-heading" dir="auto"><h3 class="heading-element" dir="auto">sqlite-comprehend entities --help</h3><a id="user-content-sqlite-comprehend-entities---help" class="anchor" aria-label="Permalink: sqlite-comprehend entities --help" href="#sqlite-comprehend-entities---help"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>

  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="Usage: sqlite-comprehend entities [OPTIONS] DATABASE TABLE COLUMNS...

    Detect entities in columns in a table

    To extract entities from columns text1 and text2 in mytable:

        sqlite-comprehend entities my.db mytable text1 text2

    To run against just a subset of the rows in the table, add:

        --where "id < :max_id" -p max_id 50

    Results will be written to a table called mytable_comprehend_entities

    To specify a different output table, use -o custom_table_name

  Options:
    --where TEXT                WHERE clause to filter table
    -p, --param <TEXT TEXT>...  Named :parameters for SQL query
    -o, --output TEXT           Custom output table
    -r, --reset                 Start from scratch, deleting previous results
    --strip-tags                Strip HTML tags before extracting entities
    --access-key TEXT           AWS access key ID
    --secret-key TEXT           AWS secret access key
    --session-token TEXT        AWS session token
    --endpoint-url TEXT         Custom endpoint URL
    -a, --auth FILENAME         Path to JSON/INI file containing credentials
    --help                      Show this message and exit.
  "><pre class="notranslate"><code>Usage: sqlite-comprehend entities [OPTIONS] DATABASE TABLE COLUMNS...

    Detect entities in columns in a table

    To extract entities from columns text1 and text2 in mytable:

        sqlite-comprehend entities my.db mytable text1 text2

    To run against just a subset of the rows in the table, add:

        --where "id < :max_id" -p max_id 50

    Results will be written to a table called mytable_comprehend_entities

    To specify a different output table, use -o custom_table_name

  Options:
    --where TEXT                WHERE clause to filter table
    -p, --param <TEXT TEXT>...  Named :parameters for SQL query
    -o, --output TEXT           Custom output table
    -r, --reset                 Start from scratch, deleting previous results
    --strip-tags                Strip HTML tags before extracting entities
    --access-key TEXT           AWS access key ID
    --secret-key TEXT           AWS secret access key
    --session-token TEXT        AWS session token
    --endpoint-url TEXT         Custom endpoint URL
    -a, --auth FILENAME         Path to JSON/INI file containing credentials
    --help                      Show this message and exit.

  </code></pre></div>

  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Schema</h2><a id="user-content-schema" class="anchor" aria-label="Permalink: Schema" href="#schema"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">Assuming an input table called <code>pages</code> the tables created by this tool will have the following schema:</p>

  <div class="highlight highlight-source-sql notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="CREATE TABLE [pages] (
     [id] INTEGER PRIMARY KEY,
     [text] TEXT
  );
  CREATE TABLE [comprehend_entity_types] (
     [id] INTEGER PRIMARY KEY,
     [value] TEXT
  );
  CREATE TABLE [comprehend_entities] (
     [id] INTEGER PRIMARY KEY,
     [name] TEXT,
     [type] INTEGER REFERENCES [comprehend_entity_types]([id])
  );
  CREATE TABLE [pages_comprehend_entities] (
     [id] INTEGER REFERENCES [pages]([id]),
     [score] FLOAT,
     [entity] INTEGER REFERENCES [comprehend_entities]([id]),
     [begin_offset] INTEGER,
     [end_offset] INTEGER
  );
  CREATE UNIQUE INDEX [idx_comprehend_entity_types_value]
      ON [comprehend_entity_types] ([value]);
  CREATE UNIQUE INDEX [idx_comprehend_entities_type_name]
      ON [comprehend_entities] ([type], [name]);
  CREATE TABLE [pages_comprehend_entities_done] (
     [id] INTEGER PRIMARY KEY REFERENCES [pages]([id])
  );"><pre>CREATE TABLE [pages] (
     [id] <span class="pl-k">INTEGER</span> <span class="pl-k">PRIMARY KEY</span>,
     [<span class="pl-k">text</span>] <span class="pl-k">TEXT</span>
  );
  CREATE TABLE [comprehend_entity_types] (
     [id] <span class="pl-k">INTEGER</span> <span class="pl-k">PRIMARY KEY</span>,
     [value] <span class="pl-k">TEXT</span>
  );
  CREATE TABLE [comprehend_entities] (
     [id] <span class="pl-k">INTEGER</span> <span class="pl-k">PRIMARY KEY</span>,
     [name] <span class="pl-k">TEXT</span>,
     [type] <span class="pl-k">INTEGER</span> <span class="pl-k">REFERENCES</span> [comprehend_entity_types]([id])
  );
  CREATE TABLE [pages_comprehend_entities] (
     [id] <span class="pl-k">INTEGER</span> <span class="pl-k">REFERENCES</span> [pages]([id]),
     [score] FLOAT,
     [entity] <span class="pl-k">INTEGER</span> <span class="pl-k">REFERENCES</span> [comprehend_entities]([id]),
     [begin_offset] <span class="pl-k">INTEGER</span>,
     [end_offset] <span class="pl-k">INTEGER</span>
  );
  CREATE UNIQUE INDEX [idx_comprehend_entity_types_value]
      <span class="pl-k">ON</span> [comprehend_entity_types] ([value]);
  CREATE UNIQUE INDEX [idx_comprehend_entities_type_name]
      <span class="pl-k">ON</span> [comprehend_entities] ([type], [name]);
  CREATE TABLE [pages_comprehend_entities_done] (
     [id] <span class="pl-k">INTEGER</span> <span class="pl-k">PRIMARY KEY</span> <span class="pl-k">REFERENCES</span> [pages]([id])
  );</pre></div>

  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Development</h2><a id="user-content-development" class="anchor" aria-label="Permalink: Development" href="#development"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">To contribute to this tool, first checkout the code. Then create a new virtual environment:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="cd sqlite-comprehend
  python -m venv venv
  source venv/bin/activate"><pre class="notranslate"><code>cd sqlite-comprehend
  python -m venv venv
  source venv/bin/activate
  </code></pre></div>
  <p dir="auto">Now install the dependencies and test dependencies:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="pip install -e '.[test]'"><pre class="notranslate"><code>pip install -e '.[test]'
  </code></pre></div>
  <p dir="auto">To run the tests:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="pytest"><pre class="notranslate"><code>pytest
  </code></pre></div>
  </article></div>


node_id: R_kgDOHwJJsQ
name: sqlite-path
full_name: asg017/sqlite-path
html_url: https://github.com/asg017/sqlite-path
description: A SQLite extension for parsing, generating, and querying paths
created_at: 2022-08-01T19:49:33Z
updated_at: 2025-04-25T17:11:46Z
pushed_at: 2023-10-07T05:57:19Z
language: C
topics: ["sqlite", "sqlite-extension"]
visibility: public
default_branch: main
permissions: {"admin": false, "maintain": false, "push": false, "triage": false, "pull": true}
readme:
  # sqlite-path

  A loadable SQLite extension for parsing, generating, and querying paths. Based on [cwalk](https://github.com/likle/cwalk)

  Try it out in your browser and learn more in [_Introducing sqlite-path: A SQLite extension for parsing and generating file paths_](https://observablehq.com/@asg017/introducing-sqlite-path) (August 2022)

  ## Usage

  ```sql
  .load ./path0
  select path_dirname('foo/bar.txt'); -- 'foo/'
  select path_basename('foo/bar.txt'); -- 'bar.txt'
  select path_extension('foo/bar.txt'); -- '.txt'

  select path_part_at('foo/bar/baz.txt', 0); -- 'foo'
  select path_part_at('foo/bar/baz.txt', 1); -- 'bar'
  select path_part_at('foo/bar/baz.txt', -1); -- 'baz.txt'
  ```

  Iterate through all parts in a path.

  ```sql

  select *
  from path_parts('/usr/bin/sqlite3');

  /*
  ┌────────┬─────────┐
  │  type  │  part   │
  ├────────┼─────────┤
  │ normal │ usr     │
  │ normal │ bin     │
  │ normal │ sqlite3 │
  └────────┴─────────┘
  */
  ```

  Inside a ZIP archive of the [SQLite source code](https://github.com/sqlite/sqlite), find the top 5 deepest `.c` source code files under the `ext/` directory (using SQLite's [ZIP support](https://www.sqlite.org/zipfile.html)).

  ```sql
  select
    name,
    path_length(name) as depth
  from zipfile('sqlite.archive.master.zip')
  where
    -- under the ext/ directory
    path_part_at(name, 1) == 'ext'
    -- ends in ".c"
    and path_extension(name) == '.c'
  order by 2 desc
  limit 5;

  /*
  ┌────────────────────────────────────────────┬───────┐
  │                    name                    │ depth │
  ├────────────────────────────────────────────┼───────┤
  │ sqlite-master/ext/fts3/tool/fts3view.c     │ 5     │
  │ sqlite-master/ext/lsm1/lsm-test/lsmtest1.c │ 5     │
  │ sqlite-master/ext/lsm1/lsm-test/lsmtest2.c │ 5     │
  │ sqlite-master/ext/lsm1/lsm-test/lsmtest3.c │ 5     │
  │ sqlite-master/ext/lsm1/lsm-test/lsmtest4.c │ 5     │
  └────────────────────────────────────────────┴───────┘
  */
  ```

  Make a histogram of the count of file extensions in the current directory, using [`fsdir()`](https://www.sqlite.org/cli.html#file_i_o_functions).

  ```sql

  select
    path_extension(name),
    count(*),
    printf('%.*c', count(*), '*') as bar
  from fsdir('.')
  where path_extension(name) is not null
  group by 1
  order by 2 desc
  limit 6;

  /*
  ┌──────────────────────┬──────────┬────────────────────────────────────┐
  │ path_extension(name) │ count(*) │                bar                 │
  ├──────────────────────┼──────────┼────────────────────────────────────┤
  │ .md                  │ 34       │ ********************************** │
  │ .sample              │ 26       │ **************************         │
  │ .c                   │ 21       │ *********************              │
  │ .css                 │ 5        │ *****                              │
  │ .yml                 │ 4        │ ****                               │
  │ .h                   │ 4        │ ****                               │
  └──────────────────────┴──────────┴────────────────────────────────────┘
  */
  ```

  ## Documentation

  See [`docs.md`](./docs.md) for a full API reference.

  ## Installing

  | Language       | Install                                                    |                                                                                                                                                                                           |
  | -------------- | ---------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
  | Python         | `pip install sqlite-path`                                   | [![PyPI](https://img.shields.io/pypi/v/sqlite-path.svg?color=blue&logo=python&logoColor=white)](https://pypi.org/project/sqlite-path/)                                                      |
  | Datasette      | `datasette install datasette-sqlite-path`                   | [![Datasette](https://img.shields.io/pypi/v/datasette-sqlite-path.svg?color=B6B6D9&label=Datasette+plugin&logoColor=white&logo=python)](https://datasette.io/plugins/datasette-sqlite-path) |
  | Node.js        | `npm install sqlite-path`                                   | [![npm](https://img.shields.io/npm/v/sqlite-path.svg?color=green&logo=nodedotjs&logoColor=white)](https://www.npmjs.com/package/sqlite-path)                                                |
  | Deno           | [`deno.land/x/sqlite_path`](https://deno.land/x/sqlite_path) | [![deno.land/x release](https://img.shields.io/github/v/release/asg017/sqlite-path?color=fef8d2&include_prereleases&label=deno.land%2Fx&logo=deno)](https://deno.land/x/sqlite_path)        |
  | Ruby           | `gem install sqlite-path`                                   | ![Gem](https://img.shields.io/gem/v/sqlite-path?color=red&logo=rubygems&logoColor=white)                                                                                                   |
  | Github Release |                                                            | ![GitHub tag (latest SemVer pre-release)](https://img.shields.io/github/v/tag/asg017/sqlite-path?color=lightgrey&include_prereleases&label=Github+release&logo=github)                     |

  <!--
  | Elixir         | [`hex.pm/packages/sqlite_path`](https://hex.pm/packages/sqlite_path) | [![Hex.pm](https://img.shields.io/hexpm/v/sqlite_path?color=purple&logo=elixir)](https://hex.pm/packages/sqlite_path)                                                                       |
  | Go             | `go get -u github.com/asg017/sqlite-path/bindings/go`               | [![Go Reference](https://pkg.go.dev/badge/github.com/asg017/sqlite-path/bindings/go.svg)](https://pkg.go.dev/github.com/asg017/sqlite-path/bindings/go)                                     |
  | Rust           | `cargo add sqlite-path`                                             | [![Crates.io](https://img.shields.io/crates/v/sqlite-path?logo=rust)](https://crates.io/crates/sqlite-path)                                                                                 |
  -->

  The [Releases page](https://github.com/asg017/sqlite-path/releases) contains pre-built binaries for Linux amd64, MacOS amd64 (no arm), and Windows.

  ### As a loadable extension

  If you want to use `sqlite-path` as a [Runtime-loadable extension](https://www.sqlite.org/loadext.html), Download the `path0.dylib` (for MacOS), `path0.so` (Linux), or `path0.dll` (Windows) file from a release and load it into your SQLite environment.

  > **Note:**
  > The `0` in the filename (`path0.dylib`/ `path0.so`/`path0.dll`) denotes the major version of `sqlite-path`. Currently `sqlite-path` is pre v1, so expect breaking changes in future versions.

  For example, if you are using the [SQLite CLI](https://www.sqlite.org/cli.html), you can load the library like so:

  ```sql
  .load ./path0
  select path_version();
  -- v0.0.1
  ```

  Or in Python, using the builtin [sqlite3 module](https://docs.python.org/3/library/sqlite3.html):

  ```python
  import sqlite3

  con = sqlite3.connect(":memory:")

  con.enable_load_extension(True)
  con.load_extension("./path0")

  print(con.execute("select path_version()").fetchone())
  # ('v0.0.1',)
  ```

  Or in Node.js using [better-sqlite3](https://github.com/WiseLibs/better-sqlite3):

  ```javascript
  const Database = require("better-sqlite3");
  const db = new Database(":memory:");

  db.loadExtension("./path0");

  console.log(db.prepare("select path_version()").get());
  // { 'html_version()': 'v0.0.1' }
  ```

  Or with [Datasette](https://datasette.io/):

  ```
  datasette data.db --load-extension ./path0
  ```

  ## See also

  - [sqlite-url](https://github.com/asg017/sqlite-url), for parsing and generating URLs (pairs well with this library)
  - [sqlite-http](https://github.com/asg017/sqlite-http), for making HTTP requests in SQLite
  - [sqlite-html](https://github.com/asg017/sqlite-html), for parsing HTML documents
  - [sqlite-lines](https://github.com/asg017/sqlite-lines), for reading large files line-by-line
  - [nalgeon/sqlean](https://github.com/nalgeon/sqlean), several pre-compiled handy SQLite functions, in C

readme_html:
  <div id="readme" class="md" data-path="README.md"><article class="markdown-body entry-content container-lg" itemprop="text"><div class="markdown-heading" dir="auto"><h1 class="heading-element" dir="auto">sqlite-path</h1><a id="user-content-sqlite-path" class="anchor" aria-label="Permalink: sqlite-path" href="#sqlite-path"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">A loadable SQLite extension for parsing, generating, and querying paths. Based on <a href="https://github.com/likle/cwalk">cwalk</a></p>
  <p dir="auto">Try it out in your browser and learn more in <a href="https://observablehq.com/@asg017/introducing-sqlite-path" rel="nofollow"><em>Introducing sqlite-path: A SQLite extension for parsing and generating file paths</em></a> (August 2022)</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Usage</h2><a id="user-content-usage" class="anchor" aria-label="Permalink: Usage" href="#usage"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <div class="highlight highlight-source-sql notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content=".load ./path0
  select path_dirname('foo/bar.txt'); -- 'foo/'
  select path_basename('foo/bar.txt'); -- 'bar.txt'
  select path_extension('foo/bar.txt'); -- '.txt'

  select path_part_at('foo/bar/baz.txt', 0); -- 'foo'
  select path_part_at('foo/bar/baz.txt', 1); -- 'bar'
  select path_part_at('foo/bar/baz.txt', -1); -- 'baz.txt'"><pre>.load .<span class="pl-k">/</span>path0
  <span class="pl-k">select</span> path_dirname(<span class="pl-s"><span class="pl-pds">'</span>foo/bar.txt<span class="pl-pds">'</span></span>); <span class="pl-c"><span class="pl-c">--</span> 'foo/'</span>
  <span class="pl-k">select</span> path_basename(<span class="pl-s"><span class="pl-pds">'</span>foo/bar.txt<span class="pl-pds">'</span></span>); <span class="pl-c"><span class="pl-c">--</span> 'bar.txt'</span>
  <span class="pl-k">select</span> path_extension(<span class="pl-s"><span class="pl-pds">'</span>foo/bar.txt<span class="pl-pds">'</span></span>); <span class="pl-c"><span class="pl-c">--</span> '.txt'</span>

  <span class="pl-k">select</span> path_part_at(<span class="pl-s"><span class="pl-pds">'</span>foo/bar/baz.txt<span class="pl-pds">'</span></span>, <span class="pl-c1">0</span>); <span class="pl-c"><span class="pl-c">--</span> 'foo'</span>
  <span class="pl-k">select</span> path_part_at(<span class="pl-s"><span class="pl-pds">'</span>foo/bar/baz.txt<span class="pl-pds">'</span></span>, <span class="pl-c1">1</span>); <span class="pl-c"><span class="pl-c">--</span> 'bar'</span>
  <span class="pl-k">select</span> path_part_at(<span class="pl-s"><span class="pl-pds">'</span>foo/bar/baz.txt<span class="pl-pds">'</span></span>, <span class="pl-k">-</span><span class="pl-c1">1</span>); <span class="pl-c"><span class="pl-c">--</span> 'baz.txt'</span></pre></div>
  <p dir="auto">Iterate through all parts in a path.</p>
  <div class="highlight highlight-source-sql notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="
  select *
  from path_parts('/usr/bin/sqlite3');

  /*
  ┌────────┬─────────┐
  │  type  │  part   │
  ├────────┼─────────┤
  │ normal │ usr     │
  │ normal │ bin     │
  │ normal │ sqlite3 │
  └────────┴─────────┘
  */"><pre><span class="pl-k">select</span> <span class="pl-k">*</span>
  <span class="pl-k">from</span> path_parts(<span class="pl-s"><span class="pl-pds">'</span>/usr/bin/sqlite3<span class="pl-pds">'</span></span>);

  <span class="pl-c"><span class="pl-c">/*</span></span>
  <span class="pl-c">┌────────┬─────────┐</span>
  <span class="pl-c">│  type  │  part   │</span>
  <span class="pl-c">├────────┼─────────┤</span>
  <span class="pl-c">│ normal │ usr     │</span>
  <span class="pl-c">│ normal │ bin     │</span>
  <span class="pl-c">│ normal │ sqlite3 │</span>
  <span class="pl-c">└────────┴─────────┘</span>
  <span class="pl-c"><span class="pl-c">*/</span></span></pre></div>
  <p dir="auto">Inside a ZIP archive of the <a href="https://github.com/sqlite/sqlite">SQLite source code</a>, find the top 5 deepest <code>.c</code> source code files under the <code>ext/</code> directory (using SQLite's <a href="https://www.sqlite.org/zipfile.html" rel="nofollow">ZIP support</a>).</p>
  <div class="highlight highlight-source-sql notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="select
    name,
    path_length(name) as depth
  from zipfile('sqlite.archive.master.zip')
  where
    -- under the ext/ directory
    path_part_at(name, 1) == 'ext'
    -- ends in ".c"
    and path_extension(name) == '.c'
  order by 2 desc
  limit 5;

  /*
  ┌────────────────────────────────────────────┬───────┐
  │                    name                    │ depth │
  ├────────────────────────────────────────────┼───────┤
  │ sqlite-master/ext/fts3/tool/fts3view.c     │ 5     │
  │ sqlite-master/ext/lsm1/lsm-test/lsmtest1.c │ 5     │
  │ sqlite-master/ext/lsm1/lsm-test/lsmtest2.c │ 5     │
  │ sqlite-master/ext/lsm1/lsm-test/lsmtest3.c │ 5     │
  │ sqlite-master/ext/lsm1/lsm-test/lsmtest4.c │ 5     │
  └────────────────────────────────────────────┴───────┘
  */"><pre><span class="pl-k">select</span>
    name,
    path_length(name) <span class="pl-k">as</span> depth
  <span class="pl-k">from</span> zipfile(<span class="pl-s"><span class="pl-pds">'</span>sqlite.archive.master.zip<span class="pl-pds">'</span></span>)
  <span class="pl-k">where</span>
    <span class="pl-c"><span class="pl-c">--</span> under the ext/ directory</span>
    path_part_at(name, <span class="pl-c1">1</span>) <span class="pl-k">==</span> <span class="pl-s"><span class="pl-pds">'</span>ext<span class="pl-pds">'</span></span>
    <span class="pl-c"><span class="pl-c">--</span> ends in ".c"</span>
    <span class="pl-k">and</span> path_extension(name) <span class="pl-k">==</span> <span class="pl-s"><span class="pl-pds">'</span>.c<span class="pl-pds">'</span></span>
  <span class="pl-k">order by</span> <span class="pl-c1">2</span> <span class="pl-k">desc</span>
  <span class="pl-k">limit</span> <span class="pl-c1">5</span>;

  <span class="pl-c"><span class="pl-c">/*</span></span>
  <span class="pl-c">┌────────────────────────────────────────────┬───────┐</span>
  <span class="pl-c">│                    name                    │ depth │</span>
  <span class="pl-c">├────────────────────────────────────────────┼───────┤</span>
  <span class="pl-c">│ sqlite-master/ext/fts3/tool/fts3view.c     │ 5     │</span>
  <span class="pl-c">│ sqlite-master/ext/lsm1/lsm-test/lsmtest1.c │ 5     │</span>
  <span class="pl-c">│ sqlite-master/ext/lsm1/lsm-test/lsmtest2.c │ 5     │</span>
  <span class="pl-c">│ sqlite-master/ext/lsm1/lsm-test/lsmtest3.c │ 5     │</span>
  <span class="pl-c">│ sqlite-master/ext/lsm1/lsm-test/lsmtest4.c │ 5     │</span>
  <span class="pl-c">└────────────────────────────────────────────┴───────┘</span>
  <span class="pl-c"><span class="pl-c">*/</span></span></pre></div>
  <p dir="auto">Make a histogram of the count of file extensions in the current directory, using <a href="https://www.sqlite.org/cli.html#file_i_o_functions" rel="nofollow"><code>fsdir()</code></a>.</p>
  <div class="highlight highlight-source-sql notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="
  select
    path_extension(name),
    count(*),
    printf('%.*c', count(*), '*') as bar
  from fsdir('.')
  where path_extension(name) is not null
  group by 1
  order by 2 desc
  limit 6;

  /*
  ┌──────────────────────┬──────────┬────────────────────────────────────┐
  │ path_extension(name) │ count(*) │                bar                 │
  ├──────────────────────┼──────────┼────────────────────────────────────┤
  │ .md                  │ 34       │ ********************************** │
  │ .sample              │ 26       │ **************************         │
  │ .c                   │ 21       │ *********************              │
  │ .css                 │ 5        │ *****                              │
  │ .yml                 │ 4        │ ****                               │
  │ .h                   │ 4        │ ****                               │
  └──────────────────────┴──────────┴────────────────────────────────────┘
  */"><pre><span class="pl-k">select</span>
    path_extension(name),
    <span class="pl-c1">count</span>(<span class="pl-k">*</span>),
    printf(<span class="pl-s"><span class="pl-pds">'</span>%.*c<span class="pl-pds">'</span></span>, <span class="pl-c1">count</span>(<span class="pl-k">*</span>), <span class="pl-s"><span class="pl-pds">'</span>*<span class="pl-pds">'</span></span>) <span class="pl-k">as</span> bar
  <span class="pl-k">from</span> fsdir(<span class="pl-s"><span class="pl-pds">'</span>.<span class="pl-pds">'</span></span>)
  <span class="pl-k">where</span> path_extension(name) <span class="pl-k">is not null</span>
  <span class="pl-k">group by</span> <span class="pl-c1">1</span>
  <span class="pl-k">order by</span> <span class="pl-c1">2</span> <span class="pl-k">desc</span>
  <span class="pl-k">limit</span> <span class="pl-c1">6</span>;

  <span class="pl-c"><span class="pl-c">/*</span></span>
  <span class="pl-c">┌──────────────────────┬──────────┬────────────────────────────────────┐</span>
  <span class="pl-c">│ path_extension(name) │ count(*) │                bar                 │</span>
  <span class="pl-c">├──────────────────────┼──────────┼────────────────────────────────────┤</span>
  <span class="pl-c">│ .md                  │ 34       │ ********************************** │</span>
  <span class="pl-c">│ .sample              │ 26       │ **************************         │</span>
  <span class="pl-c">│ .c                   │ 21       │ *********************              │</span>
  <span class="pl-c">│ .css                 │ 5        │ *****                              │</span>
  <span class="pl-c">│ .yml                 │ 4        │ ****                               │</span>
  <span class="pl-c">│ .h                   │ 4        │ ****                               │</span>
  <span class="pl-c">└──────────────────────┴──────────┴────────────────────────────────────┘</span>
  <span class="pl-c"><span class="pl-c">*/</span></span></pre></div>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Documentation</h2><a id="user-content-documentation" class="anchor" aria-label="Permalink: Documentation" href="#documentation"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">See <a href="./docs.md"><code>docs.md</code></a> for a full API reference.</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Installing</h2><a id="user-content-installing" class="anchor" aria-label="Permalink: Installing" href="#installing"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <markdown-accessiblity-table><table>
  <thead>
  <tr>
  <th>Language</th>
  <th>Install</th>
  <th></th>
  </tr>
  </thead>
  <tbody>
  <tr>
  <td>Python</td>
  <td><code>pip install sqlite-path</code></td>
  <td><a href="https://pypi.org/project/sqlite-path/" rel="nofollow"><img src="https://img.shields.io/pypi/v/sqlite-path.svg?color=blue&logo=python&logoColor=white" alt="PyPI" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/598acdaadb0d523312f045308abfb19c733a30afcda88f86a4feb1a61aaee277/68747470733a2f2f696d672e736869656c64732e696f2f707970692f762f73716c6974652d706174682e7376673f636f6c6f723d626c7565266c6f676f3d707974686f6e266c6f676f436f6c6f723d7768697465"></a></td>
  </tr>
  <tr>
  <td>Datasette</td>
  <td><code>datasette install datasette-sqlite-path</code></td>
  <td><a href="https://datasette.io/plugins/datasette-sqlite-path" rel="nofollow"><img src="https://img.shields.io/pypi/v/datasette-sqlite-path.svg?color=B6B6D9&label=Datasette+plugin&logoColor=white&logo=python" alt="Datasette" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/86597069807ce52c39659ef3d1608b82e725fd9dfab6f7c6dcab99ebd64ab7d4/68747470733a2f2f696d672e736869656c64732e696f2f707970692f762f6461746173657474652d73716c6974652d706174682e7376673f636f6c6f723d423642364439266c6162656c3d4461746173657474652b706c7567696e266c6f676f436f6c6f723d7768697465266c6f676f3d707974686f6e"></a></td>
  </tr>
  <tr>
  <td>Node.js</td>
  <td><code>npm install sqlite-path</code></td>
  <td><a href="https://www.npmjs.com/package/sqlite-path" rel="nofollow"><img src="https://img.shields.io/npm/v/sqlite-path.svg?color=green&logo=nodedotjs&logoColor=white" alt="npm" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/0b07aa339dd15653ab10e7338e23e148ad3fd789e04df040ac0a3b1c39786459/68747470733a2f2f696d672e736869656c64732e696f2f6e706d2f762f73716c6974652d706174682e7376673f636f6c6f723d677265656e266c6f676f3d6e6f6465646f746a73266c6f676f436f6c6f723d7768697465"></a></td>
  </tr>
  <tr>
  <td>Deno</td>
  <td><a href="https://deno.land/x/sqlite_path" rel="nofollow"><code>deno.land/x/sqlite_path</code></a></td>
  <td><a href="https://deno.land/x/sqlite_path" rel="nofollow"><img src="https://img.shields.io/github/v/release/asg017/sqlite-path?color=fef8d2&include_prereleases&label=deno.land%2Fx&logo=deno" alt="deno.land/x release" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/f29de819f7e598daa7d3e25f368c0e7aa0e8d5238f8baba64580ec50891b94f1/68747470733a2f2f696d672e736869656c64732e696f2f6769746875622f762f72656c656173652f6173673031372f73716c6974652d706174683f636f6c6f723d66656638643226696e636c7564655f70726572656c6561736573266c6162656c3d64656e6f2e6c616e6425324678266c6f676f3d64656e6f"></a></td>
  </tr>
  <tr>
  <td>Ruby</td>
  <td><code>gem install sqlite-path</code></td>
  <td><a target="_blank" rel="noopener noreferrer nofollow" href="https://camo.githubusercontent.com/5408da59a81f898fce3ae078a41c28e42620afc229088397a25ef128273ef244/68747470733a2f2f696d672e736869656c64732e696f2f67656d2f762f73716c6974652d706174683f636f6c6f723d726564266c6f676f3d7275627967656d73266c6f676f436f6c6f723d7768697465"><img src="https://img.shields.io/gem/v/sqlite-path?color=red&logo=rubygems&logoColor=white" alt="Gem" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/5408da59a81f898fce3ae078a41c28e42620afc229088397a25ef128273ef244/68747470733a2f2f696d672e736869656c64732e696f2f67656d2f762f73716c6974652d706174683f636f6c6f723d726564266c6f676f3d7275627967656d73266c6f676f436f6c6f723d7768697465"></a></td>
  </tr>
  <tr>
  <td>Github Release</td>
  <td></td>
  <td><a target="_blank" rel="noopener noreferrer nofollow" href="https://camo.githubusercontent.com/9d29374088cbc979bbe2e87663edfe0a9e8b3db3269fee1102c4b5ac5c6645dc/68747470733a2f2f696d672e736869656c64732e696f2f6769746875622f762f7461672f6173673031372f73716c6974652d706174683f636f6c6f723d6c696768746772657926696e636c7564655f70726572656c6561736573266c6162656c3d4769746875622b72656c65617365266c6f676f3d676974687562"><img src="https://img.shields.io/github/v/tag/asg017/sqlite-path?color=lightgrey&include_prereleases&label=Github+release&logo=github" alt="GitHub tag (latest SemVer pre-release)" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/9d29374088cbc979bbe2e87663edfe0a9e8b3db3269fee1102c4b5ac5c6645dc/68747470733a2f2f696d672e736869656c64732e696f2f6769746875622f762f7461672f6173673031372f73716c6974652d706174683f636f6c6f723d6c696768746772657926696e636c7564655f70726572656c6561736573266c6162656c3d4769746875622b72656c65617365266c6f676f3d676974687562"></a></td>
  </tr>
  </tbody>
  </table></markdown-accessiblity-table>

  <p dir="auto">The <a href="https://github.com/asg017/sqlite-path/releases">Releases page</a> contains pre-built binaries for Linux amd64, MacOS amd64 (no arm), and Windows.</p>
  <div class="markdown-heading" dir="auto"><h3 class="heading-element" dir="auto">As a loadable extension</h3><a id="user-content-as-a-loadable-extension" class="anchor" aria-label="Permalink: As a loadable extension" href="#as-a-loadable-extension"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">If you want to use <code>sqlite-path</code> as a <a href="https://www.sqlite.org/loadext.html" rel="nofollow">Runtime-loadable extension</a>, Download the <code>path0.dylib</code> (for MacOS), <code>path0.so</code> (Linux), or <code>path0.dll</code> (Windows) file from a release and load it into your SQLite environment.</p>
  <blockquote>
  <p dir="auto"><strong>Note:</strong>
  The <code>0</code> in the filename (<code>path0.dylib</code>/ <code>path0.so</code>/<code>path0.dll</code>) denotes the major version of <code>sqlite-path</code>. Currently <code>sqlite-path</code> is pre v1, so expect breaking changes in future versions.</p>
  </blockquote>
  <p dir="auto">For example, if you are using the <a href="https://www.sqlite.org/cli.html" rel="nofollow">SQLite CLI</a>, you can load the library like so:</p>
  <div class="highlight highlight-source-sql notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content=".load ./path0
  select path_version();
  -- v0.0.1"><pre>.load .<span class="pl-k">/</span>path0
  <span class="pl-k">select</span> path_version();
  <span class="pl-c"><span class="pl-c">--</span> v0.0.1</span></pre></div>
  <p dir="auto">Or in Python, using the builtin <a href="https://docs.python.org/3/library/sqlite3.html" rel="nofollow">sqlite3 module</a>:</p>
  <div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="import sqlite3

  con = sqlite3.connect(":memory:")

  con.enable_load_extension(True)
  con.load_extension("./path0")

  print(con.execute("select path_version()").fetchone())
  # ('v0.0.1',)"><pre><span class="pl-k">import</span> <span class="pl-s1">sqlite3</span>

  <span class="pl-s1">con</span> <span class="pl-c1">=</span> <span class="pl-s1">sqlite3</span>.<span class="pl-c1">connect</span>(<span class="pl-s">":memory:"</span>)

  <span class="pl-s1">con</span>.<span class="pl-c1">enable_load_extension</span>(<span class="pl-c1">True</span>)
  <span class="pl-s1">con</span>.<span class="pl-c1">load_extension</span>(<span class="pl-s">"./path0"</span>)

  <span class="pl-en">print</span>(<span class="pl-s1">con</span>.<span class="pl-c1">execute</span>(<span class="pl-s">"select path_version()"</span>).<span class="pl-c1">fetchone</span>())
  <span class="pl-c"># ('v0.0.1',)</span></pre></div>
  <p dir="auto">Or in Node.js using <a href="https://github.com/WiseLibs/better-sqlite3">better-sqlite3</a>:</p>
  <div class="highlight highlight-source-js notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="const Database = require("better-sqlite3");
  const db = new Database(":memory:");

  db.loadExtension("./path0");

  console.log(db.prepare("select path_version()").get());
  // { 'html_version()': 'v0.0.1' }"><pre><span class="pl-k">const</span> <span class="pl-v">Database</span> <span class="pl-c1">=</span> <span class="pl-en">require</span><span class="pl-kos">(</span><span class="pl-s">"better-sqlite3"</span><span class="pl-kos">)</span><span class="pl-kos">;</span>
  <span class="pl-k">const</span> <span class="pl-s1">db</span> <span class="pl-c1">=</span> <span class="pl-k">new</span> <span class="pl-v">Database</span><span class="pl-kos">(</span><span class="pl-s">":memory:"</span><span class="pl-kos">)</span><span class="pl-kos">;</span>

  <span class="pl-s1">db</span><span class="pl-kos">.</span><span class="pl-en">loadExtension</span><span class="pl-kos">(</span><span class="pl-s">"./path0"</span><span class="pl-kos">)</span><span class="pl-kos">;</span>

  <span class="pl-smi">console</span><span class="pl-kos">.</span><span class="pl-en">log</span><span class="pl-kos">(</span><span class="pl-s1">db</span><span class="pl-kos">.</span><span class="pl-en">prepare</span><span class="pl-kos">(</span><span class="pl-s">"select path_version()"</span><span class="pl-kos">)</span><span class="pl-kos">.</span><span class="pl-en">get</span><span class="pl-kos">(</span><span class="pl-kos">)</span><span class="pl-kos">)</span><span class="pl-kos">;</span>
  <span class="pl-c">// { 'html_version()': 'v0.0.1' }</span></pre></div>
  <p dir="auto">Or with <a href="https://datasette.io/" rel="nofollow">Datasette</a>:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="datasette data.db --load-extension ./path0"><pre class="notranslate"><code>datasette data.db --load-extension ./path0
  </code></pre></div>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">See also</h2><a id="user-content-see-also" class="anchor" aria-label="Permalink: See also" href="#see-also"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <ul dir="auto">
  <li><a href="https://github.com/asg017/sqlite-url">sqlite-url</a>, for parsing and generating URLs (pairs well with this library)</li>
  <li><a href="https://github.com/asg017/sqlite-http">sqlite-http</a>, for making HTTP requests in SQLite</li>
  <li><a href="https://github.com/asg017/sqlite-html">sqlite-html</a>, for parsing HTML documents</li>
  <li><a href="https://github.com/asg017/sqlite-lines">sqlite-lines</a>, for reading large files line-by-line</li>
  <li><a href="https://github.com/nalgeon/sqlean">nalgeon/sqlean</a>, several pre-compiled handy SQLite functions, in C</li>
  </ul>
  </article></div>


node_id: R_kgDOHwJOSQ
name: sqlite-url
full_name: asg017/sqlite-url
html_url: https://github.com/asg017/sqlite-url
description: A SQLite extension for parsing, generating, and querying URLs and query strings
created_at: 2022-08-01T19:54:03Z
updated_at: 2025-05-11T08:50:30Z
pushed_at: 2025-02-08T22:39:54Z
language: C
topics: ["sqlite", "sqlite-extension"]
visibility: public
default_branch: main
permissions: {"admin": false, "maintain": false, "push": false, "triage": false, "pull": true}
readme:
  # sqlite-url

  A SQLite extension for parsing and generating URLs and query strings. Based on libcurl's [URL API](https://curl.se/libcurl/c/libcurl-url.html)

  Try it out in your browser and learn more in [Introducing sqlite-url: A SQLite extension for parsing and generating URLs](https://observablehq.com/@asg017/introducing-sqlite-url) (September 2022)

  ## Usage

  ```sql
  .load ./url0
  select url_valid('https://sqlite.org'); -- 1
  ```

  Extract specific parts from a given URL.

  ```sql
  select url_scheme('https://www.sqlite.org/vtab.html#usage'); -- 'https'
  select url_host('https://www.sqlite.org/vtab.html#usage'); -- 'www.sqlite.org'
  select url_path('https://www.sqlite.org/vtab.html#usage'); -- '/vtab.html'
  select url_fragment('https://www.sqlite.org/vtab.html#usage'); -- 'usage'
  ```

  Generate a URL programmatically.

  ```sql
  select url(null,
    'scheme', 'https',
    'host', 'alexgarcia.url',
    'fragment', 'yeet'
  ); -- 'https://alexgarcia.url/#yeet'
  ```

  Iterate through all parameters in a URL's query string.

  ```sql

  select *
  from url_query_each(
    url_query('https://api.census.gov/data/2020/acs/acs5?get=B01001_001E&for=county:*&in=state:06')
  );
  /*
  ┌──────┬─────────────┐
  │ name │    value    │
  ├──────┼─────────────┤
  │ get  │ B01001_001E │
  │ for  │ county:*    │
  │ in   │ state:06    │
  └──────┴─────────────┘
  */
  ```

  Use with [sqlite-http](https://github.com/asg017/sqlite-http) to generate URLs to request.

  ```sql
  select http_get_body(
    url(
      'https://api.census.gov',
      'path', '/data/2020/acs/acs5',
      'query', url_querystring(
        'get', 'B01001_001E',
        'for', 'county:*',
        'in', 'state:06'
      )
    )
  );
  /*
  ┌────────────────────────────────────┐
  │           http_get_body(           │
  ├────────────────────────────────────┤
  │ [["B01001_001E","state","county"], │
  │ ["1661584","06","001"],            │
  │ ["1159","06","003"],               │
  │ ["223344","06","007"],             │
  │ ["21491","06","011"],              │
  │ ["1147788","06","013"],            │
  │ ["190345","06","017"],             │
  │               ...                  │
  │ ["845599","06","111"]]             │
  └────────────────────────────────────┘
  */
  ```

  Use with [sqlite-path](https://github.com/asg017/sqlite-path) to safely generate paths for a URL.

  ```sql

  select url(
    'https://github.com',
    'path', path_join('/', 'asg017', 'sqlite-url', 'issues', '1')
  );
  -- 'https://github.com/asg017/sqlite-url/issues/1'
  ```

  ## Documentation

  See [`docs.md`](./docs.md) for a full API reference.

  ## Installing

  | Language       | Install                                                    |                                                                                                                                                                                           |
  | -------------- | ---------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
  | Python         | `pip install sqlite-url`                                   | [![PyPI](https://img.shields.io/pypi/v/sqlite-url.svg?color=blue&logo=python&logoColor=white)](https://pypi.org/project/sqlite-url/)                                                      |
  | Datasette      | `datasette install datasette-sqlite-url`                   | [![Datasette](https://img.shields.io/pypi/v/datasette-sqlite-url.svg?color=B6B6D9&label=Datasette+plugin&logoColor=white&logo=python)](https://datasette.io/plugins/datasette-sqlite-url) |
  | Node.js        | `npm install sqlite-url`                                   | [![npm](https://img.shields.io/npm/v/sqlite-url.svg?color=green&logo=nodedotjs&logoColor=white)](https://www.npmjs.com/package/sqlite-url)                                                |
  | Deno           | [`deno.land/x/sqlite_url`](https://deno.land/x/sqlite_url) | [![deno.land/x release](https://img.shields.io/github/v/release/asg017/sqlite-url?color=fef8d2&include_prereleases&label=deno.land%2Fx&logo=deno)](https://deno.land/x/sqlite_url)        |
  | Ruby           | `gem install sqlite-url`                                   | ![Gem](https://img.shields.io/gem/v/sqlite-url?color=red&logo=rubygems&logoColor=white)                                                                                                   |
  | Github Release |                                                            | ![GitHub tag (latest SemVer pre-release)](https://img.shields.io/github/v/tag/asg017/sqlite-url?color=lightgrey&include_prereleases&label=Github+release&logo=github)                     |

  <!--
  | Elixir         | [`hex.pm/packages/sqlite_url`](https://hex.pm/packages/sqlite_url) | [![Hex.pm](https://img.shields.io/hexpm/v/sqlite_url?color=purple&logo=elixir)](https://hex.pm/packages/sqlite_url)                                                                       |
  | Go             | `go get -u github.com/asg017/sqlite-url/bindings/go`               | [![Go Reference](https://pkg.go.dev/badge/github.com/asg017/sqlite-url/bindings/go.svg)](https://pkg.go.dev/github.com/asg017/sqlite-url/bindings/go)                                     |
  | Rust           | `cargo add sqlite-url`                                             | [![Crates.io](https://img.shields.io/crates/v/sqlite-url?logo=rust)](https://crates.io/crates/sqlite-url)                                                                                 |
  -->

  The [Releases page](https://github.com/asg017/sqlite-url/releases) contains pre-built binaries for Linux amd6 and MacOS amd64 (no arm).

  ### As a loadable extension

  If you want to use `sqlite-url` as a [Runtime-loadable extension](https://www.sqlite.org/loadext.html), Download the `url0.dylib` (for MacOS) or `url0.so` (Linux) file from a release and load it into your SQLite environment.

  > **Note:**
  > The `0` in the filename (`url0.dylib`/ `url0.so`) denotes the major version of `sqlite-url`. Currently `sqlite-url` is pre v1, so expect breaking changes in future versions.

  For example, if you are using the [SQLite CLI](https://www.sqlite.org/cli.html), you can load the library like so:

  ```sql
  .load ./url0
  select url_version();
  -- v0.0.1
  ```

  Or in Python, using the builtin [sqlite3 module](https://docs.python.org/3/library/sqlite3.html):

  ```python
  import sqlite3

  con = sqlite3.connect(":memory:")

  con.enable_load_extension(True)
  con.load_extension("./url0")

  print(con.execute("select url_version()").fetchone())
  # ('v0.0.1',)
  ```

  Or in Node.js using [better-sqlite3](https://github.com/WiseLibs/better-sqlite3):

  ```javascript
  const Database = require("better-sqlite3");
  const db = new Database(":memory:");

  db.loadExtension("./url0");

  console.log(db.prepare("select url_version()").get());
  // { 'html_version()': 'v0.0.1' }
  ```

  Or with [Datasette](https://datasette.io/):

  ```
  datasette data.db --load-extension ./url0
  ```

  ## See also

  - [sqlite-path](https://github.com/asg017/sqlite-path), parsing/generating paths (pairs well with `url_path()` and `url()`)
  - [sqlite-http](https://github.com/asg017/sqlite-http), for making HTTP requests in SQLite
  - [sqlite-html](https://github.com/asg017/sqlite-html), for parsing HTML documents
  - [sqlite-lines](https://github.com/asg017/sqlite-lines), for reading large files line-by-line
  - [nalgeon/sqlean](https://github.com/nalgeon/sqlean), several pre-compiled handy SQLite functions, in C

readme_html:
  <div id="readme" class="md" data-path="README.md"><article class="markdown-body entry-content container-lg" itemprop="text"><div class="markdown-heading" dir="auto"><h1 class="heading-element" dir="auto">sqlite-url</h1><a id="user-content-sqlite-url" class="anchor" aria-label="Permalink: sqlite-url" href="#sqlite-url"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">A SQLite extension for parsing and generating URLs and query strings. Based on libcurl's <a href="https://curl.se/libcurl/c/libcurl-url.html" rel="nofollow">URL API</a></p>
  <p dir="auto">Try it out in your browser and learn more in <a href="https://observablehq.com/@asg017/introducing-sqlite-url" rel="nofollow">Introducing sqlite-url: A SQLite extension for parsing and generating URLs</a> (September 2022)</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Usage</h2><a id="user-content-usage" class="anchor" aria-label="Permalink: Usage" href="#usage"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <div class="highlight highlight-source-sql notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content=".load ./url0
  select url_valid('https://sqlite.org'); -- 1"><pre>.load .<span class="pl-k">/</span>url0
  <span class="pl-k">select</span> url_valid(<span class="pl-s"><span class="pl-pds">'</span>https://sqlite.org<span class="pl-pds">'</span></span>); <span class="pl-c"><span class="pl-c">--</span> 1</span></pre></div>
  <p dir="auto">Extract specific parts from a given URL.</p>
  <div class="highlight highlight-source-sql notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="select url_scheme('https://www.sqlite.org/vtab.html#usage'); -- 'https'
  select url_host('https://www.sqlite.org/vtab.html#usage'); -- 'www.sqlite.org'
  select url_path('https://www.sqlite.org/vtab.html#usage'); -- '/vtab.html'
  select url_fragment('https://www.sqlite.org/vtab.html#usage'); -- 'usage'"><pre><span class="pl-k">select</span> url_scheme(<span class="pl-s"><span class="pl-pds">'</span>https://www.sqlite.org/vtab.html#usage<span class="pl-pds">'</span></span>); <span class="pl-c"><span class="pl-c">--</span> 'https'</span>
  <span class="pl-k">select</span> url_host(<span class="pl-s"><span class="pl-pds">'</span>https://www.sqlite.org/vtab.html#usage<span class="pl-pds">'</span></span>); <span class="pl-c"><span class="pl-c">--</span> 'www.sqlite.org'</span>
  <span class="pl-k">select</span> url_path(<span class="pl-s"><span class="pl-pds">'</span>https://www.sqlite.org/vtab.html#usage<span class="pl-pds">'</span></span>); <span class="pl-c"><span class="pl-c">--</span> '/vtab.html'</span>
  <span class="pl-k">select</span> url_fragment(<span class="pl-s"><span class="pl-pds">'</span>https://www.sqlite.org/vtab.html#usage<span class="pl-pds">'</span></span>); <span class="pl-c"><span class="pl-c">--</span> 'usage'</span></pre></div>
  <p dir="auto">Generate a URL programmatically.</p>
  <div class="highlight highlight-source-sql notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="select url(null,
    'scheme', 'https',
    'host', 'alexgarcia.url',
    'fragment', 'yeet'
  ); -- 'https://alexgarcia.url/#yeet'"><pre><span class="pl-k">select</span> url(<span class="pl-k">null</span>,
    <span class="pl-s"><span class="pl-pds">'</span>scheme<span class="pl-pds">'</span></span>, <span class="pl-s"><span class="pl-pds">'</span>https<span class="pl-pds">'</span></span>,
    <span class="pl-s"><span class="pl-pds">'</span>host<span class="pl-pds">'</span></span>, <span class="pl-s"><span class="pl-pds">'</span>alexgarcia.url<span class="pl-pds">'</span></span>,
    <span class="pl-s"><span class="pl-pds">'</span>fragment<span class="pl-pds">'</span></span>, <span class="pl-s"><span class="pl-pds">'</span>yeet<span class="pl-pds">'</span></span>
  ); <span class="pl-c"><span class="pl-c">--</span> 'https://alexgarcia.url/#yeet'</span></pre></div>
  <p dir="auto">Iterate through all parameters in a URL's query string.</p>
  <div class="highlight highlight-source-sql notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="
  select *
  from url_query_each(
    url_query('https://api.census.gov/data/2020/acs/acs5?get=B01001_001E&for=county:*&in=state:06')
  );
  /*
  ┌──────┬─────────────┐
  │ name │    value    │
  ├──────┼─────────────┤
  │ get  │ B01001_001E │
  │ for  │ county:*    │
  │ in   │ state:06    │
  └──────┴─────────────┘
  */"><pre><span class="pl-k">select</span> <span class="pl-k">*</span>
  <span class="pl-k">from</span> url_query_each(
    url_query(<span class="pl-s"><span class="pl-pds">'</span>https://api.census.gov/data/2020/acs/acs5?get=B01001_001E&for=county:*&in=state:06<span class="pl-pds">'</span></span>)
  );
  <span class="pl-c"><span class="pl-c">/*</span></span>
  <span class="pl-c">┌──────┬─────────────┐</span>
  <span class="pl-c">│ name │    value    │</span>
  <span class="pl-c">├──────┼─────────────┤</span>
  <span class="pl-c">│ get  │ B01001_001E │</span>
  <span class="pl-c">│ for  │ county:*    │</span>
  <span class="pl-c">│ in   │ state:06    │</span>
  <span class="pl-c">└──────┴─────────────┘</span>
  <span class="pl-c"><span class="pl-c">*/</span></span></pre></div>
  <p dir="auto">Use with <a href="https://github.com/asg017/sqlite-http">sqlite-http</a> to generate URLs to request.</p>
  <div class="highlight highlight-source-sql notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="select http_get_body(
    url(
      'https://api.census.gov',
      'path', '/data/2020/acs/acs5',
      'query', url_querystring(
        'get', 'B01001_001E',
        'for', 'county:*',
        'in', 'state:06'
      )
    )
  );
  /*
  ┌────────────────────────────────────┐
  │           http_get_body(           │
  ├────────────────────────────────────┤
  │ [["B01001_001E","state","county"], │
  │ ["1661584","06","001"],            │
  │ ["1159","06","003"],               │
  │ ["223344","06","007"],             │
  │ ["21491","06","011"],              │
  │ ["1147788","06","013"],            │
  │ ["190345","06","017"],             │
  │               ...                  │
  │ ["845599","06","111"]]             │
  └────────────────────────────────────┘
  */"><pre><span class="pl-k">select</span> http_get_body(
    url(
      <span class="pl-s"><span class="pl-pds">'</span>https://api.census.gov<span class="pl-pds">'</span></span>,
      <span class="pl-s"><span class="pl-pds">'</span>path<span class="pl-pds">'</span></span>, <span class="pl-s"><span class="pl-pds">'</span>/data/2020/acs/acs5<span class="pl-pds">'</span></span>,
      <span class="pl-s"><span class="pl-pds">'</span>query<span class="pl-pds">'</span></span>, url_querystring(
        <span class="pl-s"><span class="pl-pds">'</span>get<span class="pl-pds">'</span></span>, <span class="pl-s"><span class="pl-pds">'</span>B01001_001E<span class="pl-pds">'</span></span>,
        <span class="pl-s"><span class="pl-pds">'</span>for<span class="pl-pds">'</span></span>, <span class="pl-s"><span class="pl-pds">'</span>county:*<span class="pl-pds">'</span></span>,
        <span class="pl-s"><span class="pl-pds">'</span>in<span class="pl-pds">'</span></span>, <span class="pl-s"><span class="pl-pds">'</span>state:06<span class="pl-pds">'</span></span>
      )
    )
  );
  <span class="pl-c"><span class="pl-c">/*</span></span>
  <span class="pl-c">┌────────────────────────────────────┐</span>
  <span class="pl-c">│           http_get_body(           │</span>
  <span class="pl-c">├────────────────────────────────────┤</span>
  <span class="pl-c">│ [["B01001_001E","state","county"], │</span>
  <span class="pl-c">│ ["1661584","06","001"],            │</span>
  <span class="pl-c">│ ["1159","06","003"],               │</span>
  <span class="pl-c">│ ["223344","06","007"],             │</span>
  <span class="pl-c">│ ["21491","06","011"],              │</span>
  <span class="pl-c">│ ["1147788","06","013"],            │</span>
  <span class="pl-c">│ ["190345","06","017"],             │</span>
  <span class="pl-c">│               ...                  │</span>
  <span class="pl-c">│ ["845599","06","111"]]             │</span>
  <span class="pl-c">└────────────────────────────────────┘</span>
  <span class="pl-c"><span class="pl-c">*/</span></span></pre></div>
  <p dir="auto">Use with <a href="https://github.com/asg017/sqlite-path">sqlite-path</a> to safely generate paths for a URL.</p>
  <div class="highlight highlight-source-sql notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="
  select url(
    'https://github.com',
    'path', path_join('/', 'asg017', 'sqlite-url', 'issues', '1')
  );
  -- 'https://github.com/asg017/sqlite-url/issues/1'"><pre><span class="pl-k">select</span> url(
    <span class="pl-s"><span class="pl-pds">'</span>https://github.com<span class="pl-pds">'</span></span>,
    <span class="pl-s"><span class="pl-pds">'</span>path<span class="pl-pds">'</span></span>, path_join(<span class="pl-s"><span class="pl-pds">'</span>/<span class="pl-pds">'</span></span>, <span class="pl-s"><span class="pl-pds">'</span>asg017<span class="pl-pds">'</span></span>, <span class="pl-s"><span class="pl-pds">'</span>sqlite-url<span class="pl-pds">'</span></span>, <span class="pl-s"><span class="pl-pds">'</span>issues<span class="pl-pds">'</span></span>, <span class="pl-s"><span class="pl-pds">'</span>1<span class="pl-pds">'</span></span>)
  );
  <span class="pl-c"><span class="pl-c">--</span> 'https://github.com/asg017/sqlite-url/issues/1'</span></pre></div>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Documentation</h2><a id="user-content-documentation" class="anchor" aria-label="Permalink: Documentation" href="#documentation"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">See <a href="./docs.md"><code>docs.md</code></a> for a full API reference.</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Installing</h2><a id="user-content-installing" class="anchor" aria-label="Permalink: Installing" href="#installing"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <markdown-accessiblity-table><table>
  <thead>
  <tr>
  <th>Language</th>
  <th>Install</th>
  <th></th>
  </tr>
  </thead>
  <tbody>
  <tr>
  <td>Python</td>
  <td><code>pip install sqlite-url</code></td>
  <td><a href="https://pypi.org/project/sqlite-url/" rel="nofollow"><img src="https://img.shields.io/pypi/v/sqlite-url.svg?color=blue&logo=python&logoColor=white" alt="PyPI" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/4e72d73c8d461c5eee3d97daac39404c874396309a4a9aae401361470e2beaf7/68747470733a2f2f696d672e736869656c64732e696f2f707970692f762f73716c6974652d75726c2e7376673f636f6c6f723d626c7565266c6f676f3d707974686f6e266c6f676f436f6c6f723d7768697465"></a></td>
  </tr>
  <tr>
  <td>Datasette</td>
  <td><code>datasette install datasette-sqlite-url</code></td>
  <td><a href="https://datasette.io/plugins/datasette-sqlite-url" rel="nofollow"><img src="https://img.shields.io/pypi/v/datasette-sqlite-url.svg?color=B6B6D9&label=Datasette+plugin&logoColor=white&logo=python" alt="Datasette" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/c17b58829bed4bf46717cf7225da127bc868f841ba227c581cc114ab9ce41d32/68747470733a2f2f696d672e736869656c64732e696f2f707970692f762f6461746173657474652d73716c6974652d75726c2e7376673f636f6c6f723d423642364439266c6162656c3d4461746173657474652b706c7567696e266c6f676f436f6c6f723d7768697465266c6f676f3d707974686f6e"></a></td>
  </tr>
  <tr>
  <td>Node.js</td>
  <td><code>npm install sqlite-url</code></td>
  <td><a href="https://www.npmjs.com/package/sqlite-url" rel="nofollow"><img src="https://img.shields.io/npm/v/sqlite-url.svg?color=green&logo=nodedotjs&logoColor=white" alt="npm" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/c1808b6072a5291f3a8080fcc08d45e3b7cf9827b40ab4b78a24d36a258fc01a/68747470733a2f2f696d672e736869656c64732e696f2f6e706d2f762f73716c6974652d75726c2e7376673f636f6c6f723d677265656e266c6f676f3d6e6f6465646f746a73266c6f676f436f6c6f723d7768697465"></a></td>
  </tr>
  <tr>
  <td>Deno</td>
  <td><a href="https://deno.land/x/sqlite_url" rel="nofollow"><code>deno.land/x/sqlite_url</code></a></td>
  <td><a href="https://deno.land/x/sqlite_url" rel="nofollow"><img src="https://img.shields.io/github/v/release/asg017/sqlite-url?color=fef8d2&include_prereleases&label=deno.land%2Fx&logo=deno" alt="deno.land/x release" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/7f88391fd035393a21e17501691f8a45581a36f2261ffa38508658807b4760ef/68747470733a2f2f696d672e736869656c64732e696f2f6769746875622f762f72656c656173652f6173673031372f73716c6974652d75726c3f636f6c6f723d66656638643226696e636c7564655f70726572656c6561736573266c6162656c3d64656e6f2e6c616e6425324678266c6f676f3d64656e6f"></a></td>
  </tr>
  <tr>
  <td>Ruby</td>
  <td><code>gem install sqlite-url</code></td>
  <td><a target="_blank" rel="noopener noreferrer nofollow" href="https://camo.githubusercontent.com/a17d3b249daf5110f258ff6418f9108c60bb6f4b06037d6a045c5abf107222b5/68747470733a2f2f696d672e736869656c64732e696f2f67656d2f762f73716c6974652d75726c3f636f6c6f723d726564266c6f676f3d7275627967656d73266c6f676f436f6c6f723d7768697465"><img src="https://img.shields.io/gem/v/sqlite-url?color=red&logo=rubygems&logoColor=white" alt="Gem" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/a17d3b249daf5110f258ff6418f9108c60bb6f4b06037d6a045c5abf107222b5/68747470733a2f2f696d672e736869656c64732e696f2f67656d2f762f73716c6974652d75726c3f636f6c6f723d726564266c6f676f3d7275627967656d73266c6f676f436f6c6f723d7768697465"></a></td>
  </tr>
  <tr>
  <td>Github Release</td>
  <td></td>
  <td><a target="_blank" rel="noopener noreferrer nofollow" href="https://camo.githubusercontent.com/4c74e627a021ec5048625a2638b23350ce05f24c377e21d787bfce25f420a166/68747470733a2f2f696d672e736869656c64732e696f2f6769746875622f762f7461672f6173673031372f73716c6974652d75726c3f636f6c6f723d6c696768746772657926696e636c7564655f70726572656c6561736573266c6162656c3d4769746875622b72656c65617365266c6f676f3d676974687562"><img src="https://img.shields.io/github/v/tag/asg017/sqlite-url?color=lightgrey&include_prereleases&label=Github+release&logo=github" alt="GitHub tag (latest SemVer pre-release)" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/4c74e627a021ec5048625a2638b23350ce05f24c377e21d787bfce25f420a166/68747470733a2f2f696d672e736869656c64732e696f2f6769746875622f762f7461672f6173673031372f73716c6974652d75726c3f636f6c6f723d6c696768746772657926696e636c7564655f70726572656c6561736573266c6162656c3d4769746875622b72656c65617365266c6f676f3d676974687562"></a></td>
  </tr>
  </tbody>
  </table></markdown-accessiblity-table>

  <p dir="auto">The <a href="https://github.com/asg017/sqlite-url/releases">Releases page</a> contains pre-built binaries for Linux amd6 and MacOS amd64 (no arm).</p>
  <div class="markdown-heading" dir="auto"><h3 class="heading-element" dir="auto">As a loadable extension</h3><a id="user-content-as-a-loadable-extension" class="anchor" aria-label="Permalink: As a loadable extension" href="#as-a-loadable-extension"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">If you want to use <code>sqlite-url</code> as a <a href="https://www.sqlite.org/loadext.html" rel="nofollow">Runtime-loadable extension</a>, Download the <code>url0.dylib</code> (for MacOS) or <code>url0.so</code> (Linux) file from a release and load it into your SQLite environment.</p>
  <blockquote>
  <p dir="auto"><strong>Note:</strong>
  The <code>0</code> in the filename (<code>url0.dylib</code>/ <code>url0.so</code>) denotes the major version of <code>sqlite-url</code>. Currently <code>sqlite-url</code> is pre v1, so expect breaking changes in future versions.</p>
  </blockquote>
  <p dir="auto">For example, if you are using the <a href="https://www.sqlite.org/cli.html" rel="nofollow">SQLite CLI</a>, you can load the library like so:</p>
  <div class="highlight highlight-source-sql notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content=".load ./url0
  select url_version();
  -- v0.0.1"><pre>.load .<span class="pl-k">/</span>url0
  <span class="pl-k">select</span> url_version();
  <span class="pl-c"><span class="pl-c">--</span> v0.0.1</span></pre></div>
  <p dir="auto">Or in Python, using the builtin <a href="https://docs.python.org/3/library/sqlite3.html" rel="nofollow">sqlite3 module</a>:</p>
  <div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="import sqlite3

  con = sqlite3.connect(":memory:")

  con.enable_load_extension(True)
  con.load_extension("./url0")

  print(con.execute("select url_version()").fetchone())
  # ('v0.0.1',)"><pre><span class="pl-k">import</span> <span class="pl-s1">sqlite3</span>

  <span class="pl-s1">con</span> <span class="pl-c1">=</span> <span class="pl-s1">sqlite3</span>.<span class="pl-c1">connect</span>(<span class="pl-s">":memory:"</span>)

  <span class="pl-s1">con</span>.<span class="pl-c1">enable_load_extension</span>(<span class="pl-c1">True</span>)
  <span class="pl-s1">con</span>.<span class="pl-c1">load_extension</span>(<span class="pl-s">"./url0"</span>)

  <span class="pl-en">print</span>(<span class="pl-s1">con</span>.<span class="pl-c1">execute</span>(<span class="pl-s">"select url_version()"</span>).<span class="pl-c1">fetchone</span>())
  <span class="pl-c"># ('v0.0.1',)</span></pre></div>
  <p dir="auto">Or in Node.js using <a href="https://github.com/WiseLibs/better-sqlite3">better-sqlite3</a>:</p>
  <div class="highlight highlight-source-js notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="const Database = require("better-sqlite3");
  const db = new Database(":memory:");

  db.loadExtension("./url0");

  console.log(db.prepare("select url_version()").get());
  // { 'html_version()': 'v0.0.1' }"><pre><span class="pl-k">const</span> <span class="pl-v">Database</span> <span class="pl-c1">=</span> <span class="pl-en">require</span><span class="pl-kos">(</span><span class="pl-s">"better-sqlite3"</span><span class="pl-kos">)</span><span class="pl-kos">;</span>
  <span class="pl-k">const</span> <span class="pl-s1">db</span> <span class="pl-c1">=</span> <span class="pl-k">new</span> <span class="pl-v">Database</span><span class="pl-kos">(</span><span class="pl-s">":memory:"</span><span class="pl-kos">)</span><span class="pl-kos">;</span>

  <span class="pl-s1">db</span><span class="pl-kos">.</span><span class="pl-en">loadExtension</span><span class="pl-kos">(</span><span class="pl-s">"./url0"</span><span class="pl-kos">)</span><span class="pl-kos">;</span>

  <span class="pl-smi">console</span><span class="pl-kos">.</span><span class="pl-en">log</span><span class="pl-kos">(</span><span class="pl-s1">db</span><span class="pl-kos">.</span><span class="pl-en">prepare</span><span class="pl-kos">(</span><span class="pl-s">"select url_version()"</span><span class="pl-kos">)</span><span class="pl-kos">.</span><span class="pl-en">get</span><span class="pl-kos">(</span><span class="pl-kos">)</span><span class="pl-kos">)</span><span class="pl-kos">;</span>
  <span class="pl-c">// { 'html_version()': 'v0.0.1' }</span></pre></div>
  <p dir="auto">Or with <a href="https://datasette.io/" rel="nofollow">Datasette</a>:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="datasette data.db --load-extension ./url0"><pre class="notranslate"><code>datasette data.db --load-extension ./url0
  </code></pre></div>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">See also</h2><a id="user-content-see-also" class="anchor" aria-label="Permalink: See also" href="#see-also"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <ul dir="auto">
  <li><a href="https://github.com/asg017/sqlite-path">sqlite-path</a>, parsing/generating paths (pairs well with <code>url_path()</code> and <code>url()</code>)</li>
  <li><a href="https://github.com/asg017/sqlite-http">sqlite-http</a>, for making HTTP requests in SQLite</li>
  <li><a href="https://github.com/asg017/sqlite-html">sqlite-html</a>, for parsing HTML documents</li>
  <li><a href="https://github.com/asg017/sqlite-lines">sqlite-lines</a>, for reading large files line-by-line</li>
  <li><a href="https://github.com/nalgeon/sqlean">nalgeon/sqlean</a>, several pre-compiled handy SQLite functions, in C</li>
  </ul>
  </article></div>


node_id: R_kgDOH2kMZA
name: datasette-mp3-audio
full_name: simonw/datasette-mp3-audio
html_url: https://github.com/simonw/datasette-mp3-audio
description: Turn .mp3 URLs into an audio player in the Datasette interface
created_at: 2022-08-20T16:20:06Z
updated_at: 2024-03-02T13:39:56Z
pushed_at: 2022-08-21T06:43:00Z
language: Python
license: apache-2.0
topics: []
visibility: public
default_branch: main
permissions: {"admin": false, "maintain": false, "push": false, "triage": false, "pull": true}
template_repository:
[cell repeats an earlier one in this file: 6377 characters]

readme:
  # datasette-mp3-audio

  [![PyPI](https://img.shields.io/pypi/v/datasette-mp3-audio.svg)](https://pypi.org/project/datasette-mp3-audio/)
  [![Changelog](https://img.shields.io/github/v/release/simonw/datasette-mp3-audio?include_prereleases&label=changelog)](https://github.com/simonw/datasette-mp3-audio/releases)
  [![Tests](https://github.com/simonw/datasette-mp3-audio/workflows/Test/badge.svg)](https://github.com/simonw/datasette-mp3-audio/actions?query=workflow%3ATest)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/datasette-mp3-audio/blob/main/LICENSE)

  Turn .mp3 URLs into an audio player in the Datasette interface

  ## Installation

  Install this plugin in the same environment as Datasette.

      datasette install datasette-mp3-audio

  ## Demo

  Try this plugin at [https://scotrail.datasette.io/scotrail/announcements](https://scotrail.datasette.io/scotrail/announcements)

  The demo uses ScotRail train announcements from [matteason/scotrail-announcements-june-2022](https://github.com/matteason/scotrail-announcements-june-2022).

  ## Usage

  Once installed, any cells with a value that ends in `.mp3` and starts with either `http://` or `/` or `https://` will be turned into an embedded HTML audio element like this:

  ```html
  <audio controls src="... value ..."><a href="...">Download MP3</a></audio>
  ```

  A "Play X MP3s on this page" button will be added to athe top of any table page listing more than one MP3.

  ## Development

  To set up this plugin locally, first checkout the code. Then create a new virtual environment:

      cd datasette-mp3-audio
      python3 -m venv venv
      source venv/bin/activate

  Now install the dependencies and test dependencies:

      pip install -e '.[test]'

  To run the tests:

      pytest

readme_html:
  <div id="readme" class="md" data-path="README.md"><article class="markdown-body entry-content container-lg" itemprop="text"><div class="markdown-heading" dir="auto"><h1 class="heading-element" dir="auto">datasette-mp3-audio</h1><a id="user-content-datasette-mp3-audio" class="anchor" aria-label="Permalink: datasette-mp3-audio" href="#datasette-mp3-audio"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto"><a href="https://pypi.org/project/datasette-mp3-audio/" rel="nofollow"><img src="https://img.shields.io/pypi/v/datasette-mp3-audio.svg" alt="PyPI" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/67d100c83a7a79a2a99c3d66cb6e00db364c380e8f53c82e9b7968d90a65d991/68747470733a2f2f696d672e736869656c64732e696f2f707970692f762f6461746173657474652d6d70332d617564696f2e737667"></a>
  <a href="https://github.com/simonw/datasette-mp3-audio/releases"><img src="https://img.shields.io/github/v/release/simonw/datasette-mp3-audio?include_prereleases&label=changelog" alt="Changelog" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/d97518ee8e06e67d076dd2049331e0c3e3130e413c49f55e814e0ff42aa856fd/68747470733a2f2f696d672e736869656c64732e696f2f6769746875622f762f72656c656173652f73696d6f6e772f6461746173657474652d6d70332d617564696f3f696e636c7564655f70726572656c6561736573266c6162656c3d6368616e67656c6f67"></a>
  <a href="https://github.com/simonw/datasette-mp3-audio/actions?query=workflow%3ATest"><img src="https://github.com/simonw/datasette-mp3-audio/workflows/Test/badge.svg" alt="Tests" style="max-width: 100%;"></a>
  <a href="https://github.com/simonw/datasette-mp3-audio/blob/main/LICENSE"><img src="https://img.shields.io/badge/license-Apache%202.0-blue.svg" alt="License" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/c355f200ea90fddaa407b6eaab303663a669248ea3ca7b1fcf77dbe04ff5f48c/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6c6963656e73652d417061636865253230322e302d626c75652e737667"></a></p>
  <p dir="auto">Turn .mp3 URLs into an audio player in the Datasette interface</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Installation</h2><a id="user-content-installation" class="anchor" aria-label="Permalink: Installation" href="#installation"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">Install this plugin in the same environment as Datasette.</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="datasette install datasette-mp3-audio"><pre class="notranslate"><code>datasette install datasette-mp3-audio
  </code></pre></div>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Demo</h2><a id="user-content-demo" class="anchor" aria-label="Permalink: Demo" href="#demo"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">Try this plugin at <a href="https://scotrail.datasette.io/scotrail/announcements" rel="nofollow">https://scotrail.datasette.io/scotrail/announcements</a></p>
  <p dir="auto">The demo uses ScotRail train announcements from <a href="https://github.com/matteason/scotrail-announcements-june-2022">matteason/scotrail-announcements-june-2022</a>.</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Usage</h2><a id="user-content-usage" class="anchor" aria-label="Permalink: Usage" href="#usage"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">Once installed, any cells with a value that ends in <code>.mp3</code> and starts with either <code>http://</code> or <code>/</code> or <code>https://</code> will be turned into an embedded HTML audio element like this:</p>
  <div class="highlight highlight-text-html-basic notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="<audio controls src="... value ..."><a href="...">Download MP3</a></audio>"><pre><span class="pl-kos"><</span><span class="pl-ent">audio</span> <span class="pl-c1">controls</span> <span class="pl-c1">src</span>="<span class="pl-s">... value ...</span>"<span class="pl-kos">></span><span class="pl-kos"><</span><span class="pl-ent">a</span> <span class="pl-c1">href</span>="<span class="pl-s">...</span>"<span class="pl-kos">></span>Download MP3<span class="pl-kos"></</span><span class="pl-ent">a</span><span class="pl-kos">></span><span class="pl-kos"></</span><span class="pl-ent">audio</span><span class="pl-kos">></span></pre></div>
  <p dir="auto">A "Play X MP3s on this page" button will be added to athe top of any table page listing more than one MP3.</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Development</h2><a id="user-content-development" class="anchor" aria-label="Permalink: Development" href="#development"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">To set up this plugin locally, first checkout the code. Then create a new virtual environment:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="cd datasette-mp3-audio
  python3 -m venv venv
  source venv/bin/activate"><pre class="notranslate"><code>cd datasette-mp3-audio
  python3 -m venv venv
  source venv/bin/activate
  </code></pre></div>
  <p dir="auto">Now install the dependencies and test dependencies:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="pip install -e '.[test]'"><pre class="notranslate"><code>pip install -e '.[test]'
  </code></pre></div>
  <p dir="auto">To run the tests:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="pytest"><pre class="notranslate"><code>pytest
  </code></pre></div>
  </article></div>


node_id: R_kgDOH4H41Q
name: datasette-multiline-links
full_name: simonw/datasette-multiline-links
html_url: https://github.com/simonw/datasette-multiline-links
description: Make multiple newline separated URLs clickable in Datasette
created_at: 2022-08-24T22:16:30Z
updated_at: 2024-05-29T18:03:56Z
pushed_at: 2022-08-24T22:56:16Z
language: Python
license: apache-2.0
topics: []
visibility: public
default_branch: main
permissions: {"admin": false, "maintain": false, "push": false, "triage": false, "pull": true}
template_repository:
[cell repeats an earlier one in this file: 6377 characters]

readme:
  # datasette-multiline-links

  [![PyPI](https://img.shields.io/pypi/v/datasette-multiline-links.svg)](https://pypi.org/project/datasette-multiline-links/)
  [![Changelog](https://img.shields.io/github/v/release/simonw/datasette-multiline-links?include_prereleases&label=changelog)](https://github.com/simonw/datasette-multiline-links/releases)
  [![Tests](https://github.com/simonw/datasette-multiline-links/workflows/Test/badge.svg)](https://github.com/simonw/datasette-multiline-links/actions?query=workflow%3ATest)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/datasette-multiline-links/blob/main/LICENSE)

  Make multiple newline separated URLs clickable in Datasette

  ## Installation

  Install this plugin in the same environment as Datasette.

      datasette install datasette-multiline-links

  ## Demo

  Try this plugin out against a [Google Sheets spreadsheet](https://docs.google.com/spreadsheets/d/1wZhPLMCHKJvwOkP4juclhjFgqIY8fQFMemwKL2c64vk) of previously featured datasets from [Data is Plural](https://www.data-is-plural.com/) using [Datasette Lite](https://lite.datasette.io/) here:

  * <a href="https://lite.datasette.io/?install=datasette-multiline-links&csv=https://docs.google.com/spreadsheets/d/1wZhPLMCHKJvwOkP4juclhjFgqIY8fQFMemwKL2c64vk/export?format=csv#/data?sql=select+edition%2C+headline%2C+text%2C+links%2C+hattips+from+export+where%0Atext+like+'%25'+||+%3Aq+||+'%25'+or+headline+like+'%25'+||+%3Aq+||+'%25'+order+by+edition+desc&q=loans">Demo this plugin in Datasette Lite</a>

  ## Usage

  Once installed, if a cell has contents like this:
  ```
  https://example.com
  Not a link
  https://google.com
  ```
  It will be rendered as:
  ```html
  <a href="https://example.com">https://example.com</a>
  Not a link
  <a href="https://google.com">https://google.com</a>
  ```
  ## Development

  To set up this plugin locally, first checkout the code. Then create a new virtual environment:

      cd datasette-multiline-links
      python3 -m venv venv
      source venv/bin/activate

  Now install the dependencies and test dependencies:

      pip install -e '.[test]'

  To run the tests:

      pytest

readme_html:
  <div id="readme" class="md" data-path="README.md"><article class="markdown-body entry-content container-lg" itemprop="text"><div class="markdown-heading" dir="auto"><h1 class="heading-element" dir="auto">datasette-multiline-links</h1><a id="user-content-datasette-multiline-links" class="anchor" aria-label="Permalink: datasette-multiline-links" href="#datasette-multiline-links"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto"><a href="https://pypi.org/project/datasette-multiline-links/" rel="nofollow"><img src="https://img.shields.io/pypi/v/datasette-multiline-links.svg" alt="PyPI" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/967cf0650d73e1bc89b693c90b74824d5d8fb5f3fe7053d52fd3f145ad698525/68747470733a2f2f696d672e736869656c64732e696f2f707970692f762f6461746173657474652d6d756c74696c696e652d6c696e6b732e737667"></a>
  <a href="https://github.com/simonw/datasette-multiline-links/releases"><img src="https://img.shields.io/github/v/release/simonw/datasette-multiline-links?include_prereleases&label=changelog" alt="Changelog" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/baf3086ba9a303361394941cab66dd1520e030cde1bd2ccf1bdba4ede3afd400/68747470733a2f2f696d672e736869656c64732e696f2f6769746875622f762f72656c656173652f73696d6f6e772f6461746173657474652d6d756c74696c696e652d6c696e6b733f696e636c7564655f70726572656c6561736573266c6162656c3d6368616e67656c6f67"></a>
  <a href="https://github.com/simonw/datasette-multiline-links/actions?query=workflow%3ATest"><img src="https://github.com/simonw/datasette-multiline-links/workflows/Test/badge.svg" alt="Tests" style="max-width: 100%;"></a>
  <a href="https://github.com/simonw/datasette-multiline-links/blob/main/LICENSE"><img src="https://img.shields.io/badge/license-Apache%202.0-blue.svg" alt="License" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/c355f200ea90fddaa407b6eaab303663a669248ea3ca7b1fcf77dbe04ff5f48c/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6c6963656e73652d417061636865253230322e302d626c75652e737667"></a></p>
  <p dir="auto">Make multiple newline separated URLs clickable in Datasette</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Installation</h2><a id="user-content-installation" class="anchor" aria-label="Permalink: Installation" href="#installation"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">Install this plugin in the same environment as Datasette.</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="datasette install datasette-multiline-links"><pre class="notranslate"><code>datasette install datasette-multiline-links
  </code></pre></div>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Demo</h2><a id="user-content-demo" class="anchor" aria-label="Permalink: Demo" href="#demo"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">Try this plugin out against a <a href="https://docs.google.com/spreadsheets/d/1wZhPLMCHKJvwOkP4juclhjFgqIY8fQFMemwKL2c64vk" rel="nofollow">Google Sheets spreadsheet</a> of previously featured datasets from <a href="https://www.data-is-plural.com/" rel="nofollow">Data is Plural</a> using <a href="https://lite.datasette.io/" rel="nofollow">Datasette Lite</a> here:</p>
  <ul dir="auto">
  <li><a href="https://lite.datasette.io/?install=datasette-multiline-links&csv=https://docs.google.com/spreadsheets/d/1wZhPLMCHKJvwOkP4juclhjFgqIY8fQFMemwKL2c64vk/export?format=csv#/data?sql=select+edition%2C+headline%2C+text%2C+links%2C+hattips+from+export+where%0Atext+like+'%25'+||+%3Aq+||+'%25'+or+headline+like+'%25'+||+%3Aq+||+'%25'+order+by+edition+desc&q=loans" rel="nofollow">Demo this plugin in Datasette Lite</a></li>
  </ul>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Usage</h2><a id="user-content-usage" class="anchor" aria-label="Permalink: Usage" href="#usage"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">Once installed, if a cell has contents like this:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="https://example.com
  Not a link
  https://google.com"><pre class="notranslate"><code>https://example.com
  Not a link
  https://google.com
  </code></pre></div>
  <p dir="auto">It will be rendered as:</p>
  <div class="highlight highlight-text-html-basic notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="<a href="https://example.com">https://example.com</a>
  Not a link
  <a href="https://google.com">https://google.com</a>"><pre><span class="pl-kos"><</span><span class="pl-ent">a</span> <span class="pl-c1">href</span>="<span class="pl-s">https://example.com</span>"<span class="pl-kos">></span>https://example.com<span class="pl-kos"></</span><span class="pl-ent">a</span><span class="pl-kos">></span>
  Not a link
  <span class="pl-kos"><</span><span class="pl-ent">a</span> <span class="pl-c1">href</span>="<span class="pl-s">https://google.com</span>"<span class="pl-kos">></span>https://google.com<span class="pl-kos"></</span><span class="pl-ent">a</span><span class="pl-kos">></span></pre></div>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Development</h2><a id="user-content-development" class="anchor" aria-label="Permalink: Development" href="#development"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">To set up this plugin locally, first checkout the code. Then create a new virtual environment:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="cd datasette-multiline-links
  python3 -m venv venv
  source venv/bin/activate"><pre class="notranslate"><code>cd datasette-multiline-links
  python3 -m venv venv
  source venv/bin/activate
  </code></pre></div>
  <p dir="auto">Now install the dependencies and test dependencies:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="pip install -e '.[test]'"><pre class="notranslate"><code>pip install -e '.[test]'
  </code></pre></div>
  <p dir="auto">To run the tests:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="pytest"><pre class="notranslate"><code>pytest
  </code></pre></div>
  </article></div>


node_id: R_kgDOH50_7g
name: datasette-sitemap
full_name: simonw/datasette-sitemap
html_url: https://github.com/simonw/datasette-sitemap
description: Generate sitemap.xml for Datasette sites
created_at: 2022-08-29T21:22:46Z
updated_at: 2023-04-04T16:17:18Z
pushed_at: 2022-08-30T17:59:53Z
language: Python
license: apache-2.0
topics: []
visibility: public
default_branch: main
permissions: {"admin": false, "maintain": false, "push": false, "triage": false, "pull": true}
template_repository:
[cell repeats an earlier one in this file: 6377 characters]

readme:
  # datasette-sitemap

  [![PyPI](https://img.shields.io/pypi/v/datasette-sitemap.svg)](https://pypi.org/project/datasette-sitemap/)
  [![Changelog](https://img.shields.io/github/v/release/simonw/datasette-sitemap?include_prereleases&label=changelog)](https://github.com/simonw/datasette-sitemap/releases)
  [![Tests](https://github.com/simonw/datasette-sitemap/workflows/Test/badge.svg)](https://github.com/simonw/datasette-sitemap/actions?query=workflow%3ATest)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/datasette-sitemap/blob/main/LICENSE)

  Generate sitemap.xml for Datasette sites

  ## Installation

  Install this plugin in the same environment as Datasette.

      datasette install datasette-sitemap

  ## Demo

  This plugin is used for the sitemap on [til.simonwillison.net](https://til.simonwillison.net/):

  - https://til.simonwillison.net/sitemap.xml

  Here's [the configuration](https://github.com/simonw/til/blob/d4f67743a90a67100b46145986b2dec6f8d96583/metadata.yaml#L14-L16) used for that sitemap.

  ## Usage

  Once configured, this plugin adds a sitemap at `/sitemap.xml` with a list of URLs.

  This list is defined using a SQL query in `metadata.json` (or `.yml`) that looks like this:

  ```json
  {
    "plugins": {
      "datasette-sitemap": {
        "query": "select '/' || id as path from my_table"
      }
    }
  }
  ```

  Using `metadata.yml` allows for multi-line SQL queries which can be easier to maintain:

  ```yaml
  plugins:
    datasette-sitemap:
      query: |
        select
          '/' || id as path
        from
          my_table
  ```
  The SQL query must return a column called `path`. The values in this column must begin with a `/`. They will be used to generate a sitemap that looks like this:

  ```xml
  <?xml version="1.0" encoding="UTF-8"?>
  <urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
    <url><loc>https://example.com/1</loc></url>
    <url><loc>https://example.com/2</loc></url>
  </urlset>
  ```
  You can use ``UNION`` in your SQL query to combine results from multiple tables, or include literal paths that you want to include in the index:

  ```sql
  select
    '/data/table1/' || id as path
    from table1
  union
  select
    '/data/table2/' || id as path
    from table2
  union
  select
    '/about' as path
  ```
  If your Datasette instance has multiple databases you can configure the database to query using the `database` configuration property.

  By default the domain name for the genearted URLs in the sitemap will be detected from the incoming request.

  You can set `base_url` instead to override this. This should not include a trailing slash.

  This example shows both of those settings, running the query against the `content` database and setting a custom base URL:

  ```yaml
  plugins:
    datasette-sitemap:
      query: |
        select '/plugins/' || name as path from plugins
        union
        select '/tools/' || name as path from tools
        union
        select '/news' as path
      database: content
      base_url: https://datasette.io
  ```
  [Try that query](https://datasette.io/content?sql=select+%27%2Fplugins%2F%27+||+name+as+path+from+plugins%0D%0Aunion%0D%0Aselect+%27%2Ftools%2F%27+||+name+as+path+from+tools%0D%0Aunion%0D%0Aselect+%27%2Fnews%27+as+path%0D%0A).

  ## robots.txt

  This plugin adds a `robots.txt` file pointing to the sitemap:

  ```
  Sitemap: http://example.com/sitemap.xml
  ```

  You can take full control of the sitemap by installing and configuring the [datasette-block-robots](https://datasette.io/plugins/datasette-block-robots) plugin.

  This plugin will add the `Sitemap:` line even if you are using `datasette-block-robots` for the rest of your `robots.txt` file.

  ## Adding paths to the sitemap from other plugins

  This plugin adds a new [plugin hook](https://docs.datasette.io/en/stable/plugin_hooks.html) to Datasete called `sitemap_extra_paths()` which can be used by other plugins to add their own additional lines to the `sitemap.xml` file.

  The hook accepts these optional parameters:

  - `datasette`: The current [Datasette instance](https://docs.datasette.io/en/stable/internals.html#datasette-class). You can use this to execute SQL queries or read plugin configuration settings.
  - `request`: The [Request object](https://docs.datasette.io/en/stable/internals.html#request-object) representing the incoming request to `/sitemap.xml`.

  The hook should return a list of strings, each representing a path to be added to the sitemap. Each path must begin with a `/`.

  It can also return an `async def` function, which will be awaited and used to generate a list of lines. Use this option if you need to make `await` calls inside you hook implementation.

  This example uses the hook to add two extra paths, one of which came from a SQL query:

  ```python
  from datasette import hookimpl

  @hookimpl
  def sitemap_extra_paths(datasette):
      async def inner():
          db = datasette.get_database()
          path_from_db = (await db.execute("select '/example'")).single_value()
          return ["/about", path_from_db]
      return inner
  ```

  ## Development

  To set up this plugin locally, first checkout the code. Then create a new virtual environment:

      cd datasette-sitemap
      python3 -m venv venv
      source venv/bin/activate

  Now install the dependencies and test dependencies:

      pip install -e '.[test]'

  To run the tests:

      pytest

readme_html:
  <div id="readme" class="md" data-path="README.md"><article class="markdown-body entry-content container-lg" itemprop="text"><div class="markdown-heading" dir="auto"><h1 class="heading-element" dir="auto">datasette-sitemap</h1><a id="user-content-datasette-sitemap" class="anchor" aria-label="Permalink: datasette-sitemap" href="#datasette-sitemap"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto"><a href="https://pypi.org/project/datasette-sitemap/" rel="nofollow"><img src="https://img.shields.io/pypi/v/datasette-sitemap.svg" alt="PyPI" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/e1d9239c79e422fa34fb9a7102734a61cb1fba929cdb26e8b6a05d2282757c0b/68747470733a2f2f696d672e736869656c64732e696f2f707970692f762f6461746173657474652d736974656d61702e737667"></a>
  <a href="https://github.com/simonw/datasette-sitemap/releases"><img src="https://img.shields.io/github/v/release/simonw/datasette-sitemap?include_prereleases&label=changelog" alt="Changelog" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/d31e561fa81e176c94d5e7ffbdae2ab885d31c38ce5049c04bef4711e6b0bbbb/68747470733a2f2f696d672e736869656c64732e696f2f6769746875622f762f72656c656173652f73696d6f6e772f6461746173657474652d736974656d61703f696e636c7564655f70726572656c6561736573266c6162656c3d6368616e67656c6f67"></a>
  <a href="https://github.com/simonw/datasette-sitemap/actions?query=workflow%3ATest"><img src="https://github.com/simonw/datasette-sitemap/workflows/Test/badge.svg" alt="Tests" style="max-width: 100%;"></a>
  <a href="https://github.com/simonw/datasette-sitemap/blob/main/LICENSE"><img src="https://img.shields.io/badge/license-Apache%202.0-blue.svg" alt="License" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/c355f200ea90fddaa407b6eaab303663a669248ea3ca7b1fcf77dbe04ff5f48c/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6c6963656e73652d417061636865253230322e302d626c75652e737667"></a></p>
  <p dir="auto">Generate sitemap.xml for Datasette sites</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Installation</h2><a id="user-content-installation" class="anchor" aria-label="Permalink: Installation" href="#installation"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">Install this plugin in the same environment as Datasette.</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="datasette install datasette-sitemap"><pre class="notranslate"><code>datasette install datasette-sitemap
  </code></pre></div>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Demo</h2><a id="user-content-demo" class="anchor" aria-label="Permalink: Demo" href="#demo"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">This plugin is used for the sitemap on <a href="https://til.simonwillison.net/" rel="nofollow">til.simonwillison.net</a>:</p>
  <ul dir="auto">
  <li><a href="https://til.simonwillison.net/sitemap.xml" rel="nofollow">https://til.simonwillison.net/sitemap.xml</a></li>
  </ul>
  <p dir="auto">Here's <a href="https://github.com/simonw/til/blob/d4f67743a90a67100b46145986b2dec6f8d96583/metadata.yaml#L14-L16">the configuration</a> used for that sitemap.</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Usage</h2><a id="user-content-usage" class="anchor" aria-label="Permalink: Usage" href="#usage"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">Once configured, this plugin adds a sitemap at <code>/sitemap.xml</code> with a list of URLs.</p>
  <p dir="auto">This list is defined using a SQL query in <code>metadata.json</code> (or <code>.yml</code>) that looks like this:</p>
  <div class="highlight highlight-source-json notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="{
    "plugins": {
      "datasette-sitemap": {
        "query": "select '/' || id as path from my_table"
      }
    }
  }"><pre>{
    <span class="pl-ent">"plugins"</span>: {
      <span class="pl-ent">"datasette-sitemap"</span>: {
        <span class="pl-ent">"query"</span>: <span class="pl-s"><span class="pl-pds">"</span>select '/' || id as path from my_table<span class="pl-pds">"</span></span>
      }
    }
  }</pre></div>
  <p dir="auto">Using <code>metadata.yml</code> allows for multi-line SQL queries which can be easier to maintain:</p>
  <div class="highlight highlight-source-yaml notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="plugins:
    datasette-sitemap:
      query: |
        select
          '/' || id as path
        from
          my_table"><pre><span class="pl-ent">plugins</span>:
    <span class="pl-ent">datasette-sitemap</span>:
      <span class="pl-ent">query</span>: <span class="pl-s">|</span>
  <span class="pl-s">      select</span>
  <span class="pl-s">        '/' || id as path</span>
  <span class="pl-s">      from</span>
  <span class="pl-s">        my_table</span></pre></div>
  <p dir="auto">The SQL query must return a column called <code>path</code>. The values in this column must begin with a <code>/</code>. They will be used to generate a sitemap that looks like this:</p>
  <div class="highlight highlight-text-xml notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="<?xml version="1.0" encoding="UTF-8"?>
  <urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
    <url><loc>https://example.com/1</loc></url>
    <url><loc>https://example.com/2</loc></url>
  </urlset>"><pre><?<span class="pl-ent">xml</span><span class="pl-e"> version</span>=<span class="pl-s"><span class="pl-pds">"</span>1.0<span class="pl-pds">"</span></span><span class="pl-e"> encoding</span>=<span class="pl-s"><span class="pl-pds">"</span>UTF-8<span class="pl-pds">"</span></span>?>
  <<span class="pl-ent">urlset</span> <span class="pl-e">xmlns</span>=<span class="pl-s"><span class="pl-pds">"</span>http://www.sitemaps.org/schemas/sitemap/0.9<span class="pl-pds">"</span></span>>
    <<span class="pl-ent">url</span>><<span class="pl-ent">loc</span>>https://example.com/1</<span class="pl-ent">loc</span>></<span class="pl-ent">url</span>>
    <<span class="pl-ent">url</span>><<span class="pl-ent">loc</span>>https://example.com/2</<span class="pl-ent">loc</span>></<span class="pl-ent">url</span>>
  </<span class="pl-ent">urlset</span>></pre></div>
  <p dir="auto">You can use <code>UNION</code> in your SQL query to combine results from multiple tables, or include literal paths that you want to include in the index:</p>
  <div class="highlight highlight-source-sql notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="select
    '/data/table1/' || id as path
    from table1
  union
  select
    '/data/table2/' || id as path
    from table2
  union
  select
    '/about' as path"><pre><span class="pl-k">select</span>
    <span class="pl-s"><span class="pl-pds">'</span>/data/table1/<span class="pl-pds">'</span></span> <span class="pl-k">||</span> id <span class="pl-k">as</span> <span class="pl-k">path</span>
    <span class="pl-k">from</span> table1
  <span class="pl-k">union</span>
  <span class="pl-k">select</span>
    <span class="pl-s"><span class="pl-pds">'</span>/data/table2/<span class="pl-pds">'</span></span> <span class="pl-k">||</span> id <span class="pl-k">as</span> <span class="pl-k">path</span>
    <span class="pl-k">from</span> table2
  <span class="pl-k">union</span>
  <span class="pl-k">select</span>
    <span class="pl-s"><span class="pl-pds">'</span>/about<span class="pl-pds">'</span></span> <span class="pl-k">as</span> <span class="pl-k">path</span></pre></div>
  <p dir="auto">If your Datasette instance has multiple databases you can configure the database to query using the <code>database</code> configuration property.</p>
  <p dir="auto">By default the domain name for the genearted URLs in the sitemap will be detected from the incoming request.</p>
  <p dir="auto">You can set <code>base_url</code> instead to override this. This should not include a trailing slash.</p>
  <p dir="auto">This example shows both of those settings, running the query against the <code>content</code> database and setting a custom base URL:</p>
  <div class="highlight highlight-source-yaml notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="plugins:
    datasette-sitemap:
      query: |
        select '/plugins/' || name as path from plugins
        union
        select '/tools/' || name as path from tools
        union
        select '/news' as path
      database: content
      base_url: https://datasette.io"><pre><span class="pl-ent">plugins</span>:
    <span class="pl-ent">datasette-sitemap</span>:
      <span class="pl-ent">query</span>: <span class="pl-s">|</span>
  <span class="pl-s">      select '/plugins/' || name as path from plugins</span>
  <span class="pl-s">      union</span>
  <span class="pl-s">      select '/tools/' || name as path from tools</span>
  <span class="pl-s">      union</span>
  <span class="pl-s">      select '/news' as path</span>
  <span class="pl-s"></span>    <span class="pl-ent">database</span>: <span class="pl-s">content</span>
      <span class="pl-ent">base_url</span>: <span class="pl-s">https://datasette.io</span></pre></div>
  <p dir="auto"><a href="https://datasette.io/content?sql=select+%27%2Fplugins%2F%27+%7C%7C+name+as+path+from+plugins%0D%0Aunion%0D%0Aselect+%27%2Ftools%2F%27+%7C%7C+name+as+path+from+tools%0D%0Aunion%0D%0Aselect+%27%2Fnews%27+as+path%0D%0A" rel="nofollow">Try that query</a>.</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">robots.txt</h2><a id="user-content-robotstxt" class="anchor" aria-label="Permalink: robots.txt" href="#robotstxt"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">This plugin adds a <code>robots.txt</code> file pointing to the sitemap:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="Sitemap: http://example.com/sitemap.xml"><pre class="notranslate"><code>Sitemap: http://example.com/sitemap.xml
  </code></pre></div>
  <p dir="auto">You can take full control of the sitemap by installing and configuring the <a href="https://datasette.io/plugins/datasette-block-robots" rel="nofollow">datasette-block-robots</a> plugin.</p>
  <p dir="auto">This plugin will add the <code>Sitemap:</code> line even if you are using <code>datasette-block-robots</code> for the rest of your <code>robots.txt</code> file.</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Adding paths to the sitemap from other plugins</h2><a id="user-content-adding-paths-to-the-sitemap-from-other-plugins" class="anchor" aria-label="Permalink: Adding paths to the sitemap from other plugins" href="#adding-paths-to-the-sitemap-from-other-plugins"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">This plugin adds a new <a href="https://docs.datasette.io/en/stable/plugin_hooks.html" rel="nofollow">plugin hook</a> to Datasete called <code>sitemap_extra_paths()</code> which can be used by other plugins to add their own additional lines to the <code>sitemap.xml</code> file.</p>
  <p dir="auto">The hook accepts these optional parameters:</p>
  <ul dir="auto">
  <li><code>datasette</code>: The current <a href="https://docs.datasette.io/en/stable/internals.html#datasette-class" rel="nofollow">Datasette instance</a>. You can use this to execute SQL queries or read plugin configuration settings.</li>
  <li><code>request</code>: The <a href="https://docs.datasette.io/en/stable/internals.html#request-object" rel="nofollow">Request object</a> representing the incoming request to <code>/sitemap.xml</code>.</li>
  </ul>
  <p dir="auto">The hook should return a list of strings, each representing a path to be added to the sitemap. Each path must begin with a <code>/</code>.</p>
  <p dir="auto">It can also return an <code>async def</code> function, which will be awaited and used to generate a list of lines. Use this option if you need to make <code>await</code> calls inside you hook implementation.</p>
  <p dir="auto">This example uses the hook to add two extra paths, one of which came from a SQL query:</p>
  <div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="from datasette import hookimpl

  @hookimpl
  def sitemap_extra_paths(datasette):
      async def inner():
          db = datasette.get_database()
          path_from_db = (await db.execute("select '/example'")).single_value()
          return ["/about", path_from_db]
      return inner"><pre><span class="pl-k">from</span> <span class="pl-s1">datasette</span> <span class="pl-k">import</span> <span class="pl-s1">hookimpl</span>

  <span class="pl-en">@<span class="pl-s1">hookimpl</span></span>
  <span class="pl-k">def</span> <span class="pl-en">sitemap_extra_paths</span>(<span class="pl-s1">datasette</span>):
      <span class="pl-k">async</span> <span class="pl-k">def</span> <span class="pl-en">inner</span>():
          <span class="pl-s1">db</span> <span class="pl-c1">=</span> <span class="pl-s1">datasette</span>.<span class="pl-c1">get_database</span>()
          <span class="pl-s1">path_from_db</span> <span class="pl-c1">=</span> (<span class="pl-k">await</span> <span class="pl-s1">db</span>.<span class="pl-c1">execute</span>(<span class="pl-s">"select '/example'"</span>)).<span class="pl-c1">single_value</span>()
          <span class="pl-k">return</span> [<span class="pl-s">"/about"</span>, <span class="pl-s1">path_from_db</span>]
      <span class="pl-k">return</span> <span class="pl-s1">inner</span></pre></div>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Development</h2><a id="user-content-development" class="anchor" aria-label="Permalink: Development" href="#development"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">To set up this plugin locally, first checkout the code. Then create a new virtual environment:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="cd datasette-sitemap
  python3 -m venv venv
  source venv/bin/activate"><pre class="notranslate"><code>cd datasette-sitemap
  python3 -m venv venv
  source venv/bin/activate
  </code></pre></div>
  <p dir="auto">Now install the dependencies and test dependencies:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="pip install -e '.[test]'"><pre class="notranslate"><code>pip install -e '.[test]'
  </code></pre></div>
  <p dir="auto">To run the tests:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="pytest"><pre class="notranslate"><code>pytest
  </code></pre></div>
  </article></div>


node_id: R_kgDOH7vZQw
name: datasette-render-image-tags
full_name: simonw/datasette-render-image-tags
html_url: https://github.com/simonw/datasette-render-image-tags
description: Turn any URLs ending in .jpg/.png/.gif into img tags with width 200
created_at: 2022-09-04T00:36:44Z
updated_at: 2025-11-02T21:10:02Z
pushed_at: 2025-11-02T21:09:58Z
language: Python
license: apache-2.0
topics: []
visibility: public
default_branch: main
permissions: {"admin": false, "maintain": false, "push": false, "triage": false, "pull": false}
template_repository:
  {"id": 400878073, "node_id": "MDEwOlJlcG9zaXRvcnk0MDA4NzgwNzM=", "name": "datasette-plugin-template-repository", "full_name": "simonw/datasette-plugin-template-repository", "private": false, "owner": {"login": "simonw", "id": 9599, "node_id": "MDQ6VXNlcjk1OTk=", "avatar_url": "https://avatars.githubusercontent.com/u/9599?v=4", "gravatar_id": "", "url": "https://api.github.com/users/simonw", "html_url": "https://github.com/simonw", "followers_url": "https://api.github.com/users/simonw/followers", "following_url": "https://api.github.com/users/simonw/following{/other_user}", "gists_url": "https://api.github.com/users/simonw/gists{/gist_id}", "starred_url": "https://api.github.com/users/simonw/starred{/owner}{/repo}", "subscriptions_url": "https://api.github.com/users/simonw/subscriptions", "organizations_url": "https://api.github.com/users/simonw/orgs", "repos_url": "https://api.github.com/users/simonw/repos", "events_url": "https://api.github.com/users/simonw/events{/privacy}", "received_events_url": "https://api.github.com/users/simonw/received_events", "type": "User", "user_view_type": "public", "site_admin": false}, "html_url": "https://github.com/simonw/datasette-plugin-template-repository", "description": "GitHub template repository for creating new Datasette plugins, using the simonw/datasette-plugin cookiecutter template", "fork": false, "url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository", "forks_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/forks", "keys_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/keys{/key_id}", "collaborators_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/collaborators{/collaborator}", "teams_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/teams", "hooks_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/hooks", "issue_events_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/issues/events{/number}", "events_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/events", "assignees_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/assignees{/user}", "branches_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/branches{/branch}", "tags_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/tags", "blobs_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/git/blobs{/sha}", "git_tags_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/git/tags{/sha}", "git_refs_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/git/refs{/sha}", "trees_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/git/trees{/sha}", "statuses_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/statuses/{sha}", "languages_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/languages", "stargazers_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/stargazers", "contributors_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/contributors", "subscribers_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/subscribers", "subscription_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/subscription", "commits_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/commits{/sha}", "git_commits_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/git/commits{/sha}", "comments_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/comments{/number}", "issue_comment_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/issues/comments{/number}", "contents_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/contents/{+path}", "compare_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/compare/{base}...{head}", "merges_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/merges", "archive_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/{archive_format}{/ref}", "downloads_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/downloads", "issues_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/issues{/number}", "pulls_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/pulls{/number}", "milestones_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/milestones{/number}", "notifications_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/notifications{?since,all,participating}", "labels_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/labels{/name}", "releases_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/releases{/id}", "deployments_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/deployments", "created_at": "2021-08-28T19:50:28Z", "updated_at": "2025-02-18T20:42:29Z", "pushed_at": "2025-02-18T20:42:25Z", "git_url": "git://github.com/simonw/datasette-plugin-template-repository.git", "ssh_url": "git@github.com:simonw/datasette-plugin-template-repository.git", "clone_url": "https://github.com/simonw/datasette-plugin-template-repository.git", "svn_url": "https://github.com/simonw/datasette-plugin-template-repository", "homepage": "", "size": 24, "stargazers_count": 24, "watchers_count": 24, "language": null, "has_issues": true, "has_projects": true, "has_downloads": true, "has_wiki": true, "has_pages": false, "has_discussions": false, "forks_count": 0, "mirror_url": null, "archived": false, "disabled": false, "open_issues_count": 0, "license": null, "allow_forking": true, "is_template": true, "web_commit_signoff_required": false, "topics": [], "visibility": "public", "forks": 0, "open_issues": 0, "watchers": 24, "default_branch": "main", "permissions": {"admin": false, "maintain": false, "push": false, "triage": false, "pull": false}, "temp_clone_token": ""}

readme:
  # datasette-render-image-tags

  [![PyPI](https://img.shields.io/pypi/v/datasette-render-image-tags.svg)](https://pypi.org/project/datasette-render-image-tags/)
  [![Changelog](https://img.shields.io/github/v/release/simonw/datasette-render-image-tags?include_prereleases&label=changelog)](https://github.com/simonw/datasette-render-image-tags/releases)
  [![Tests](https://github.com/simonw/datasette-render-image-tags/workflows/Test/badge.svg)](https://github.com/simonw/datasette-render-image-tags/actions?query=workflow%3ATest)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/datasette-render-image-tags/blob/main/LICENSE)

  Turn any URLs ending in .jpg/.png/.gif into img tags with width 200

  ## Installation

  Install this plugin in the same environment as Datasette.

      datasette install datasette-render-image-tags

  ## Usage

  Once installed, any cells contaning a URL that ends with `.png` or `.jpg` or `.jpeg` or `.gif` will be rendered using an image tag, with a width of 200px.

  ## Development

  To set up this plugin locally, first checkout the code. Then create a new virtual environment:

      cd datasette-render-image-tags
      python3 -m venv venv
      source venv/bin/activate

  Now install the dependencies and test dependencies:

      pip install -e '.[test]'

  To run the tests:

      pytest

readme_html:
  <div id="readme" class="md" data-path="README.md"><article class="markdown-body entry-content container-lg" itemprop="text"><div class="markdown-heading" dir="auto"><h1 class="heading-element" dir="auto">datasette-render-image-tags</h1><a id="user-content-datasette-render-image-tags" class="anchor" aria-label="Permalink: datasette-render-image-tags" href="#datasette-render-image-tags"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto"><a href="https://pypi.org/project/datasette-render-image-tags/" rel="nofollow"><img src="https://img.shields.io/pypi/v/datasette-render-image-tags.svg" alt="PyPI" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/b43174a134cd9482d7e156864975d4e903b6267661b39b5f78a3f619eaeea432/68747470733a2f2f696d672e736869656c64732e696f2f707970692f762f6461746173657474652d72656e6465722d696d6167652d746167732e737667"></a>
  <a href="https://github.com/simonw/datasette-render-image-tags/releases"><img src="https://img.shields.io/github/v/release/simonw/datasette-render-image-tags?include_prereleases&label=changelog" alt="Changelog" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/a67869047595092c9c2b30b425b7cd5ca5bbb8fdd1b7404d63a53e9c68c402c0/68747470733a2f2f696d672e736869656c64732e696f2f6769746875622f762f72656c656173652f73696d6f6e772f6461746173657474652d72656e6465722d696d6167652d746167733f696e636c7564655f70726572656c6561736573266c6162656c3d6368616e67656c6f67"></a>
  <a href="https://github.com/simonw/datasette-render-image-tags/actions?query=workflow%3ATest"><img src="https://github.com/simonw/datasette-render-image-tags/workflows/Test/badge.svg" alt="Tests" style="max-width: 100%;"></a>
  <a href="https://github.com/simonw/datasette-render-image-tags/blob/main/LICENSE"><img src="https://img.shields.io/badge/license-Apache%202.0-blue.svg" alt="License" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/c355f200ea90fddaa407b6eaab303663a669248ea3ca7b1fcf77dbe04ff5f48c/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6c6963656e73652d417061636865253230322e302d626c75652e737667"></a></p>
  <p dir="auto">Turn any URLs ending in .jpg/.png/.gif into img tags with width 200</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Installation</h2><a id="user-content-installation" class="anchor" aria-label="Permalink: Installation" href="#installation"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">Install this plugin in the same environment as Datasette.</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="datasette install datasette-render-image-tags"><pre class="notranslate"><code>datasette install datasette-render-image-tags
  </code></pre></div>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Usage</h2><a id="user-content-usage" class="anchor" aria-label="Permalink: Usage" href="#usage"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">Once installed, any cells contaning a URL that ends with <code>.png</code> or <code>.jpg</code> or <code>.jpeg</code> or <code>.gif</code> will be rendered using an image tag, with a width of 200px.</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Development</h2><a id="user-content-development" class="anchor" aria-label="Permalink: Development" href="#development"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">To set up this plugin locally, first checkout the code. Then create a new virtual environment:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="cd datasette-render-image-tags
  python3 -m venv venv
  source venv/bin/activate"><pre class="notranslate"><code>cd datasette-render-image-tags
  python3 -m venv venv
  source venv/bin/activate
  </code></pre></div>
  <p dir="auto">Now install the dependencies and test dependencies:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="pip install -e '.[test]'"><pre class="notranslate"><code>pip install -e '.[test]'
  </code></pre></div>
  <p dir="auto">To run the tests:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="pytest"><pre class="notranslate"><code>pytest
  </code></pre></div>
  </article></div>


node_id: R_kgDOH99rFw
name: datasette-sandstorm-support
full_name: simonw/datasette-sandstorm-support
html_url: https://github.com/simonw/datasette-sandstorm-support
description: Authentication and permissions for Datasette on Sandstorm
created_at: 2022-09-09T17:11:04Z
updated_at: 2022-09-20T07:41:41Z
pushed_at: 2022-09-16T17:14:12Z
language: Python
license: apache-2.0
topics: []
visibility: public
default_branch: main
permissions: {"admin": false, "maintain": false, "push": false, "triage": false, "pull": true}
template_repository:
[cell repeats an earlier one in this file: 6377 characters]

readme:
  # datasette-sandstorm-support

  [![PyPI](https://img.shields.io/pypi/v/datasette-sandstorm-support.svg)](https://pypi.org/project/datasette-sandstorm-support/)
  [![Changelog](https://img.shields.io/github/v/release/simonw/datasette-sandstorm-support?include_prereleases&label=changelog)](https://github.com/simonw/datasette-sandstorm-support/releases)
  [![Tests](https://github.com/simonw/datasette-sandstorm-support/workflows/Test/badge.svg)](https://github.com/simonw/datasette-sandstorm-support/actions?query=workflow%3ATest)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/datasette-sandstorm-support/blob/main/LICENSE)

  Authentication and permissions for Datasette on Sandstorm

  ## Installation

  Install this plugin in the same environment as Datasette.

      datasette install datasette-sandstorm-support

  ## Usage

  This plugin is part of [datasette-sandstorm](https://github.com/ocdtrekkie/datasette-sandstorm).

  ## Development

  To set up this plugin locally, first checkout the code. Then create a new virtual environment:

      cd datasette-sandstorm-support
      python3 -m venv venv
      source venv/bin/activate

  Now install the dependencies and test dependencies:

      pip install -e '.[test]'

  To run the tests:

      pytest

readme_html:
  <div id="readme" class="md" data-path="README.md"><article class="markdown-body entry-content container-lg" itemprop="text"><div class="markdown-heading" dir="auto"><h1 class="heading-element" dir="auto">datasette-sandstorm-support</h1><a id="user-content-datasette-sandstorm-support" class="anchor" aria-label="Permalink: datasette-sandstorm-support" href="#datasette-sandstorm-support"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto"><a href="https://pypi.org/project/datasette-sandstorm-support/" rel="nofollow"><img src="https://img.shields.io/pypi/v/datasette-sandstorm-support.svg" alt="PyPI" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/3b6e677dbb27de7f9d46cac481286ebaaeb15ea4d345a944726c123c35985b75/68747470733a2f2f696d672e736869656c64732e696f2f707970692f762f6461746173657474652d73616e6473746f726d2d737570706f72742e737667"></a>
  <a href="https://github.com/simonw/datasette-sandstorm-support/releases"><img src="https://img.shields.io/github/v/release/simonw/datasette-sandstorm-support?include_prereleases&label=changelog" alt="Changelog" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/7db14d0d799f7f3b490d1a66d5be20a04b72c4274adeed8a3ca9c9127b3707d7/68747470733a2f2f696d672e736869656c64732e696f2f6769746875622f762f72656c656173652f73696d6f6e772f6461746173657474652d73616e6473746f726d2d737570706f72743f696e636c7564655f70726572656c6561736573266c6162656c3d6368616e67656c6f67"></a>
  <a href="https://github.com/simonw/datasette-sandstorm-support/actions?query=workflow%3ATest"><img src="https://github.com/simonw/datasette-sandstorm-support/workflows/Test/badge.svg" alt="Tests" style="max-width: 100%;"></a>
  <a href="https://github.com/simonw/datasette-sandstorm-support/blob/main/LICENSE"><img src="https://img.shields.io/badge/license-Apache%202.0-blue.svg" alt="License" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/c355f200ea90fddaa407b6eaab303663a669248ea3ca7b1fcf77dbe04ff5f48c/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6c6963656e73652d417061636865253230322e302d626c75652e737667"></a></p>
  <p dir="auto">Authentication and permissions for Datasette on Sandstorm</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Installation</h2><a id="user-content-installation" class="anchor" aria-label="Permalink: Installation" href="#installation"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">Install this plugin in the same environment as Datasette.</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="datasette install datasette-sandstorm-support"><pre class="notranslate"><code>datasette install datasette-sandstorm-support
  </code></pre></div>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Usage</h2><a id="user-content-usage" class="anchor" aria-label="Permalink: Usage" href="#usage"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">This plugin is part of <a href="https://github.com/ocdtrekkie/datasette-sandstorm">datasette-sandstorm</a>.</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Development</h2><a id="user-content-development" class="anchor" aria-label="Permalink: Development" href="#development"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">To set up this plugin locally, first checkout the code. Then create a new virtual environment:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="cd datasette-sandstorm-support
  python3 -m venv venv
  source venv/bin/activate"><pre class="notranslate"><code>cd datasette-sandstorm-support
  python3 -m venv venv
  source venv/bin/activate
  </code></pre></div>
  <p dir="auto">Now install the dependencies and test dependencies:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="pip install -e '.[test]'"><pre class="notranslate"><code>pip install -e '.[test]'
  </code></pre></div>
  <p dir="auto">To run the tests:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="pytest"><pre class="notranslate"><code>pytest
  </code></pre></div>
  </article></div>


node_id: R_kgDOIIe2Hg
name: datasette-public
full_name: datasette/datasette-public
html_url: https://github.com/datasette/datasette-public
description: Make selected Datasette databases and tables visible to the public
created_at: 2022-10-05T00:03:28Z
updated_at: 2026-04-17T04:28:32Z
pushed_at: 2026-04-17T04:30:22Z
language: Python
license: apache-2.0
topics: []
visibility: public
default_branch: main
permissions: {"admin": false, "maintain": false, "push": false, "triage": false, "pull": false}
custom_properties: {}
pull_request_creation_policy: all
readme:
  # datasette-public

  [![PyPI](https://img.shields.io/pypi/v/datasette-public.svg)](https://pypi.org/project/datasette-public/)
  [![Changelog](https://img.shields.io/github/v/release/datasette/datasette-public?include_prereleases&label=changelog)](https://github.com/datasette/datasette-public/releases)
  [![Tests](https://github.com/datasette/datasette-public/workflows/Test/badge.svg)](https://github.com/datasette/datasette-public/actions?query=workflow%3ATest)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/datasette/datasette-public/blob/main/LICENSE)

  Make selected Datasette databases and tables visible to the public

  ## Installation

  Install this plugin in the same environment as Datasette.
  ```bash
  datasette install datasette-public
  ```
  ## Usage

  This plugin can only be used with Datasette 1.0a+ and requires Datasette to be run with a persistent internal database:

  ```bash
  datasette --internal internal.db data.db
  ```
  To grant `datasette-public` permission to the root user run the following:

  ```bash
  datasette --internal internal.db data.db --root \
    -s permissions.datasette-public.id root
  ```

  New database and table action menu items allow users with the `datasette-public` permission to toggle both tables and databases between public and private.

  For databases, users can also select if the ability to execute arbitrary SQL should be exposed to the public.

  If a table is public but the database is private, users will not we able to use the `?_where=` parameter on that table.

  The interfaces for managing the visibility of databases and tables include an audit log of changes that have been made to their public status.

  ## Internals

  This plugin uses three tables in the internal database:

  - `public_databases` - stores the public status of databases and if execute SQL is enabled
  - `public_tables` - stores the public status of tables
  - `public_audit_log` - stores the history of changes to the public status of databases and tables

  ## Development

  To set up this plugin locally, first checkout the code. Then create a new virtual environment:
  ```bash
  cd datasette-public
  python -m venv venv
  source venv/bin/activate
  ```
  Now install the dependencies and test dependencies:
  ```bash
  pip install -e '.[test]'
  ```
  In local development it's useful to run Datasette with everything made private by default:
  ```bash
  datasette data.db \
    --internal internal.db \
    -s allow.id root \
    -s permissions.datasette-public.id root \
    --root \
    --secret fixed \
    --reload
  ```

  To run the tests:
  ```bash
  pytest
  ```

readme_html:
  <div id="readme" class="md" data-path="README.md"><article class="markdown-body entry-content container-lg" itemprop="text"><div class="markdown-heading" dir="auto"><h1 class="heading-element" dir="auto">datasette-public</h1><a id="user-content-datasette-public" class="anchor" aria-label="Permalink: datasette-public" href="#datasette-public"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto"><a href="https://pypi.org/project/datasette-public/" rel="nofollow"><img src="https://img.shields.io/pypi/v/datasette-public.svg" alt="PyPI" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/def9338f60a73b1af5e5d8357d1118fd3b52886039bdc417945721c0d087aaf3/68747470733a2f2f696d672e736869656c64732e696f2f707970692f762f6461746173657474652d7075626c69632e737667"></a>
  <a href="https://github.com/datasette/datasette-public/releases"><img src="https://img.shields.io/github/v/release/datasette/datasette-public?include_prereleases&label=changelog" alt="Changelog" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/7efafea3fda3f7fd0e0c3e6aa7808751e684141da2393b5d0a47aff80bb8d802/68747470733a2f2f696d672e736869656c64732e696f2f6769746875622f762f72656c656173652f6461746173657474652f6461746173657474652d7075626c69633f696e636c7564655f70726572656c6561736573266c6162656c3d6368616e67656c6f67"></a>
  <a href="https://github.com/datasette/datasette-public/actions?query=workflow%3ATest"><img src="https://github.com/datasette/datasette-public/workflows/Test/badge.svg" alt="Tests" style="max-width: 100%;"></a>
  <a href="https://github.com/datasette/datasette-public/blob/main/LICENSE"><img src="https://img.shields.io/badge/license-Apache%202.0-blue.svg" alt="License" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/c355f200ea90fddaa407b6eaab303663a669248ea3ca7b1fcf77dbe04ff5f48c/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6c6963656e73652d417061636865253230322e302d626c75652e737667"></a></p>
  <p dir="auto">Make selected Datasette databases and tables visible to the public</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Installation</h2><a id="user-content-installation" class="anchor" aria-label="Permalink: Installation" href="#installation"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">Install this plugin in the same environment as Datasette.</p>
  <div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="datasette install datasette-public"><pre>datasette install datasette-public</pre></div>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Usage</h2><a id="user-content-usage" class="anchor" aria-label="Permalink: Usage" href="#usage"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">This plugin can only be used with Datasette 1.0a+ and requires Datasette to be run with a persistent internal database:</p>
  <div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="datasette --internal internal.db data.db"><pre>datasette --internal internal.db data.db</pre></div>
  <p dir="auto">To grant <code>datasette-public</code> permission to the root user run the following:</p>
  <div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="datasette --internal internal.db data.db --root \
    -s permissions.datasette-public.id root"><pre>datasette --internal internal.db data.db --root \
    -s permissions.datasette-public.id root</pre></div>
  <p dir="auto">New database and table action menu items allow users with the <code>datasette-public</code> permission to toggle both tables and databases between public and private.</p>
  <p dir="auto">For databases, users can also select if the ability to execute arbitrary SQL should be exposed to the public.</p>
  <p dir="auto">If a table is public but the database is private, users will not we able to use the <code>?_where=</code> parameter on that table.</p>
  <p dir="auto">The interfaces for managing the visibility of databases and tables include an audit log of changes that have been made to their public status.</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Internals</h2><a id="user-content-internals" class="anchor" aria-label="Permalink: Internals" href="#internals"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">This plugin uses three tables in the internal database:</p>
  <ul dir="auto">
  <li><code>public_databases</code> - stores the public status of databases and if execute SQL is enabled</li>
  <li><code>public_tables</code> - stores the public status of tables</li>
  <li><code>public_audit_log</code> - stores the history of changes to the public status of databases and tables</li>
  </ul>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Development</h2><a id="user-content-development" class="anchor" aria-label="Permalink: Development" href="#development"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">To set up this plugin locally, first checkout the code. Then create a new virtual environment:</p>
  <div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="cd datasette-public
  python -m venv venv
  source venv/bin/activate"><pre><span class="pl-c1">cd</span> datasette-public
  python -m venv venv
  <span class="pl-c1">source</span> venv/bin/activate</pre></div>
  <p dir="auto">Now install the dependencies and test dependencies:</p>
  <div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="pip install -e '.[test]'"><pre>pip install -e <span class="pl-s"><span class="pl-pds">'</span>.[test]<span class="pl-pds">'</span></span></pre></div>
  <p dir="auto">In local development it's useful to run Datasette with everything made private by default:</p>
  <div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="datasette data.db \
    --internal internal.db \
    -s allow.id root \
    -s permissions.datasette-public.id root \
    --root \
    --secret fixed \
    --reload"><pre>datasette data.db \
    --internal internal.db \
    -s allow.id root \
    -s permissions.datasette-public.id root \
    --root \
    --secret fixed \
    --reload</pre></div>
  <p dir="auto">To run the tests:</p>
  <div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="pytest"><pre>pytest</pre></div>
  </article></div>


