# repos

node_id: MDEwOlJlcG9zaXRvcnkyNzQyNjQ0ODQ=
name: sqlite-generate
full_name: simonw/sqlite-generate
html_url: https://github.com/simonw/sqlite-generate
description: Tool for generating demo SQLite databases
created_at: 2020-06-22T23:36:44Z
updated_at: 2025-04-13T14:23:55Z
pushed_at: 2021-02-27T15:25:24Z
homepage: https://sqlite-generate-demo.datasette.io/
language: Python
license: apache-2.0
topics: ["datasette-io", "datasette-tool", "sqlite"]
visibility: public
default_branch: main
permissions: {"admin": false, "maintain": false, "push": false, "triage": false, "pull": true}
readme:
  # sqlite-generate

  [![PyPI](https://img.shields.io/pypi/v/sqlite-generate.svg)](https://pypi.org/project/sqlite-generate/)
  [![Changelog](https://img.shields.io/github/v/release/simonw/sqlite-generate?label=changelog)](https://github.com/simonw/sqlite-generate/releases)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/sqlite-generate/blob/master/LICENSE)

  Tool for generating demo SQLite databases

  ## Installation

  Install this plugin using `pip`:

      $ pip install sqlite-generate

  ## Demo

  You can see a demo of the database generated using this command running in [Datasette](https://github.com/simonw/datasette) at https://sqlite-generate-demo.datasette.io/

  The demo is generated using the following command:

      sqlite-generate demo.db --seed seed --fts --columns=10 --fks=0,3 --pks=0,2

  ## Usage

  To generate a SQLite database file called `data.db` with 10 randomly named tables in it, run the following:

      sqlite-generate data.db

  You can use the `--tables` option to generate a different number of tables:

      sqlite-generate data.db --tables 20

  You can run the command against the same database file multiple times to keep adding new tables, using different settings for each batch of generated tables.

  By default each table will contain a random number of rows between 0 and 200. You can customize this with the `--rows` option:

      sqlite-generate data.db --rows 20

  This will insert 20 rows into each table.

      sqlite-generate data.db --rows 500,2000

  This inserts a random number of rows between 500 and 2000 into each table.

  Each table will have 5 columns. You can change this using `--columns`:

      sqlite-generate data.db --columns 10

  `--columns` can also accept a range:

      sqlite-generate data.db --columns 5,15

  You can control the random number seed used with the `--seed` option. This will result in the exact same database file being created by multiple runs of the tool:

      sqlite-generate data.db --seed=myseed

  By default each table will contain between 0 and 2 foreign key columns to other tables. You can control this using the `--fks` option, with either a single number or a range:

      sqlite-generate data.db --columns=20 --fks=5,15

  Each table will have a single primary key column called `id`. You can use the `--pks=` option to change the number of primary key columns on each table. Drop it to 0 to generate [rowid tables](https://www.sqlite.org/rowidtable.html). Increase it above 1 to generate tables with compound primary keys. Or use a range to get a random selection of different primary key layouts:

      sqlite-generate data.db --pks=0,2

  To configure [SQLite full-text search](https://www.sqlite.org/fts5.html) for all columns of type text, use `--fts`:

      sqlite-generate data.db --fts

  This will use FTS5 by default. To use [FTS4](https://www.sqlite.org/fts3.html) instead, use `--fts4`.

  ## Development

  To contribute to this tool, first checkout the code. Then create a new virtual environment:

      cd sqlite-generate
      python -mvenv venv
      source venv/bin/activate

  Or if you are using `pipenv`:

      pipenv shell

  Now install the dependencies and tests:

      pip install -e '.[test]'

  To run the tests:

      pytest

readme_html:
  <div id="readme" class="md" data-path="README.md"><article class="markdown-body entry-content container-lg" itemprop="text"><div class="markdown-heading" dir="auto"><h1 class="heading-element" dir="auto">sqlite-generate</h1><a id="user-content-sqlite-generate" class="anchor" aria-label="Permalink: sqlite-generate" href="#sqlite-generate"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto"><a href="https://pypi.org/project/sqlite-generate/" rel="nofollow"><img src="https://img.shields.io/pypi/v/sqlite-generate.svg" alt="PyPI" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/953c31d7a2678ffbaf79ad0d17856c6dc79fd8d0fc33b2338a2cc8844f1e41fc/68747470733a2f2f696d672e736869656c64732e696f2f707970692f762f73716c6974652d67656e65726174652e737667"></a>
  <a href="https://github.com/simonw/sqlite-generate/releases"><img src="https://img.shields.io/github/v/release/simonw/sqlite-generate?label=changelog" alt="Changelog" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/f1e22813ca5bf3559ace213b8ffa730e7a7eac15fffa580ea6ceb26538d95ca1/68747470733a2f2f696d672e736869656c64732e696f2f6769746875622f762f72656c656173652f73696d6f6e772f73716c6974652d67656e65726174653f6c6162656c3d6368616e67656c6f67"></a>
  <a href="https://github.com/simonw/sqlite-generate/blob/master/LICENSE"><img src="https://img.shields.io/badge/license-Apache%202.0-blue.svg" alt="License" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/c355f200ea90fddaa407b6eaab303663a669248ea3ca7b1fcf77dbe04ff5f48c/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6c6963656e73652d417061636865253230322e302d626c75652e737667"></a></p>
  <p dir="auto">Tool for generating demo SQLite databases</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Installation</h2><a id="user-content-installation" class="anchor" aria-label="Permalink: Installation" href="#installation"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">Install this plugin using <code>pip</code>:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ pip install sqlite-generate"><pre class="notranslate"><code>$ pip install sqlite-generate
  </code></pre></div>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Demo</h2><a id="user-content-demo" class="anchor" aria-label="Permalink: Demo" href="#demo"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">You can see a demo of the database generated using this command running in <a href="https://github.com/simonw/datasette">Datasette</a> at <a href="https://sqlite-generate-demo.datasette.io/" rel="nofollow">https://sqlite-generate-demo.datasette.io/</a></p>
  <p dir="auto">The demo is generated using the following command:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="sqlite-generate demo.db --seed seed --fts --columns=10 --fks=0,3 --pks=0,2"><pre class="notranslate"><code>sqlite-generate demo.db --seed seed --fts --columns=10 --fks=0,3 --pks=0,2
  </code></pre></div>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Usage</h2><a id="user-content-usage" class="anchor" aria-label="Permalink: Usage" href="#usage"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">To generate a SQLite database file called <code>data.db</code> with 10 randomly named tables in it, run the following:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="sqlite-generate data.db"><pre class="notranslate"><code>sqlite-generate data.db
  </code></pre></div>
  <p dir="auto">You can use the <code>--tables</code> option to generate a different number of tables:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="sqlite-generate data.db --tables 20"><pre class="notranslate"><code>sqlite-generate data.db --tables 20
  </code></pre></div>
  <p dir="auto">You can run the command against the same database file multiple times to keep adding new tables, using different settings for each batch of generated tables.</p>
  <p dir="auto">By default each table will contain a random number of rows between 0 and 200. You can customize this with the <code>--rows</code> option:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="sqlite-generate data.db --rows 20"><pre class="notranslate"><code>sqlite-generate data.db --rows 20
  </code></pre></div>
  <p dir="auto">This will insert 20 rows into each table.</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="sqlite-generate data.db --rows 500,2000"><pre class="notranslate"><code>sqlite-generate data.db --rows 500,2000
  </code></pre></div>
  <p dir="auto">This inserts a random number of rows between 500 and 2000 into each table.</p>
  <p dir="auto">Each table will have 5 columns. You can change this using <code>--columns</code>:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="sqlite-generate data.db --columns 10"><pre class="notranslate"><code>sqlite-generate data.db --columns 10
  </code></pre></div>
  <p dir="auto"><code>--columns</code> can also accept a range:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="sqlite-generate data.db --columns 5,15"><pre class="notranslate"><code>sqlite-generate data.db --columns 5,15
  </code></pre></div>
  <p dir="auto">You can control the random number seed used with the <code>--seed</code> option. This will result in the exact same database file being created by multiple runs of the tool:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="sqlite-generate data.db --seed=myseed"><pre class="notranslate"><code>sqlite-generate data.db --seed=myseed
  </code></pre></div>
  <p dir="auto">By default each table will contain between 0 and 2 foreign key columns to other tables. You can control this using the <code>--fks</code> option, with either a single number or a range:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="sqlite-generate data.db --columns=20 --fks=5,15"><pre class="notranslate"><code>sqlite-generate data.db --columns=20 --fks=5,15
  </code></pre></div>
  <p dir="auto">Each table will have a single primary key column called <code>id</code>. You can use the <code>--pks=</code> option to change the number of primary key columns on each table. Drop it to 0 to generate <a href="https://www.sqlite.org/rowidtable.html" rel="nofollow">rowid tables</a>. Increase it above 1 to generate tables with compound primary keys. Or use a range to get a random selection of different primary key layouts:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="sqlite-generate data.db --pks=0,2"><pre class="notranslate"><code>sqlite-generate data.db --pks=0,2
  </code></pre></div>
  <p dir="auto">To configure <a href="https://www.sqlite.org/fts5.html" rel="nofollow">SQLite full-text search</a> for all columns of type text, use <code>--fts</code>:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="sqlite-generate data.db --fts"><pre class="notranslate"><code>sqlite-generate data.db --fts
  </code></pre></div>
  <p dir="auto">This will use FTS5 by default. To use <a href="https://www.sqlite.org/fts3.html" rel="nofollow">FTS4</a> instead, use <code>--fts4</code>.</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Development</h2><a id="user-content-development" class="anchor" aria-label="Permalink: Development" href="#development"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">To contribute to this tool, first checkout the code. Then create a new virtual environment:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="cd sqlite-generate
  python -mvenv venv
  source venv/bin/activate"><pre class="notranslate"><code>cd sqlite-generate
  python -mvenv venv
  source venv/bin/activate
  </code></pre></div>
  <p dir="auto">Or if you are using <code>pipenv</code>:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="pipenv shell"><pre class="notranslate"><code>pipenv shell
  </code></pre></div>
  <p dir="auto">Now install the dependencies and tests:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="pip install -e '.[test]'"><pre class="notranslate"><code>pip install -e '.[test]'
  </code></pre></div>
  <p dir="auto">To run the tests:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="pytest"><pre class="notranslate"><code>pytest
  </code></pre></div>
  </article></div>


node_id: MDEwOlJlcG9zaXRvcnkyNzQyOTM1OTc=
name: datasette-block-robots
full_name: simonw/datasette-block-robots
html_url: https://github.com/simonw/datasette-block-robots
description: Datasette plugin that blocks robots and crawlers using robots.txt
created_at: 2020-06-23T02:52:23Z
updated_at: 2025-02-28T21:23:38Z
pushed_at: 2022-08-30T16:25:38Z
homepage: https://datasette.io/plugins/datasette-block-robots
language: Python
topics: ["datasette", "datasette-io", "datasette-plugin", "robots-txt"]
visibility: public
default_branch: main
permissions: {"admin": false, "maintain": false, "push": false, "triage": false, "pull": true}
readme:
  # datasette-block-robots

  [![PyPI](https://img.shields.io/pypi/v/datasette-block-robots.svg)](https://pypi.org/project/datasette-block-robots/)
  [![Changelog](https://img.shields.io/github/v/release/simonw/datasette-block-robots?label=changelog)](https://github.com/simonw/datasette-block-robots/releases)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/datasette-block-robots/blob/master/LICENSE)

  Datasette plugin that blocks robots and crawlers using robots.txt

  ## Installation

  Install this plugin in the same environment as Datasette.

      $ pip install datasette-block-robots

  ## Usage

  Having installed the plugin, `/robots.txt` on your Datasette instance will return the following:

      User-agent: *
      Disallow: /

  This will request all robots and crawlers not to visit any of the pages on your site.

  Here's a demo of the plugin in action: https://sqlite-generate-demo.datasette.io/robots.txt

  ## Configuration

  By default the plugin will block all access to the site, using `Disallow: /`.

  If you want the index page to be indexed by search engines without crawling the database, table or row pages themselves, you can use the following:

  ```json
  {
      "plugins": {
          "datasette-block-robots": {
              "allow_only_index": true
          }
      }
  }
  ```
  This will return a `/robots.txt` like so:

      User-agent: *
      Disallow: /db1
      Disallow: /db2

  With a `Disallow` line for every attached database.

  To block access to specific areas of the site using custom paths, add this to your `metadata.json` configuration file:

  ```json
  {
      "plugins": {
          "datasette-block-robots": {
              "disallow": ["/mydatabase/mytable"]
          }
      }
  }
  ```
  This will result in a `/robots.txt` that looks like this:

      User-agent: *
      Disallow: /mydatabase/mytable

  Alternatively you can set the full contents of the `robots.txt` file using the `literal` configuration option. Here's how to do that if you are using YAML rather than JSON and have a `metadata.yml` file:

  ```yaml
  plugins:
      datasette-block-robots:
          literal: |-
              User-agent: *
              Disallow: /
              User-agent: Bingbot
              User-agent: Googlebot
              Disallow:
  ```
  This example would block all crawlers with the exception of Googlebot and Bingbot, which are allowed to crawl the entire site.

  ## Extending this with other plugins

  This plugin adds a new [plugin hook](https://docs.datasette.io/en/stable/plugin_hooks.html) to Datasete called `block_robots_extra_lines()` which can be used by other plugins to add their own additional lines to the `robots.txt` file.

  The hook can optionally accept these parameters:

  - `datasette`: The current [Datasette instance](https://docs.datasette.io/en/stable/internals.html#datasette-class). You can use this to execute SQL queries or read plugin configuration settings.
  - `request`: The [Request object](https://docs.datasette.io/en/stable/internals.html#request-object) representing the incoming request to `/robots.txt`.

  The hook should return a list of strings, each representing a line to be added to the `robots.txt` file.

  It can also return an `async def` function, which will be awaited and used to generate a list of lines. Use this option if you need to make `await` calls inside you hook implementation.

  This example uses the hook to add a `Sitemap: http://example.com/sitemap.xml` line to the `robots.txt` file:

  ```python
  from datasette import hookimpl

  @hookimpl
  def block_robots_extra_lines(datasette, request):
      return [
          "Sitemap: {}".format(datasette.absolute_url(request, "/sitemap.xml")),
      ]
  ```
  This example blocks access to paths based on a database query:

  ```python
  @hookimpl
  def block_robots_extra_lines(datasette):
      async def inner():
          db = datasette.get_database()
          result = await db.execute("select path from mytable")
          return [
              "Disallow: /{}".format(row["path"]) for row in result
          ]
      return inner
  ```
  [datasette-sitemap](https://datasette.io/plugins/datasette-sitemap) is an example of a plugin that uses this hook.

  ## Development

  To set up this plugin locally, first checkout the code. Then create a new virtual environment:

      cd datasette-block-robots
      python3 -mvenv venv
      source venv/bin/activate

  Or if you are using `pipenv`:

      pipenv shell

  Now install the dependencies and tests:

      pip install -e '.[test]'

  To run the tests:

      pytest

readme_html:
  <div id="readme" class="md" data-path="README.md"><article class="markdown-body entry-content container-lg" itemprop="text"><div class="markdown-heading" dir="auto"><h1 class="heading-element" dir="auto">datasette-block-robots</h1><a id="user-content-datasette-block-robots" class="anchor" aria-label="Permalink: datasette-block-robots" href="#datasette-block-robots"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto"><a href="https://pypi.org/project/datasette-block-robots/" rel="nofollow"><img src="https://img.shields.io/pypi/v/datasette-block-robots.svg" alt="PyPI" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/8efd47a6daf08da79cd6d29e7623bc9a890d8c37d67ac84774a1b07386ea0b03/68747470733a2f2f696d672e736869656c64732e696f2f707970692f762f6461746173657474652d626c6f636b2d726f626f74732e737667"></a>
  <a href="https://github.com/simonw/datasette-block-robots/releases"><img src="https://img.shields.io/github/v/release/simonw/datasette-block-robots?label=changelog" alt="Changelog" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/b71913d9c1c4d99fa2af79ad19cb8f4172022f93ea113c8fe3a8c153bb9f3676/68747470733a2f2f696d672e736869656c64732e696f2f6769746875622f762f72656c656173652f73696d6f6e772f6461746173657474652d626c6f636b2d726f626f74733f6c6162656c3d6368616e67656c6f67"></a>
  <a href="https://github.com/simonw/datasette-block-robots/blob/master/LICENSE"><img src="https://img.shields.io/badge/license-Apache%202.0-blue.svg" alt="License" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/c355f200ea90fddaa407b6eaab303663a669248ea3ca7b1fcf77dbe04ff5f48c/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6c6963656e73652d417061636865253230322e302d626c75652e737667"></a></p>
  <p dir="auto">Datasette plugin that blocks robots and crawlers using robots.txt</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Installation</h2><a id="user-content-installation" class="anchor" aria-label="Permalink: Installation" href="#installation"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">Install this plugin in the same environment as Datasette.</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ pip install datasette-block-robots"><pre class="notranslate"><code>$ pip install datasette-block-robots
  </code></pre></div>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Usage</h2><a id="user-content-usage" class="anchor" aria-label="Permalink: Usage" href="#usage"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">Having installed the plugin, <code>/robots.txt</code> on your Datasette instance will return the following:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="User-agent: *
  Disallow: /"><pre class="notranslate"><code>User-agent: *
  Disallow: /
  </code></pre></div>
  <p dir="auto">This will request all robots and crawlers not to visit any of the pages on your site.</p>
  <p dir="auto">Here's a demo of the plugin in action: <a href="https://sqlite-generate-demo.datasette.io/robots.txt" rel="nofollow">https://sqlite-generate-demo.datasette.io/robots.txt</a></p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Configuration</h2><a id="user-content-configuration" class="anchor" aria-label="Permalink: Configuration" href="#configuration"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">By default the plugin will block all access to the site, using <code>Disallow: /</code>.</p>
  <p dir="auto">If you want the index page to be indexed by search engines without crawling the database, table or row pages themselves, you can use the following:</p>
  <div class="highlight highlight-source-json notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="{
      "plugins": {
          "datasette-block-robots": {
              "allow_only_index": true
          }
      }
  }"><pre>{
      <span class="pl-ent">"plugins"</span>: {
          <span class="pl-ent">"datasette-block-robots"</span>: {
              <span class="pl-ent">"allow_only_index"</span>: <span class="pl-c1">true</span>
          }
      }
  }</pre></div>
  <p dir="auto">This will return a <code>/robots.txt</code> like so:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="User-agent: *
  Disallow: /db1
  Disallow: /db2"><pre class="notranslate"><code>User-agent: *
  Disallow: /db1
  Disallow: /db2
  </code></pre></div>
  <p dir="auto">With a <code>Disallow</code> line for every attached database.</p>
  <p dir="auto">To block access to specific areas of the site using custom paths, add this to your <code>metadata.json</code> configuration file:</p>
  <div class="highlight highlight-source-json notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="{
      "plugins": {
          "datasette-block-robots": {
              "disallow": ["/mydatabase/mytable"]
          }
      }
  }"><pre>{
      <span class="pl-ent">"plugins"</span>: {
          <span class="pl-ent">"datasette-block-robots"</span>: {
              <span class="pl-ent">"disallow"</span>: [<span class="pl-s"><span class="pl-pds">"</span>/mydatabase/mytable<span class="pl-pds">"</span></span>]
          }
      }
  }</pre></div>
  <p dir="auto">This will result in a <code>/robots.txt</code> that looks like this:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="User-agent: *
  Disallow: /mydatabase/mytable"><pre class="notranslate"><code>User-agent: *
  Disallow: /mydatabase/mytable
  </code></pre></div>
  <p dir="auto">Alternatively you can set the full contents of the <code>robots.txt</code> file using the <code>literal</code> configuration option. Here's how to do that if you are using YAML rather than JSON and have a <code>metadata.yml</code> file:</p>
  <div class="highlight highlight-source-yaml notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="plugins:
      datasette-block-robots:
          literal: |-
              User-agent: *
              Disallow: /
              User-agent: Bingbot
              User-agent: Googlebot
              Disallow:"><pre><span class="pl-ent">plugins</span>:
      <span class="pl-ent">datasette-block-robots</span>:
          <span class="pl-ent">literal</span>: <span class="pl-s">|-</span>
  <span class="pl-s">            User-agent: *</span>
  <span class="pl-s">            Disallow: /</span>
  <span class="pl-s">            User-agent: Bingbot</span>
  <span class="pl-s">            User-agent: Googlebot</span>
  <span class="pl-s">            Disallow:</span></pre></div>
  <p dir="auto">This example would block all crawlers with the exception of Googlebot and Bingbot, which are allowed to crawl the entire site.</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Extending this with other plugins</h2><a id="user-content-extending-this-with-other-plugins" class="anchor" aria-label="Permalink: Extending this with other plugins" href="#extending-this-with-other-plugins"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">This plugin adds a new <a href="https://docs.datasette.io/en/stable/plugin_hooks.html" rel="nofollow">plugin hook</a> to Datasete called <code>block_robots_extra_lines()</code> which can be used by other plugins to add their own additional lines to the <code>robots.txt</code> file.</p>
  <p dir="auto">The hook can optionally accept these parameters:</p>
  <ul dir="auto">
  <li><code>datasette</code>: The current <a href="https://docs.datasette.io/en/stable/internals.html#datasette-class" rel="nofollow">Datasette instance</a>. You can use this to execute SQL queries or read plugin configuration settings.</li>
  <li><code>request</code>: The <a href="https://docs.datasette.io/en/stable/internals.html#request-object" rel="nofollow">Request object</a> representing the incoming request to <code>/robots.txt</code>.</li>
  </ul>
  <p dir="auto">The hook should return a list of strings, each representing a line to be added to the <code>robots.txt</code> file.</p>
  <p dir="auto">It can also return an <code>async def</code> function, which will be awaited and used to generate a list of lines. Use this option if you need to make <code>await</code> calls inside you hook implementation.</p>
  <p dir="auto">This example uses the hook to add a <code>Sitemap: http://example.com/sitemap.xml</code> line to the <code>robots.txt</code> file:</p>
  <div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="from datasette import hookimpl

  @hookimpl
  def block_robots_extra_lines(datasette, request):
      return [
          "Sitemap: {}".format(datasette.absolute_url(request, "/sitemap.xml")),
      ]"><pre><span class="pl-k">from</span> <span class="pl-s1">datasette</span> <span class="pl-k">import</span> <span class="pl-s1">hookimpl</span>

  <span class="pl-en">@<span class="pl-s1">hookimpl</span></span>
  <span class="pl-k">def</span> <span class="pl-en">block_robots_extra_lines</span>(<span class="pl-s1">datasette</span>, <span class="pl-s1">request</span>):
      <span class="pl-k">return</span> [
          <span class="pl-s">"Sitemap: {}"</span>.<span class="pl-c1">format</span>(<span class="pl-s1">datasette</span>.<span class="pl-c1">absolute_url</span>(<span class="pl-s1">request</span>, <span class="pl-s">"/sitemap.xml"</span>)),
      ]</pre></div>
  <p dir="auto">This example blocks access to paths based on a database query:</p>
  <div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="@hookimpl
  def block_robots_extra_lines(datasette):
      async def inner():
          db = datasette.get_database()
          result = await db.execute("select path from mytable")
          return [
              "Disallow: /{}".format(row["path"]) for row in result
          ]
      return inner"><pre><span class="pl-en">@<span class="pl-s1">hookimpl</span></span>
  <span class="pl-k">def</span> <span class="pl-en">block_robots_extra_lines</span>(<span class="pl-s1">datasette</span>):
      <span class="pl-k">async</span> <span class="pl-k">def</span> <span class="pl-en">inner</span>():
          <span class="pl-s1">db</span> <span class="pl-c1">=</span> <span class="pl-s1">datasette</span>.<span class="pl-c1">get_database</span>()
          <span class="pl-s1">result</span> <span class="pl-c1">=</span> <span class="pl-k">await</span> <span class="pl-s1">db</span>.<span class="pl-c1">execute</span>(<span class="pl-s">"select path from mytable"</span>)
          <span class="pl-k">return</span> [
              <span class="pl-s">"Disallow: /{}"</span>.<span class="pl-c1">format</span>(<span class="pl-s1">row</span>[<span class="pl-s">"path"</span>]) <span class="pl-k">for</span> <span class="pl-s1">row</span> <span class="pl-c1">in</span> <span class="pl-s1">result</span>
          ]
      <span class="pl-k">return</span> <span class="pl-s1">inner</span></pre></div>
  <p dir="auto"><a href="https://datasette.io/plugins/datasette-sitemap" rel="nofollow">datasette-sitemap</a> is an example of a plugin that uses this hook.</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Development</h2><a id="user-content-development" class="anchor" aria-label="Permalink: Development" href="#development"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">To set up this plugin locally, first checkout the code. Then create a new virtual environment:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="cd datasette-block-robots
  python3 -mvenv venv
  source venv/bin/activate"><pre class="notranslate"><code>cd datasette-block-robots
  python3 -mvenv venv
  source venv/bin/activate
  </code></pre></div>
  <p dir="auto">Or if you are using <code>pipenv</code>:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="pipenv shell"><pre class="notranslate"><code>pipenv shell
  </code></pre></div>
  <p dir="auto">Now install the dependencies and tests:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="pip install -e '.[test]'"><pre class="notranslate"><code>pip install -e '.[test]'
  </code></pre></div>
  <p dir="auto">To run the tests:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="pytest"><pre class="notranslate"><code>pytest
  </code></pre></div>
  </article></div>


node_id: MDEwOlJlcG9zaXRvcnkyNzU2MTU5NDc=
name: datasette-glitch
full_name: simonw/datasette-glitch
html_url: https://github.com/simonw/datasette-glitch
description: Utilities to help run Datasette on Glitch
created_at: 2020-06-28T15:41:25Z
updated_at: 2020-07-01T22:48:35Z
pushed_at: 2020-07-01T22:49:22Z
language: Python
topics: ["datasette", "datasette-io", "datasette-plugin", "glitch"]
visibility: public
default_branch: main
permissions: {"admin": false, "maintain": false, "push": false, "triage": false, "pull": true}
readme:
  # datasette-glitch

  [![PyPI](https://img.shields.io/pypi/v/datasette-glitch.svg)](https://pypi.org/project/datasette-glitch/)
  [![Changelog](https://img.shields.io/github/v/release/simonw/datasette-glitch?label=changelog)](https://github.com/simonw/datasette-glitch/releases)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/datasette-glitch/blob/master/LICENSE)

  Utilities to help run Datasette on Glitch

  ## Installation

  Install this plugin in the same environment as Datasette.

      $ pip install datasette-glitch

  ## Usage

  This plugin outputs a special link which will sign you into Datasette as the root user.

  Click Tools -> Logs in the Glitch editor interface after your app starts to see the link.

readme_html:
  <div id="readme" class="md" data-path="README.md"><article class="markdown-body entry-content container-lg" itemprop="text"><div class="markdown-heading" dir="auto"><h1 class="heading-element" dir="auto">datasette-glitch</h1><a id="user-content-datasette-glitch" class="anchor" aria-label="Permalink: datasette-glitch" href="#datasette-glitch"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto"><a href="https://pypi.org/project/datasette-glitch/" rel="nofollow"><img src="https://img.shields.io/pypi/v/datasette-glitch.svg" alt="PyPI" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/5e39d15b2491512f3d02238a08071bf5b0fa83c9ce8fdf1816a7e06ce7fcc1e3/68747470733a2f2f696d672e736869656c64732e696f2f707970692f762f6461746173657474652d676c697463682e737667"></a>
  <a href="https://github.com/simonw/datasette-glitch/releases"><img src="https://img.shields.io/github/v/release/simonw/datasette-glitch?label=changelog" alt="Changelog" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/506f0d68a9adb34d5bb5cb970426c8069910984923c7edc25a886967f6fac79f/68747470733a2f2f696d672e736869656c64732e696f2f6769746875622f762f72656c656173652f73696d6f6e772f6461746173657474652d676c697463683f6c6162656c3d6368616e67656c6f67"></a>
  <a href="https://github.com/simonw/datasette-glitch/blob/master/LICENSE"><img src="https://img.shields.io/badge/license-Apache%202.0-blue.svg" alt="License" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/c355f200ea90fddaa407b6eaab303663a669248ea3ca7b1fcf77dbe04ff5f48c/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6c6963656e73652d417061636865253230322e302d626c75652e737667"></a></p>
  <p dir="auto">Utilities to help run Datasette on Glitch</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Installation</h2><a id="user-content-installation" class="anchor" aria-label="Permalink: Installation" href="#installation"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">Install this plugin in the same environment as Datasette.</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ pip install datasette-glitch"><pre class="notranslate"><code>$ pip install datasette-glitch
  </code></pre></div>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Usage</h2><a id="user-content-usage" class="anchor" aria-label="Permalink: Usage" href="#usage"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">This plugin outputs a special link which will sign you into Datasette as the root user.</p>
  <p dir="auto">Click Tools -> Logs in the Glitch editor interface after your app starts to see the link.</p>
  </article></div>


node_id: MDEwOlJlcG9zaXRvcnkyNzU2MjQzNDY=
name: datasette-init
full_name: simonw/datasette-init
html_url: https://github.com/simonw/datasette-init
description: Ensure specific tables and views exist on startup
created_at: 2020-06-28T16:26:29Z
updated_at: 2023-03-13T10:42:25Z
pushed_at: 2020-07-01T22:47:09Z
language: Python
topics: []
visibility: public
default_branch: main
permissions: {"admin": false, "maintain": false, "push": false, "triage": false, "pull": true}
readme:
  # datasette-init

  [![PyPI](https://img.shields.io/pypi/v/datasette-init.svg)](https://pypi.org/project/datasette-init/)
  [![Changelog](https://img.shields.io/github/v/release/simonw/datasette-init?label=changelog)](https://github.com/simonw/datasette-init/releases)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/datasette-init/blob/master/LICENSE)

  Ensure specific tables and views exist on startup

  ## Installation

  Install this plugin in the same environment as Datasette.

      $ pip install datasette-init

  ## Usage

  This plugin is configured using `metadata.json` (or `metadata.yaml`). 

  ### Creating tables

  Add a block like this that specifies the tables you would like to ensure exist:

  ```json
  {
    "plugins": {
      "datasette-init": {
        "my_database": {
          "tables": {
            "dogs": {
              "columns": {
                "id": "integer",
                "name": "text",
                "age": "integer",
                "weight": "float"
              },
              "pk": "id"
            }
          }
        }
      }
    }
  }
  ```

  Any tables that do not yet exist will be created when Datasette first starts.

  Valid column types are `"integer"`, `"text"`, `"float"` and `"blob"`.

  The `"pk"` is optional, and is used to define the primary key. To define a compound primary key (across more than one column) use a list of column names here:
  ```json
      "pk": ["id1", "id2"]
  ```

  ### Creating views

  The plugin can also be used to create views:

  ```json
  {
    "plugins": {
      "datasette-init": {
        "my_database": {
          "views": {
            "my_view": "select 1 + 1"
          }
        }
      }
    }
  }
  ```

  Each view in the ``"views"`` block will be created when the Database first starts. If a view with the same name already exists it will be replaced with the new definition.


  ## Development

  To set up this plugin locally, first checkout the code. Then create a new virtual environment:

      cd datasette-init
      python3 -mvenv venv
      source venv/bin/activate

  Or if you are using `pipenv`:

      pipenv shell

  Now install the dependencies and tests:

      pip install -e '.[test]'

  To run the tests:

      pytest

readme_html:
  <div id="readme" class="md" data-path="README.md"><article class="markdown-body entry-content container-lg" itemprop="text"><div class="markdown-heading" dir="auto"><h1 class="heading-element" dir="auto">datasette-init</h1><a id="user-content-datasette-init" class="anchor" aria-label="Permalink: datasette-init" href="#datasette-init"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto"><a href="https://pypi.org/project/datasette-init/" rel="nofollow"><img src="https://img.shields.io/pypi/v/datasette-init.svg" alt="PyPI" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/1f40c9f5e0ef4882356318f8bd0efecbeb650d29e44635046c5d570b76229c4e/68747470733a2f2f696d672e736869656c64732e696f2f707970692f762f6461746173657474652d696e69742e737667"></a>
  <a href="https://github.com/simonw/datasette-init/releases"><img src="https://img.shields.io/github/v/release/simonw/datasette-init?label=changelog" alt="Changelog" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/009aa2787ea61dd787b8e0e84f2d8748df63746c6abef6c8d46daf5508f337d3/68747470733a2f2f696d672e736869656c64732e696f2f6769746875622f762f72656c656173652f73696d6f6e772f6461746173657474652d696e69743f6c6162656c3d6368616e67656c6f67"></a>
  <a href="https://github.com/simonw/datasette-init/blob/master/LICENSE"><img src="https://img.shields.io/badge/license-Apache%202.0-blue.svg" alt="License" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/c355f200ea90fddaa407b6eaab303663a669248ea3ca7b1fcf77dbe04ff5f48c/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6c6963656e73652d417061636865253230322e302d626c75652e737667"></a></p>
  <p dir="auto">Ensure specific tables and views exist on startup</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Installation</h2><a id="user-content-installation" class="anchor" aria-label="Permalink: Installation" href="#installation"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">Install this plugin in the same environment as Datasette.</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ pip install datasette-init"><pre class="notranslate"><code>$ pip install datasette-init
  </code></pre></div>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Usage</h2><a id="user-content-usage" class="anchor" aria-label="Permalink: Usage" href="#usage"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">This plugin is configured using <code>metadata.json</code> (or <code>metadata.yaml</code>).</p>
  <div class="markdown-heading" dir="auto"><h3 class="heading-element" dir="auto">Creating tables</h3><a id="user-content-creating-tables" class="anchor" aria-label="Permalink: Creating tables" href="#creating-tables"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">Add a block like this that specifies the tables you would like to ensure exist:</p>
  <div class="highlight highlight-source-json notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="{
    "plugins": {
      "datasette-init": {
        "my_database": {
          "tables": {
            "dogs": {
              "columns": {
                "id": "integer",
                "name": "text",
                "age": "integer",
                "weight": "float"
              },
              "pk": "id"
            }
          }
        }
      }
    }
  }"><pre>{
    <span class="pl-ent">"plugins"</span>: {
      <span class="pl-ent">"datasette-init"</span>: {
        <span class="pl-ent">"my_database"</span>: {
          <span class="pl-ent">"tables"</span>: {
            <span class="pl-ent">"dogs"</span>: {
              <span class="pl-ent">"columns"</span>: {
                <span class="pl-ent">"id"</span>: <span class="pl-s"><span class="pl-pds">"</span>integer<span class="pl-pds">"</span></span>,
                <span class="pl-ent">"name"</span>: <span class="pl-s"><span class="pl-pds">"</span>text<span class="pl-pds">"</span></span>,
                <span class="pl-ent">"age"</span>: <span class="pl-s"><span class="pl-pds">"</span>integer<span class="pl-pds">"</span></span>,
                <span class="pl-ent">"weight"</span>: <span class="pl-s"><span class="pl-pds">"</span>float<span class="pl-pds">"</span></span>
              },
              <span class="pl-ent">"pk"</span>: <span class="pl-s"><span class="pl-pds">"</span>id<span class="pl-pds">"</span></span>
            }
          }
        }
      }
    }
  }</pre></div>
  <p dir="auto">Any tables that do not yet exist will be created when Datasette first starts.</p>
  <p dir="auto">Valid column types are <code>"integer"</code>, <code>"text"</code>, <code>"float"</code> and <code>"blob"</code>.</p>
  <p dir="auto">The <code>"pk"</code> is optional, and is used to define the primary key. To define a compound primary key (across more than one column) use a list of column names here:</p>
  <div class="highlight highlight-source-json notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="    "pk": ["id1", "id2"]"><pre>    <span class="pl-ent">"pk"</span>: [<span class="pl-s"><span class="pl-pds">"</span>id1<span class="pl-pds">"</span></span>, <span class="pl-s"><span class="pl-pds">"</span>id2<span class="pl-pds">"</span></span>]</pre></div>
  <div class="markdown-heading" dir="auto"><h3 class="heading-element" dir="auto">Creating views</h3><a id="user-content-creating-views" class="anchor" aria-label="Permalink: Creating views" href="#creating-views"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">The plugin can also be used to create views:</p>
  <div class="highlight highlight-source-json notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="{
    "plugins": {
      "datasette-init": {
        "my_database": {
          "views": {
            "my_view": "select 1 + 1"
          }
        }
      }
    }
  }"><pre>{
    <span class="pl-ent">"plugins"</span>: {
      <span class="pl-ent">"datasette-init"</span>: {
        <span class="pl-ent">"my_database"</span>: {
          <span class="pl-ent">"views"</span>: {
            <span class="pl-ent">"my_view"</span>: <span class="pl-s"><span class="pl-pds">"</span>select 1 + 1<span class="pl-pds">"</span></span>
          }
        }
      }
    }
  }</pre></div>
  <p dir="auto">Each view in the <code>"views"</code> block will be created when the Database first starts. If a view with the same name already exists it will be replaced with the new definition.</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Development</h2><a id="user-content-development" class="anchor" aria-label="Permalink: Development" href="#development"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">To set up this plugin locally, first checkout the code. Then create a new virtual environment:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="cd datasette-init
  python3 -mvenv venv
  source venv/bin/activate"><pre class="notranslate"><code>cd datasette-init
  python3 -mvenv venv
  source venv/bin/activate
  </code></pre></div>
  <p dir="auto">Or if you are using <code>pipenv</code>:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="pipenv shell"><pre class="notranslate"><code>pipenv shell
  </code></pre></div>
  <p dir="auto">Now install the dependencies and tests:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="pip install -e '.[test]'"><pre class="notranslate"><code>pip install -e '.[test]'
  </code></pre></div>
  <p dir="auto">To run the tests:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="pytest"><pre class="notranslate"><code>pytest
  </code></pre></div>
  </article></div>


node_id: MDEwOlJlcG9zaXRvcnkyNzU3MTEyNTQ=
name: datasette-write
full_name: simonw/datasette-write
html_url: https://github.com/simonw/datasette-write
description: Datasette plugin providing a UI for executing SQL writes against the database
created_at: 2020-06-29T02:27:31Z
updated_at: 2025-11-11T01:49:49Z
pushed_at: 2025-11-11T01:50:21Z
homepage: https://datasette.io/plugins/datasette-write
language: Python
topics: ["datasette-io", "datasette-plugin"]
visibility: public
default_branch: main
permissions: {"admin": false, "maintain": false, "push": false, "triage": false, "pull": false}
pull_request_creation_policy: all
readme:
  # datasette-write

  [![PyPI](https://img.shields.io/pypi/v/datasette-write.svg)](https://pypi.org/project/datasette-write/)
  [![Changelog](https://img.shields.io/github/v/release/simonw/datasette-write?label=changelog)](https://github.com/simonw/datasette-write/releases)
  [![Tests](https://github.com/simonw/datasette-write/workflows/Test/badge.svg)](https://github.com/simonw/datasette-write/actions?query=workflow%3ATest)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/datasette-write/blob/master/LICENSE)

  Datasette plugin providing a UI for writing to a database

  ## Installation

  Install this plugin in the same environment as Datasette.
  ```bash
  pip install datasette-write
  ```
  ## Usage

  Having installed the plugin, visit `/db/-/write` on your Datasette instance to submit SQL queries that will be executed against a write connection to the specified database.

  By default only the `root` user can access the page - so you'll need to run Datasette with the `--root` option and click on the link shown in the terminal to sign in and access the page.

  The `datasette-write` permission governs access. You can use permission plugins such as [datasette-permissions-sql](https://github.com/simonw/datasette-permissions-sql) to grant additional access to the write interface.

  Pass `?sql=...` in the query string to pre-populate the SQL editor with a query.

  ## Parameterized queries

  SQL queries can include parameters like this:
  ```sql
  insert into news (title, body)
      values (:title, :body_textarea)
  ```
  These will be converted into form fields on the `/db/-/write` page.

  If a parameter name ends with `_textarea` it will be rendered as a multi-line textarea instead of a text input.

  If a parameter name ends with `_hidden` it will be rendered as a hidden input.

  ## Updating rows with SQL

  On Datasette 1.0a13 and higher a row actions menu item will be added to the row page linking to a SQL query for updating that row, for users with the `datasette-write` permission.

  ## Development

  To set up this plugin locally, first checkout the code. Then create a new virtual environment:
  ```bash
  cd datasette-write
  python3 -mvenv venv
  source venv/bin/activate
  ```
  Or if you are using `pipenv`:
  ```bash
  pipenv shell
  ```
  Now install the dependencies and tests:
  ```bash
  pip install -e '.[test]'
  ```
  To run the tests:
  ```bash
  pytest
  ```

readme_html:
  <div id="readme" class="md" data-path="README.md"><article class="markdown-body entry-content container-lg" itemprop="text"><div class="markdown-heading" dir="auto"><h1 class="heading-element" dir="auto">datasette-write</h1><a id="user-content-datasette-write" class="anchor" aria-label="Permalink: datasette-write" href="#datasette-write"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto"><a href="https://pypi.org/project/datasette-write/" rel="nofollow"><img src="https://img.shields.io/pypi/v/datasette-write.svg" alt="PyPI" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/466788ab2a0d2a06a6d66ed747c7c3d2212a9731aed50fb24ea371a3161f79fc/68747470733a2f2f696d672e736869656c64732e696f2f707970692f762f6461746173657474652d77726974652e737667"></a>
  <a href="https://github.com/simonw/datasette-write/releases"><img src="https://img.shields.io/github/v/release/simonw/datasette-write?label=changelog" alt="Changelog" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/151078f37bd49fca1cf1eca6dac061f0be05a0bd6f7664d5955eb90ddc7aadb5/68747470733a2f2f696d672e736869656c64732e696f2f6769746875622f762f72656c656173652f73696d6f6e772f6461746173657474652d77726974653f6c6162656c3d6368616e67656c6f67"></a>
  <a href="https://github.com/simonw/datasette-write/actions?query=workflow%3ATest"><img src="https://github.com/simonw/datasette-write/workflows/Test/badge.svg" alt="Tests" style="max-width: 100%;"></a>
  <a href="https://github.com/simonw/datasette-write/blob/master/LICENSE"><img src="https://img.shields.io/badge/license-Apache%202.0-blue.svg" alt="License" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/c355f200ea90fddaa407b6eaab303663a669248ea3ca7b1fcf77dbe04ff5f48c/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6c6963656e73652d417061636865253230322e302d626c75652e737667"></a></p>
  <p dir="auto">Datasette plugin providing a UI for writing to a database</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Installation</h2><a id="user-content-installation" class="anchor" aria-label="Permalink: Installation" href="#installation"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">Install this plugin in the same environment as Datasette.</p>
  <div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="pip install datasette-write"><pre>pip install datasette-write</pre></div>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Usage</h2><a id="user-content-usage" class="anchor" aria-label="Permalink: Usage" href="#usage"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">Having installed the plugin, visit <code>/db/-/write</code> on your Datasette instance to submit SQL queries that will be executed against a write connection to the specified database.</p>
  <p dir="auto">By default only the <code>root</code> user can access the page - so you'll need to run Datasette with the <code>--root</code> option and click on the link shown in the terminal to sign in and access the page.</p>
  <p dir="auto">The <code>datasette-write</code> permission governs access. You can use permission plugins such as <a href="https://github.com/simonw/datasette-permissions-sql">datasette-permissions-sql</a> to grant additional access to the write interface.</p>
  <p dir="auto">Pass <code>?sql=...</code> in the query string to pre-populate the SQL editor with a query.</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Parameterized queries</h2><a id="user-content-parameterized-queries" class="anchor" aria-label="Permalink: Parameterized queries" href="#parameterized-queries"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">SQL queries can include parameters like this:</p>
  <div class="highlight highlight-source-sql notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="insert into news (title, body)
      values (:title, :body_textarea)"><pre><span class="pl-k">insert into</span> news (title, body)
      <span class="pl-k">values</span> (:title, :body_textarea)</pre></div>
  <p dir="auto">These will be converted into form fields on the <code>/db/-/write</code> page.</p>
  <p dir="auto">If a parameter name ends with <code>_textarea</code> it will be rendered as a multi-line textarea instead of a text input.</p>
  <p dir="auto">If a parameter name ends with <code>_hidden</code> it will be rendered as a hidden input.</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Updating rows with SQL</h2><a id="user-content-updating-rows-with-sql" class="anchor" aria-label="Permalink: Updating rows with SQL" href="#updating-rows-with-sql"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">On Datasette 1.0a13 and higher a row actions menu item will be added to the row page linking to a SQL query for updating that row, for users with the <code>datasette-write</code> permission.</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Development</h2><a id="user-content-development" class="anchor" aria-label="Permalink: Development" href="#development"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">To set up this plugin locally, first checkout the code. Then create a new virtual environment:</p>
  <div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="cd datasette-write
  python3 -mvenv venv
  source venv/bin/activate"><pre><span class="pl-c1">cd</span> datasette-write
  python3 -mvenv venv
  <span class="pl-c1">source</span> venv/bin/activate</pre></div>
  <p dir="auto">Or if you are using <code>pipenv</code>:</p>
  <div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="pipenv shell"><pre>pipenv shell</pre></div>
  <p dir="auto">Now install the dependencies and tests:</p>
  <div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="pip install -e '.[test]'"><pre>pip install -e <span class="pl-s"><span class="pl-pds">'</span>.[test]<span class="pl-pds">'</span></span></pre></div>
  <p dir="auto">To run the tests:</p>
  <div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="pytest"><pre>pytest</pre></div>
  </article></div>


node_id: MDEwOlJlcG9zaXRvcnkyNzkzNTcxMjM=
name: datasette-auth-passwords
full_name: simonw/datasette-auth-passwords
html_url: https://github.com/simonw/datasette-auth-passwords
description: Datasette plugin for authentication using passwords
created_at: 2020-07-13T16:34:39Z
updated_at: 2025-03-19T14:59:43Z
pushed_at: 2024-09-03T17:53:00Z
homepage: https://datasette-auth-passwords-demo.datasette.io
language: Python
topics: ["datasette", "datasette-io", "datasette-plugin"]
visibility: public
default_branch: main
permissions: {"admin": false, "maintain": false, "push": false, "triage": false, "pull": true}
readme:
  # datasette-auth-passwords

  [![PyPI](https://img.shields.io/pypi/v/datasette-auth-passwords.svg)](https://pypi.org/project/datasette-auth-passwords/)
  [![Changelog](https://img.shields.io/github/v/release/simonw/datasette-auth-passwords?label=changelog)](https://github.com/simonw/datasette-auth-passwords/releases)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/datasette-auth-passwords/blob/master/LICENSE)

  Datasette plugin for authenticating access using passwords

  ## Installation

  Install this plugin in the same environment as Datasette.

  ```bash
  datasette install datasette-auth-passwords
  ```

  ## Demo

  A demo of this plugin is running at https://datasette-auth-passwords-demo.datasette.io/

  The demo is configured to show the `public.db` database to everyone, but the `private.db` database only to logged in users.

  You can log in at https://datasette-auth-passwords-demo.datasette.io/-/login with username `root` and password `password!`.

  ## Usage

  This plugin works based on a list of username/password accounts that are hard-coded into the plugin configuration.

  First, you'll need to create a password hash. There are three ways to do that:

  - Install the plugin, then use the interactive tool located at `/-/password-tool`
  - Use the hosted version of that tool at https://datasette-auth-passwords-demo.datasette.io/-/password-tool
  - Use the `datasette hash-password` command, described below

  Now add the following to your `metadata.json`:

  ```json
  {
      "plugins": {
          "datasette-auth-passwords": {
              "someusername_password_hash": {
                  "$env": "PASSWORD_HASH_1"
              }
          }
      }
  }
  ```

  The password hash can now be specified in an environment variable when you run Datasette. You can do that like so:

      PASSWORD_HASH_1='pbkdf2_sha256$...' \
          datasette -m metadata.json

  Be sure to use single quotes here otherwise the `$` symbols in the password hash may be incorrectly interpreted by your shell.

  You will now be able to log in to your instance using the form at `/-/login` with `someusername` as the username and the password that you used to create your hash as the password.

  You can include as many accounts as you like in the configuration, each with different usernames.

  ### datasette hash-password

  The plugin exposes a new CLI command, `datasette hash-password`. You can run this without arguments to interactively create a new password hash:
  ```bash
  datasette hash-password
  ```
  ```
  Password: 
  Repeat for confirmation: 
  pbkdf2_sha256$260000$1513...
  ```
  Or if you want to use it as part of a script, you can add the `--no-confirm` option to generate a hash directly from a value passed to standard input:
  ```bash
  echo 'my password' | datasette hash-password --no-confirm
  ```
  ```
  pbkdf2_sha256$260000$daa...
  ```
  ### Specifying actors

  By default, a logged in user will result in an [actor block](https://datasette.readthedocs.io/en/stable/authentication.html#actors) that just contains their username:

  ```json
  {
      "id": "someusername"
  }
  ```

  You can customize the actor that will be used for a username by including an `"actors"` configuration block, like this:

  ```json
  {
      "plugins": {
          "datasette-auth-passwords": {
              "someusername_password_hash": {
                  "$env": "PASSWORD_HASH_1"
              },
              "actors": {
                  "someusername": {
                      "id": "someusername",
                      "name": "Some user"
                  }
              }
          }
      }
  }
  ```
  ### HTTP Basic authentication option

  This plugin defaults to implementing login using an HTML form that sets a signed authentication cookie.

  You can alternatively configure it to use [HTTP Basic authentication](https://developer.mozilla.org/en-US/docs/Web/HTTP/Authentication#basic_authentication_scheme) instead.

  Do this by adding `"http_basic_auth": true` to the `datasette-auth-passwords` block in your plugin configuration.

  This option introduces the following behaviour:

  - Account usernames and passwords are configured in the same way as form-based authentication
  - Every page within Datasette - even pages that normally do not use authentication, such as static assets - will display a browser login prompt
  - Users will be unable to log out without closing their browser entirely

  There is a demo of this mode at https://datasette-auth-passwords-http-basic-demo.datasette.io/ - sign in with username `root` and password `password!`

  ### Using with datasette publish

  If you are publishing data using a [datasette publish](https://datasette.readthedocs.io/en/stable/publish.html#datasette-publish) command you can use the `--plugin-secret` option to securely configure your password hashes (see [secret configuration values](https://datasette.readthedocs.io/en/stable/plugins.html#secret-configuration-values)).

  You would run the command something like this:

      datasette publish cloudrun mydatabase.db \
          --install datasette-auth-passwords \
          --plugin-secret datasette-auth-passwords root_password_hash 'pbkdf2_sha256$...' \
          --service datasette-auth-passwords-demo

  This will allow you to log in as username `root` using the password that you used to create the hash.

  ## Development

  To set up this plugin locally, first checkout the code. Then create a new virtual environment:

      cd datasette-auth-passwords
      python3 -mvenv venv
      source venv/bin/activate

  Or if you are using `pipenv`:

      pipenv shell

  Now install the dependencies and tests:

      pip install -e '.[test]'

  To run the tests:

      pytest

readme_html:
  <div id="readme" class="md" data-path="README.md"><article class="markdown-body entry-content container-lg" itemprop="text"><div class="markdown-heading" dir="auto"><h1 class="heading-element" dir="auto">datasette-auth-passwords</h1><a id="user-content-datasette-auth-passwords" class="anchor" aria-label="Permalink: datasette-auth-passwords" href="#datasette-auth-passwords"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto"><a href="https://pypi.org/project/datasette-auth-passwords/" rel="nofollow"><img src="https://img.shields.io/pypi/v/datasette-auth-passwords.svg" alt="PyPI" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/f76dc83d569eb67312dcc1fd9e8557149ea1fb7f3800d8cb306b29ad3c5c8dc7/68747470733a2f2f696d672e736869656c64732e696f2f707970692f762f6461746173657474652d617574682d70617373776f7264732e737667"></a>
  <a href="https://github.com/simonw/datasette-auth-passwords/releases"><img src="https://img.shields.io/github/v/release/simonw/datasette-auth-passwords?label=changelog" alt="Changelog" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/f30a5b377cd3ea5de3942ad5588ba4a7b85cdd0901347e8d3238e376a3bb0ab5/68747470733a2f2f696d672e736869656c64732e696f2f6769746875622f762f72656c656173652f73696d6f6e772f6461746173657474652d617574682d70617373776f7264733f6c6162656c3d6368616e67656c6f67"></a>
  <a href="https://github.com/simonw/datasette-auth-passwords/blob/master/LICENSE"><img src="https://img.shields.io/badge/license-Apache%202.0-blue.svg" alt="License" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/c355f200ea90fddaa407b6eaab303663a669248ea3ca7b1fcf77dbe04ff5f48c/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6c6963656e73652d417061636865253230322e302d626c75652e737667"></a></p>
  <p dir="auto">Datasette plugin for authenticating access using passwords</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Installation</h2><a id="user-content-installation" class="anchor" aria-label="Permalink: Installation" href="#installation"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">Install this plugin in the same environment as Datasette.</p>
  <div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="datasette install datasette-auth-passwords"><pre>datasette install datasette-auth-passwords</pre></div>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Demo</h2><a id="user-content-demo" class="anchor" aria-label="Permalink: Demo" href="#demo"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">A demo of this plugin is running at <a href="https://datasette-auth-passwords-demo.datasette.io/" rel="nofollow">https://datasette-auth-passwords-demo.datasette.io/</a></p>
  <p dir="auto">The demo is configured to show the <code>public.db</code> database to everyone, but the <code>private.db</code> database only to logged in users.</p>
  <p dir="auto">You can log in at <a href="https://datasette-auth-passwords-demo.datasette.io/-/login" rel="nofollow">https://datasette-auth-passwords-demo.datasette.io/-/login</a> with username <code>root</code> and password <code>password!</code>.</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Usage</h2><a id="user-content-usage" class="anchor" aria-label="Permalink: Usage" href="#usage"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">This plugin works based on a list of username/password accounts that are hard-coded into the plugin configuration.</p>
  <p dir="auto">First, you'll need to create a password hash. There are three ways to do that:</p>
  <ul dir="auto">
  <li>Install the plugin, then use the interactive tool located at <code>/-/password-tool</code></li>
  <li>Use the hosted version of that tool at <a href="https://datasette-auth-passwords-demo.datasette.io/-/password-tool" rel="nofollow">https://datasette-auth-passwords-demo.datasette.io/-/password-tool</a></li>
  <li>Use the <code>datasette hash-password</code> command, described below</li>
  </ul>
  <p dir="auto">Now add the following to your <code>metadata.json</code>:</p>
  <div class="highlight highlight-source-json notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="{
      "plugins": {
          "datasette-auth-passwords": {
              "someusername_password_hash": {
                  "$env": "PASSWORD_HASH_1"
              }
          }
      }
  }"><pre>{
      <span class="pl-ent">"plugins"</span>: {
          <span class="pl-ent">"datasette-auth-passwords"</span>: {
              <span class="pl-ent">"someusername_password_hash"</span>: {
                  <span class="pl-ent">"$env"</span>: <span class="pl-s"><span class="pl-pds">"</span>PASSWORD_HASH_1<span class="pl-pds">"</span></span>
              }
          }
      }
  }</pre></div>
  <p dir="auto">The password hash can now be specified in an environment variable when you run Datasette. You can do that like so:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="PASSWORD_HASH_1='pbkdf2_sha256$...' \
      datasette -m metadata.json"><pre class="notranslate"><code>PASSWORD_HASH_1='pbkdf2_sha256$...' \
      datasette -m metadata.json
  </code></pre></div>
  <p dir="auto">Be sure to use single quotes here otherwise the <code>$</code> symbols in the password hash may be incorrectly interpreted by your shell.</p>
  <p dir="auto">You will now be able to log in to your instance using the form at <code>/-/login</code> with <code>someusername</code> as the username and the password that you used to create your hash as the password.</p>
  <p dir="auto">You can include as many accounts as you like in the configuration, each with different usernames.</p>
  <div class="markdown-heading" dir="auto"><h3 class="heading-element" dir="auto">datasette hash-password</h3><a id="user-content-datasette-hash-password" class="anchor" aria-label="Permalink: datasette hash-password" href="#datasette-hash-password"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">The plugin exposes a new CLI command, <code>datasette hash-password</code>. You can run this without arguments to interactively create a new password hash:</p>
  <div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="datasette hash-password"><pre>datasette hash-password</pre></div>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="Password: 
  Repeat for confirmation: 
  pbkdf2_sha256$260000$1513..."><pre class="notranslate"><code>Password: 
  Repeat for confirmation: 
  pbkdf2_sha256$260000$1513...
  </code></pre></div>
  <p dir="auto">Or if you want to use it as part of a script, you can add the <code>--no-confirm</code> option to generate a hash directly from a value passed to standard input:</p>
  <div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="echo 'my password' | datasette hash-password --no-confirm"><pre><span class="pl-c1">echo</span> <span class="pl-s"><span class="pl-pds">'</span>my password<span class="pl-pds">'</span></span> <span class="pl-k">|</span> datasette hash-password --no-confirm</pre></div>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="pbkdf2_sha256$260000$daa..."><pre class="notranslate"><code>pbkdf2_sha256$260000$daa...
  </code></pre></div>
  <div class="markdown-heading" dir="auto"><h3 class="heading-element" dir="auto">Specifying actors</h3><a id="user-content-specifying-actors" class="anchor" aria-label="Permalink: Specifying actors" href="#specifying-actors"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">By default, a logged in user will result in an <a href="https://datasette.readthedocs.io/en/stable/authentication.html#actors" rel="nofollow">actor block</a> that just contains their username:</p>
  <div class="highlight highlight-source-json notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="{
      "id": "someusername"
  }"><pre>{
      <span class="pl-ent">"id"</span>: <span class="pl-s"><span class="pl-pds">"</span>someusername<span class="pl-pds">"</span></span>
  }</pre></div>
  <p dir="auto">You can customize the actor that will be used for a username by including an <code>"actors"</code> configuration block, like this:</p>
  <div class="highlight highlight-source-json notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="{
      "plugins": {
          "datasette-auth-passwords": {
              "someusername_password_hash": {
                  "$env": "PASSWORD_HASH_1"
              },
              "actors": {
                  "someusername": {
                      "id": "someusername",
                      "name": "Some user"
                  }
              }
          }
      }
  }"><pre>{
      <span class="pl-ent">"plugins"</span>: {
          <span class="pl-ent">"datasette-auth-passwords"</span>: {
              <span class="pl-ent">"someusername_password_hash"</span>: {
                  <span class="pl-ent">"$env"</span>: <span class="pl-s"><span class="pl-pds">"</span>PASSWORD_HASH_1<span class="pl-pds">"</span></span>
              },
              <span class="pl-ent">"actors"</span>: {
                  <span class="pl-ent">"someusername"</span>: {
                      <span class="pl-ent">"id"</span>: <span class="pl-s"><span class="pl-pds">"</span>someusername<span class="pl-pds">"</span></span>,
                      <span class="pl-ent">"name"</span>: <span class="pl-s"><span class="pl-pds">"</span>Some user<span class="pl-pds">"</span></span>
                  }
              }
          }
      }
  }</pre></div>
  <div class="markdown-heading" dir="auto"><h3 class="heading-element" dir="auto">HTTP Basic authentication option</h3><a id="user-content-http-basic-authentication-option" class="anchor" aria-label="Permalink: HTTP Basic authentication option" href="#http-basic-authentication-option"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">This plugin defaults to implementing login using an HTML form that sets a signed authentication cookie.</p>
  <p dir="auto">You can alternatively configure it to use <a href="https://developer.mozilla.org/en-US/docs/Web/HTTP/Authentication#basic_authentication_scheme" rel="nofollow">HTTP Basic authentication</a> instead.</p>
  <p dir="auto">Do this by adding <code>"http_basic_auth": true</code> to the <code>datasette-auth-passwords</code> block in your plugin configuration.</p>
  <p dir="auto">This option introduces the following behaviour:</p>
  <ul dir="auto">
  <li>Account usernames and passwords are configured in the same way as form-based authentication</li>
  <li>Every page within Datasette - even pages that normally do not use authentication, such as static assets - will display a browser login prompt</li>
  <li>Users will be unable to log out without closing their browser entirely</li>
  </ul>
  <p dir="auto">There is a demo of this mode at <a href="https://datasette-auth-passwords-http-basic-demo.datasette.io/" rel="nofollow">https://datasette-auth-passwords-http-basic-demo.datasette.io/</a> - sign in with username <code>root</code> and password <code>password!</code></p>
  <div class="markdown-heading" dir="auto"><h3 class="heading-element" dir="auto">Using with datasette publish</h3><a id="user-content-using-with-datasette-publish" class="anchor" aria-label="Permalink: Using with datasette publish" href="#using-with-datasette-publish"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">If you are publishing data using a <a href="https://datasette.readthedocs.io/en/stable/publish.html#datasette-publish" rel="nofollow">datasette publish</a> command you can use the <code>--plugin-secret</code> option to securely configure your password hashes (see <a href="https://datasette.readthedocs.io/en/stable/plugins.html#secret-configuration-values" rel="nofollow">secret configuration values</a>).</p>
  <p dir="auto">You would run the command something like this:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="datasette publish cloudrun mydatabase.db \
      --install datasette-auth-passwords \
      --plugin-secret datasette-auth-passwords root_password_hash 'pbkdf2_sha256$...' \
      --service datasette-auth-passwords-demo"><pre class="notranslate"><code>datasette publish cloudrun mydatabase.db \
      --install datasette-auth-passwords \
      --plugin-secret datasette-auth-passwords root_password_hash 'pbkdf2_sha256$...' \
      --service datasette-auth-passwords-demo
  </code></pre></div>
  <p dir="auto">This will allow you to log in as username <code>root</code> using the password that you used to create the hash.</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Development</h2><a id="user-content-development" class="anchor" aria-label="Permalink: Development" href="#development"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">To set up this plugin locally, first checkout the code. Then create a new virtual environment:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="cd datasette-auth-passwords
  python3 -mvenv venv
  source venv/bin/activate"><pre class="notranslate"><code>cd datasette-auth-passwords
  python3 -mvenv venv
  source venv/bin/activate
  </code></pre></div>
  <p dir="auto">Or if you are using <code>pipenv</code>:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="pipenv shell"><pre class="notranslate"><code>pipenv shell
  </code></pre></div>
  <p dir="auto">Now install the dependencies and tests:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="pip install -e '.[test]'"><pre class="notranslate"><code>pip install -e '.[test]'
  </code></pre></div>
  <p dir="auto">To run the tests:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="pytest"><pre class="notranslate"><code>pytest
  </code></pre></div>
  </article></div>


node_id: MDEwOlJlcG9zaXRvcnkyODA1MDAwMjc=
name: datasette-insert
full_name: simonw/datasette-insert
html_url: https://github.com/simonw/datasette-insert
description: Datasette plugin for inserting and updating data
created_at: 2020-07-17T18:40:34Z
updated_at: 2024-09-04T07:37:41Z
pushed_at: 2024-03-29T19:04:49Z
language: Python
topics: ["datasette", "datasette-io", "datasette-plugin"]
visibility: public
default_branch: main
permissions: {"admin": false, "maintain": false, "push": false, "triage": false, "pull": true}
readme:
  # datasette-insert

  [![PyPI](https://img.shields.io/pypi/v/datasette-insert.svg)](https://pypi.org/project/datasette-insert/)
  [![Changelog](https://img.shields.io/github/v/release/simonw/datasette-insert?include_prereleases&label=changelog)](https://github.com/simonw/datasette-insert/releases)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/datasette-insert/blob/master/LICENSE)

  Datasette plugin for inserting and updating data

  **No longer necessary with Datasette 1.0**. Use the [JSON write API](https://docs.datasette.io/en/latest/json_api.html#the-json-write-api) instead.

  ## Installation

  Install this plugin in the same environment as Datasette.

      $ pip install datasette-insert

  This plugin should always be deployed with additional configuration to prevent unauthenticated access, see notes below.

  If you are trying it out on your own local machine, you can `pip install` the [datasette-insert-unsafe](https://github.com/simonw/datasette-insert-unsafe) plugin to allow access without needing to set up authentication or permissions separately.

  ## Inserting data and creating tables

  Start datasette and make sure it has a writable SQLite database attached to it. If you have not yet created a database file you can use this:

      datasette data.db --create

  The `--create` option will create a new empty `data.db` database file if it does not already exist.

  The plugin adds an endpoint that allows data to be inserted or updated and tables to be created by POSTing JSON data to the following URL:

      /-/insert/name-of-database/name-of-table

  The JSON should look like this:

  ```json
  [
      {
          "id": 1,
          "name": "Cleopaws",
          "age": 5
      },
      {
          "id": 2,
          "name": "Pancakes",
          "age": 5
      }
  ]
  ```

  The first time data is posted to the URL a table of that name will be created if it does not aready exist, with the desired columns.

  You can specify which column should be used as the primary key using the `?pk=` URL argument.

  Here's how to POST to a database and create a new table using the Python `requests` library:

  ```python
  import requests

  requests.post("http://localhost:8001/-/insert/data/dogs?pk=id", json=[
      {
          "id": 1,
          "name": "Cleopaws",
          "age": 5
      },
      {
          "id": 2,
          "name": "Pancakes",
          "age": 4
      }
  ])
  ```
  And here's how to do the same thing using `curl`:

  ```
  curl --request POST \
    --data '[
        {
          "id": 1,
          "name": "Cleopaws",
          "age": 5
        },
        {
          "id": 2,
          "name": "Pancakes",
          "age": 4
        }
      ]' \
      'http://localhost:8001/-/insert/data/dogs?pk=id'
  ```
  Or by piping in JSON like so:

      cat dogs.json | curl --request POST -d @- \
          'http://localhost:8001/-/insert/data/dogs?pk=id'

  ### Inserting a single row

  If you are inserting a single row you can optionally send it as a dictionary rather than a list with a single item:

  ```
  curl --request POST \
    --data '{
        "id": 1,
        "name": "Cleopaws",
        "age": 5
      }' \
      'http://localhost:8001/-/insert/data/dogs?pk=id'
  ```

  ### Automatically adding new columns

  If you send data to an existing table with keys that are not reflected by the existing columns, you will get an HTTP 400 error with a JSON response like this:

  ```json
  {
      "status": 400,
      "error": "Unknown keys: 'foo'",
      "error_code": "unknown_keys"
  }
  ```

  If you add `?alter=1` to the URL you are posting to any missing columns will be automatically added:

  ```
  curl --request POST \
    --data '[
        {
          "id": 3,
          "name": "Boris",
          "age": 1,
          "breed": "Husky"
        }
      ]' \
      'http://localhost:8001/-/insert/data/dogs?alter=1'
  ```

  ## Upserting data

  An "upsert" operation can be used to partially update a record. With upserts you can send a subset of the keys and, if the ID matches the specified primary key, they will be used to update an existing record.

  Upserts can be sent to the `/-/upsert` API endpoint.

  This example will update the dog with ID=1's age from 5 to 7:
  ```
  curl --request POST \
    --data '{
        "id": 1,
        "age": 7
      }' \
      'http://localhost:3322/-/upsert/data/dogs?pk=id'
  ```
  Like the `/-/insert` endpoint, the `/-/upsert` endpoint can accept an array of objects too. It also supports the `?alter=1` option.

  ## Permissions and authentication

  This plugin defaults to denying all access, to help ensure people don't accidentally deploy it on the open internet in an unsafe configuration.

  You can read about [Datasette's approach to authentication](https://datasette.readthedocs.io/en/stable/authentication.html) in the Datasette manual.

  You can install the `datasette-insert-unsafe` plugin to run in unsafe mode, where all access is allowed by default.

  I recommend using this plugin in conjunction with [datasette-auth-tokens](https://github.com/simonw/datasette-auth-tokens), which provides a mechanism for making authenticated calls using API tokens.

  You can then use ["allow" blocks](https://datasette.readthedocs.io/en/stable/authentication.html#defining-permissions-with-allow-blocks) in the `datasette-insert` plugin configuration to specify which authenticated tokens are allowed to make use of the API.

  Here's an example `metadata.json` file which restricts access to the `/-/insert` API to an API token defined in an `INSERT_TOKEN` environment variable:

  ```json
  {
      "plugins": {
          "datasette-insert": {
              "allow": {
                  "bot": "insert-bot"
              }
          },
          "datasette-auth-tokens": {
              "tokens": [
                  {
                      "token": {
                          "$env": "INSERT_TOKEN"
                      },
                      "actor": {
                          "bot": "insert-bot"
                      }
                  }
              ]
          }
      }
  }
  ```
  With this configuration in place you can start Datasette like this:

      INSERT_TOKEN=abc123 datasette data.db -m metadata.json

  You can now send data to the API using `curl` like this:

  ```
  curl --request POST \
    -H "Authorization: Bearer abc123" \
    --data '[
        {
          "id": 3,
          "name": "Boris",
          "age": 1,
          "breed": "Husky"
        }
      ]' \
      'http://localhost:8001/-/insert/data/dogs'
  ```

  Or using the Python `requests` library like so:

  ```python
  requests.post(
      "http://localhost:8001/-/insert/data/dogs",
      json={"id": 1, "name": "Cleopaws", "age": 5},
      headers={"Authorization": "bearer abc123"},
  )
  ```

  ### Finely grained permissions

  Using an `"allow"` block as described above grants full permission to the features enabled by the API.

  The API implements several new Datasett permissions, which other plugins can use to make more finely grained decisions.

  The full set of permissions are as follows:

  - `insert:all` - all permissions - this is used by the `"allow"` block described above. Argument: `database_name`
  - `insert:insert-update` - the ability to insert data into an existing table, or to update data by its primary key. Arguments: `(database_name, table_name)`
  - `insert:create-table` - the ability to create a new table. Argument: `database_name`
  - `insert:alter-table` - the ability to add columns to an existing table (using `?alter=1`). Arguments: `(database_name, table_name)`

  You can use plugins like [datasette-permissions-sql](https://github.com/simonw/datasette-permissions-sql) to hook into these more detailed permissions for finely grained control over what actions each authenticated actor can take.

  Plugins that implement the [permission_allowed()](https://datasette.readthedocs.io/en/stable/plugin_hooks.html#plugin-hook-permission-allowed) plugin hook can take full control over these permission decisions.

  ## CORS

  If you start Datasette with the `datasette --cors` option the following HTTP headers will be added to resources served by this plugin:

      Access-Control-Allow-Origin: *
      Access-Control-Allow-Headers: content-type,authorization
      Access-Control-Allow-Methods: POST

  ## Development

  To set up this plugin locally, first checkout the code. Then create a new virtual environment:

      cd datasette-insert
      python3 -m venv venv
      source venv/bin/activate

  Now install the dependencies and tests:

      pip install -e '.[test]'

  To run the tests:

      pytest

readme_html:
  <div id="readme" class="md" data-path="README.md"><article class="markdown-body entry-content container-lg" itemprop="text"><div class="markdown-heading" dir="auto"><h1 class="heading-element" dir="auto">datasette-insert</h1><a id="user-content-datasette-insert" class="anchor" aria-label="Permalink: datasette-insert" href="#datasette-insert"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto"><a href="https://pypi.org/project/datasette-insert/" rel="nofollow"><img src="https://img.shields.io/pypi/v/datasette-insert.svg" alt="PyPI" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/42198b67f4ca0d480bfc98732016f006690b019434109b7bcc3afbfe9cfe70f0/68747470733a2f2f696d672e736869656c64732e696f2f707970692f762f6461746173657474652d696e736572742e737667"></a>
  <a href="https://github.com/simonw/datasette-insert/releases"><img src="https://img.shields.io/github/v/release/simonw/datasette-insert?include_prereleases&label=changelog" alt="Changelog" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/066a8cf461e60749bda905e31920cb9fca8d178e96b0d9e6f406453a0cbfd21f/68747470733a2f2f696d672e736869656c64732e696f2f6769746875622f762f72656c656173652f73696d6f6e772f6461746173657474652d696e736572743f696e636c7564655f70726572656c6561736573266c6162656c3d6368616e67656c6f67"></a>
  <a href="https://github.com/simonw/datasette-insert/blob/master/LICENSE"><img src="https://img.shields.io/badge/license-Apache%202.0-blue.svg" alt="License" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/c355f200ea90fddaa407b6eaab303663a669248ea3ca7b1fcf77dbe04ff5f48c/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6c6963656e73652d417061636865253230322e302d626c75652e737667"></a></p>
  <p dir="auto">Datasette plugin for inserting and updating data</p>
  <p dir="auto"><strong>No longer necessary with Datasette 1.0</strong>. Use the <a href="https://docs.datasette.io/en/latest/json_api.html#the-json-write-api" rel="nofollow">JSON write API</a> instead.</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Installation</h2><a id="user-content-installation" class="anchor" aria-label="Permalink: Installation" href="#installation"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">Install this plugin in the same environment as Datasette.</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ pip install datasette-insert"><pre class="notranslate"><code>$ pip install datasette-insert
  </code></pre></div>
  <p dir="auto">This plugin should always be deployed with additional configuration to prevent unauthenticated access, see notes below.</p>
  <p dir="auto">If you are trying it out on your own local machine, you can <code>pip install</code> the <a href="https://github.com/simonw/datasette-insert-unsafe">datasette-insert-unsafe</a> plugin to allow access without needing to set up authentication or permissions separately.</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Inserting data and creating tables</h2><a id="user-content-inserting-data-and-creating-tables" class="anchor" aria-label="Permalink: Inserting data and creating tables" href="#inserting-data-and-creating-tables"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">Start datasette and make sure it has a writable SQLite database attached to it. If you have not yet created a database file you can use this:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="datasette data.db --create"><pre class="notranslate"><code>datasette data.db --create
  </code></pre></div>
  <p dir="auto">The <code>--create</code> option will create a new empty <code>data.db</code> database file if it does not already exist.</p>
  <p dir="auto">The plugin adds an endpoint that allows data to be inserted or updated and tables to be created by POSTing JSON data to the following URL:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="/-/insert/name-of-database/name-of-table"><pre class="notranslate"><code>/-/insert/name-of-database/name-of-table
  </code></pre></div>
  <p dir="auto">The JSON should look like this:</p>
  <div class="highlight highlight-source-json notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="[
      {
          "id": 1,
          "name": "Cleopaws",
          "age": 5
      },
      {
          "id": 2,
          "name": "Pancakes",
          "age": 5
      }
  ]"><pre>[
      {
          <span class="pl-ent">"id"</span>: <span class="pl-c1">1</span>,
          <span class="pl-ent">"name"</span>: <span class="pl-s"><span class="pl-pds">"</span>Cleopaws<span class="pl-pds">"</span></span>,
          <span class="pl-ent">"age"</span>: <span class="pl-c1">5</span>
      },
      {
          <span class="pl-ent">"id"</span>: <span class="pl-c1">2</span>,
          <span class="pl-ent">"name"</span>: <span class="pl-s"><span class="pl-pds">"</span>Pancakes<span class="pl-pds">"</span></span>,
          <span class="pl-ent">"age"</span>: <span class="pl-c1">5</span>
      }
  ]</pre></div>
  <p dir="auto">The first time data is posted to the URL a table of that name will be created if it does not aready exist, with the desired columns.</p>
  <p dir="auto">You can specify which column should be used as the primary key using the <code>?pk=</code> URL argument.</p>
  <p dir="auto">Here's how to POST to a database and create a new table using the Python <code>requests</code> library:</p>
  <div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="import requests

  requests.post("http://localhost:8001/-/insert/data/dogs?pk=id", json=[
      {
          "id": 1,
          "name": "Cleopaws",
          "age": 5
      },
      {
          "id": 2,
          "name": "Pancakes",
          "age": 4
      }
  ])"><pre><span class="pl-k">import</span> <span class="pl-s1">requests</span>

  <span class="pl-s1">requests</span>.<span class="pl-c1">post</span>(<span class="pl-s">"http://localhost:8001/-/insert/data/dogs?pk=id"</span>, <span class="pl-s1">json</span><span class="pl-c1">=</span>[
      {
          <span class="pl-s">"id"</span>: <span class="pl-c1">1</span>,
          <span class="pl-s">"name"</span>: <span class="pl-s">"Cleopaws"</span>,
          <span class="pl-s">"age"</span>: <span class="pl-c1">5</span>
      },
      {
          <span class="pl-s">"id"</span>: <span class="pl-c1">2</span>,
          <span class="pl-s">"name"</span>: <span class="pl-s">"Pancakes"</span>,
          <span class="pl-s">"age"</span>: <span class="pl-c1">4</span>
      }
  ])</pre></div>
  <p dir="auto">And here's how to do the same thing using <code>curl</code>:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="curl --request POST \
    --data '[
        {
          "id": 1,
          "name": "Cleopaws",
          "age": 5
        },
        {
          "id": 2,
          "name": "Pancakes",
          "age": 4
        }
      ]' \
      'http://localhost:8001/-/insert/data/dogs?pk=id'"><pre class="notranslate"><code>curl --request POST \
    --data '[
        {
          "id": 1,
          "name": "Cleopaws",
          "age": 5
        },
        {
          "id": 2,
          "name": "Pancakes",
          "age": 4
        }
      ]' \
      'http://localhost:8001/-/insert/data/dogs?pk=id'
  </code></pre></div>
  <p dir="auto">Or by piping in JSON like so:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="cat dogs.json | curl --request POST -d @- \
      'http://localhost:8001/-/insert/data/dogs?pk=id'"><pre class="notranslate"><code>cat dogs.json | curl --request POST -d @- \
      'http://localhost:8001/-/insert/data/dogs?pk=id'
  </code></pre></div>
  <div class="markdown-heading" dir="auto"><h3 class="heading-element" dir="auto">Inserting a single row</h3><a id="user-content-inserting-a-single-row" class="anchor" aria-label="Permalink: Inserting a single row" href="#inserting-a-single-row"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">If you are inserting a single row you can optionally send it as a dictionary rather than a list with a single item:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="curl --request POST \
    --data '{
        "id": 1,
        "name": "Cleopaws",
        "age": 5
      }' \
      'http://localhost:8001/-/insert/data/dogs?pk=id'"><pre class="notranslate"><code>curl --request POST \
    --data '{
        "id": 1,
        "name": "Cleopaws",
        "age": 5
      }' \
      'http://localhost:8001/-/insert/data/dogs?pk=id'
  </code></pre></div>
  <div class="markdown-heading" dir="auto"><h3 class="heading-element" dir="auto">Automatically adding new columns</h3><a id="user-content-automatically-adding-new-columns" class="anchor" aria-label="Permalink: Automatically adding new columns" href="#automatically-adding-new-columns"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">If you send data to an existing table with keys that are not reflected by the existing columns, you will get an HTTP 400 error with a JSON response like this:</p>
  <div class="highlight highlight-source-json notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="{
      "status": 400,
      "error": "Unknown keys: 'foo'",
      "error_code": "unknown_keys"
  }"><pre>{
      <span class="pl-ent">"status"</span>: <span class="pl-c1">400</span>,
      <span class="pl-ent">"error"</span>: <span class="pl-s"><span class="pl-pds">"</span>Unknown keys: 'foo'<span class="pl-pds">"</span></span>,
      <span class="pl-ent">"error_code"</span>: <span class="pl-s"><span class="pl-pds">"</span>unknown_keys<span class="pl-pds">"</span></span>
  }</pre></div>
  <p dir="auto">If you add <code>?alter=1</code> to the URL you are posting to any missing columns will be automatically added:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="curl --request POST \
    --data '[
        {
          "id": 3,
          "name": "Boris",
          "age": 1,
          "breed": "Husky"
        }
      ]' \
      'http://localhost:8001/-/insert/data/dogs?alter=1'"><pre class="notranslate"><code>curl --request POST \
    --data '[
        {
          "id": 3,
          "name": "Boris",
          "age": 1,
          "breed": "Husky"
        }
      ]' \
      'http://localhost:8001/-/insert/data/dogs?alter=1'
  </code></pre></div>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Upserting data</h2><a id="user-content-upserting-data" class="anchor" aria-label="Permalink: Upserting data" href="#upserting-data"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">An "upsert" operation can be used to partially update a record. With upserts you can send a subset of the keys and, if the ID matches the specified primary key, they will be used to update an existing record.</p>
  <p dir="auto">Upserts can be sent to the <code>/-/upsert</code> API endpoint.</p>
  <p dir="auto">This example will update the dog with ID=1's age from 5 to 7:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="curl --request POST \
    --data '{
        "id": 1,
        "age": 7
      }' \
      'http://localhost:3322/-/upsert/data/dogs?pk=id'"><pre class="notranslate"><code>curl --request POST \
    --data '{
        "id": 1,
        "age": 7
      }' \
      'http://localhost:3322/-/upsert/data/dogs?pk=id'
  </code></pre></div>
  <p dir="auto">Like the <code>/-/insert</code> endpoint, the <code>/-/upsert</code> endpoint can accept an array of objects too. It also supports the <code>?alter=1</code> option.</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Permissions and authentication</h2><a id="user-content-permissions-and-authentication" class="anchor" aria-label="Permalink: Permissions and authentication" href="#permissions-and-authentication"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">This plugin defaults to denying all access, to help ensure people don't accidentally deploy it on the open internet in an unsafe configuration.</p>
  <p dir="auto">You can read about <a href="https://datasette.readthedocs.io/en/stable/authentication.html" rel="nofollow">Datasette's approach to authentication</a> in the Datasette manual.</p>
  <p dir="auto">You can install the <code>datasette-insert-unsafe</code> plugin to run in unsafe mode, where all access is allowed by default.</p>
  <p dir="auto">I recommend using this plugin in conjunction with <a href="https://github.com/simonw/datasette-auth-tokens">datasette-auth-tokens</a>, which provides a mechanism for making authenticated calls using API tokens.</p>
  <p dir="auto">You can then use <a href="https://datasette.readthedocs.io/en/stable/authentication.html#defining-permissions-with-allow-blocks" rel="nofollow">"allow" blocks</a> in the <code>datasette-insert</code> plugin configuration to specify which authenticated tokens are allowed to make use of the API.</p>
  <p dir="auto">Here's an example <code>metadata.json</code> file which restricts access to the <code>/-/insert</code> API to an API token defined in an <code>INSERT_TOKEN</code> environment variable:</p>
  <div class="highlight highlight-source-json notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="{
      "plugins": {
          "datasette-insert": {
              "allow": {
                  "bot": "insert-bot"
              }
          },
          "datasette-auth-tokens": {
              "tokens": [
                  {
                      "token": {
                          "$env": "INSERT_TOKEN"
                      },
                      "actor": {
                          "bot": "insert-bot"
                      }
                  }
              ]
          }
      }
  }"><pre>{
      <span class="pl-ent">"plugins"</span>: {
          <span class="pl-ent">"datasette-insert"</span>: {
              <span class="pl-ent">"allow"</span>: {
                  <span class="pl-ent">"bot"</span>: <span class="pl-s"><span class="pl-pds">"</span>insert-bot<span class="pl-pds">"</span></span>
              }
          },
          <span class="pl-ent">"datasette-auth-tokens"</span>: {
              <span class="pl-ent">"tokens"</span>: [
                  {
                      <span class="pl-ent">"token"</span>: {
                          <span class="pl-ent">"$env"</span>: <span class="pl-s"><span class="pl-pds">"</span>INSERT_TOKEN<span class="pl-pds">"</span></span>
                      },
                      <span class="pl-ent">"actor"</span>: {
                          <span class="pl-ent">"bot"</span>: <span class="pl-s"><span class="pl-pds">"</span>insert-bot<span class="pl-pds">"</span></span>
                      }
                  }
              ]
          }
      }
  }</pre></div>
  <p dir="auto">With this configuration in place you can start Datasette like this:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="INSERT_TOKEN=abc123 datasette data.db -m metadata.json"><pre class="notranslate"><code>INSERT_TOKEN=abc123 datasette data.db -m metadata.json
  </code></pre></div>
  <p dir="auto">You can now send data to the API using <code>curl</code> like this:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="curl --request POST \
    -H "Authorization: Bearer abc123" \
    --data '[
        {
          "id": 3,
          "name": "Boris",
          "age": 1,
          "breed": "Husky"
        }
      ]' \
      'http://localhost:8001/-/insert/data/dogs'"><pre class="notranslate"><code>curl --request POST \
    -H "Authorization: Bearer abc123" \
    --data '[
        {
          "id": 3,
          "name": "Boris",
          "age": 1,
          "breed": "Husky"
        }
      ]' \
      'http://localhost:8001/-/insert/data/dogs'
  </code></pre></div>
  <p dir="auto">Or using the Python <code>requests</code> library like so:</p>
  <div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="requests.post(
      "http://localhost:8001/-/insert/data/dogs",
      json={"id": 1, "name": "Cleopaws", "age": 5},
      headers={"Authorization": "bearer abc123"},
  )"><pre><span class="pl-s1">requests</span>.<span class="pl-c1">post</span>(
      <span class="pl-s">"http://localhost:8001/-/insert/data/dogs"</span>,
      <span class="pl-s1">json</span><span class="pl-c1">=</span>{<span class="pl-s">"id"</span>: <span class="pl-c1">1</span>, <span class="pl-s">"name"</span>: <span class="pl-s">"Cleopaws"</span>, <span class="pl-s">"age"</span>: <span class="pl-c1">5</span>},
      <span class="pl-s1">headers</span><span class="pl-c1">=</span>{<span class="pl-s">"Authorization"</span>: <span class="pl-s">"bearer abc123"</span>},
  )</pre></div>
  <div class="markdown-heading" dir="auto"><h3 class="heading-element" dir="auto">Finely grained permissions</h3><a id="user-content-finely-grained-permissions" class="anchor" aria-label="Permalink: Finely grained permissions" href="#finely-grained-permissions"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">Using an <code>"allow"</code> block as described above grants full permission to the features enabled by the API.</p>
  <p dir="auto">The API implements several new Datasett permissions, which other plugins can use to make more finely grained decisions.</p>
  <p dir="auto">The full set of permissions are as follows:</p>
  <ul dir="auto">
  <li><code>insert:all</code> - all permissions - this is used by the <code>"allow"</code> block described above. Argument: <code>database_name</code></li>
  <li><code>insert:insert-update</code> - the ability to insert data into an existing table, or to update data by its primary key. Arguments: <code>(database_name, table_name)</code></li>
  <li><code>insert:create-table</code> - the ability to create a new table. Argument: <code>database_name</code></li>
  <li><code>insert:alter-table</code> - the ability to add columns to an existing table (using <code>?alter=1</code>). Arguments: <code>(database_name, table_name)</code></li>
  </ul>
  <p dir="auto">You can use plugins like <a href="https://github.com/simonw/datasette-permissions-sql">datasette-permissions-sql</a> to hook into these more detailed permissions for finely grained control over what actions each authenticated actor can take.</p>
  <p dir="auto">Plugins that implement the <a href="https://datasette.readthedocs.io/en/stable/plugin_hooks.html#plugin-hook-permission-allowed" rel="nofollow">permission_allowed()</a> plugin hook can take full control over these permission decisions.</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">CORS</h2><a id="user-content-cors" class="anchor" aria-label="Permalink: CORS" href="#cors"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">If you start Datasette with the <code>datasette --cors</code> option the following HTTP headers will be added to resources served by this plugin:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="Access-Control-Allow-Origin: *
  Access-Control-Allow-Headers: content-type,authorization
  Access-Control-Allow-Methods: POST"><pre class="notranslate"><code>Access-Control-Allow-Origin: *
  Access-Control-Allow-Headers: content-type,authorization
  Access-Control-Allow-Methods: POST
  </code></pre></div>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Development</h2><a id="user-content-development" class="anchor" aria-label="Permalink: Development" href="#development"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">To set up this plugin locally, first checkout the code. Then create a new virtual environment:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="cd datasette-insert
  python3 -m venv venv
  source venv/bin/activate"><pre class="notranslate"><code>cd datasette-insert
  python3 -m venv venv
  source venv/bin/activate
  </code></pre></div>
  <p dir="auto">Now install the dependencies and tests:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="pip install -e '.[test]'"><pre class="notranslate"><code>pip install -e '.[test]'
  </code></pre></div>
  <p dir="auto">To run the tests:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="pytest"><pre class="notranslate"><code>pytest
  </code></pre></div>
  </article></div>


node_id: MDEwOlJlcG9zaXRvcnkyODE0ODEzNDc=
name: datasette-copyable
full_name: simonw/datasette-copyable
html_url: https://github.com/simonw/datasette-copyable
description: Datasette plugin for outputting tables in formats suitable for copy and paste
created_at: 2020-07-21T19:04:08Z
updated_at: 2024-08-20T10:35:06Z
pushed_at: 2024-02-17T23:11:29Z
language: Python
topics: ["datasette", "datasette-io", "datasette-plugin"]
visibility: public
default_branch: main
permissions: {"admin": false, "maintain": false, "push": false, "triage": false, "pull": true}
readme:
  # datasette-copyable

  [![PyPI](https://img.shields.io/pypi/v/datasette-copyable.svg)](https://pypi.org/project/datasette-copyable/)
  [![Changelog](https://img.shields.io/github/v/release/simonw/datasette-copyable?include_prereleases&label=changelog)](https://github.com/simonw/datasette-copyable/releases)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/datasette-copyable/blob/master/LICENSE)

  Datasette plugin for outputting tables in formats suitable for copy and paste

  ## Installation

  Install this plugin in the same environment as Datasette.

      $ pip install datasette-copyable

  ## Demo

  You can try this plugin on [fivethirtyeight.datasettes.com](https://fivethirtyeight.datasettes.com/) - browse for tables or queries there and look for the "copyable" link. Here's an example for a table of [airline safety data](https://fivethirtyeight.datasettes.com/fivethirtyeight/airline-safety~2Fairline-safety.copyable).

  ## Usage

  This plugin adds a `.copyable` output extension to every table, view and query.

  Navigating to this page will show an interface allowing you to select a format for copying and pasting the demo. The default is TSV, which is suitable for copying into Google Sheets or Excel.

  You can add `?_raw=1` to get back just the raw data.

  ## Development

  To set up this plugin locally, first checkout the code. Then create a new virtual environment:

      cd datasette-copyable
      python3 -mvenv venv
      source venv/bin/activate

  Or if you are using `pipenv`:

      pipenv shell

  Now install the dependencies and tests:

      pip install -e '.[test]'

  To run the tests:

      pytest

readme_html:
  <div id="readme" class="md" data-path="README.md"><article class="markdown-body entry-content container-lg" itemprop="text"><div class="markdown-heading" dir="auto"><h1 class="heading-element" dir="auto">datasette-copyable</h1><a id="user-content-datasette-copyable" class="anchor" aria-label="Permalink: datasette-copyable" href="#datasette-copyable"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto"><a href="https://pypi.org/project/datasette-copyable/" rel="nofollow"><img src="https://img.shields.io/pypi/v/datasette-copyable.svg" alt="PyPI" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/ec53aa4a62e0409db87c37843e4ca333771900b6e4fc1bdd03608365d92e3182/68747470733a2f2f696d672e736869656c64732e696f2f707970692f762f6461746173657474652d636f707961626c652e737667"></a>
  <a href="https://github.com/simonw/datasette-copyable/releases"><img src="https://img.shields.io/github/v/release/simonw/datasette-copyable?include_prereleases&label=changelog" alt="Changelog" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/c4b42d959b6f7e56edc8b9325d460b6b34986482411ba875cddd028e3e8c6c8f/68747470733a2f2f696d672e736869656c64732e696f2f6769746875622f762f72656c656173652f73696d6f6e772f6461746173657474652d636f707961626c653f696e636c7564655f70726572656c6561736573266c6162656c3d6368616e67656c6f67"></a>
  <a href="https://github.com/simonw/datasette-copyable/blob/master/LICENSE"><img src="https://img.shields.io/badge/license-Apache%202.0-blue.svg" alt="License" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/c355f200ea90fddaa407b6eaab303663a669248ea3ca7b1fcf77dbe04ff5f48c/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6c6963656e73652d417061636865253230322e302d626c75652e737667"></a></p>
  <p dir="auto">Datasette plugin for outputting tables in formats suitable for copy and paste</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Installation</h2><a id="user-content-installation" class="anchor" aria-label="Permalink: Installation" href="#installation"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">Install this plugin in the same environment as Datasette.</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ pip install datasette-copyable"><pre class="notranslate"><code>$ pip install datasette-copyable
  </code></pre></div>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Demo</h2><a id="user-content-demo" class="anchor" aria-label="Permalink: Demo" href="#demo"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">You can try this plugin on <a href="https://fivethirtyeight.datasettes.com/" rel="nofollow">fivethirtyeight.datasettes.com</a> - browse for tables or queries there and look for the "copyable" link. Here's an example for a table of <a href="https://fivethirtyeight.datasettes.com/fivethirtyeight/airline-safety~2Fairline-safety.copyable" rel="nofollow">airline safety data</a>.</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Usage</h2><a id="user-content-usage" class="anchor" aria-label="Permalink: Usage" href="#usage"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">This plugin adds a <code>.copyable</code> output extension to every table, view and query.</p>
  <p dir="auto">Navigating to this page will show an interface allowing you to select a format for copying and pasting the demo. The default is TSV, which is suitable for copying into Google Sheets or Excel.</p>
  <p dir="auto">You can add <code>?_raw=1</code> to get back just the raw data.</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Development</h2><a id="user-content-development" class="anchor" aria-label="Permalink: Development" href="#development"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">To set up this plugin locally, first checkout the code. Then create a new virtual environment:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="cd datasette-copyable
  python3 -mvenv venv
  source venv/bin/activate"><pre class="notranslate"><code>cd datasette-copyable
  python3 -mvenv venv
  source venv/bin/activate
  </code></pre></div>
  <p dir="auto">Or if you are using <code>pipenv</code>:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="pipenv shell"><pre class="notranslate"><code>pipenv shell
  </code></pre></div>
  <p dir="auto">Now install the dependencies and tests:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="pip install -e '.[test]'"><pre class="notranslate"><code>pip install -e '.[test]'
  </code></pre></div>
  <p dir="auto">To run the tests:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="pytest"><pre class="notranslate"><code>pytest
  </code></pre></div>
  </article></div>


node_id: MDEwOlJlcG9zaXRvcnkyODQzODMyNjU=
name: datasette-graphql
full_name: simonw/datasette-graphql
html_url: https://github.com/simonw/datasette-graphql
description: Datasette plugin providing an automatic GraphQL API for your SQLite databases
created_at: 2020-08-02T03:31:58Z
updated_at: 2026-04-08T18:31:48Z
pushed_at: 2026-04-08T18:32:44Z
homepage: https://datasette-graphql-demo.datasette.io/
language: Python
license: apache-2.0
topics: ["datasette", "datasette-io", "datasette-plugin", "graphql", "sqlite"]
visibility: public
default_branch: main
permissions: {"admin": false, "maintain": false, "push": false, "triage": false, "pull": false}
pull_request_creation_policy: all
readme:
  # datasette-graphql

  [![PyPI](https://img.shields.io/pypi/v/datasette-graphql.svg)](https://pypi.org/project/datasette-graphql/)
  [![Changelog](https://img.shields.io/github/v/release/simonw/datasette-graphql?include_prereleases&label=changelog)](https://github.com/simonw/datasette-graphql/releases)
  [![Tests](https://github.com/simonw/datasette-graphql/workflows/Test/badge.svg)](https://github.com/simonw/datasette-graphql/actions?query=workflow%3ATest)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/datasette-graphql/blob/main/LICENSE)

  **Datasette plugin providing an automatic GraphQL API for your SQLite databases**

  Read more about this project: [GraphQL in Datasette with the new datasette-graphql plugin](https://simonwillison.net/2020/Aug/7/datasette-graphql/)

  Try out a live demo at [datasette-graphql-demo.datasette.io/graphql](https://datasette-graphql-demo.datasette.io/graphql?query=%7B%0A%20%20repos(first%3A10%2C%20search%3A%20%22sql%22%2C%20sort_desc%3A%20created_at)%20%7B%0A%20%20%20%20totalCount%0A%20%20%20%20pageInfo%20%7B%0A%20%20%20%20%20%20endCursor%0A%20%20%20%20%20%20hasNextPage%0A%20%20%20%20%7D%0A%20%20%20%20nodes%20%7B%0A%20%20%20%20%20%20full_name%0A%20%20%20%20%20%20description_%0A%20%20%20%20%09stargazers_count%0A%20%20%20%20%20%20created_at%0A%20%20%20%20%20%20owner%20%7B%0A%20%20%20%20%20%20%20%20name%0A%20%20%20%20%20%20%20%20html_url%0A%20%20%20%20%20%20%7D%0A%20%20%20%20%7D%0A%20%20%7D%0A%7D%0A)

  <!-- toc -->

  - [Installation](#installation)
  - [Configuration](#configuration)
  - [Usage](#usage)
    * [Querying for tables and columns](#querying-for-tables-and-columns)
    * [Fetching a single record](#fetching-a-single-record)
    * [Accessing nested objects](#accessing-nested-objects)
    * [Accessing related objects](#accessing-related-objects)
    * [Filtering tables](#filtering-tables)
    * [Sorting](#sorting)
    * [Pagination](#pagination)
    * [Search](#search)
    * [Columns containing JSON strings](#columns-containing-json-strings)
    * [Auto camelCase](#auto-camelcase)
    * [CORS](#cors)
    * [Execution limits](#execution-limits)
  - [The graphql() template function](#the-graphql-template-function)
  - [Adding custom fields with plugins](#adding-custom-fields-with-plugins)
  - [Development](#development)

  <!-- tocstop -->

  ![Animated demo showing autocomplete while typing a GraphQL query into the GraphiQL interface](https://static.simonwillison.net/static/2020/graphiql.gif)

  ## Installation

  Install this plugin in the same environment as Datasette.

      $ datasette install datasette-graphql

  ## Configuration

  By default this plugin adds the GraphQL API at `/graphql`. You can configure a different path using the `path` plugin setting, for example by adding this to `metadata.json`:
  ```json
  {
    "plugins": {
      "datasette-graphql": {
        "path": "/-/graphql"
      }
    }
  }
  ```
  This will set the GraphQL API to live at `/-/graphql` instead.

  ## Usage

  This plugin sets up `/graphql` as a GraphQL endpoint for the first attached database.

  If you have multiple attached databases each will get its own endpoint at `/graphql/name_of_database`.

  The automatically generated GraphQL schema is available at `/graphql/name_of_database.graphql` - here's [an example](https://datasette-graphql-demo.datasette.io/graphql/github.graphql).

  ### Querying for tables and columns

  Individual tables (and SQL views) can be queried like this:

  ```graphql
  {
    repos {
      nodes {
        id
        full_name
        description_
      }
    }
  }
  ```
  [Try this query](https://datasette-graphql-demo.datasette.io/graphql?query=%0A%7B%0A%20%20repos%20%7B%0A%20%20%20%20nodes%20%7B%0A%20%20%20%20%20%20id%0A%20%20%20%20%20%20full_name%0A%20%20%20%20%20%20description_%0A%20%20%20%20%7D%0A%20%20%7D%0A%7D%0A)

  In this example query the underlying database table is called `repos` and its columns include `id`, `full_name` and `description`. Since `description` is a reserved word the query needs to ask for `description_` instead.

  ### Fetching a single record

  If you only want to fetch a single record - for example if you want to fetch a row by its primary key - you can use the `tablename_row` field:

  ```graphql
  {
    repos_row(id: 107914493) {
      id
      full_name
      description_
    }
  }
  ```
  [Try this query](https://datasette-graphql-demo.datasette.io/graphql?query=%0A%7B%0A%20%20repos_row%28id%3A%20107914493%29%20%7B%0A%20%20%20%20id%0A%20%20%20%20full_name%0A%20%20%20%20description_%0A%20%20%7D%0A%7D%0A)

  The `tablename_row` field accepts the primary key column (or columns) as arguments. It also supports the same `filter:`, `search:`, `sort:` and `sort_desc:` arguments as the `tablename` field, described below.

  ### Accessing nested objects

  If a column is a foreign key to another table, you can request columns from the table pointed to by that foreign key using a nested query like this:

  ```graphql
  {
    repos {
      nodes {
        id
        full_name
        owner {
          id
          login
        }
      }
    }
  }
  ```
  [Try this query](https://datasette-graphql-demo.datasette.io/graphql?query=%0A%7B%0A%20%20repos%20%7B%0A%20%20%20%20nodes%20%7B%0A%20%20%20%20%20%20id%0A%20%20%20%20%20%20full_name%0A%20%20%20%20%20%20owner%20%7B%0A%20%20%20%20%20%20%20%20id%0A%20%20%20%20%20%20%20%20login%0A%20%20%20%20%20%20%7D%0A%20%20%20%20%7D%0A%20%20%7D%0A%7D%0A)

  ### Accessing related objects

  If another table has a foreign key back to the table you are accessing, you can fetch rows from that related table.

  Consider a `users` table which is related to `repos` - a repo has a foreign key back to the user that owns the repository. The `users` object type will have a `repos_by_owner_list` field which can be used to access those related repos:

  ```graphql
  {
    users(first: 1, search: "simonw") {
      nodes {
        name
        repos_by_owner_list(first: 5) {
          totalCount
          nodes {
            full_name
          }
        }
      }
    }
  }
  ```
  [Try this query](https://datasette-graphql-demo.datasette.io/graphql?query=%0A%7B%0A%20%20users%28first%3A%201%2C%20search%3A%20%22simonw%22%29%20%7B%0A%20%20%20%20nodes%20%7B%0A%20%20%20%20%20%20name%0A%20%20%20%20%20%20repos_by_owner_list%28first%3A%205%29%20%7B%0A%20%20%20%20%20%20%20%20totalCount%0A%20%20%20%20%20%20%20%20nodes%20%7B%0A%20%20%20%20%20%20%20%20%20%20full_name%0A%20%20%20%20%20%20%20%20%7D%0A%20%20%20%20%20%20%7D%0A%20%20%20%20%7D%0A%20%20%7D%0A%7D%0A)


  ### Filtering tables

  You can filter the rows returned for a specific table using the `filter:` argument. This accepts a filter object mapping columns to operations. For example, to return just repositories with the Apache 2 license and more than 10 stars:

  ```graphql
  {
    repos(filter: {license: {eq: "apache-2.0"}, stargazers_count: {gt: 10}}) {
      nodes {
        full_name
        stargazers_count
        license {
          key
        }
      }
    }
  }
  ```
  [Try this query](https://datasette-graphql-demo.datasette.io/graphql?query=%0A%7B%0A%20%20repos%28filter%3A%20%7Blicense%3A%20%7Beq%3A%20%22apache-2.0%22%7D%2C%20stargazers_count%3A%20%7Bgt%3A%2010%7D%7D%29%20%7B%0A%20%20%20%20nodes%20%7B%0A%20%20%20%20%20%20full_name%0A%20%20%20%20%20%20stargazers_count%0A%20%20%20%20%20%20license%20%7B%0A%20%20%20%20%20%20%20%20key%0A%20%20%20%20%20%20%7D%0A%20%20%20%20%7D%0A%20%20%7D%0A%7D%0A)

  See [table filters examples](https://github.com/simonw/datasette-graphql/blob/main/examples/filters.md) for more operations, and [column filter arguments](https://docs.datasette.io/en/stable/json_api.html#column-filter-arguments) in the Datasette documentation for details of how those operations work.

  These same filters can be used on nested relationships, like so:

  ```graphql
  {
    users_row(id: 9599) {
      name
      repos_by_owner_list(filter: {name: {startswith: "datasette-"}}) {
        totalCount
        nodes {
          full_name
        }
      }
    }
  }
  ```
  [Try this query](https://datasette-graphql-demo.datasette.io/graphql?query=%0A%7B%0A%20%20users_row%28id%3A%209599%29%20%7B%0A%20%20%20%20name%0A%20%20%20%20repos_by_owner_list%28filter%3A%20%7Bname%3A%20%7Bstartswith%3A%20%22datasette-%22%7D%7D%29%20%7B%0A%20%20%20%20%20%20totalCount%0A%20%20%20%20%20%20nodes%20%7B%0A%20%20%20%20%20%20%20%20full_name%0A%20%20%20%20%20%20%7D%0A%20%20%20%20%7D%0A%20%20%7D%0A%7D%0A)


  The `where:` argument can be used as an alternative to `filter:` when the thing you are expressing is too complex to be modeled using a filter expression. It accepts a string fragment of SQL that will be included in the `WHERE` clause of the SQL query.

  ```graphql
  {
    repos(where: "name='sqlite-utils' or name like 'datasette-%'") {
      totalCount
      nodes {
        full_name
      }
    }
  }
  ```
  [Try this query](https://datasette-graphql-demo.datasette.io/graphql?query=%0A%7B%0A%20%20repos%28where%3A%20%22name%3D%27sqlite-utils%27%20or%20name%20like%20%27datasette-%25%27%22%29%20%7B%0A%20%20%20%20totalCount%0A%20%20%20%20nodes%20%7B%0A%20%20%20%20%20%20full_name%0A%20%20%20%20%7D%0A%20%20%7D%0A%7D%0A)

  ### Sorting

  You can set a sort order for results from a table using the `sort:` or `sort_desc:` arguments. The value for this argument should be the name of the column you wish to sort (or sort-descending) by.

  ```graphql
  {
    repos(sort_desc: stargazers_count) {
      nodes {
        full_name
        stargazers_count
      }
    }
  }
  ```
  [Try this query](https://datasette-graphql-demo.datasette.io/graphql?query=%0A%7B%0A%20%20repos%28sort_desc%3A%20stargazers_count%29%20%7B%0A%20%20%20%20nodes%20%7B%0A%20%20%20%20%20%20full_name%0A%20%20%20%20%20%20stargazers_count%0A%20%20%20%20%7D%0A%20%20%7D%0A%7D%0A)

  ### Pagination

  By default the first 10 rows will be returned. You can control this using the `first:` argument.

  ```graphql
  {
    repos(first: 20) {
      totalCount
      pageInfo {
        hasNextPage
        endCursor
      }
      nodes {
        full_name
        stargazers_count
        license {
          key
        }
      }
    }
  }
  ```
  [Try this query](https://datasette-graphql-demo.datasette.io/graphql?query=%0A%7B%0A%20%20repos%28first%3A%2020%29%20%7B%0A%20%20%20%20totalCount%0A%20%20%20%20pageInfo%20%7B%0A%20%20%20%20%20%20hasNextPage%0A%20%20%20%20%20%20endCursor%0A%20%20%20%20%7D%0A%20%20%20%20nodes%20%7B%0A%20%20%20%20%20%20full_name%0A%20%20%20%20%20%20stargazers_count%0A%20%20%20%20%20%20license%20%7B%0A%20%20%20%20%20%20%20%20key%0A%20%20%20%20%20%20%7D%0A%20%20%20%20%7D%0A%20%20%7D%0A%7D%0A)

  The `totalCount` field returns the total number of records that match the query.

  Requesting the `pageInfo.endCursor` field provides you with the value you need to request the next page. You can pass this to the `after:` argument to request the next page.

  ```graphql
  {
    repos(first: 20, after: "134874019") {
      totalCount
      pageInfo {
        hasNextPage
        endCursor
      }
      nodes {
        full_name
        stargazers_count
        license {
          key
        }
      }
    }
  }
  ```
  [Try this query](https://datasette-graphql-demo.datasette.io/graphql?query=%0A%7B%0A%20%20repos%28first%3A%2020%2C%20after%3A%20%22134874019%22%29%20%7B%0A%20%20%20%20totalCount%0A%20%20%20%20pageInfo%20%7B%0A%20%20%20%20%20%20hasNextPage%0A%20%20%20%20%20%20endCursor%0A%20%20%20%20%7D%0A%20%20%20%20nodes%20%7B%0A%20%20%20%20%20%20full_name%0A%20%20%20%20%20%20stargazers_count%0A%20%20%20%20%20%20license%20%7B%0A%20%20%20%20%20%20%20%20key%0A%20%20%20%20%20%20%7D%0A%20%20%20%20%7D%0A%20%20%7D%0A%7D%0A)

  The `hasNextPage` field tells you if there are any more records.

  ### Search

  If a table has been configured to use SQLite full-text search you can execute searches against it using the `search:` argument:

  ```graphql
  {
    repos(search: "datasette") {
      totalCount
      pageInfo {
        hasNextPage
        endCursor
      }
      nodes {
        full_name
        description_
      }
    }
  }
  ```
  [Try this query](https://datasette-graphql-demo.datasette.io/graphql?query=%0A%7B%0A%20%20repos%28search%3A%20%22datasette%22%29%20%7B%0A%20%20%20%20totalCount%0A%20%20%20%20pageInfo%20%7B%0A%20%20%20%20%20%20hasNextPage%0A%20%20%20%20%20%20endCursor%0A%20%20%20%20%7D%0A%20%20%20%20nodes%20%7B%0A%20%20%20%20%20%20full_name%0A%20%20%20%20%20%20description_%0A%20%20%20%20%7D%0A%20%20%7D%0A%7D%0A)

  The [sqlite-utils](https://sqlite-utils.datasette.io/) Python library and CLI tool can be used to add full-text search to an existing database table.

  ### Columns containing JSON strings

  If your table has a column that contains data encoded as JSON, `datasette-graphql` will make that column available as an encoded JSON string. Clients calling your API will need to parse the string as JSON in order to access the data.

  You can return the data as a nested structure by configuring that column to be treated as a JSON column. The [plugin configuration](https://docs.datasette.io/en/stable/plugins.html#plugin-configuration) for that in `metadata.json` looks like this:

  ```json
  {
      "databases": {
          "test": {
              "tables": {
                  "repos": {
                      "plugins": {
                          "datasette-graphql": {
                              "json_columns": [
                                  "tags"
                              ]
                          }
                      }
                  }
              }
          }
      }
  }
  ```

  ### Auto camelCase

  The names of your columns and tables default to being matched by their representations in GraphQL.

  If you have tables with `names_like_this` you may want to work with them in GraphQL using `namesLikeThis`, for consistency with GraphQL and JavaScript conventions.

  You can turn on automatic camelCase using the `"auto_camelcase"` plugin configuration setting in `metadata.json`, like this:

  ```json
  {
      "plugins": {
          "datasette-graphql": {
              "auto_camelcase": true
          }
      }
  }
  ```

  ### CORS

  This plugin obeys the `--cors` option passed to the `datasette` command-line tool. If you pass `--cors` it adds the following CORS HTTP headers to allow JavaScript running on other domains to access the GraphQL API:

      access-control-allow-headers: content-type
      access-control-allow-method: POST
      access-control-allow-origin: *

  ### Execution limits

  The plugin implements two limits by default:

  - The total time spent executing all of the underlying SQL queries that make up the GraphQL execution must not exceed 1000ms (one second)
  - The total number of SQL table queries executed as a result of nested GraphQL fields must not exceed 100

  These limits can be customized using the `num_queries_limit` and `time_limit_ms` plugin configuration settings, for example in `metadata.json`:

  ```json
  {
      "plugins": {
          "datasette-graphql": {
              "num_queries_limit": 200,
              "time_limit_ms": 5000
          }
      }
  }
  ```
  Setting these to `0` will disable the limit checks entirely.

  ## The graphql() template function

  The plugin also makes a Jinja template function available called `graphql()`. You can use that function in your Datasette [custom templates](https://docs.datasette.io/en/stable/custom_templates.html#custom-templates) like so:

  ```html+jinja
  {% set users = graphql("""
  {
      users {
          nodes {
              name
              points
              score
          }
      }
  }
  """)["users"] %}
  {% for user in users.nodes %}
      <p>{{ user.name }} - points: {{ user.points }}, score = {{ user.score }}</p>
  {% endfor %}
  ```

  The function executes a GraphQL query against the generated schema and returns the results. You can assign those results to a variable in your template and then loop through and display them.

  By default the query will be run against the first attached database. You can use the optional second argument to the function to specify a different database - for example, to run against an attached `github.db` database you would do this:

  ```html+jinja
  {% set user = graphql("""
  {
      users_row(id:9599) {
          name
          login
          avatar_url
      }
  }
  """, "github")["users_row"] %}

  <h1>Hello, {{ user.name }}</h1>
  ```

  You can use [GraphQL variables](https://graphql.org/learn/queries/#variables) in these template calls by passing them to the `variables=` argument:

  ```html+jinja
  {% set user = graphql("""
  query ($id: Int) {
      users_row(id: $id) {
          name
          login
          avatar_url
      }
  }
  """, database="github", variables={"id": 9599})["users_row"] %}

  <h1>Hello, {{ user.name }}</h1>
  ```
  ## Adding custom fields with plugins

  `datasette-graphql` adds a new [plugin hook](https://docs.datasette.io/en/stable/writing_plugins.html) to Datasette which can be used to add custom fields to your GraphQL schema.

  The plugin hook looks like this:

  ```python
  @hookimpl
  def graphql_extra_fields(datasette, database):
      "A list of (name, field_type) tuples to include in the GraphQL schema"
  ```

  You can use this hook to return a list of tuples describing additional fields that should be exposed in your schema. Each tuple should consist of a string naming the new field, plus a [Graphene Field object](https://docs.graphene-python.org/en/latest/types/objecttypes/) that specifies the schema and provides a `resolver` function.

  This example implementation uses `pkg_resources` to return a list of currently installed Python packages:

  ```python
  import graphene
  from datasette import hookimpl
  import pkg_resources


  @hookimpl
  def graphql_extra_fields():
      class Package(graphene.ObjectType):
          "An installed package"
          name = graphene.String()
          version = graphene.String()

      def resolve_packages(root, info):
          return [
              {"name": d.project_name, "version": d.version}
              for d in pkg_resources.working_set
          ]

      return [
          (
              "packages",
              graphene.Field(
                  graphene.List(Package),
                  description="List of installed packages",
                  resolver=resolve_packages,
              ),
          ),
      ]
  ```

  With this plugin installed, the following GraphQL query can be used to retrieve a list of installed packages:

  ```graphql
  {
    packages {
      name
      version
    }
  }
  ```

  ## Development

  To set up this plugin locally, first checkout the code. Then create a new virtual environment:

      cd datasette-graphql
      python3 -mvenv venv
      source venv/bin/activate

  Or if you are using `pipenv`:

      pipenv shell

  Now install the dependencies and tests:

      pip install -e '.[test]'

  To run the tests:

      pytest

readme_html:
  <div id="readme" class="md" data-path="README.md"><article class="markdown-body entry-content container-lg" itemprop="text"><div class="markdown-heading" dir="auto"><h1 class="heading-element" dir="auto">datasette-graphql</h1><a id="user-content-datasette-graphql" class="anchor" aria-label="Permalink: datasette-graphql" href="#datasette-graphql"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto"><a href="https://pypi.org/project/datasette-graphql/" rel="nofollow"><img src="https://img.shields.io/pypi/v/datasette-graphql.svg" alt="PyPI" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/2e6f4569a022c397e96f99d2ff1fb6ac6144ee1d015b2ef9c3c524c7c390be7d/68747470733a2f2f696d672e736869656c64732e696f2f707970692f762f6461746173657474652d6772617068716c2e737667"></a>
  <a href="https://github.com/simonw/datasette-graphql/releases"><img src="https://img.shields.io/github/v/release/simonw/datasette-graphql?include_prereleases&label=changelog" alt="Changelog" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/94d60a0a452c14932972afad3f6edfa1822c6193369b6d31dc751b88a5be95e1/68747470733a2f2f696d672e736869656c64732e696f2f6769746875622f762f72656c656173652f73696d6f6e772f6461746173657474652d6772617068716c3f696e636c7564655f70726572656c6561736573266c6162656c3d6368616e67656c6f67"></a>
  <a href="https://github.com/simonw/datasette-graphql/actions?query=workflow%3ATest"><img src="https://github.com/simonw/datasette-graphql/workflows/Test/badge.svg" alt="Tests" style="max-width: 100%;"></a>
  <a href="https://github.com/simonw/datasette-graphql/blob/main/LICENSE"><img src="https://img.shields.io/badge/license-Apache%202.0-blue.svg" alt="License" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/c355f200ea90fddaa407b6eaab303663a669248ea3ca7b1fcf77dbe04ff5f48c/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6c6963656e73652d417061636865253230322e302d626c75652e737667"></a></p>
  <p dir="auto"><strong>Datasette plugin providing an automatic GraphQL API for your SQLite databases</strong></p>
  <p dir="auto">Read more about this project: <a href="https://simonwillison.net/2020/Aug/7/datasette-graphql/" rel="nofollow">GraphQL in Datasette with the new datasette-graphql plugin</a></p>
  <p dir="auto">Try out a live demo at <a href="https://datasette-graphql-demo.datasette.io/graphql?query=%7B%0A%20%20repos(first%3A10%2C%20search%3A%20%22sql%22%2C%20sort_desc%3A%20created_at)%20%7B%0A%20%20%20%20totalCount%0A%20%20%20%20pageInfo%20%7B%0A%20%20%20%20%20%20endCursor%0A%20%20%20%20%20%20hasNextPage%0A%20%20%20%20%7D%0A%20%20%20%20nodes%20%7B%0A%20%20%20%20%20%20full_name%0A%20%20%20%20%20%20description_%0A%20%20%20%20%09stargazers_count%0A%20%20%20%20%20%20created_at%0A%20%20%20%20%20%20owner%20%7B%0A%20%20%20%20%20%20%20%20name%0A%20%20%20%20%20%20%20%20html_url%0A%20%20%20%20%20%20%7D%0A%20%20%20%20%7D%0A%20%20%7D%0A%7D%0A" rel="nofollow">datasette-graphql-demo.datasette.io/graphql</a></p>

  <ul dir="auto">
  <li><a href="#installation">Installation</a></li>
  <li><a href="#configuration">Configuration</a></li>
  <li><a href="#usage">Usage</a>
  <ul dir="auto">
  <li><a href="#querying-for-tables-and-columns">Querying for tables and columns</a></li>
  <li><a href="#fetching-a-single-record">Fetching a single record</a></li>
  <li><a href="#accessing-nested-objects">Accessing nested objects</a></li>
  <li><a href="#accessing-related-objects">Accessing related objects</a></li>
  <li><a href="#filtering-tables">Filtering tables</a></li>
  <li><a href="#sorting">Sorting</a></li>
  <li><a href="#pagination">Pagination</a></li>
  <li><a href="#search">Search</a></li>
  <li><a href="#columns-containing-json-strings">Columns containing JSON strings</a></li>
  <li><a href="#auto-camelcase">Auto camelCase</a></li>
  <li><a href="#cors">CORS</a></li>
  <li><a href="#execution-limits">Execution limits</a></li>
  </ul>
  </li>
  <li><a href="#the-graphql-template-function">The graphql() template function</a></li>
  <li><a href="#adding-custom-fields-with-plugins">Adding custom fields with plugins</a></li>
  <li><a href="#development">Development</a></li>
  </ul>

  <p dir="auto"><a target="_blank" rel="noopener noreferrer nofollow" href="https://camo.githubusercontent.com/accf29f83b9e94f31b93301c5af63ccd929258a9704d4138e72352c1f5d66850/68747470733a2f2f7374617469632e73696d6f6e77696c6c69736f6e2e6e65742f7374617469632f323032302f677261706869716c2e676966"><img src="https://static.simonwillison.net/static/2020/graphiql.gif" alt="Animated demo showing autocomplete while typing a GraphQL query into the GraphiQL interface" data-animated-image="" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/accf29f83b9e94f31b93301c5af63ccd929258a9704d4138e72352c1f5d66850/68747470733a2f2f7374617469632e73696d6f6e77696c6c69736f6e2e6e65742f7374617469632f323032302f677261706869716c2e676966"></a></p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Installation</h2><a id="user-content-installation" class="anchor" aria-label="Permalink: Installation" href="#installation"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">Install this plugin in the same environment as Datasette.</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ datasette install datasette-graphql"><pre class="notranslate"><code>$ datasette install datasette-graphql
  </code></pre></div>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Configuration</h2><a id="user-content-configuration" class="anchor" aria-label="Permalink: Configuration" href="#configuration"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">By default this plugin adds the GraphQL API at <code>/graphql</code>. You can configure a different path using the <code>path</code> plugin setting, for example by adding this to <code>metadata.json</code>:</p>
  <div class="highlight highlight-source-json notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="{
    "plugins": {
      "datasette-graphql": {
        "path": "/-/graphql"
      }
    }
  }"><pre>{
    <span class="pl-ent">"plugins"</span>: {
      <span class="pl-ent">"datasette-graphql"</span>: {
        <span class="pl-ent">"path"</span>: <span class="pl-s"><span class="pl-pds">"</span>/-/graphql<span class="pl-pds">"</span></span>
      }
    }
  }</pre></div>
  <p dir="auto">This will set the GraphQL API to live at <code>/-/graphql</code> instead.</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Usage</h2><a id="user-content-usage" class="anchor" aria-label="Permalink: Usage" href="#usage"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">This plugin sets up <code>/graphql</code> as a GraphQL endpoint for the first attached database.</p>
  <p dir="auto">If you have multiple attached databases each will get its own endpoint at <code>/graphql/name_of_database</code>.</p>
  <p dir="auto">The automatically generated GraphQL schema is available at <code>/graphql/name_of_database.graphql</code> - here's <a href="https://datasette-graphql-demo.datasette.io/graphql/github.graphql" rel="nofollow">an example</a>.</p>
  <div class="markdown-heading" dir="auto"><h3 class="heading-element" dir="auto">Querying for tables and columns</h3><a id="user-content-querying-for-tables-and-columns" class="anchor" aria-label="Permalink: Querying for tables and columns" href="#querying-for-tables-and-columns"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">Individual tables (and SQL views) can be queried like this:</p>
  <div class="highlight highlight-source-graphql notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="{
    repos {
      nodes {
        id
        full_name
        description_
      }
    }
  }"><pre>{
    <span class="pl-v">repos</span> {
      <span class="pl-v">nodes</span> {
        <span class="pl-v">id</span>
        <span class="pl-v">full_name</span>
        <span class="pl-v">description_</span>
      }
    }
  }</pre></div>
  <p dir="auto"><a href="https://datasette-graphql-demo.datasette.io/graphql?query=%0A%7B%0A%20%20repos%20%7B%0A%20%20%20%20nodes%20%7B%0A%20%20%20%20%20%20id%0A%20%20%20%20%20%20full_name%0A%20%20%20%20%20%20description_%0A%20%20%20%20%7D%0A%20%20%7D%0A%7D%0A" rel="nofollow">Try this query</a></p>
  <p dir="auto">In this example query the underlying database table is called <code>repos</code> and its columns include <code>id</code>, <code>full_name</code> and <code>description</code>. Since <code>description</code> is a reserved word the query needs to ask for <code>description_</code> instead.</p>
  <div class="markdown-heading" dir="auto"><h3 class="heading-element" dir="auto">Fetching a single record</h3><a id="user-content-fetching-a-single-record" class="anchor" aria-label="Permalink: Fetching a single record" href="#fetching-a-single-record"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">If you only want to fetch a single record - for example if you want to fetch a row by its primary key - you can use the <code>tablename_row</code> field:</p>
  <div class="highlight highlight-source-graphql notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="{
    repos_row(id: 107914493) {
      id
      full_name
      description_
    }
  }"><pre>{
    <span class="pl-v">repos_row</span>(<span class="pl-v">id</span>: <span class="pl-c1">107914493</span>) {
      <span class="pl-v">id</span>
      <span class="pl-v">full_name</span>
      <span class="pl-v">description_</span>
    }
  }</pre></div>
  <p dir="auto"><a href="https://datasette-graphql-demo.datasette.io/graphql?query=%0A%7B%0A%20%20repos_row%28id%3A%20107914493%29%20%7B%0A%20%20%20%20id%0A%20%20%20%20full_name%0A%20%20%20%20description_%0A%20%20%7D%0A%7D%0A" rel="nofollow">Try this query</a></p>
  <p dir="auto">The <code>tablename_row</code> field accepts the primary key column (or columns) as arguments. It also supports the same <code>filter:</code>, <code>search:</code>, <code>sort:</code> and <code>sort_desc:</code> arguments as the <code>tablename</code> field, described below.</p>
  <div class="markdown-heading" dir="auto"><h3 class="heading-element" dir="auto">Accessing nested objects</h3><a id="user-content-accessing-nested-objects" class="anchor" aria-label="Permalink: Accessing nested objects" href="#accessing-nested-objects"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">If a column is a foreign key to another table, you can request columns from the table pointed to by that foreign key using a nested query like this:</p>
  <div class="highlight highlight-source-graphql notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="{
    repos {
      nodes {
        id
        full_name
        owner {
          id
          login
        }
      }
    }
  }"><pre>{
    <span class="pl-v">repos</span> {
      <span class="pl-v">nodes</span> {
        <span class="pl-v">id</span>
        <span class="pl-v">full_name</span>
        <span class="pl-v">owner</span> {
          <span class="pl-v">id</span>
          <span class="pl-v">login</span>
        }
      }
    }
  }</pre></div>
  <p dir="auto"><a href="https://datasette-graphql-demo.datasette.io/graphql?query=%0A%7B%0A%20%20repos%20%7B%0A%20%20%20%20nodes%20%7B%0A%20%20%20%20%20%20id%0A%20%20%20%20%20%20full_name%0A%20%20%20%20%20%20owner%20%7B%0A%20%20%20%20%20%20%20%20id%0A%20%20%20%20%20%20%20%20login%0A%20%20%20%20%20%20%7D%0A%20%20%20%20%7D%0A%20%20%7D%0A%7D%0A" rel="nofollow">Try this query</a></p>
  <div class="markdown-heading" dir="auto"><h3 class="heading-element" dir="auto">Accessing related objects</h3><a id="user-content-accessing-related-objects" class="anchor" aria-label="Permalink: Accessing related objects" href="#accessing-related-objects"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">If another table has a foreign key back to the table you are accessing, you can fetch rows from that related table.</p>
  <p dir="auto">Consider a <code>users</code> table which is related to <code>repos</code> - a repo has a foreign key back to the user that owns the repository. The <code>users</code> object type will have a <code>repos_by_owner_list</code> field which can be used to access those related repos:</p>
  <div class="highlight highlight-source-graphql notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="{
    users(first: 1, search: "simonw") {
      nodes {
        name
        repos_by_owner_list(first: 5) {
          totalCount
          nodes {
            full_name
          }
        }
      }
    }
  }"><pre>{
    <span class="pl-v">users</span>(<span class="pl-v">first</span>: <span class="pl-c1">1</span>, <span class="pl-v">search</span>: <span class="pl-s"><span class="pl-pds">"</span>simonw<span class="pl-pds">"</span></span>) {
      <span class="pl-v">nodes</span> {
        <span class="pl-v">name</span>
        <span class="pl-v">repos_by_owner_list</span>(<span class="pl-v">first</span>: <span class="pl-c1">5</span>) {
          <span class="pl-v">totalCount</span>
          <span class="pl-v">nodes</span> {
            <span class="pl-v">full_name</span>
          }
        }
      }
    }
  }</pre></div>
  <p dir="auto"><a href="https://datasette-graphql-demo.datasette.io/graphql?query=%0A%7B%0A%20%20users%28first%3A%201%2C%20search%3A%20%22simonw%22%29%20%7B%0A%20%20%20%20nodes%20%7B%0A%20%20%20%20%20%20name%0A%20%20%20%20%20%20repos_by_owner_list%28first%3A%205%29%20%7B%0A%20%20%20%20%20%20%20%20totalCount%0A%20%20%20%20%20%20%20%20nodes%20%7B%0A%20%20%20%20%20%20%20%20%20%20full_name%0A%20%20%20%20%20%20%20%20%7D%0A%20%20%20%20%20%20%7D%0A%20%20%20%20%7D%0A%20%20%7D%0A%7D%0A" rel="nofollow">Try this query</a></p>
  <div class="markdown-heading" dir="auto"><h3 class="heading-element" dir="auto">Filtering tables</h3><a id="user-content-filtering-tables" class="anchor" aria-label="Permalink: Filtering tables" href="#filtering-tables"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">You can filter the rows returned for a specific table using the <code>filter:</code> argument. This accepts a filter object mapping columns to operations. For example, to return just repositories with the Apache 2 license and more than 10 stars:</p>
  <div class="highlight highlight-source-graphql notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="{
    repos(filter: {license: {eq: "apache-2.0"}, stargazers_count: {gt: 10}}) {
      nodes {
        full_name
        stargazers_count
        license {
          key
        }
      }
    }
  }"><pre>{
    <span class="pl-v">repos</span>(<span class="pl-v">filter</span>: {<span class="pl-c1"><span class="pl-s">license</span></span>: {<span class="pl-c1"><span class="pl-s">eq</span></span>: <span class="pl-s"><span class="pl-pds">"</span>apache-2.0<span class="pl-pds">"</span></span>}, <span class="pl-c1"><span class="pl-s">stargazers_count</span></span>: {<span class="pl-c1"><span class="pl-s">gt</span></span>: <span class="pl-c1">10</span>}}) {
      <span class="pl-v">nodes</span> {
        <span class="pl-v">full_name</span>
        <span class="pl-v">stargazers_count</span>
        <span class="pl-v">license</span> {
          <span class="pl-v">key</span>
        }
      }
    }
  }</pre></div>
  <p dir="auto"><a href="https://datasette-graphql-demo.datasette.io/graphql?query=%0A%7B%0A%20%20repos%28filter%3A%20%7Blicense%3A%20%7Beq%3A%20%22apache-2.0%22%7D%2C%20stargazers_count%3A%20%7Bgt%3A%2010%7D%7D%29%20%7B%0A%20%20%20%20nodes%20%7B%0A%20%20%20%20%20%20full_name%0A%20%20%20%20%20%20stargazers_count%0A%20%20%20%20%20%20license%20%7B%0A%20%20%20%20%20%20%20%20key%0A%20%20%20%20%20%20%7D%0A%20%20%20%20%7D%0A%20%20%7D%0A%7D%0A" rel="nofollow">Try this query</a></p>
  <p dir="auto">See <a href="https://github.com/simonw/datasette-graphql/blob/main/examples/filters.md">table filters examples</a> for more operations, and <a href="https://docs.datasette.io/en/stable/json_api.html#column-filter-arguments" rel="nofollow">column filter arguments</a> in the Datasette documentation for details of how those operations work.</p>
  <p dir="auto">These same filters can be used on nested relationships, like so:</p>
  <div class="highlight highlight-source-graphql notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="{
    users_row(id: 9599) {
      name
      repos_by_owner_list(filter: {name: {startswith: "datasette-"}}) {
        totalCount
        nodes {
          full_name
        }
      }
    }
  }"><pre>{
    <span class="pl-v">users_row</span>(<span class="pl-v">id</span>: <span class="pl-c1">9599</span>) {
      <span class="pl-v">name</span>
      <span class="pl-v">repos_by_owner_list</span>(<span class="pl-v">filter</span>: {<span class="pl-c1"><span class="pl-s">name</span></span>: {<span class="pl-c1"><span class="pl-s">startswith</span></span>: <span class="pl-s"><span class="pl-pds">"</span>datasette-<span class="pl-pds">"</span></span>}}) {
        <span class="pl-v">totalCount</span>
        <span class="pl-v">nodes</span> {
          <span class="pl-v">full_name</span>
        }
      }
    }
  }</pre></div>
  <p dir="auto"><a href="https://datasette-graphql-demo.datasette.io/graphql?query=%0A%7B%0A%20%20users_row%28id%3A%209599%29%20%7B%0A%20%20%20%20name%0A%20%20%20%20repos_by_owner_list%28filter%3A%20%7Bname%3A%20%7Bstartswith%3A%20%22datasette-%22%7D%7D%29%20%7B%0A%20%20%20%20%20%20totalCount%0A%20%20%20%20%20%20nodes%20%7B%0A%20%20%20%20%20%20%20%20full_name%0A%20%20%20%20%20%20%7D%0A%20%20%20%20%7D%0A%20%20%7D%0A%7D%0A" rel="nofollow">Try this query</a></p>
  <p dir="auto">The <code>where:</code> argument can be used as an alternative to <code>filter:</code> when the thing you are expressing is too complex to be modeled using a filter expression. It accepts a string fragment of SQL that will be included in the <code>WHERE</code> clause of the SQL query.</p>
  <div class="highlight highlight-source-graphql notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="{
    repos(where: "name='sqlite-utils' or name like 'datasette-%'") {
      totalCount
      nodes {
        full_name
      }
    }
  }"><pre>{
    <span class="pl-v">repos</span>(<span class="pl-v">where</span>: <span class="pl-s"><span class="pl-pds">"</span>name='sqlite-utils' or name like 'datasette-%'<span class="pl-pds">"</span></span>) {
      <span class="pl-v">totalCount</span>
      <span class="pl-v">nodes</span> {
        <span class="pl-v">full_name</span>
      }
    }
  }</pre></div>
  <p dir="auto"><a href="https://datasette-graphql-demo.datasette.io/graphql?query=%0A%7B%0A%20%20repos%28where%3A%20%22name%3D%27sqlite-utils%27%20or%20name%20like%20%27datasette-%25%27%22%29%20%7B%0A%20%20%20%20totalCount%0A%20%20%20%20nodes%20%7B%0A%20%20%20%20%20%20full_name%0A%20%20%20%20%7D%0A%20%20%7D%0A%7D%0A" rel="nofollow">Try this query</a></p>
  <div class="markdown-heading" dir="auto"><h3 class="heading-element" dir="auto">Sorting</h3><a id="user-content-sorting" class="anchor" aria-label="Permalink: Sorting" href="#sorting"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">You can set a sort order for results from a table using the <code>sort:</code> or <code>sort_desc:</code> arguments. The value for this argument should be the name of the column you wish to sort (or sort-descending) by.</p>
  <div class="highlight highlight-source-graphql notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="{
    repos(sort_desc: stargazers_count) {
      nodes {
        full_name
        stargazers_count
      }
    }
  }"><pre>{
    <span class="pl-v">repos</span>(<span class="pl-v">sort_desc</span>:<span class="pl-c1"> stargazers_count</span>) {
      <span class="pl-v">nodes</span> {
        <span class="pl-v">full_name</span>
        <span class="pl-v">stargazers_count</span>
      }
    }
  }</pre></div>
  <p dir="auto"><a href="https://datasette-graphql-demo.datasette.io/graphql?query=%0A%7B%0A%20%20repos%28sort_desc%3A%20stargazers_count%29%20%7B%0A%20%20%20%20nodes%20%7B%0A%20%20%20%20%20%20full_name%0A%20%20%20%20%20%20stargazers_count%0A%20%20%20%20%7D%0A%20%20%7D%0A%7D%0A" rel="nofollow">Try this query</a></p>
  <div class="markdown-heading" dir="auto"><h3 class="heading-element" dir="auto">Pagination</h3><a id="user-content-pagination" class="anchor" aria-label="Permalink: Pagination" href="#pagination"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">By default the first 10 rows will be returned. You can control this using the <code>first:</code> argument.</p>
  <div class="highlight highlight-source-graphql notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="{
    repos(first: 20) {
      totalCount
      pageInfo {
        hasNextPage
        endCursor
      }
      nodes {
        full_name
        stargazers_count
        license {
          key
        }
      }
    }
  }"><pre>{
    <span class="pl-v">repos</span>(<span class="pl-v">first</span>: <span class="pl-c1">20</span>) {
      <span class="pl-v">totalCount</span>
      <span class="pl-v">pageInfo</span> {
        <span class="pl-v">hasNextPage</span>
        <span class="pl-v">endCursor</span>
      }
      <span class="pl-v">nodes</span> {
        <span class="pl-v">full_name</span>
        <span class="pl-v">stargazers_count</span>
        <span class="pl-v">license</span> {
          <span class="pl-v">key</span>
        }
      }
    }
  }</pre></div>
  <p dir="auto"><a href="https://datasette-graphql-demo.datasette.io/graphql?query=%0A%7B%0A%20%20repos%28first%3A%2020%29%20%7B%0A%20%20%20%20totalCount%0A%20%20%20%20pageInfo%20%7B%0A%20%20%20%20%20%20hasNextPage%0A%20%20%20%20%20%20endCursor%0A%20%20%20%20%7D%0A%20%20%20%20nodes%20%7B%0A%20%20%20%20%20%20full_name%0A%20%20%20%20%20%20stargazers_count%0A%20%20%20%20%20%20license%20%7B%0A%20%20%20%20%20%20%20%20key%0A%20%20%20%20%20%20%7D%0A%20%20%20%20%7D%0A%20%20%7D%0A%7D%0A" rel="nofollow">Try this query</a></p>
  <p dir="auto">The <code>totalCount</code> field returns the total number of records that match the query.</p>
  <p dir="auto">Requesting the <code>pageInfo.endCursor</code> field provides you with the value you need to request the next page. You can pass this to the <code>after:</code> argument to request the next page.</p>
  <div class="highlight highlight-source-graphql notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="{
    repos(first: 20, after: "134874019") {
      totalCount
      pageInfo {
        hasNextPage
        endCursor
      }
      nodes {
        full_name
        stargazers_count
        license {
          key
        }
      }
    }
  }"><pre>{
    <span class="pl-v">repos</span>(<span class="pl-v">first</span>: <span class="pl-c1">20</span>, <span class="pl-v">after</span>: <span class="pl-s"><span class="pl-pds">"</span>134874019<span class="pl-pds">"</span></span>) {
      <span class="pl-v">totalCount</span>
      <span class="pl-v">pageInfo</span> {
        <span class="pl-v">hasNextPage</span>
        <span class="pl-v">endCursor</span>
      }
      <span class="pl-v">nodes</span> {
        <span class="pl-v">full_name</span>
        <span class="pl-v">stargazers_count</span>
        <span class="pl-v">license</span> {
          <span class="pl-v">key</span>
        }
      }
    }
  }</pre></div>
  <p dir="auto"><a href="https://datasette-graphql-demo.datasette.io/graphql?query=%0A%7B%0A%20%20repos%28first%3A%2020%2C%20after%3A%20%22134874019%22%29%20%7B%0A%20%20%20%20totalCount%0A%20%20%20%20pageInfo%20%7B%0A%20%20%20%20%20%20hasNextPage%0A%20%20%20%20%20%20endCursor%0A%20%20%20%20%7D%0A%20%20%20%20nodes%20%7B%0A%20%20%20%20%20%20full_name%0A%20%20%20%20%20%20stargazers_count%0A%20%20%20%20%20%20license%20%7B%0A%20%20%20%20%20%20%20%20key%0A%20%20%20%20%20%20%7D%0A%20%20%20%20%7D%0A%20%20%7D%0A%7D%0A" rel="nofollow">Try this query</a></p>
  <p dir="auto">The <code>hasNextPage</code> field tells you if there are any more records.</p>
  <div class="markdown-heading" dir="auto"><h3 class="heading-element" dir="auto">Search</h3><a id="user-content-search" class="anchor" aria-label="Permalink: Search" href="#search"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">If a table has been configured to use SQLite full-text search you can execute searches against it using the <code>search:</code> argument:</p>
  <div class="highlight highlight-source-graphql notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="{
    repos(search: "datasette") {
      totalCount
      pageInfo {
        hasNextPage
        endCursor
      }
      nodes {
        full_name
        description_
      }
    }
  }"><pre>{
    <span class="pl-v">repos</span>(<span class="pl-v">search</span>: <span class="pl-s"><span class="pl-pds">"</span>datasette<span class="pl-pds">"</span></span>) {
      <span class="pl-v">totalCount</span>
      <span class="pl-v">pageInfo</span> {
        <span class="pl-v">hasNextPage</span>
        <span class="pl-v">endCursor</span>
      }
      <span class="pl-v">nodes</span> {
        <span class="pl-v">full_name</span>
        <span class="pl-v">description_</span>
      }
    }
  }</pre></div>
  <p dir="auto"><a href="https://datasette-graphql-demo.datasette.io/graphql?query=%0A%7B%0A%20%20repos%28search%3A%20%22datasette%22%29%20%7B%0A%20%20%20%20totalCount%0A%20%20%20%20pageInfo%20%7B%0A%20%20%20%20%20%20hasNextPage%0A%20%20%20%20%20%20endCursor%0A%20%20%20%20%7D%0A%20%20%20%20nodes%20%7B%0A%20%20%20%20%20%20full_name%0A%20%20%20%20%20%20description_%0A%20%20%20%20%7D%0A%20%20%7D%0A%7D%0A" rel="nofollow">Try this query</a></p>
  <p dir="auto">The <a href="https://sqlite-utils.datasette.io/" rel="nofollow">sqlite-utils</a> Python library and CLI tool can be used to add full-text search to an existing database table.</p>
  <div class="markdown-heading" dir="auto"><h3 class="heading-element" dir="auto">Columns containing JSON strings</h3><a id="user-content-columns-containing-json-strings" class="anchor" aria-label="Permalink: Columns containing JSON strings" href="#columns-containing-json-strings"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">If your table has a column that contains data encoded as JSON, <code>datasette-graphql</code> will make that column available as an encoded JSON string. Clients calling your API will need to parse the string as JSON in order to access the data.</p>
  <p dir="auto">You can return the data as a nested structure by configuring that column to be treated as a JSON column. The <a href="https://docs.datasette.io/en/stable/plugins.html#plugin-configuration" rel="nofollow">plugin configuration</a> for that in <code>metadata.json</code> looks like this:</p>
  <div class="highlight highlight-source-json notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="{
      "databases": {
          "test": {
              "tables": {
                  "repos": {
                      "plugins": {
                          "datasette-graphql": {
                              "json_columns": [
                                  "tags"
                              ]
                          }
                      }
                  }
              }
          }
      }
  }"><pre>{
      <span class="pl-ent">"databases"</span>: {
          <span class="pl-ent">"test"</span>: {
              <span class="pl-ent">"tables"</span>: {
                  <span class="pl-ent">"repos"</span>: {
                      <span class="pl-ent">"plugins"</span>: {
                          <span class="pl-ent">"datasette-graphql"</span>: {
                              <span class="pl-ent">"json_columns"</span>: [
                                  <span class="pl-s"><span class="pl-pds">"</span>tags<span class="pl-pds">"</span></span>
                              ]
                          }
                      }
                  }
              }
          }
      }
  }</pre></div>
  <div class="markdown-heading" dir="auto"><h3 class="heading-element" dir="auto">Auto camelCase</h3><a id="user-content-auto-camelcase" class="anchor" aria-label="Permalink: Auto camelCase" href="#auto-camelcase"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">The names of your columns and tables default to being matched by their representations in GraphQL.</p>
  <p dir="auto">If you have tables with <code>names_like_this</code> you may want to work with them in GraphQL using <code>namesLikeThis</code>, for consistency with GraphQL and JavaScript conventions.</p>
  <p dir="auto">You can turn on automatic camelCase using the <code>"auto_camelcase"</code> plugin configuration setting in <code>metadata.json</code>, like this:</p>
  <div class="highlight highlight-source-json notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="{
      "plugins": {
          "datasette-graphql": {
              "auto_camelcase": true
          }
      }
  }"><pre>{
      <span class="pl-ent">"plugins"</span>: {
          <span class="pl-ent">"datasette-graphql"</span>: {
              <span class="pl-ent">"auto_camelcase"</span>: <span class="pl-c1">true</span>
          }
      }
  }</pre></div>
  <div class="markdown-heading" dir="auto"><h3 class="heading-element" dir="auto">CORS</h3><a id="user-content-cors" class="anchor" aria-label="Permalink: CORS" href="#cors"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">This plugin obeys the <code>--cors</code> option passed to the <code>datasette</code> command-line tool. If you pass <code>--cors</code> it adds the following CORS HTTP headers to allow JavaScript running on other domains to access the GraphQL API:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="access-control-allow-headers: content-type
  access-control-allow-method: POST
  access-control-allow-origin: *"><pre class="notranslate"><code>access-control-allow-headers: content-type
  access-control-allow-method: POST
  access-control-allow-origin: *
  </code></pre></div>
  <div class="markdown-heading" dir="auto"><h3 class="heading-element" dir="auto">Execution limits</h3><a id="user-content-execution-limits" class="anchor" aria-label="Permalink: Execution limits" href="#execution-limits"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">The plugin implements two limits by default:</p>
  <ul dir="auto">
  <li>The total time spent executing all of the underlying SQL queries that make up the GraphQL execution must not exceed 1000ms (one second)</li>
  <li>The total number of SQL table queries executed as a result of nested GraphQL fields must not exceed 100</li>
  </ul>
  <p dir="auto">These limits can be customized using the <code>num_queries_limit</code> and <code>time_limit_ms</code> plugin configuration settings, for example in <code>metadata.json</code>:</p>
  <div class="highlight highlight-source-json notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="{
      "plugins": {
          "datasette-graphql": {
              "num_queries_limit": 200,
              "time_limit_ms": 5000
          }
      }
  }"><pre>{
      <span class="pl-ent">"plugins"</span>: {
          <span class="pl-ent">"datasette-graphql"</span>: {
              <span class="pl-ent">"num_queries_limit"</span>: <span class="pl-c1">200</span>,
              <span class="pl-ent">"time_limit_ms"</span>: <span class="pl-c1">5000</span>
          }
      }
  }</pre></div>
  <p dir="auto">Setting these to <code>0</code> will disable the limit checks entirely.</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">The graphql() template function</h2><a id="user-content-the-graphql-template-function" class="anchor" aria-label="Permalink: The graphql() template function" href="#the-graphql-template-function"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">The plugin also makes a Jinja template function available called <code>graphql()</code>. You can use that function in your Datasette <a href="https://docs.datasette.io/en/stable/custom_templates.html#custom-templates" rel="nofollow">custom templates</a> like so:</p>
  <div class="highlight highlight-text-html-django notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="{% set users = graphql("""
  {
      users {
          nodes {
              name
              points
              score
          }
      }
  }
  """)["users"] %}
  {% for user in users.nodes %}
      <p>{{ user.name }} - points: {{ user.points }}, score = {{ user.score }}</p>
  {% endfor %}"><pre><span class="pl-e">{%</span> <span class="pl-s">set</span> <span class="pl-s">users</span> = <span class="pl-s">graphql</span>(<span class="pl-s">"""</span>
  <span class="pl-s">{</span>
  <span class="pl-s">    users {</span>
  <span class="pl-s">        nodes {</span>
  <span class="pl-s">            name</span>
  <span class="pl-s">            points</span>
  <span class="pl-s">            score</span>
  <span class="pl-s">        }</span>
  <span class="pl-s">    }</span>
  <span class="pl-s">}</span>
  <span class="pl-s">"""</span>)[<span class="pl-s">"users"</span>] <span class="pl-e">%}</span>
  <span class="pl-e">{%</span> <span class="pl-k">for</span> <span class="pl-s">user</span> <span class="pl-k">in</span> <span class="pl-s">users</span>.<span class="pl-s">nodes</span> <span class="pl-e">%}</span>
      <<span class="pl-ent">p</span>>{{ user.name }} - points: {{ user.points }}, score = {{ user.score }}</<span class="pl-ent">p</span>>
  <span class="pl-e">{%</span> <span class="pl-k">endfor</span> <span class="pl-e">%}</span></pre></div>
  <p dir="auto">The function executes a GraphQL query against the generated schema and returns the results. You can assign those results to a variable in your template and then loop through and display them.</p>
  <p dir="auto">By default the query will be run against the first attached database. You can use the optional second argument to the function to specify a different database - for example, to run against an attached <code>github.db</code> database you would do this:</p>
  <div class="highlight highlight-text-html-django notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="{% set user = graphql("""
  {
      users_row(id:9599) {
          name
          login
          avatar_url
      }
  }
  """, "github")["users_row"] %}

  <h1>Hello, {{ user.name }}</h1>"><pre><span class="pl-e">{%</span> <span class="pl-s">set</span> <span class="pl-s">user</span> = <span class="pl-s">graphql</span>(<span class="pl-s">"""</span>
  <span class="pl-s">{</span>
  <span class="pl-s">    users_row(id:9599) {</span>
  <span class="pl-s">        name</span>
  <span class="pl-s">        login</span>
  <span class="pl-s">        avatar_url</span>
  <span class="pl-s">    }</span>
  <span class="pl-s">}</span>
  <span class="pl-s">"""</span>, <span class="pl-s">"github"</span>)[<span class="pl-s">"users_row"</span>] <span class="pl-e">%}</span>

  <<span class="pl-ent">h1</span>>Hello, {{ user.name }}</<span class="pl-ent">h1</span>></pre></div>
  <p dir="auto">You can use <a href="https://graphql.org/learn/queries/#variables" rel="nofollow">GraphQL variables</a> in these template calls by passing them to the <code>variables=</code> argument:</p>
  <div class="highlight highlight-text-html-django notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="{% set user = graphql("""
  query ($id: Int) {
      users_row(id: $id) {
          name
          login
          avatar_url
      }
  }
  """, database="github", variables={"id": 9599})["users_row"] %}

  <h1>Hello, {{ user.name }}</h1>"><pre><span class="pl-e">{%</span> <span class="pl-s">set</span> <span class="pl-s">user</span> = <span class="pl-s">graphql</span>(<span class="pl-s">"""</span>
  <span class="pl-s">query ($id: Int) {</span>
  <span class="pl-s">    users_row(id: $id) {</span>
  <span class="pl-s">        name</span>
  <span class="pl-s">        login</span>
  <span class="pl-s">        avatar_url</span>
  <span class="pl-s">    }</span>
  <span class="pl-s">}</span>
  <span class="pl-s">"""</span>, <span class="pl-s">database</span>=<span class="pl-s">"github"</span>, <span class="pl-s">variables</span>={<span class="pl-s">"id"</span>: 9599})[<span class="pl-s">"users_row"</span>] <span class="pl-e">%}</span>

  <<span class="pl-ent">h1</span>>Hello, {{ user.name }}</<span class="pl-ent">h1</span>></pre></div>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Adding custom fields with plugins</h2><a id="user-content-adding-custom-fields-with-plugins" class="anchor" aria-label="Permalink: Adding custom fields with plugins" href="#adding-custom-fields-with-plugins"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto"><code>datasette-graphql</code> adds a new <a href="https://docs.datasette.io/en/stable/writing_plugins.html" rel="nofollow">plugin hook</a> to Datasette which can be used to add custom fields to your GraphQL schema.</p>
  <p dir="auto">The plugin hook looks like this:</p>
  <div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="@hookimpl
  def graphql_extra_fields(datasette, database):
      "A list of (name, field_type) tuples to include in the GraphQL schema""><pre><span class="pl-en">@<span class="pl-s1">hookimpl</span></span>
  <span class="pl-k">def</span> <span class="pl-en">graphql_extra_fields</span>(<span class="pl-s1">datasette</span>, <span class="pl-s1">database</span>):
      <span class="pl-s">"A list of (name, field_type) tuples to include in the GraphQL schema"</span></pre></div>
  <p dir="auto">You can use this hook to return a list of tuples describing additional fields that should be exposed in your schema. Each tuple should consist of a string naming the new field, plus a <a href="https://docs.graphene-python.org/en/latest/types/objecttypes/" rel="nofollow">Graphene Field object</a> that specifies the schema and provides a <code>resolver</code> function.</p>
  <p dir="auto">This example implementation uses <code>pkg_resources</code> to return a list of currently installed Python packages:</p>
  <div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="import graphene
  from datasette import hookimpl
  import pkg_resources


  @hookimpl
  def graphql_extra_fields():
      class Package(graphene.ObjectType):
          "An installed package"
          name = graphene.String()
          version = graphene.String()

      def resolve_packages(root, info):
          return [
              {"name": d.project_name, "version": d.version}
              for d in pkg_resources.working_set
          ]

      return [
          (
              "packages",
              graphene.Field(
                  graphene.List(Package),
                  description="List of installed packages",
                  resolver=resolve_packages,
              ),
          ),
      ]"><pre><span class="pl-k">import</span> <span class="pl-s1">graphene</span>
  <span class="pl-k">from</span> <span class="pl-s1">datasette</span> <span class="pl-k">import</span> <span class="pl-s1">hookimpl</span>
  <span class="pl-k">import</span> <span class="pl-s1">pkg_resources</span>


  <span class="pl-en">@<span class="pl-s1">hookimpl</span></span>
  <span class="pl-k">def</span> <span class="pl-en">graphql_extra_fields</span>():
      <span class="pl-k">class</span> <span class="pl-v">Package</span>(<span class="pl-s1">graphene</span>.<span class="pl-c1">ObjectType</span>):
          <span class="pl-s">"An installed package"</span>
          <span class="pl-s1">name</span> <span class="pl-c1">=</span> <span class="pl-s1">graphene</span>.<span class="pl-c1">String</span>()
          <span class="pl-s1">version</span> <span class="pl-c1">=</span> <span class="pl-s1">graphene</span>.<span class="pl-c1">String</span>()

      <span class="pl-k">def</span> <span class="pl-en">resolve_packages</span>(<span class="pl-s1">root</span>, <span class="pl-s1">info</span>):
          <span class="pl-k">return</span> [
              {<span class="pl-s">"name"</span>: <span class="pl-s1">d</span>.<span class="pl-c1">project_name</span>, <span class="pl-s">"version"</span>: <span class="pl-s1">d</span>.<span class="pl-c1">version</span>}
              <span class="pl-k">for</span> <span class="pl-s1">d</span> <span class="pl-c1">in</span> <span class="pl-s1">pkg_resources</span>.<span class="pl-c1">working_set</span>
          ]

      <span class="pl-k">return</span> [
          (
              <span class="pl-s">"packages"</span>,
              <span class="pl-s1">graphene</span>.<span class="pl-c1">Field</span>(
                  <span class="pl-s1">graphene</span>.<span class="pl-c1">List</span>(<span class="pl-v">Package</span>),
                  <span class="pl-s1">description</span><span class="pl-c1">=</span><span class="pl-s">"List of installed packages"</span>,
                  <span class="pl-s1">resolver</span><span class="pl-c1">=</span><span class="pl-s1">resolve_packages</span>,
              ),
          ),
      ]</pre></div>
  <p dir="auto">With this plugin installed, the following GraphQL query can be used to retrieve a list of installed packages:</p>
  <div class="highlight highlight-source-graphql notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="{
    packages {
      name
      version
    }
  }"><pre>{
    <span class="pl-v">packages</span> {
      <span class="pl-v">name</span>
      <span class="pl-v">version</span>
    }
  }</pre></div>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Development</h2><a id="user-content-development" class="anchor" aria-label="Permalink: Development" href="#development"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">To set up this plugin locally, first checkout the code. Then create a new virtual environment:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="cd datasette-graphql
  python3 -mvenv venv
  source venv/bin/activate"><pre class="notranslate"><code>cd datasette-graphql
  python3 -mvenv venv
  source venv/bin/activate
  </code></pre></div>
  <p dir="auto">Or if you are using <code>pipenv</code>:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="pipenv shell"><pre class="notranslate"><code>pipenv shell
  </code></pre></div>
  <p dir="auto">Now install the dependencies and tests:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="pip install -e '.[test]'"><pre class="notranslate"><code>pip install -e '.[test]'
  </code></pre></div>
  <p dir="auto">To run the tests:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="pytest"><pre class="notranslate"><code>pytest
  </code></pre></div>
  </article></div>


node_id: MDEwOlJlcG9zaXRvcnkyODg2Mjk3NjY=
name: datasette-schema-versions
full_name: simonw/datasette-schema-versions
html_url: https://github.com/simonw/datasette-schema-versions
description: Datasette plugin that shows the schema version of every attached database
created_at: 2020-08-19T04:04:39Z
updated_at: 2024-03-01T02:39:30Z
pushed_at: 2024-03-01T02:45:22Z
language: Python
topics: ["datasette", "datasette-io", "datasette-plugin"]
visibility: public
default_branch: main
permissions: {"admin": false, "maintain": false, "push": false, "triage": false, "pull": true}
readme:
  # datasette-schema-versions

  [![PyPI](https://img.shields.io/pypi/v/datasette-schema-versions.svg)](https://pypi.org/project/datasette-schema-versions/)
  [![Changelog](https://img.shields.io/github/v/release/simonw/datasette-schema-versions?include_prereleases&label=changelog)](https://github.com/simonw/datasette-schema-versions/releases)
  [![Tests](https://github.com/simonw/datasette-schema-versions/workflows/Test/badge.svg)](https://github.com/simonw/datasette-schema-versions/actions?query=workflow%3ATest)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/datasette-schema-versions/blob/main/LICENSE)

  Datasette plugin that shows the schema version of every attached database

  ## Installation

  Install this plugin in the same environment as Datasette.
  ```bash
  datasette install datasette-schema-versions
  ```
  ## Usage

  Visit `/-/schema-versions` on your Datasette instance to see a numeric version for the schema for each of your databases.

  Any changes you make to the schema will increase this version number.

readme_html:
  <div id="readme" class="md" data-path="README.md"><article class="markdown-body entry-content container-lg" itemprop="text"><div class="markdown-heading" dir="auto"><h1 class="heading-element" dir="auto">datasette-schema-versions</h1><a id="user-content-datasette-schema-versions" class="anchor" aria-label="Permalink: datasette-schema-versions" href="#datasette-schema-versions"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto"><a href="https://pypi.org/project/datasette-schema-versions/" rel="nofollow"><img src="https://img.shields.io/pypi/v/datasette-schema-versions.svg" alt="PyPI" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/180fc579e2598a915d9ae9a2cc94a21b38e47fc4378483176370abe970d66bf8/68747470733a2f2f696d672e736869656c64732e696f2f707970692f762f6461746173657474652d736368656d612d76657273696f6e732e737667"></a>
  <a href="https://github.com/simonw/datasette-schema-versions/releases"><img src="https://img.shields.io/github/v/release/simonw/datasette-schema-versions?include_prereleases&label=changelog" alt="Changelog" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/f63d0832ccf4c251c79b2d20ca32dcb8651767daaba3a308d5962e8168d17b09/68747470733a2f2f696d672e736869656c64732e696f2f6769746875622f762f72656c656173652f73696d6f6e772f6461746173657474652d736368656d612d76657273696f6e733f696e636c7564655f70726572656c6561736573266c6162656c3d6368616e67656c6f67"></a>
  <a href="https://github.com/simonw/datasette-schema-versions/actions?query=workflow%3ATest"><img src="https://github.com/simonw/datasette-schema-versions/workflows/Test/badge.svg" alt="Tests" style="max-width: 100%;"></a>
  <a href="https://github.com/simonw/datasette-schema-versions/blob/main/LICENSE"><img src="https://img.shields.io/badge/license-Apache%202.0-blue.svg" alt="License" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/c355f200ea90fddaa407b6eaab303663a669248ea3ca7b1fcf77dbe04ff5f48c/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6c6963656e73652d417061636865253230322e302d626c75652e737667"></a></p>
  <p dir="auto">Datasette plugin that shows the schema version of every attached database</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Installation</h2><a id="user-content-installation" class="anchor" aria-label="Permalink: Installation" href="#installation"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">Install this plugin in the same environment as Datasette.</p>
  <div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="datasette install datasette-schema-versions"><pre>datasette install datasette-schema-versions</pre></div>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Usage</h2><a id="user-content-usage" class="anchor" aria-label="Permalink: Usage" href="#usage"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">Visit <code>/-/schema-versions</code> on your Datasette instance to see a numeric version for the schema for each of your databases.</p>
  <p dir="auto">Any changes you make to the schema will increase this version number.</p>
  </article></div>


node_id: MDEwOlJlcG9zaXRvcnkyOTEzMzkwODY=
name: airtable-export
full_name: simonw/airtable-export
html_url: https://github.com/simonw/airtable-export
description: Export Airtable data to YAML, JSON or SQLite files on disk
created_at: 2020-08-29T19:51:37Z
updated_at: 2025-05-13T15:33:00Z
pushed_at: 2024-04-26T01:24:23Z
homepage: https://datasette.io/tools/airtable-export
language: Python
license: apache-2.0
topics: ["airtable", "airtable-api", "datasette-io", "datasette-tool", "yaml"]
visibility: public
default_branch: main
permissions: {"admin": false, "maintain": false, "push": false, "triage": false, "pull": true}
readme:
  # airtable-export

  [![PyPI](https://img.shields.io/pypi/v/airtable-export.svg)](https://pypi.org/project/airtable-export/)
  [![Changelog](https://img.shields.io/github/v/release/simonw/airtable-export?include_prereleases&label=changelog)](https://github.com/simonw/airtable-export/releases)
  [![Tests](https://github.com/simonw/airtable-export/workflows/Test/badge.svg)](https://github.com/simonw/airtable-export/actions?query=workflow%3ATest)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/airtable-export/blob/master/LICENSE)

  Export Airtable data to files on disk

  ## Installation

  Install this tool using `pip`:

      $ pip install airtable-export

  ## Usage

  You will need to the following information:

  - Your Airtable base ID - this is a string starting with `app...`
  - Your Airtable personal access token - this is a string starting with `pat...`

  If you just want to export a subset of your tables you also need to know the names of those tables.

  You can export all of your data to a folder called `export/` by running the following:

      airtable-export export base_id --key=key

  This example would files for each of your tables, for example: `export/table1.yml` and `export/table2.yml`.

  Rather than passing the API key using the `--key` option you can set it as an environment variable called `AIRTABLE_KEY`.

  To export only specified tables, pass their names as additional arguments:

      airtable-export export base_id table1 table2 --key=key

  ## Export options

  By default the tool exports your data as YAML.

  You can also export as JSON or as [newline delimited JSON](http://ndjson.org/) using the `--json` or `--ndjson` options:

      airtable-export export base_id --key=key --ndjson

  You can pass multiple format options at once. This command will create a `.json`, `.yml` and `.ndjson` file for each exported table:

      airtable-export export base_id \
          --key=key --ndjson --yaml --json

  If you import all tables, or if you add the `--schema` option, a JSON schema for the base will be written to `output-dir/_schema.json`.

  ### SQLite database export

  You can export tables to a SQLite database file using the `--sqlite database.db` option:

      airtable-export export base_id \
          --key=key --sqlite database.db

  This can be combined with other format options. If you only specify `--sqlite` the export directory argument will be ignored.

  The SQLite database will have a table created for each table you export. Those tables will have a primary key column called `airtable_id`.

  If you run this command against an existing SQLite database records with matching primary keys will be over-written by new records from the export.

  ## Request options

  By default the tool uses [python-httpx](https://www.python-httpx.org)'s default configurations.

  You can override the `user-agent` using the `--user-agent` option:

      airtable-export export base_id table1 table2 --key=key --user-agent "Airtable Export Robot"

  You can override the [timeout during a network read operation](https://www.python-httpx.org/advanced/#fine-tuning-the-configuration) using the `--http-read-timeout` option. If not set, this defaults to 5s.

      airtable-export export base_id table1 table2 --key=key --http-read-timeout 60

  ## Running this using GitHub Actions

  [GitHub Actions](https://github.com/features/actions) is GitHub's workflow automation product. You can use it to run `airtable-export` in order to back up your Airtable data to a GitHub repository. Doing this gives you a visible commit history of changes you make to your Airtable data - like [this one](https://github.com/natbat/rockybeaches/commits/main/airtable).

  To run this for your own Airtable database you'll first need to add the following secrets to your GitHub repository:

  <dl>
    <dt>AIRTABLE_BASE_ID</dt>
    <dd>The base ID, a string beginning `app...`</dd>
    <dt>AIRTABLE_KEY</dt>
    <dd>Your Airtable API key</dd>
    <dt>AIRTABLE_TABLES</dt>
    <dd>A space separated list of the Airtable tables that you want to backup. If any of these contain spaces you will need to enclose them in single quotes, e.g. <samp>'My table with spaces in the name' OtherTableWithNoSpaces</samp></dd>
  </dl>

  Once you have set those secrets, add the following as a file called `.github/workflows/backup-airtable.yml`:
  ```yaml
  name: Backup Airtable

  on:
    workflow_dispatch:
    schedule:
    - cron: '32 0 * * *'

  jobs:
    build:
      runs-on: ubuntu-latest
      steps:
      - name: Check out repo
        uses: actions/checkout@v2
      - name: Set up Python
        uses: actions/setup-python@v2
        with:
          python-version: 3.8
      - uses: actions/cache@v2
        name: Configure pip caching
        with:
          path: ~/.cache/pip
          key: ${{ runner.os }}-pip-
          restore-keys: |
            ${{ runner.os }}-pip-
      - name: Install airtable-export
        run: |
          pip install airtable-export
      - name: Backup Airtable to backups/
        env:
          AIRTABLE_BASE_ID: ${{ secrets.AIRTABLE_BASE_ID }}
          AIRTABLE_KEY: ${{ secrets.AIRTABLE_KEY }}
          AIRTABLE_TABLES: ${{ secrets.AIRTABLE_TABLES }}
        run: |-
          airtable-export backups $AIRTABLE_BASE_ID $AIRTABLE_TABLES -v
      - name: Commit and push if it changed
        run: |-
          git config user.name "Automated"
          git config user.email "actions@users.noreply.github.com"
          git add -A
          timestamp=$(date -u)
          git commit -m "Latest data: ${timestamp}" || exit 0
          git push
  ```
  This will run once a day (at 32 minutes past midnight UTC) and will also run if you manually click the "Run workflow" button, see [GitHub Actions: Manual triggers with workflow_dispatch](https://github.blog/changelog/2020-07-06-github-actions-manual-triggers-with-workflow_dispatch/).

  ## Development

  To contribute to this tool, first checkout the code. Then create a new virtual environment:

      cd airtable-export
      python -mvenv venv
      source venv/bin/activate

  Or if you are using `pipenv`:

      pipenv shell

  Now install the dependencies and tests:

      pip install -e '.[test]'

  To run the tests:

      pytest

readme_html:
  <div id="readme" class="md" data-path="README.md"><article class="markdown-body entry-content container-lg" itemprop="text"><div class="markdown-heading" dir="auto"><h1 class="heading-element" dir="auto">airtable-export</h1><a id="user-content-airtable-export" class="anchor" aria-label="Permalink: airtable-export" href="#airtable-export"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto"><a href="https://pypi.org/project/airtable-export/" rel="nofollow"><img src="https://img.shields.io/pypi/v/airtable-export.svg" alt="PyPI" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/f2798e847a4bea29ff82f8fa99309141a7232916cfd7d9a785afa5dd844beed5/68747470733a2f2f696d672e736869656c64732e696f2f707970692f762f6169727461626c652d6578706f72742e737667"></a>
  <a href="https://github.com/simonw/airtable-export/releases"><img src="https://img.shields.io/github/v/release/simonw/airtable-export?include_prereleases&label=changelog" alt="Changelog" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/7262286038227e419aae5764d45a725588159e717c4e3899ffadb8afd394ab4d/68747470733a2f2f696d672e736869656c64732e696f2f6769746875622f762f72656c656173652f73696d6f6e772f6169727461626c652d6578706f72743f696e636c7564655f70726572656c6561736573266c6162656c3d6368616e67656c6f67"></a>
  <a href="https://github.com/simonw/airtable-export/actions?query=workflow%3ATest"><img src="https://github.com/simonw/airtable-export/workflows/Test/badge.svg" alt="Tests" style="max-width: 100%;"></a>
  <a href="https://github.com/simonw/airtable-export/blob/master/LICENSE"><img src="https://img.shields.io/badge/license-Apache%202.0-blue.svg" alt="License" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/c355f200ea90fddaa407b6eaab303663a669248ea3ca7b1fcf77dbe04ff5f48c/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6c6963656e73652d417061636865253230322e302d626c75652e737667"></a></p>
  <p dir="auto">Export Airtable data to files on disk</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Installation</h2><a id="user-content-installation" class="anchor" aria-label="Permalink: Installation" href="#installation"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">Install this tool using <code>pip</code>:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ pip install airtable-export"><pre class="notranslate"><code>$ pip install airtable-export
  </code></pre></div>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Usage</h2><a id="user-content-usage" class="anchor" aria-label="Permalink: Usage" href="#usage"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">You will need to the following information:</p>
  <ul dir="auto">
  <li>Your Airtable base ID - this is a string starting with <code>app...</code></li>
  <li>Your Airtable personal access token - this is a string starting with <code>pat...</code></li>
  </ul>
  <p dir="auto">If you just want to export a subset of your tables you also need to know the names of those tables.</p>
  <p dir="auto">You can export all of your data to a folder called <code>export/</code> by running the following:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="airtable-export export base_id --key=key"><pre class="notranslate"><code>airtable-export export base_id --key=key
  </code></pre></div>
  <p dir="auto">This example would files for each of your tables, for example: <code>export/table1.yml</code> and <code>export/table2.yml</code>.</p>
  <p dir="auto">Rather than passing the API key using the <code>--key</code> option you can set it as an environment variable called <code>AIRTABLE_KEY</code>.</p>
  <p dir="auto">To export only specified tables, pass their names as additional arguments:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="airtable-export export base_id table1 table2 --key=key"><pre class="notranslate"><code>airtable-export export base_id table1 table2 --key=key
  </code></pre></div>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Export options</h2><a id="user-content-export-options" class="anchor" aria-label="Permalink: Export options" href="#export-options"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">By default the tool exports your data as YAML.</p>
  <p dir="auto">You can also export as JSON or as <a href="http://ndjson.org/" rel="nofollow">newline delimited JSON</a> using the <code>--json</code> or <code>--ndjson</code> options:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="airtable-export export base_id --key=key --ndjson"><pre class="notranslate"><code>airtable-export export base_id --key=key --ndjson
  </code></pre></div>
  <p dir="auto">You can pass multiple format options at once. This command will create a <code>.json</code>, <code>.yml</code> and <code>.ndjson</code> file for each exported table:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="airtable-export export base_id \
      --key=key --ndjson --yaml --json"><pre class="notranslate"><code>airtable-export export base_id \
      --key=key --ndjson --yaml --json
  </code></pre></div>
  <p dir="auto">If you import all tables, or if you add the <code>--schema</code> option, a JSON schema for the base will be written to <code>output-dir/_schema.json</code>.</p>
  <div class="markdown-heading" dir="auto"><h3 class="heading-element" dir="auto">SQLite database export</h3><a id="user-content-sqlite-database-export" class="anchor" aria-label="Permalink: SQLite database export" href="#sqlite-database-export"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">You can export tables to a SQLite database file using the <code>--sqlite database.db</code> option:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="airtable-export export base_id \
      --key=key --sqlite database.db"><pre class="notranslate"><code>airtable-export export base_id \
      --key=key --sqlite database.db
  </code></pre></div>
  <p dir="auto">This can be combined with other format options. If you only specify <code>--sqlite</code> the export directory argument will be ignored.</p>
  <p dir="auto">The SQLite database will have a table created for each table you export. Those tables will have a primary key column called <code>airtable_id</code>.</p>
  <p dir="auto">If you run this command against an existing SQLite database records with matching primary keys will be over-written by new records from the export.</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Request options</h2><a id="user-content-request-options" class="anchor" aria-label="Permalink: Request options" href="#request-options"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">By default the tool uses <a href="https://www.python-httpx.org" rel="nofollow">python-httpx</a>'s default configurations.</p>
  <p dir="auto">You can override the <code>user-agent</code> using the <code>--user-agent</code> option:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="airtable-export export base_id table1 table2 --key=key --user-agent "Airtable Export Robot""><pre class="notranslate"><code>airtable-export export base_id table1 table2 --key=key --user-agent "Airtable Export Robot"
  </code></pre></div>
  <p dir="auto">You can override the <a href="https://www.python-httpx.org/advanced/#fine-tuning-the-configuration" rel="nofollow">timeout during a network read operation</a> using the <code>--http-read-timeout</code> option. If not set, this defaults to 5s.</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="airtable-export export base_id table1 table2 --key=key --http-read-timeout 60"><pre class="notranslate"><code>airtable-export export base_id table1 table2 --key=key --http-read-timeout 60
  </code></pre></div>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Running this using GitHub Actions</h2><a id="user-content-running-this-using-github-actions" class="anchor" aria-label="Permalink: Running this using GitHub Actions" href="#running-this-using-github-actions"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto"><a href="https://github.com/features/actions">GitHub Actions</a> is GitHub's workflow automation product. You can use it to run <code>airtable-export</code> in order to back up your Airtable data to a GitHub repository. Doing this gives you a visible commit history of changes you make to your Airtable data - like <a href="https://github.com/natbat/rockybeaches/commits/main/airtable">this one</a>.</p>
  <p dir="auto">To run this for your own Airtable database you'll first need to add the following secrets to your GitHub repository:</p>
  <dl>
    <dt>AIRTABLE_BASE_ID</dt>
    <dd>The base ID, a string beginning `app...`</dd>
    <dt>AIRTABLE_KEY</dt>
    <dd>Your Airtable API key</dd>
    <dt>AIRTABLE_TABLES</dt>
    <dd>A space separated list of the Airtable tables that you want to backup. If any of these contain spaces you will need to enclose them in single quotes, e.g. <samp>'My table with spaces in the name' OtherTableWithNoSpaces</samp></dd>
  </dl>
  <p dir="auto">Once you have set those secrets, add the following as a file called <code>.github/workflows/backup-airtable.yml</code>:</p>
  <div class="highlight highlight-source-yaml notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="name: Backup Airtable

  on:
    workflow_dispatch:
    schedule:
    - cron: '32 0 * * *'

  jobs:
    build:
      runs-on: ubuntu-latest
      steps:
      - name: Check out repo
        uses: actions/checkout@v2
      - name: Set up Python
        uses: actions/setup-python@v2
        with:
          python-version: 3.8
      - uses: actions/cache@v2
        name: Configure pip caching
        with:
          path: ~/.cache/pip
          key: ${{ runner.os }}-pip-
          restore-keys: |
            ${{ runner.os }}-pip-
      - name: Install airtable-export
        run: |
          pip install airtable-export
      - name: Backup Airtable to backups/
        env:
          AIRTABLE_BASE_ID: ${{ secrets.AIRTABLE_BASE_ID }}
          AIRTABLE_KEY: ${{ secrets.AIRTABLE_KEY }}
          AIRTABLE_TABLES: ${{ secrets.AIRTABLE_TABLES }}
        run: |-
          airtable-export backups $AIRTABLE_BASE_ID $AIRTABLE_TABLES -v
      - name: Commit and push if it changed
        run: |-
          git config user.name "Automated"
          git config user.email "actions@users.noreply.github.com"
          git add -A
          timestamp=$(date -u)
          git commit -m "Latest data: ${timestamp}" || exit 0
          git push"><pre><span class="pl-ent">name</span>: <span class="pl-s">Backup Airtable</span>

  <span class="pl-ent">on</span>:
    <span class="pl-ent">workflow_dispatch</span>:
    <span class="pl-ent">schedule</span>:
    - <span class="pl-ent">cron</span>: <span class="pl-s"><span class="pl-pds">'</span>32 0 * * *<span class="pl-pds">'</span></span>

  <span class="pl-ent">jobs</span>:
    <span class="pl-ent">build</span>:
      <span class="pl-ent">runs-on</span>: <span class="pl-s">ubuntu-latest</span>
      <span class="pl-ent">steps</span>:
      - <span class="pl-ent">name</span>: <span class="pl-s">Check out repo</span>
        <span class="pl-ent">uses</span>: <span class="pl-s">actions/checkout@v2</span>
      - <span class="pl-ent">name</span>: <span class="pl-s">Set up Python</span>
        <span class="pl-ent">uses</span>: <span class="pl-s">actions/setup-python@v2</span>
        <span class="pl-ent">with</span>:
          <span class="pl-ent">python-version</span>: <span class="pl-c1">3.8</span>
      - <span class="pl-ent">uses</span>: <span class="pl-s">actions/cache@v2</span>
        <span class="pl-ent">name</span>: <span class="pl-s">Configure pip caching</span>
        <span class="pl-ent">with</span>:
          <span class="pl-ent">path</span>: <span class="pl-s">~/.cache/pip</span>
          <span class="pl-ent">key</span>: <span class="pl-s">${{ runner.os }}-pip-</span>
          <span class="pl-ent">restore-keys</span>: <span class="pl-s">|</span>
  <span class="pl-s">          ${{ runner.os }}-pip-</span>
  <span class="pl-s"></span>    - <span class="pl-ent">name</span>: <span class="pl-s">Install airtable-export</span>
        <span class="pl-ent">run</span>: <span class="pl-s">|</span>
  <span class="pl-s">        pip install airtable-export</span>
  <span class="pl-s"></span>    - <span class="pl-ent">name</span>: <span class="pl-s">Backup Airtable to backups/</span>
        <span class="pl-ent">env</span>:
          <span class="pl-ent">AIRTABLE_BASE_ID</span>: <span class="pl-s">${{ secrets.AIRTABLE_BASE_ID }}</span>
          <span class="pl-ent">AIRTABLE_KEY</span>: <span class="pl-s">${{ secrets.AIRTABLE_KEY }}</span>
          <span class="pl-ent">AIRTABLE_TABLES</span>: <span class="pl-s">${{ secrets.AIRTABLE_TABLES }}</span>
        <span class="pl-ent">run</span>: <span class="pl-s">|-</span>
  <span class="pl-s">        airtable-export backups $AIRTABLE_BASE_ID $AIRTABLE_TABLES -v</span>
  <span class="pl-s"></span>    - <span class="pl-ent">name</span>: <span class="pl-s">Commit and push if it changed</span>
        <span class="pl-ent">run</span>: <span class="pl-s">|-</span>
  <span class="pl-s">        git config user.name "Automated"</span>
  <span class="pl-s">        git config user.email "actions@users.noreply.github.com"</span>
  <span class="pl-s">        git add -A</span>
  <span class="pl-s">        timestamp=$(date -u)</span>
  <span class="pl-s">        git commit -m "Latest data: ${timestamp}" || exit 0</span>
  <span class="pl-s">        git push</span></pre></div>
  <p dir="auto">This will run once a day (at 32 minutes past midnight UTC) and will also run if you manually click the "Run workflow" button, see <a href="https://github.blog/changelog/2020-07-06-github-actions-manual-triggers-with-workflow_dispatch/" rel="nofollow">GitHub Actions: Manual triggers with workflow_dispatch</a>.</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Development</h2><a id="user-content-development" class="anchor" aria-label="Permalink: Development" href="#development"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">To contribute to this tool, first checkout the code. Then create a new virtual environment:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="cd airtable-export
  python -mvenv venv
  source venv/bin/activate"><pre class="notranslate"><code>cd airtable-export
  python -mvenv venv
  source venv/bin/activate
  </code></pre></div>
  <p dir="auto">Or if you are using <code>pipenv</code>:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="pipenv shell"><pre class="notranslate"><code>pipenv shell
  </code></pre></div>
  <p dir="auto">Now install the dependencies and tests:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="pip install -e '.[test]'"><pre class="notranslate"><code>pip install -e '.[test]'
  </code></pre></div>
  <p dir="auto">To run the tests:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="pytest"><pre class="notranslate"><code>pytest
  </code></pre></div>
  </article></div>


node_id: MDEwOlJlcG9zaXRvcnkyOTEzNTkzNTg=
name: datasette-yaml
full_name: simonw/datasette-yaml
html_url: https://github.com/simonw/datasette-yaml
description: Export Datasette records as YAML
created_at: 2020-08-29T22:32:15Z
updated_at: 2025-11-10T20:21:30Z
pushed_at: 2025-11-10T20:21:26Z
language: Python
topics: ["datasette", "datasette-io", "datasette-plugin", "yaml"]
visibility: public
default_branch: main
permissions: {"admin": false, "maintain": false, "push": false, "triage": false, "pull": false}
pull_request_creation_policy: all
readme:
  # datasette-yaml

  [![PyPI](https://img.shields.io/pypi/v/datasette-yaml.svg)](https://pypi.org/project/datasette-yaml/)
  [![Changelog](https://img.shields.io/github/v/release/simonw/datasette-yaml?include_prereleases&label=changelog)](https://github.com/simonw/datasette-yaml/releases)
  [![Tests](https://github.com/simonw/datasette-yaml/workflows/Test/badge.svg)](https://github.com/simonw/datasette-yaml/actions?query=workflow%3ATest)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/datasette-yaml/blob/main/LICENSE)

  Export Datasette records as YAML

  ## Installation

  Install this plugin in the same environment as Datasette.

      $ datasette install datasette-yaml

  ## Usage

  Having installed this plugin, every table and query will gain a new `.yaml` export link.

  You can also construct these URLs directly: `/dbname/tablename.yaml`

  ## Demo

  The plugin is running on [covid-19.datasettes.com](https://covid-19.datasettes.co/) - for example [/covid/latest_ny_times_counties_with_populations.yaml](https://covid-19.datasettes.com/covid/latest_ny_times_counties_with_populations.yaml)

  ## Development

  To set up this plugin locally, first checkout the code. Then create a new virtual environment:

      cd datasette-yaml
      python3 -mvenv venv
      source venv/bin/activate

  Or if you are using `pipenv`:

      pipenv shell

  Now install the dependencies and tests:

      pip install -e '.[test]'

  To run the tests:

      pytest

readme_html:
  <div id="readme" class="md" data-path="README.md"><article class="markdown-body entry-content container-lg" itemprop="text"><div class="markdown-heading" dir="auto"><h1 class="heading-element" dir="auto">datasette-yaml</h1><a id="user-content-datasette-yaml" class="anchor" aria-label="Permalink: datasette-yaml" href="#datasette-yaml"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto"><a href="https://pypi.org/project/datasette-yaml/" rel="nofollow"><img src="https://img.shields.io/pypi/v/datasette-yaml.svg" alt="PyPI" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/fcf0fcddf71374fd71dc10ee261a4fd2d409221ac00d46023856ebe70e713645/68747470733a2f2f696d672e736869656c64732e696f2f707970692f762f6461746173657474652d79616d6c2e737667"></a>
  <a href="https://github.com/simonw/datasette-yaml/releases"><img src="https://img.shields.io/github/v/release/simonw/datasette-yaml?include_prereleases&label=changelog" alt="Changelog" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/19d55cbc14c66a82c85829266fd958354fd0be6c8f4b4d8469063bed428fddc9/68747470733a2f2f696d672e736869656c64732e696f2f6769746875622f762f72656c656173652f73696d6f6e772f6461746173657474652d79616d6c3f696e636c7564655f70726572656c6561736573266c6162656c3d6368616e67656c6f67"></a>
  <a href="https://github.com/simonw/datasette-yaml/actions?query=workflow%3ATest"><img src="https://github.com/simonw/datasette-yaml/workflows/Test/badge.svg" alt="Tests" style="max-width: 100%;"></a>
  <a href="https://github.com/simonw/datasette-yaml/blob/main/LICENSE"><img src="https://img.shields.io/badge/license-Apache%202.0-blue.svg" alt="License" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/c355f200ea90fddaa407b6eaab303663a669248ea3ca7b1fcf77dbe04ff5f48c/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6c6963656e73652d417061636865253230322e302d626c75652e737667"></a></p>
  <p dir="auto">Export Datasette records as YAML</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Installation</h2><a id="user-content-installation" class="anchor" aria-label="Permalink: Installation" href="#installation"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">Install this plugin in the same environment as Datasette.</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ datasette install datasette-yaml"><pre class="notranslate"><code>$ datasette install datasette-yaml
  </code></pre></div>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Usage</h2><a id="user-content-usage" class="anchor" aria-label="Permalink: Usage" href="#usage"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">Having installed this plugin, every table and query will gain a new <code>.yaml</code> export link.</p>
  <p dir="auto">You can also construct these URLs directly: <code>/dbname/tablename.yaml</code></p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Demo</h2><a id="user-content-demo" class="anchor" aria-label="Permalink: Demo" href="#demo"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">The plugin is running on <a href="https://covid-19.datasettes.co/" rel="nofollow">covid-19.datasettes.com</a> - for example <a href="https://covid-19.datasettes.com/covid/latest_ny_times_counties_with_populations.yaml" rel="nofollow">/covid/latest_ny_times_counties_with_populations.yaml</a></p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Development</h2><a id="user-content-development" class="anchor" aria-label="Permalink: Development" href="#development"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">To set up this plugin locally, first checkout the code. Then create a new virtual environment:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="cd datasette-yaml
  python3 -mvenv venv
  source venv/bin/activate"><pre class="notranslate"><code>cd datasette-yaml
  python3 -mvenv venv
  source venv/bin/activate
  </code></pre></div>
  <p dir="auto">Or if you are using <code>pipenv</code>:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="pipenv shell"><pre class="notranslate"><code>pipenv shell
  </code></pre></div>
  <p dir="auto">Now install the dependencies and tests:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="pip install -e '.[test]'"><pre class="notranslate"><code>pip install -e '.[test]'
  </code></pre></div>
  <p dir="auto">To run the tests:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="pytest"><pre class="notranslate"><code>pytest
  </code></pre></div>
  </article></div>


node_id: MDEwOlJlcG9zaXRvcnkyOTMxNjQ0NDc=
name: datasette-backup
full_name: simonw/datasette-backup
html_url: https://github.com/simonw/datasette-backup
description: Plugin adding backup options to Datasette
created_at: 2020-09-05T22:33:29Z
updated_at: 2023-02-14T04:46:54Z
pushed_at: 2020-09-07T02:27:30Z
language: Python
topics: ["datasette", "datasette-io", "datasette-plugin"]
visibility: public
default_branch: main
permissions: {"admin": false, "maintain": false, "push": false, "triage": false, "pull": true}
readme:
  # datasette-backup

  [![PyPI](https://img.shields.io/pypi/v/datasette-backup.svg)](https://pypi.org/project/datasette-backup/)
  [![Changelog](https://img.shields.io/github/v/release/simonw/datasette-backup?include_prereleases&label=changelog)](https://github.com/simonw/datasette-backup/releases)
  [![Tests](https://github.com/simonw/datasette-backup/workflows/Test/badge.svg)](https://github.com/simonw/datasette-backup/actions?query=workflow%3ATest)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/datasette-backup/blob/main/LICENSE)

  Plugin adding backup options to Datasette

  ## Installation

  Install this plugin in the same environment as Datasette.

      $ datasette install datasette-backup

  ## Usage

  Once installed, you can download a SQL backup of any of your databases from:

      /-/backup/dbname.sql

  ## Development

  To set up this plugin locally, first checkout the code. Then create a new virtual environment:

      cd datasette-backup
      python3 -mvenv venv
      source venv/bin/activate

  Or if you are using `pipenv`:

      pipenv shell

  Now install the dependencies and tests:

      pip install -e '.[test]'

  To run the tests:

      pytest

readme_html:
  <div id="readme" class="md" data-path="README.md"><article class="markdown-body entry-content container-lg" itemprop="text"><div class="markdown-heading" dir="auto"><h1 class="heading-element" dir="auto">datasette-backup</h1><a id="user-content-datasette-backup" class="anchor" aria-label="Permalink: datasette-backup" href="#datasette-backup"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto"><a href="https://pypi.org/project/datasette-backup/" rel="nofollow"><img src="https://img.shields.io/pypi/v/datasette-backup.svg" alt="PyPI" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/66085f79ecdf26a14b2e7ab3b816afcb5fc05a234fc39b9324b88f748d2f27a0/68747470733a2f2f696d672e736869656c64732e696f2f707970692f762f6461746173657474652d6261636b75702e737667"></a>
  <a href="https://github.com/simonw/datasette-backup/releases"><img src="https://img.shields.io/github/v/release/simonw/datasette-backup?include_prereleases&label=changelog" alt="Changelog" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/69e12a10ccb57d645d3a499cfc929d28e047ef85bf134e42c1ddcf0c3537ade2/68747470733a2f2f696d672e736869656c64732e696f2f6769746875622f762f72656c656173652f73696d6f6e772f6461746173657474652d6261636b75703f696e636c7564655f70726572656c6561736573266c6162656c3d6368616e67656c6f67"></a>
  <a href="https://github.com/simonw/datasette-backup/actions?query=workflow%3ATest"><img src="https://github.com/simonw/datasette-backup/workflows/Test/badge.svg" alt="Tests" style="max-width: 100%;"></a>
  <a href="https://github.com/simonw/datasette-backup/blob/main/LICENSE"><img src="https://img.shields.io/badge/license-Apache%202.0-blue.svg" alt="License" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/c355f200ea90fddaa407b6eaab303663a669248ea3ca7b1fcf77dbe04ff5f48c/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6c6963656e73652d417061636865253230322e302d626c75652e737667"></a></p>
  <p dir="auto">Plugin adding backup options to Datasette</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Installation</h2><a id="user-content-installation" class="anchor" aria-label="Permalink: Installation" href="#installation"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">Install this plugin in the same environment as Datasette.</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ datasette install datasette-backup"><pre class="notranslate"><code>$ datasette install datasette-backup
  </code></pre></div>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Usage</h2><a id="user-content-usage" class="anchor" aria-label="Permalink: Usage" href="#usage"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">Once installed, you can download a SQL backup of any of your databases from:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="/-/backup/dbname.sql"><pre class="notranslate"><code>/-/backup/dbname.sql
  </code></pre></div>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Development</h2><a id="user-content-development" class="anchor" aria-label="Permalink: Development" href="#development"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">To set up this plugin locally, first checkout the code. Then create a new virtual environment:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="cd datasette-backup
  python3 -mvenv venv
  source venv/bin/activate"><pre class="notranslate"><code>cd datasette-backup
  python3 -mvenv venv
  source venv/bin/activate
  </code></pre></div>
  <p dir="auto">Or if you are using <code>pipenv</code>:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="pipenv shell"><pre class="notranslate"><code>pipenv shell
  </code></pre></div>
  <p dir="auto">Now install the dependencies and tests:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="pip install -e '.[test]'"><pre class="notranslate"><code>pip install -e '.[test]'
  </code></pre></div>
  <p dir="auto">To run the tests:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="pytest"><pre class="notranslate"><code>pytest
  </code></pre></div>
  </article></div>


node_id: MDEwOlJlcG9zaXRvcnkyOTMzNjE1MTQ=
name: geocode-sqlite
full_name: eyeseast/geocode-sqlite
html_url: https://github.com/eyeseast/geocode-sqlite
description: Geocode rows in a SQLite database table
created_at: 2020-09-06T21:05:39Z
updated_at: 2025-04-28T04:46:33Z
pushed_at: 2022-11-07T17:31:05Z
language: Python
license: apache-2.0
topics: []
visibility: public
default_branch: main
permissions: {"admin": false, "maintain": false, "push": false, "triage": false, "pull": true}
readme:
  # geocode-sqlite

  [![PyPI](https://img.shields.io/pypi/v/geocode-sqlite.svg)](https://pypi.org/project/geocode-sqlite/)
  [![Changelog](https://img.shields.io/github/v/release/eyeseast/geocode-sqlite?include_prereleases&label=changelog)](https://github.com/eyeseast/geocode-sqlite/releases)
  [![Tests](https://github.com/eyeseast/geocode-sqlite/workflows/Test/badge.svg)](https://github.com/eyeseast/geocode-sqlite/actions?query=workflow%3ATest)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/eyeseast/geocode-sqlite/blob/master/LICENSE)

  Geocode rows from a SQLite table

  ## Installation

  Install this tool using `pip` or `pipx`:

  ```sh
  # install inside a virtualenv
  pip install geocode-sqlite

  # install globally
  pipx install geocode-sqlite
  ```

  ## Usage

  Let's say you have a spreadsheet with addresses in it, and you'd like to map those locations.
  First, create a SQLite database and insert rows from that spreadsheet using `sqlite-utils`.

  ```sh
  sqlite-utils insert data.db data data.csv --csv
  ```

  Now, geocode it using OpenStreetMap's Nominatim geocoder.

  ```sh
  geocode-sqlite nominatim data.db data \
   --location="{address}, {city}, {state} {zip}" \
   --delay=1 \
   --user-agent="this-is-me"
  ```

  In the command above, you're using Nominatim, which is free and only asks for a unique user agent (`--user-agent`).

  This will connect to a database (`data.db`) and read all rows from the table `data` (skipping any that already
  have both a `latitude` and `longitude` column filled).

  You're also telling the geocoder how to extract a location query (`--location`) from a row of data, using Python's
  built-in string formatting, and setting a rate limit (`--delay`) of one request per second.

  For each row where geocoding succeeds, `latitude` and `longitude` will be populated. If you hit an error, or a rate limit,
  run the same query and pick up where you left off.

  The resulting table layout can be visualized with [datasette-cluster-map](https://datasette.io/plugins/datasette-cluster-map).

  Under the hood, this package uses the excellent [geopy](https://geopy.readthedocs.io/en/latest/) library, which is stable and thoroughly road-tested. If you need help understanding a particular geocoder's options, consult [geopy's documentation](https://geopy.readthedocs.io/en/latest/#module-geopy.geocoders).

  ### Supported Geocoders

  The CLI currently supports these geocoders:

  - `bing`
  - `googlev3`
  - `mapquest` (and `open-mapquest`)
  - `mapbox`
  - `nominatim`
  - `opencage`

  #### Adding new geocoders

  1. Open an issue with the name of the geocoding service as the ticket title ([example](https://github.com/eyeseast/geocode-sqlite/issues/35)). Put any noteworthy implementation details in the ticket body, like where to get an API key if one is required.
  2. Fork the repo and add a geocoder.
  3. Add an example to the `Makefile`. Add tests if there's new shared functionality.

  ### Common arguments and options

  Each geocoder needs to know where to find the data it's working with. These are the first two arguments:

  - `database`: a path to a SQLite file, which must already exist
  - `table`: the name of a table, in that database, which exists and has data to geocode

  From there, we have a set of options passed to every geocoder:

  - `location`: a [string format](https://docs.python.org/3/library/stdtypes.html#str.format) that will be expanded with each row to build a full query, to be geocoded
  - `delay`: a delay between each call (some services require this)
  - `latitude`: latitude column name
  - `longitude`: longitude column name
  - `geojson`: store results as GeoJSON, instead of in latitude and longitude columns
  - `spatialite`: store results in a SpatiaLite geometry column, instead of in latitude and longitude columns
  - `raw`: store raw geocoding results in a JSON column

  Each geocoder takes additional, specific arguments beyond these, such as API keys. Again, [geopy's documentation](https://geopy.readthedocs.io/en/latest/#module-geopy.geocoders) is an excellent resource.

  ## Using SpatiaLite

  The `--spatialite` flag will store results in a [geometry column](https://www.gaia-gis.it/gaia-sins/spatialite-cookbook-5/cookbook_topics.adminstration.html#topic_TABLE_to_SpatialTable), instead of `latitude` and `longitude` columns. This is useful if you're doing other GIS operations, such as using a [spatial index](https://www.gaia-gis.it/fossil/libspatialite/wiki?name=SpatialIndex). See the [SpatiaLite cookbook](https://www.gaia-gis.it/gaia-sins/spatialite-cookbook-5/index.html) and [functions list](https://www.gaia-gis.it/gaia-sins/spatialite-sql-latest.html) for more of what's possible.

  ## Capturing additional geocoding data

  Geocoding services typically return more data than just coordinates. This might include accuracy, normalized addresses or other context. This can be captured using the `--raw` flag. By default, this will add a `raw` column and store the full geocoding response as JSON. If you want to rename that column, pass a value, like `--raw custom_raw`.

  The shape of this response object will vary between services. You can query specific values using [SQLite's built-in JSON functions](https://www.sqlite.org/json1.html). For example, this will work with Google's geocoder:

  ```sql
  select
    json_extract(raw, '$.formatted_address') as address,
    json_extract(raw, '$.geometry.location_type') as location_type
  from
    innout_test
  ```

  Check each geocoding service's documentation for what's included in the response.

  ## Python API

  The command line interface aims to support the most common options for each geocoder. For more fine-grained control, use the Python API.

  As with the CLI, this assumes you already have a SQLite database and a table of location data.

  ```python
  from geocode_sqlite import geocode_table
  from geopy.geocoders import Nominatim

  # create a geocoder instance, with some extra options
  nominatim = Nominatim(user_agent="this-is-me", domain="nominatim.local.dev", scheme="http")

  # assuming our database is in the same directory
  count = geocode_table("data.db", "data", query_template="{address}, {city}, {state} {zip}")

  # when it's done
  print(f"Geocoded {count} rows")
  ```

  Any [geopy geocoder](https://geopy.readthedocs.io/en/latest/#module-geopy.geocoders) can be used with the Python API.

  ## Development

  To contribute to this tool, first checkout the code. Then create a new virtual environment:

  ```sh
  cd geocode-sqlite
  python -m venv .venv
  source .venv/bin/activate
  ```

  Or if you are using `pipenv`:

  ```sh
  pipenv shell
  ```

  Now install the dependencies and tests:

  ```sh
  pip install -e '.[test]'
  ```

  To run the tests:

  ```sh
  pytest
  ```

  Please remember that this library is mainly glue code between other well-tested projects, specifically: [click](https://click.palletsprojects.com/), [geopy](https://geopy.readthedocs.io/en/stable/) and [sqlite-utils](https://sqlite-utils.datasette.io/en/stable/). Tests should focus on making sure those parts fit together correctly. We can assume the parts themselves already work.

  To that end, there is a test geocoder included: `geocode_sqlite.testing.DummyGeocoder`. That geocoder works with an included dataset of In-N-Out Burger locations provided by [AllThePlaces](https://www.alltheplaces.xyz/). It works like a normal GeoPy geocoder, except it will only return results for In-N-Out locations using the included database.

readme_html:
  <div id="readme" class="md" data-path="README.md"><article class="markdown-body entry-content container-lg" itemprop="text"><div class="markdown-heading" dir="auto"><h1 class="heading-element" dir="auto">geocode-sqlite</h1><a id="user-content-geocode-sqlite" class="anchor" aria-label="Permalink: geocode-sqlite" href="#geocode-sqlite"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto"><a href="https://pypi.org/project/geocode-sqlite/" rel="nofollow"><img src="https://img.shields.io/pypi/v/geocode-sqlite.svg" alt="PyPI" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/3f0d89dfcc9572c5ed17646b06c4863dcf0fc2a9a9046ae1e083cc67468b56a4/68747470733a2f2f696d672e736869656c64732e696f2f707970692f762f67656f636f64652d73716c6974652e737667"></a>
  <a href="https://github.com/eyeseast/geocode-sqlite/releases"><img src="https://img.shields.io/github/v/release/eyeseast/geocode-sqlite?include_prereleases&label=changelog" alt="Changelog" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/4e6a9ccbb719d48cccf659276ae54018029cad6dff9af638ac85d8507cfbd171/68747470733a2f2f696d672e736869656c64732e696f2f6769746875622f762f72656c656173652f65796573656173742f67656f636f64652d73716c6974653f696e636c7564655f70726572656c6561736573266c6162656c3d6368616e67656c6f67"></a>
  <a href="https://github.com/eyeseast/geocode-sqlite/actions?query=workflow%3ATest"><img src="https://github.com/eyeseast/geocode-sqlite/workflows/Test/badge.svg" alt="Tests" style="max-width: 100%;"></a>
  <a href="https://github.com/eyeseast/geocode-sqlite/blob/master/LICENSE"><img src="https://img.shields.io/badge/license-Apache%202.0-blue.svg" alt="License" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/c355f200ea90fddaa407b6eaab303663a669248ea3ca7b1fcf77dbe04ff5f48c/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6c6963656e73652d417061636865253230322e302d626c75652e737667"></a></p>
  <p dir="auto">Geocode rows from a SQLite table</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Installation</h2><a id="user-content-installation" class="anchor" aria-label="Permalink: Installation" href="#installation"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">Install this tool using <code>pip</code> or <code>pipx</code>:</p>
  <div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="# install inside a virtualenv
  pip install geocode-sqlite

  # install globally
  pipx install geocode-sqlite"><pre><span class="pl-c"><span class="pl-c">#</span> install inside a virtualenv</span>
  pip install geocode-sqlite

  <span class="pl-c"><span class="pl-c">#</span> install globally</span>
  pipx install geocode-sqlite</pre></div>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Usage</h2><a id="user-content-usage" class="anchor" aria-label="Permalink: Usage" href="#usage"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">Let's say you have a spreadsheet with addresses in it, and you'd like to map those locations.
  First, create a SQLite database and insert rows from that spreadsheet using <code>sqlite-utils</code>.</p>
  <div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="sqlite-utils insert data.db data data.csv --csv"><pre>sqlite-utils insert data.db data data.csv --csv</pre></div>
  <p dir="auto">Now, geocode it using OpenStreetMap's Nominatim geocoder.</p>
  <div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="geocode-sqlite nominatim data.db data \
   --location="{address}, {city}, {state} {zip}" \
   --delay=1 \
   --user-agent="this-is-me""><pre>geocode-sqlite nominatim data.db data \
   --location=<span class="pl-s"><span class="pl-pds">"</span>{address}, {city}, {state} {zip}<span class="pl-pds">"</span></span> \
   --delay=1 \
   --user-agent=<span class="pl-s"><span class="pl-pds">"</span>this-is-me<span class="pl-pds">"</span></span></pre></div>
  <p dir="auto">In the command above, you're using Nominatim, which is free and only asks for a unique user agent (<code>--user-agent</code>).</p>
  <p dir="auto">This will connect to a database (<code>data.db</code>) and read all rows from the table <code>data</code> (skipping any that already
  have both a <code>latitude</code> and <code>longitude</code> column filled).</p>
  <p dir="auto">You're also telling the geocoder how to extract a location query (<code>--location</code>) from a row of data, using Python's
  built-in string formatting, and setting a rate limit (<code>--delay</code>) of one request per second.</p>
  <p dir="auto">For each row where geocoding succeeds, <code>latitude</code> and <code>longitude</code> will be populated. If you hit an error, or a rate limit,
  run the same query and pick up where you left off.</p>
  <p dir="auto">The resulting table layout can be visualized with <a href="https://datasette.io/plugins/datasette-cluster-map" rel="nofollow">datasette-cluster-map</a>.</p>
  <p dir="auto">Under the hood, this package uses the excellent <a href="https://geopy.readthedocs.io/en/latest/" rel="nofollow">geopy</a> library, which is stable and thoroughly road-tested. If you need help understanding a particular geocoder's options, consult <a href="https://geopy.readthedocs.io/en/latest/#module-geopy.geocoders" rel="nofollow">geopy's documentation</a>.</p>
  <div class="markdown-heading" dir="auto"><h3 class="heading-element" dir="auto">Supported Geocoders</h3><a id="user-content-supported-geocoders" class="anchor" aria-label="Permalink: Supported Geocoders" href="#supported-geocoders"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">The CLI currently supports these geocoders:</p>
  <ul dir="auto">
  <li><code>bing</code></li>
  <li><code>googlev3</code></li>
  <li><code>mapquest</code> (and <code>open-mapquest</code>)</li>
  <li><code>mapbox</code></li>
  <li><code>nominatim</code></li>
  <li><code>opencage</code></li>
  </ul>
  <div class="markdown-heading" dir="auto"><h4 class="heading-element" dir="auto">Adding new geocoders</h4><a id="user-content-adding-new-geocoders" class="anchor" aria-label="Permalink: Adding new geocoders" href="#adding-new-geocoders"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <ol dir="auto">
  <li>Open an issue with the name of the geocoding service as the ticket title (<a href="https://github.com/eyeseast/geocode-sqlite/issues/35" data-hovercard-type="issue" data-hovercard-url="/eyeseast/geocode-sqlite/issues/35/hovercard">example</a>). Put any noteworthy implementation details in the ticket body, like where to get an API key if one is required.</li>
  <li>Fork the repo and add a geocoder.</li>
  <li>Add an example to the <code>Makefile</code>. Add tests if there's new shared functionality.</li>
  </ol>
  <div class="markdown-heading" dir="auto"><h3 class="heading-element" dir="auto">Common arguments and options</h3><a id="user-content-common-arguments-and-options" class="anchor" aria-label="Permalink: Common arguments and options" href="#common-arguments-and-options"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">Each geocoder needs to know where to find the data it's working with. These are the first two arguments:</p>
  <ul dir="auto">
  <li><code>database</code>: a path to a SQLite file, which must already exist</li>
  <li><code>table</code>: the name of a table, in that database, which exists and has data to geocode</li>
  </ul>
  <p dir="auto">From there, we have a set of options passed to every geocoder:</p>
  <ul dir="auto">
  <li><code>location</code>: a <a href="https://docs.python.org/3/library/stdtypes.html#str.format" rel="nofollow">string format</a> that will be expanded with each row to build a full query, to be geocoded</li>
  <li><code>delay</code>: a delay between each call (some services require this)</li>
  <li><code>latitude</code>: latitude column name</li>
  <li><code>longitude</code>: longitude column name</li>
  <li><code>geojson</code>: store results as GeoJSON, instead of in latitude and longitude columns</li>
  <li><code>spatialite</code>: store results in a SpatiaLite geometry column, instead of in latitude and longitude columns</li>
  <li><code>raw</code>: store raw geocoding results in a JSON column</li>
  </ul>
  <p dir="auto">Each geocoder takes additional, specific arguments beyond these, such as API keys. Again, <a href="https://geopy.readthedocs.io/en/latest/#module-geopy.geocoders" rel="nofollow">geopy's documentation</a> is an excellent resource.</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Using SpatiaLite</h2><a id="user-content-using-spatialite" class="anchor" aria-label="Permalink: Using SpatiaLite" href="#using-spatialite"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">The <code>--spatialite</code> flag will store results in a <a href="https://www.gaia-gis.it/gaia-sins/spatialite-cookbook-5/cookbook_topics.adminstration.html#topic_TABLE_to_SpatialTable" rel="nofollow">geometry column</a>, instead of <code>latitude</code> and <code>longitude</code> columns. This is useful if you're doing other GIS operations, such as using a <a href="https://www.gaia-gis.it/fossil/libspatialite/wiki?name=SpatialIndex" rel="nofollow">spatial index</a>. See the <a href="https://www.gaia-gis.it/gaia-sins/spatialite-cookbook-5/index.html" rel="nofollow">SpatiaLite cookbook</a> and <a href="https://www.gaia-gis.it/gaia-sins/spatialite-sql-latest.html" rel="nofollow">functions list</a> for more of what's possible.</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Capturing additional geocoding data</h2><a id="user-content-capturing-additional-geocoding-data" class="anchor" aria-label="Permalink: Capturing additional geocoding data" href="#capturing-additional-geocoding-data"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">Geocoding services typically return more data than just coordinates. This might include accuracy, normalized addresses or other context. This can be captured using the <code>--raw</code> flag. By default, this will add a <code>raw</code> column and store the full geocoding response as JSON. If you want to rename that column, pass a value, like <code>--raw custom_raw</code>.</p>
  <p dir="auto">The shape of this response object will vary between services. You can query specific values using <a href="https://www.sqlite.org/json1.html" rel="nofollow">SQLite's built-in JSON functions</a>. For example, this will work with Google's geocoder:</p>
  <div class="highlight highlight-source-sql notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="select
    json_extract(raw, '$.formatted_address') as address,
    json_extract(raw, '$.geometry.location_type') as location_type
  from
    innout_test"><pre><span class="pl-k">select</span>
    json_extract(raw, <span class="pl-s"><span class="pl-pds">'</span>$.formatted_address<span class="pl-pds">'</span></span>) <span class="pl-k">as</span> address,
    json_extract(raw, <span class="pl-s"><span class="pl-pds">'</span>$.geometry.location_type<span class="pl-pds">'</span></span>) <span class="pl-k">as</span> location_type
  <span class="pl-k">from</span>
    innout_test</pre></div>
  <p dir="auto">Check each geocoding service's documentation for what's included in the response.</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Python API</h2><a id="user-content-python-api" class="anchor" aria-label="Permalink: Python API" href="#python-api"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">The command line interface aims to support the most common options for each geocoder. For more fine-grained control, use the Python API.</p>
  <p dir="auto">As with the CLI, this assumes you already have a SQLite database and a table of location data.</p>
  <div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="from geocode_sqlite import geocode_table
  from geopy.geocoders import Nominatim

  # create a geocoder instance, with some extra options
  nominatim = Nominatim(user_agent="this-is-me", domain="nominatim.local.dev", scheme="http")

  # assuming our database is in the same directory
  count = geocode_table("data.db", "data", query_template="{address}, {city}, {state} {zip}")

  # when it's done
  print(f"Geocoded {count} rows")"><pre><span class="pl-k">from</span> <span class="pl-s1">geocode_sqlite</span> <span class="pl-k">import</span> <span class="pl-s1">geocode_table</span>
  <span class="pl-k">from</span> <span class="pl-s1">geopy</span>.<span class="pl-s1">geocoders</span> <span class="pl-k">import</span> <span class="pl-v">Nominatim</span>

  <span class="pl-c"># create a geocoder instance, with some extra options</span>
  <span class="pl-s1">nominatim</span> <span class="pl-c1">=</span> <span class="pl-en">Nominatim</span>(<span class="pl-s1">user_agent</span><span class="pl-c1">=</span><span class="pl-s">"this-is-me"</span>, <span class="pl-s1">domain</span><span class="pl-c1">=</span><span class="pl-s">"nominatim.local.dev"</span>, <span class="pl-s1">scheme</span><span class="pl-c1">=</span><span class="pl-s">"http"</span>)

  <span class="pl-c"># assuming our database is in the same directory</span>
  <span class="pl-s1">count</span> <span class="pl-c1">=</span> <span class="pl-en">geocode_table</span>(<span class="pl-s">"data.db"</span>, <span class="pl-s">"data"</span>, <span class="pl-s1">query_template</span><span class="pl-c1">=</span><span class="pl-s">"{address}, {city}, {state} {zip}"</span>)

  <span class="pl-c"># when it's done</span>
  <span class="pl-en">print</span>(<span class="pl-s">f"Geocoded <span class="pl-s1"><span class="pl-kos">{</span><span class="pl-s1">count</span><span class="pl-kos">}</span></span> rows"</span>)</pre></div>
  <p dir="auto">Any <a href="https://geopy.readthedocs.io/en/latest/#module-geopy.geocoders" rel="nofollow">geopy geocoder</a> can be used with the Python API.</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Development</h2><a id="user-content-development" class="anchor" aria-label="Permalink: Development" href="#development"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">To contribute to this tool, first checkout the code. Then create a new virtual environment:</p>
  <div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="cd geocode-sqlite
  python -m venv .venv
  source .venv/bin/activate"><pre><span class="pl-c1">cd</span> geocode-sqlite
  python -m venv .venv
  <span class="pl-c1">source</span> .venv/bin/activate</pre></div>
  <p dir="auto">Or if you are using <code>pipenv</code>:</p>
  <div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="pipenv shell"><pre>pipenv shell</pre></div>
  <p dir="auto">Now install the dependencies and tests:</p>
  <div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="pip install -e '.[test]'"><pre>pip install -e <span class="pl-s"><span class="pl-pds">'</span>.[test]<span class="pl-pds">'</span></span></pre></div>
  <p dir="auto">To run the tests:</p>
  <div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="pytest"><pre>pytest</pre></div>
  <p dir="auto">Please remember that this library is mainly glue code between other well-tested projects, specifically: <a href="https://click.palletsprojects.com/" rel="nofollow">click</a>, <a href="https://geopy.readthedocs.io/en/stable/" rel="nofollow">geopy</a> and <a href="https://sqlite-utils.datasette.io/en/stable/" rel="nofollow">sqlite-utils</a>. Tests should focus on making sure those parts fit together correctly. We can assume the parts themselves already work.</p>
  <p dir="auto">To that end, there is a test geocoder included: <code>geocode_sqlite.testing.DummyGeocoder</code>. That geocoder works with an included dataset of In-N-Out Burger locations provided by <a href="https://www.alltheplaces.xyz/" rel="nofollow">AllThePlaces</a>. It works like a normal GeoPy geocoder, except it will only return results for In-N-Out locations using the included database.</p>
  </article></div>


node_id: MDEwOlJlcG9zaXRvcnkyOTQ3MDYyNjc=
name: datasette-seaborn
full_name: simonw/datasette-seaborn
html_url: https://github.com/simonw/datasette-seaborn
description: Statistical visualizations for Datasette using Seaborn
created_at: 2020-09-11T13:43:08Z
updated_at: 2025-02-28T21:22:11Z
pushed_at: 2022-03-22T01:49:36Z
homepage: https://datasette-seaborn-demo.datasette.io/
language: Python
topics: ["datasette", "datasette-io", "datasette-plugin", "seaborn", "visualization"]
visibility: public
default_branch: main
permissions: {"admin": false, "maintain": false, "push": false, "triage": false, "pull": true}
readme:
  # datasette-seaborn

  [![PyPI](https://img.shields.io/pypi/v/datasette-seaborn.svg)](https://pypi.org/project/datasette-seaborn/)
  [![Changelog](https://img.shields.io/github/v/release/simonw/datasette-seaborn?include_prereleases&label=changelog)](https://github.com/simonw/datasette-seaborn/releases)
  [![Tests](https://github.com/simonw/datasette-seaborn/workflows/Test/badge.svg)](https://github.com/simonw/datasette-seaborn/actions?query=workflow%3ATest)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/datasette-seaborn/blob/main/LICENSE)

  Statistical visualizations for Datasette using Seaborn

  ## Installation

  Install this plugin in the same environment as Datasette.

      $ datasette install datasette-seaborn

  ## Usage

  Navigate to the new `.seaborn` extension for any Datasette table.

  The `_seaborn` argument specifies a method on `sns` to execute, e.g. `?_seaborn=relplot`.

  Extra arguments to those methods can be specified using e.g. `&_seaborn_x=column_name`.

  ## Configuration

  The plugin implements a default rendering time limit of five seconds. You can customize this limit using the `render_time_limit` setting, which accepts a floating point number of seconds. Add this to your `metadata.json`:

  ```json
  {
      "plugins": {
          "datasette-seaborn": {
              "render_time_limit": 1.0
          }
      }
  }
  ```

  ## Development

  To set up this plugin locally, first checkout the code. Then create a new virtual environment:

      cd datasette-seaborn
      python3 -mvenv venv
      source venv/bin/activate

  Or if you are using `pipenv`:

      pipenv shell

  Now install the dependencies and tests:

      pip install -e '.[test]'

  To run the tests:

      pytest

readme_html:
  <div id="readme" class="md" data-path="README.md"><article class="markdown-body entry-content container-lg" itemprop="text"><div class="markdown-heading" dir="auto"><h1 class="heading-element" dir="auto">datasette-seaborn</h1><a id="user-content-datasette-seaborn" class="anchor" aria-label="Permalink: datasette-seaborn" href="#datasette-seaborn"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto"><a href="https://pypi.org/project/datasette-seaborn/" rel="nofollow"><img src="https://img.shields.io/pypi/v/datasette-seaborn.svg" alt="PyPI" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/6a0845b26051caa73afc5ddf286ac90d749e7457de7046a90fa62572aa45ae02/68747470733a2f2f696d672e736869656c64732e696f2f707970692f762f6461746173657474652d736561626f726e2e737667"></a>
  <a href="https://github.com/simonw/datasette-seaborn/releases"><img src="https://img.shields.io/github/v/release/simonw/datasette-seaborn?include_prereleases&label=changelog" alt="Changelog" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/c9256729b0195201006e9d4b29ba56c49432e3b869a3f36164c1dc6c229783f8/68747470733a2f2f696d672e736869656c64732e696f2f6769746875622f762f72656c656173652f73696d6f6e772f6461746173657474652d736561626f726e3f696e636c7564655f70726572656c6561736573266c6162656c3d6368616e67656c6f67"></a>
  <a href="https://github.com/simonw/datasette-seaborn/actions?query=workflow%3ATest"><img src="https://github.com/simonw/datasette-seaborn/workflows/Test/badge.svg" alt="Tests" style="max-width: 100%;"></a>
  <a href="https://github.com/simonw/datasette-seaborn/blob/main/LICENSE"><img src="https://img.shields.io/badge/license-Apache%202.0-blue.svg" alt="License" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/c355f200ea90fddaa407b6eaab303663a669248ea3ca7b1fcf77dbe04ff5f48c/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6c6963656e73652d417061636865253230322e302d626c75652e737667"></a></p>
  <p dir="auto">Statistical visualizations for Datasette using Seaborn</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Installation</h2><a id="user-content-installation" class="anchor" aria-label="Permalink: Installation" href="#installation"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">Install this plugin in the same environment as Datasette.</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ datasette install datasette-seaborn"><pre class="notranslate"><code>$ datasette install datasette-seaborn
  </code></pre></div>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Usage</h2><a id="user-content-usage" class="anchor" aria-label="Permalink: Usage" href="#usage"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">Navigate to the new <code>.seaborn</code> extension for any Datasette table.</p>
  <p dir="auto">The <code>_seaborn</code> argument specifies a method on <code>sns</code> to execute, e.g. <code>?_seaborn=relplot</code>.</p>
  <p dir="auto">Extra arguments to those methods can be specified using e.g. <code>&_seaborn_x=column_name</code>.</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Configuration</h2><a id="user-content-configuration" class="anchor" aria-label="Permalink: Configuration" href="#configuration"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">The plugin implements a default rendering time limit of five seconds. You can customize this limit using the <code>render_time_limit</code> setting, which accepts a floating point number of seconds. Add this to your <code>metadata.json</code>:</p>
  <div class="highlight highlight-source-json notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="{
      "plugins": {
          "datasette-seaborn": {
              "render_time_limit": 1.0
          }
      }
  }"><pre>{
      <span class="pl-ent">"plugins"</span>: {
          <span class="pl-ent">"datasette-seaborn"</span>: {
              <span class="pl-ent">"render_time_limit"</span>: <span class="pl-c1">1.0</span>
          }
      }
  }</pre></div>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Development</h2><a id="user-content-development" class="anchor" aria-label="Permalink: Development" href="#development"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">To set up this plugin locally, first checkout the code. Then create a new virtual environment:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="cd datasette-seaborn
  python3 -mvenv venv
  source venv/bin/activate"><pre class="notranslate"><code>cd datasette-seaborn
  python3 -mvenv venv
  source venv/bin/activate
  </code></pre></div>
  <p dir="auto">Or if you are using <code>pipenv</code>:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="pipenv shell"><pre class="notranslate"><code>pipenv shell
  </code></pre></div>
  <p dir="auto">Now install the dependencies and tests:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="pip install -e '.[test]'"><pre class="notranslate"><code>pip install -e '.[test]'
  </code></pre></div>
  <p dir="auto">To run the tests:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="pytest"><pre class="notranslate"><code>pytest
  </code></pre></div>
  </article></div>


node_id: MDEwOlJlcG9zaXRvcnkyOTkxNDM4NDk=
name: datasette-dateutil
full_name: simonw/datasette-dateutil
html_url: https://github.com/simonw/datasette-dateutil
description: dateutil functions for Datasette
created_at: 2020-09-28T00:14:20Z
updated_at: 2025-11-02T20:44:05Z
pushed_at: 2025-11-02T20:44:01Z
language: Python
topics: ["datasette", "datasette-io", "datasette-plugin", "dateutil"]
visibility: public
default_branch: main
permissions: {"admin": false, "maintain": false, "push": false, "triage": false, "pull": false}
readme:
  # datasette-dateutil

  [![PyPI](https://img.shields.io/pypi/v/datasette-dateutil.svg)](https://pypi.org/project/datasette-dateutil/)
  [![Changelog](https://img.shields.io/github/v/release/simonw/datasette-dateutil?include_prereleases&label=changelog)](https://github.com/simonw/datasette-dateutil/releases)
  [![Tests](https://github.com/simonw/datasette-dateutil/workflows/Test/badge.svg)](https://github.com/simonw/datasette-dateutil/actions?query=workflow%3ATest)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/datasette-dateutil/blob/main/LICENSE)

  dateutil functions for Datasette

  ## Installation

  Install this plugin in the same environment as Datasette.

      $ datasette install datasette-dateutil

  ## Usage

  This function adds custom SQL functions that expose functionality from the [dateutil](https://dateutil.readthedocs.io/) Python library.

  Once installed, the following SQL functions become available:

  ### Parsing date strings

  - `dateutil_parse(text)` - returns an ISO8601 date string parsed from the text, or `null` if the input could not be parsed. `dateutil_parse("10 october 2020 3pm")` returns `2020-10-10T15:00:00`.
  - `dateutil_parse_fuzzy(text)` - same as `dateutil_parse()` but this also works against strings that contain a date somewhere within them - that date will be returned, or `null` if no dates could be found. `dateutil_parse_fuzzy("This is due 10 september")` returns `2020-09-10T00:00:00` (but will start returning the 2021 version of that if the year is 2021).

  The `dateutil_parse()` and `dateutil_parse_fuzzy()` functions both follow the American convention of assuming that `1/2/2020` lists the month first, evaluating this example to the 2nd of January.

  If you want to assume that the day comes first, use these two functions instead:

  - `dateutil_parse_dayfirst(text)`
  - `dateutil_parse_fuzzy_dayfirst(text)`

  Here's a query demonstrating these functions:

  ```sql
  select
    dateutil_parse("10 october 2020 3pm"),
    dateutil_parse_fuzzy("This is due 10 september"),
    dateutil_parse("1/2/2020"),
    dateutil_parse("2020-03-04"),
    dateutil_parse_dayfirst("2020-03-04");
  ```

  [Try that query](https://latest-with-plugins.datasette.io/fixtures?sql=select%0D%0A++dateutil_parse%28%2210+october+2020+3pm%22%29%2C%0D%0A++dateutil_parse_fuzzy%28%22This+is+due+10+september%22%29%2C%0D%0A++dateutil_parse%28%221%2F2%2F2020%22%29%2C%0D%0A++dateutil_parse%28%222020-03-04%22%29%2C%0D%0A++dateutil_parse_dayfirst%28%222020-03-04%22%29%3B)

  ### Optional default dates

  The `dateutil_parse()`, `dateutil_parse_fuzzy()`, `dateutil_parse_dayfirst()` and `dateutil_parse_fuzzy_dayfirst()` functions all accept an optional second argument specifying a "default" datetime to consider if some of the details are missing. For example, the following:
  ```sql
  select dateutil_parse('1st october', '1985-01-01')
  ```
  Will return `1985-10-01T00:00:00` - the missing year is replaced with the year from the default date.

  [Example query demonstrating the default date argument](https://latest-with-plugins.datasette.io/fixtures?sql=with+times+as+%28%0D%0A++select%0D%0A++++datetime%28%27now%27%29+as+t%0D%0A++union%0D%0A++select%0D%0A++++datetime%28%27now%27%2C+%27-1+year%27%29%0D%0A++union%0D%0A++select%0D%0A++++datetime%28%27now%27%2C+%27-3+years%27%29%0D%0A%29%0D%0Aselect+t%2C+dateutil_parse_fuzzy%28%22This+is+due+10+september%22%2C+t%29+from+times)

  ### Calculating Easter

  - `dateutil_easter(year)` - returns the date for Easter in that year, for example `dateutil_easter("2020")` returns `2020-04-12`.

  [Example Easter query](https://latest-with-plugins.datasette.io/fixtures?sql=select%0D%0A++dateutil_easter%282019%29%2C%0D%0A++dateutil_easter%282020%29%2C%0D%0A++dateutil_easter%282021%29)

  ### JSON arrays of dates

  Several functions return JSON arrays of date strings. These can be used with SQLite's `json_each()` function to perform joins against dates from a specific date range or recurrence rule.

  These functions can return up to 10,000 results. They will return an error if more than 10,000 dates would be returned - this is to protect against denial of service attacks.

  - `dateutil_dates_between('1 january 2020', '5 jan 2020')` - given two dates (in any format that can be handled by `dateutil_parse()`) this function returns a JSON string containing the dates between those two days, inclusive. This example returns `["2020-01-01", "2020-01-02", "2020-01-03", "2020-01-04", "2020-01-05"]`.
  - `dateutil_dates_between('1 january 2020', '5 jan 2020', 0)` - set the optional third argument to `0` to specify that you would like this to be exclusive of the last day. This example returns `["2020-01-01", "2020-01-02", "2020-01-03", "2020-01-04"]`.

  [Try these queries](https://latest-with-plugins.datasette.io/fixtures?sql=select%0D%0A++dateutil_dates_between%28%271+january+2020%27%2C+%275+jan+2020%27%29%2C%0D%0A++dateutil_dates_between%28%271+january+2020%27%2C+%275+jan+2020%27%2C+0%29)

  The `dateutil_rrule()` and `dateutil_rrule_date()` functions accept the iCalendar standard `rrule` format - see [the dateutil documentation](https://dateutil.readthedocs.io/en/stable/rrule.html#rrulestr-examples) for more examples.

  This format lets you specify recurrence rules such as "the next four last mondays of the month".

  - `dateutil_rrule(rrule, optional_dtsart)` - given an rrule returns a JSON array of ISO datetimes. The second argument is optional and will be treated as the start date for the rule.
  - `dateutil_rrule_date(rrule, optional_dtsart)` - same as `dateutil_rrule()` but returns ISO dates.

  Example query:

  ```sql
  select
    dateutil_rrule('FREQ=HOURLY;COUNT=5'),
    dateutil_rrule_date(
      'FREQ=DAILY;COUNT=3',
      '1st jan 2020'
    );
  ```
  [Try the rrule example query](https://latest-with-plugins.datasette.io/fixtures?sql=select%0D%0A++dateutil_rrule('FREQ%3DHOURLY%3BCOUNT%3D5')%2C%0D%0A++dateutil_rrule_date(%0D%0A++++'FREQ%3DDAILY%3BCOUNT%3D3'%2C%0D%0A++++'1st+jan+2020'%0D%0A++)%3B)

  ### Joining data using json_each()

  SQLite's [json_each() function](https://www.sqlite.org/json1.html#jeach) can be used to turn a JSON array of dates into a table that can be joined against other data. Here's a query that returns a table showing every day in January 2019:

  ```sql
  select
    value as date
  from
    json_each(
      dateutil_dates_between('1 Jan 2019', '31 Jan 2019')
    )
  ```
  [Try that query](https://latest-with-plugins.datasette.io/fixtures?sql=select%0D%0A++value+as+date%0D%0Afrom%0D%0A++json_each%28%0D%0A++++dateutil_dates_between%28%271+Jan+2019%27%2C+%2731+Jan+2019%27%29%0D%0A++%29)

  You can run joins against this table by assigning it a name using SQLite's [support for Common Table Expressions (CTEs)](https://sqlite.org/lang_with.html).

  This example query uses `substr(created, 0, 11)` to retrieve the date portion of the `created` column in the [facetable demo table](https://latest-with-plugins.datasette.io/fixtures/facetable), then joins that against the table of days in January to calculate the count of rows created on each day. The `LEFT JOIN` against `days_in_january` ensures that days which had no created records are still returned in the results, with a count of 0.

  ```sql
  with created_dates as (
    select
      substr(created, 0, 11) as date
    from
      facetable
  ),
  days_in_january as (
    select
      value as date
    from
      json_each(
        dateutil_dates_between('1 Jan 2019', '31 Jan 2019')
      )
  )
  select
    days_in_january.date,
    count(created_dates.date) as total
  from
    days_in_january
    left join created_dates on days_in_january.date = created_dates.date
  group by
    days_in_january.date;
  ```
  [Try that query](https://latest-with-plugins.datasette.io/fixtures?sql=with+created_dates+as+%28%0D%0A++select%0D%0A++++substr%28created%2C+0%2C+11%29+as+date%0D%0A++from%0D%0A++++facetable%0D%0A%29%2C%0D%0Adays_in_january+as+%28%0D%0A++select%0D%0A++++value+as+date%0D%0A++from%0D%0A++++json_each%28%0D%0A++++++dateutil_dates_between%28%271+Jan+2019%27%2C+%2731+Jan+2019%27%29%0D%0A++++%29%0D%0A%29%0D%0Aselect%0D%0A++days_in_january.date%2C%0D%0A++count%28created_dates.date%29+as+total%0D%0Afrom%0D%0A++days_in_january%0D%0A++left+join+created_dates+on+days_in_january.date+%3D+created_dates.date%0D%0Agroup+by%0D%0A++days_in_january.date%3B#g.mark=bar&g.x_column=date&g.x_type=ordinal&g.y_column=total&g.y_type=quantitative) with a bar chart rendered using the [datasette-vega](https://github.com/simonw/datasette-vega) plugin.

  ## Development

  To set up this plugin locally, first checkout the code. Then create a new virtual environment:

      cd datasette-dateutil
      python3 -mvenv venv
      source venv/bin/activate

  Or if you are using `pipenv`:

      pipenv shell

  Now install the dependencies and tests:

      pip install -e '.[test]'

  To run the tests:

      pytest

readme_html:
  <div id="readme" class="md" data-path="README.md"><article class="markdown-body entry-content container-lg" itemprop="text"><div class="markdown-heading" dir="auto"><h1 class="heading-element" dir="auto">datasette-dateutil</h1><a id="user-content-datasette-dateutil" class="anchor" aria-label="Permalink: datasette-dateutil" href="#datasette-dateutil"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto"><a href="https://pypi.org/project/datasette-dateutil/" rel="nofollow"><img src="https://img.shields.io/pypi/v/datasette-dateutil.svg" alt="PyPI" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/b460380da3ba1e605909df42ac32b78d6435e6f1c27406319b46c98a53e44fb2/68747470733a2f2f696d672e736869656c64732e696f2f707970692f762f6461746173657474652d646174657574696c2e737667"></a>
  <a href="https://github.com/simonw/datasette-dateutil/releases"><img src="https://img.shields.io/github/v/release/simonw/datasette-dateutil?include_prereleases&label=changelog" alt="Changelog" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/dca0030e9c84bf7cbb79c2d695aa50cc493986fc4002651f178cffcab2150f93/68747470733a2f2f696d672e736869656c64732e696f2f6769746875622f762f72656c656173652f73696d6f6e772f6461746173657474652d646174657574696c3f696e636c7564655f70726572656c6561736573266c6162656c3d6368616e67656c6f67"></a>
  <a href="https://github.com/simonw/datasette-dateutil/actions?query=workflow%3ATest"><img src="https://github.com/simonw/datasette-dateutil/workflows/Test/badge.svg" alt="Tests" style="max-width: 100%;"></a>
  <a href="https://github.com/simonw/datasette-dateutil/blob/main/LICENSE"><img src="https://img.shields.io/badge/license-Apache%202.0-blue.svg" alt="License" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/c355f200ea90fddaa407b6eaab303663a669248ea3ca7b1fcf77dbe04ff5f48c/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6c6963656e73652d417061636865253230322e302d626c75652e737667"></a></p>
  <p dir="auto">dateutil functions for Datasette</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Installation</h2><a id="user-content-installation" class="anchor" aria-label="Permalink: Installation" href="#installation"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">Install this plugin in the same environment as Datasette.</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ datasette install datasette-dateutil"><pre class="notranslate"><code>$ datasette install datasette-dateutil
  </code></pre></div>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Usage</h2><a id="user-content-usage" class="anchor" aria-label="Permalink: Usage" href="#usage"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">This function adds custom SQL functions that expose functionality from the <a href="https://dateutil.readthedocs.io/" rel="nofollow">dateutil</a> Python library.</p>
  <p dir="auto">Once installed, the following SQL functions become available:</p>
  <div class="markdown-heading" dir="auto"><h3 class="heading-element" dir="auto">Parsing date strings</h3><a id="user-content-parsing-date-strings" class="anchor" aria-label="Permalink: Parsing date strings" href="#parsing-date-strings"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <ul dir="auto">
  <li><code>dateutil_parse(text)</code> - returns an ISO8601 date string parsed from the text, or <code>null</code> if the input could not be parsed. <code>dateutil_parse("10 october 2020 3pm")</code> returns <code>2020-10-10T15:00:00</code>.</li>
  <li><code>dateutil_parse_fuzzy(text)</code> - same as <code>dateutil_parse()</code> but this also works against strings that contain a date somewhere within them - that date will be returned, or <code>null</code> if no dates could be found. <code>dateutil_parse_fuzzy("This is due 10 september")</code> returns <code>2020-09-10T00:00:00</code> (but will start returning the 2021 version of that if the year is 2021).</li>
  </ul>
  <p dir="auto">The <code>dateutil_parse()</code> and <code>dateutil_parse_fuzzy()</code> functions both follow the American convention of assuming that <code>1/2/2020</code> lists the month first, evaluating this example to the 2nd of January.</p>
  <p dir="auto">If you want to assume that the day comes first, use these two functions instead:</p>
  <ul dir="auto">
  <li><code>dateutil_parse_dayfirst(text)</code></li>
  <li><code>dateutil_parse_fuzzy_dayfirst(text)</code></li>
  </ul>
  <p dir="auto">Here's a query demonstrating these functions:</p>
  <div class="highlight highlight-source-sql notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="select
    dateutil_parse("10 october 2020 3pm"),
    dateutil_parse_fuzzy("This is due 10 september"),
    dateutil_parse("1/2/2020"),
    dateutil_parse("2020-03-04"),
    dateutil_parse_dayfirst("2020-03-04");"><pre><span class="pl-k">select</span>
    dateutil_parse(<span class="pl-s"><span class="pl-pds">"</span>10 october 2020 3pm<span class="pl-pds">"</span></span>),
    dateutil_parse_fuzzy(<span class="pl-s"><span class="pl-pds">"</span>This is due 10 september<span class="pl-pds">"</span></span>),
    dateutil_parse(<span class="pl-s"><span class="pl-pds">"</span>1/2/2020<span class="pl-pds">"</span></span>),
    dateutil_parse(<span class="pl-s"><span class="pl-pds">"</span>2020-03-04<span class="pl-pds">"</span></span>),
    dateutil_parse_dayfirst(<span class="pl-s"><span class="pl-pds">"</span>2020-03-04<span class="pl-pds">"</span></span>);</pre></div>
  <p dir="auto"><a href="https://latest-with-plugins.datasette.io/fixtures?sql=select%0D%0A++dateutil_parse%28%2210+october+2020+3pm%22%29%2C%0D%0A++dateutil_parse_fuzzy%28%22This+is+due+10+september%22%29%2C%0D%0A++dateutil_parse%28%221%2F2%2F2020%22%29%2C%0D%0A++dateutil_parse%28%222020-03-04%22%29%2C%0D%0A++dateutil_parse_dayfirst%28%222020-03-04%22%29%3B" rel="nofollow">Try that query</a></p>
  <div class="markdown-heading" dir="auto"><h3 class="heading-element" dir="auto">Optional default dates</h3><a id="user-content-optional-default-dates" class="anchor" aria-label="Permalink: Optional default dates" href="#optional-default-dates"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">The <code>dateutil_parse()</code>, <code>dateutil_parse_fuzzy()</code>, <code>dateutil_parse_dayfirst()</code> and <code>dateutil_parse_fuzzy_dayfirst()</code> functions all accept an optional second argument specifying a "default" datetime to consider if some of the details are missing. For example, the following:</p>
  <div class="highlight highlight-source-sql notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="select dateutil_parse('1st october', '1985-01-01')"><pre><span class="pl-k">select</span> dateutil_parse(<span class="pl-s"><span class="pl-pds">'</span>1st october<span class="pl-pds">'</span></span>, <span class="pl-s"><span class="pl-pds">'</span>1985-01-01<span class="pl-pds">'</span></span>)</pre></div>
  <p dir="auto">Will return <code>1985-10-01T00:00:00</code> - the missing year is replaced with the year from the default date.</p>
  <p dir="auto"><a href="https://latest-with-plugins.datasette.io/fixtures?sql=with+times+as+%28%0D%0A++select%0D%0A++++datetime%28%27now%27%29+as+t%0D%0A++union%0D%0A++select%0D%0A++++datetime%28%27now%27%2C+%27-1+year%27%29%0D%0A++union%0D%0A++select%0D%0A++++datetime%28%27now%27%2C+%27-3+years%27%29%0D%0A%29%0D%0Aselect+t%2C+dateutil_parse_fuzzy%28%22This+is+due+10+september%22%2C+t%29+from+times" rel="nofollow">Example query demonstrating the default date argument</a></p>
  <div class="markdown-heading" dir="auto"><h3 class="heading-element" dir="auto">Calculating Easter</h3><a id="user-content-calculating-easter" class="anchor" aria-label="Permalink: Calculating Easter" href="#calculating-easter"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <ul dir="auto">
  <li><code>dateutil_easter(year)</code> - returns the date for Easter in that year, for example <code>dateutil_easter("2020")</code> returns <code>2020-04-12</code>.</li>
  </ul>
  <p dir="auto"><a href="https://latest-with-plugins.datasette.io/fixtures?sql=select%0D%0A++dateutil_easter%282019%29%2C%0D%0A++dateutil_easter%282020%29%2C%0D%0A++dateutil_easter%282021%29" rel="nofollow">Example Easter query</a></p>
  <div class="markdown-heading" dir="auto"><h3 class="heading-element" dir="auto">JSON arrays of dates</h3><a id="user-content-json-arrays-of-dates" class="anchor" aria-label="Permalink: JSON arrays of dates" href="#json-arrays-of-dates"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">Several functions return JSON arrays of date strings. These can be used with SQLite's <code>json_each()</code> function to perform joins against dates from a specific date range or recurrence rule.</p>
  <p dir="auto">These functions can return up to 10,000 results. They will return an error if more than 10,000 dates would be returned - this is to protect against denial of service attacks.</p>
  <ul dir="auto">
  <li><code>dateutil_dates_between('1 january 2020', '5 jan 2020')</code> - given two dates (in any format that can be handled by <code>dateutil_parse()</code>) this function returns a JSON string containing the dates between those two days, inclusive. This example returns <code>["2020-01-01", "2020-01-02", "2020-01-03", "2020-01-04", "2020-01-05"]</code>.</li>
  <li><code>dateutil_dates_between('1 january 2020', '5 jan 2020', 0)</code> - set the optional third argument to <code>0</code> to specify that you would like this to be exclusive of the last day. This example returns <code>["2020-01-01", "2020-01-02", "2020-01-03", "2020-01-04"]</code>.</li>
  </ul>
  <p dir="auto"><a href="https://latest-with-plugins.datasette.io/fixtures?sql=select%0D%0A++dateutil_dates_between%28%271+january+2020%27%2C+%275+jan+2020%27%29%2C%0D%0A++dateutil_dates_between%28%271+january+2020%27%2C+%275+jan+2020%27%2C+0%29" rel="nofollow">Try these queries</a></p>
  <p dir="auto">The <code>dateutil_rrule()</code> and <code>dateutil_rrule_date()</code> functions accept the iCalendar standard <code>rrule</code> format - see <a href="https://dateutil.readthedocs.io/en/stable/rrule.html#rrulestr-examples" rel="nofollow">the dateutil documentation</a> for more examples.</p>
  <p dir="auto">This format lets you specify recurrence rules such as "the next four last mondays of the month".</p>
  <ul dir="auto">
  <li><code>dateutil_rrule(rrule, optional_dtsart)</code> - given an rrule returns a JSON array of ISO datetimes. The second argument is optional and will be treated as the start date for the rule.</li>
  <li><code>dateutil_rrule_date(rrule, optional_dtsart)</code> - same as <code>dateutil_rrule()</code> but returns ISO dates.</li>
  </ul>
  <p dir="auto">Example query:</p>
  <div class="highlight highlight-source-sql notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="select
    dateutil_rrule('FREQ=HOURLY;COUNT=5'),
    dateutil_rrule_date(
      'FREQ=DAILY;COUNT=3',
      '1st jan 2020'
    );"><pre><span class="pl-k">select</span>
    dateutil_rrule(<span class="pl-s"><span class="pl-pds">'</span>FREQ=HOURLY;COUNT=5<span class="pl-pds">'</span></span>),
    dateutil_rrule_date(
      <span class="pl-s"><span class="pl-pds">'</span>FREQ=DAILY;COUNT=3<span class="pl-pds">'</span></span>,
      <span class="pl-s"><span class="pl-pds">'</span>1st jan 2020<span class="pl-pds">'</span></span>
    );</pre></div>
  <p dir="auto"><a href="https://latest-with-plugins.datasette.io/fixtures?sql=select%0D%0A++dateutil_rrule('FREQ%3DHOURLY%3BCOUNT%3D5')%2C%0D%0A++dateutil_rrule_date(%0D%0A++++'FREQ%3DDAILY%3BCOUNT%3D3'%2C%0D%0A++++'1st+jan+2020'%0D%0A++)%3B" rel="nofollow">Try the rrule example query</a></p>
  <div class="markdown-heading" dir="auto"><h3 class="heading-element" dir="auto">Joining data using json_each()</h3><a id="user-content-joining-data-using-json_each" class="anchor" aria-label="Permalink: Joining data using json_each()" href="#joining-data-using-json_each"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">SQLite's <a href="https://www.sqlite.org/json1.html#jeach" rel="nofollow">json_each() function</a> can be used to turn a JSON array of dates into a table that can be joined against other data. Here's a query that returns a table showing every day in January 2019:</p>
  <div class="highlight highlight-source-sql notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="select
    value as date
  from
    json_each(
      dateutil_dates_between('1 Jan 2019', '31 Jan 2019')
    )"><pre><span class="pl-k">select</span>
    value <span class="pl-k">as</span> <span class="pl-k">date</span>
  <span class="pl-k">from</span>
    json_each(
      dateutil_dates_between(<span class="pl-s"><span class="pl-pds">'</span>1 Jan 2019<span class="pl-pds">'</span></span>, <span class="pl-s"><span class="pl-pds">'</span>31 Jan 2019<span class="pl-pds">'</span></span>)
    )</pre></div>
  <p dir="auto"><a href="https://latest-with-plugins.datasette.io/fixtures?sql=select%0D%0A++value+as+date%0D%0Afrom%0D%0A++json_each%28%0D%0A++++dateutil_dates_between%28%271+Jan+2019%27%2C+%2731+Jan+2019%27%29%0D%0A++%29" rel="nofollow">Try that query</a></p>
  <p dir="auto">You can run joins against this table by assigning it a name using SQLite's <a href="https://sqlite.org/lang_with.html" rel="nofollow">support for Common Table Expressions (CTEs)</a>.</p>
  <p dir="auto">This example query uses <code>substr(created, 0, 11)</code> to retrieve the date portion of the <code>created</code> column in the <a href="https://latest-with-plugins.datasette.io/fixtures/facetable" rel="nofollow">facetable demo table</a>, then joins that against the table of days in January to calculate the count of rows created on each day. The <code>LEFT JOIN</code> against <code>days_in_january</code> ensures that days which had no created records are still returned in the results, with a count of 0.</p>
  <div class="highlight highlight-source-sql notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="with created_dates as (
    select
      substr(created, 0, 11) as date
    from
      facetable
  ),
  days_in_january as (
    select
      value as date
    from
      json_each(
        dateutil_dates_between('1 Jan 2019', '31 Jan 2019')
      )
  )
  select
    days_in_january.date,
    count(created_dates.date) as total
  from
    days_in_january
    left join created_dates on days_in_january.date = created_dates.date
  group by
    days_in_january.date;"><pre>with created_dates <span class="pl-k">as</span> (
    <span class="pl-k">select</span>
      substr(created, <span class="pl-c1">0</span>, <span class="pl-c1">11</span>) <span class="pl-k">as</span> <span class="pl-k">date</span>
    <span class="pl-k">from</span>
      facetable
  ),
  days_in_january <span class="pl-k">as</span> (
    <span class="pl-k">select</span>
      value <span class="pl-k">as</span> <span class="pl-k">date</span>
    <span class="pl-k">from</span>
      json_each(
        dateutil_dates_between(<span class="pl-s"><span class="pl-pds">'</span>1 Jan 2019<span class="pl-pds">'</span></span>, <span class="pl-s"><span class="pl-pds">'</span>31 Jan 2019<span class="pl-pds">'</span></span>)
      )
  )
  <span class="pl-k">select</span>
    <span class="pl-c1">days_in_january</span>.<span class="pl-c1">date</span>,
    <span class="pl-c1">count</span>(<span class="pl-c1">created_dates</span>.<span class="pl-c1">date</span>) <span class="pl-k">as</span> total
  <span class="pl-k">from</span>
    days_in_january
    <span class="pl-k">left join</span> created_dates <span class="pl-k">on</span> <span class="pl-c1">days_in_january</span>.<span class="pl-c1">date</span> <span class="pl-k">=</span> <span class="pl-c1">created_dates</span>.<span class="pl-c1">date</span>
  <span class="pl-k">group by</span>
    <span class="pl-c1">days_in_january</span>.<span class="pl-c1">date</span>;</pre></div>
  <p dir="auto"><a href="https://latest-with-plugins.datasette.io/fixtures?sql=with+created_dates+as+%28%0D%0A++select%0D%0A++++substr%28created%2C+0%2C+11%29+as+date%0D%0A++from%0D%0A++++facetable%0D%0A%29%2C%0D%0Adays_in_january+as+%28%0D%0A++select%0D%0A++++value+as+date%0D%0A++from%0D%0A++++json_each%28%0D%0A++++++dateutil_dates_between%28%271+Jan+2019%27%2C+%2731+Jan+2019%27%29%0D%0A++++%29%0D%0A%29%0D%0Aselect%0D%0A++days_in_january.date%2C%0D%0A++count%28created_dates.date%29+as+total%0D%0Afrom%0D%0A++days_in_january%0D%0A++left+join+created_dates+on+days_in_january.date+%3D+created_dates.date%0D%0Agroup+by%0D%0A++days_in_january.date%3B#g.mark=bar&g.x_column=date&g.x_type=ordinal&g.y_column=total&g.y_type=quantitative" rel="nofollow">Try that query</a> with a bar chart rendered using the <a href="https://github.com/simonw/datasette-vega">datasette-vega</a> plugin.</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Development</h2><a id="user-content-development" class="anchor" aria-label="Permalink: Development" href="#development"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">To set up this plugin locally, first checkout the code. Then create a new virtual environment:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="cd datasette-dateutil
  python3 -mvenv venv
  source venv/bin/activate"><pre class="notranslate"><code>cd datasette-dateutil
  python3 -mvenv venv
  source venv/bin/activate
  </code></pre></div>
  <p dir="auto">Or if you are using <code>pipenv</code>:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="pipenv shell"><pre class="notranslate"><code>pipenv shell
  </code></pre></div>
  <p dir="auto">Now install the dependencies and tests:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="pip install -e '.[test]'"><pre class="notranslate"><code>pip install -e '.[test]'
  </code></pre></div>
  <p dir="auto">To run the tests:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="pytest"><pre class="notranslate"><code>pytest
  </code></pre></div>
  </article></div>


node_id: MDEwOlJlcG9zaXRvcnkyOTkxOTgzNjk=
name: datasette-import-table
full_name: datasette/datasette-import-table
html_url: https://github.com/datasette/datasette-import-table
description: Datasette plugin for importing tables from other Datasette instances
created_at: 2020-09-28T05:30:07Z
updated_at: 2025-01-30T17:20:37Z
pushed_at: 2025-01-30T17:20:33Z
language: Python
topics: ["datasette", "datasette-io", "datasette-plugin"]
visibility: public
default_branch: main
permissions: {"admin": false, "maintain": false, "push": false, "triage": false, "pull": true}
custom_properties: {}
readme:
  # datasette-import-table

  [![PyPI](https://img.shields.io/pypi/v/datasette-import-table.svg)](https://pypi.org/project/datasette-import-table/)
  [![Changelog](https://img.shields.io/github/v/release/datasette/datasette-import-table?include_prereleases&label=changelog)](https://github.com/datasette/datasette-import-table/releases)
  [![Tests](https://github.com/datasette/datasette-import-table/workflows/Test/badge.svg)](https://github.com/datasette/datasette-import-table/actions?query=workflow%3ATest)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/datasette/datasette-import-table/blob/main/LICENSE)

  Datasette plugin for importing tables from other Datasette instances

  ## Installation

  Install this plugin in the same environment as Datasette.

      $ datasette install datasette-import-table

  ## Usage

  Visit `/-/import-table` for the interface. Paste in the URL to a table page on another Datasette instance and click the button to import that table.

  By default only [the root actor](https://datasette.readthedocs.io/en/stable/authentication.html#using-the-root-actor) can access the page - so you'll need to run Datasette with the `--root` option and click on the link shown in the terminal to sign in and access the page.

  The `import-table` permission governs access. You can use permission plugins such as [datasette-permissions-sql](https://github.com/simonw/datasette-permissions-sql) to grant additional access to the write interface.

  ## Development

  To set up this plugin locally, first checkout the code. Then create a new virtual environment:

      cd datasette-import-table
      python3 -mvenv venv
      source venv/bin/activate

  Or if you are using `pipenv`:

      pipenv shell

  Now install the dependencies and tests:

      pip install -e '.[test]'

  To run the tests:

      pytest

readme_html:
  <div id="readme" class="md" data-path="README.md"><article class="markdown-body entry-content container-lg" itemprop="text"><div class="markdown-heading" dir="auto"><h1 class="heading-element" dir="auto">datasette-import-table</h1><a id="user-content-datasette-import-table" class="anchor" aria-label="Permalink: datasette-import-table" href="#datasette-import-table"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto"><a href="https://pypi.org/project/datasette-import-table/" rel="nofollow"><img src="https://img.shields.io/pypi/v/datasette-import-table.svg" alt="PyPI" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/33515b4248af780ed8b6e90fcb15f7da506259500826e1576aee45a25aa33e09/68747470733a2f2f696d672e736869656c64732e696f2f707970692f762f6461746173657474652d696d706f72742d7461626c652e737667"></a>
  <a href="https://github.com/datasette/datasette-import-table/releases"><img src="https://img.shields.io/github/v/release/datasette/datasette-import-table?include_prereleases&label=changelog" alt="Changelog" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/92ca849a4bb11c4aa4a3c8df51da5507b492f6f842882df8479e37094238749d/68747470733a2f2f696d672e736869656c64732e696f2f6769746875622f762f72656c656173652f6461746173657474652f6461746173657474652d696d706f72742d7461626c653f696e636c7564655f70726572656c6561736573266c6162656c3d6368616e67656c6f67"></a>
  <a href="https://github.com/datasette/datasette-import-table/actions?query=workflow%3ATest"><img src="https://github.com/datasette/datasette-import-table/workflows/Test/badge.svg" alt="Tests" style="max-width: 100%;"></a>
  <a href="https://github.com/datasette/datasette-import-table/blob/main/LICENSE"><img src="https://img.shields.io/badge/license-Apache%202.0-blue.svg" alt="License" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/c355f200ea90fddaa407b6eaab303663a669248ea3ca7b1fcf77dbe04ff5f48c/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6c6963656e73652d417061636865253230322e302d626c75652e737667"></a></p>
  <p dir="auto">Datasette plugin for importing tables from other Datasette instances</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Installation</h2><a id="user-content-installation" class="anchor" aria-label="Permalink: Installation" href="#installation"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">Install this plugin in the same environment as Datasette.</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ datasette install datasette-import-table"><pre class="notranslate"><code>$ datasette install datasette-import-table
  </code></pre></div>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Usage</h2><a id="user-content-usage" class="anchor" aria-label="Permalink: Usage" href="#usage"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">Visit <code>/-/import-table</code> for the interface. Paste in the URL to a table page on another Datasette instance and click the button to import that table.</p>
  <p dir="auto">By default only <a href="https://datasette.readthedocs.io/en/stable/authentication.html#using-the-root-actor" rel="nofollow">the root actor</a> can access the page - so you'll need to run Datasette with the <code>--root</code> option and click on the link shown in the terminal to sign in and access the page.</p>
  <p dir="auto">The <code>import-table</code> permission governs access. You can use permission plugins such as <a href="https://github.com/simonw/datasette-permissions-sql">datasette-permissions-sql</a> to grant additional access to the write interface.</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Development</h2><a id="user-content-development" class="anchor" aria-label="Permalink: Development" href="#development"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">To set up this plugin locally, first checkout the code. Then create a new virtual environment:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="cd datasette-import-table
  python3 -mvenv venv
  source venv/bin/activate"><pre class="notranslate"><code>cd datasette-import-table
  python3 -mvenv venv
  source venv/bin/activate
  </code></pre></div>
  <p dir="auto">Or if you are using <code>pipenv</code>:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="pipenv shell"><pre class="notranslate"><code>pipenv shell
  </code></pre></div>
  <p dir="auto">Now install the dependencies and tests:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="pip install -e '.[test]'"><pre class="notranslate"><code>pip install -e '.[test]'
  </code></pre></div>
  <p dir="auto">To run the tests:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="pytest"><pre class="notranslate"><code>pytest
  </code></pre></div>
  </article></div>


node_id: MDEwOlJlcG9zaXRvcnkzMDMyMTgzNjk=
name: evernote-to-sqlite
full_name: dogsheep/evernote-to-sqlite
html_url: https://github.com/dogsheep/evernote-to-sqlite
description: Tools for converting Evernote content to SQLite
created_at: 2020-10-11T21:45:49Z
updated_at: 2025-03-28T17:23:11Z
pushed_at: 2021-11-02T20:04:32Z
language: Python
license: apache-2.0
topics: ["datasette-io", "datasette-tool", "dogsheep", "evernote", "sqlite"]
visibility: public
default_branch: main
permissions: {"admin": false, "maintain": false, "push": false, "triage": false, "pull": true}
custom_properties: {}
readme:
  # evernote-to-sqlite

  [![PyPI](https://img.shields.io/pypi/v/evernote-to-sqlite.svg)](https://pypi.org/project/evernote-to-sqlite/)
  [![Changelog](https://img.shields.io/github/v/release/dogsheep/evernote-to-sqlite?include_prereleases&label=changelog)](https://github.com/dogsheep/evernote-to-sqlite/releases)
  [![Tests](https://github.com/dogsheep/evernote-to-sqlite/workflows/Test/badge.svg)](https://github.com/dogsheep/evernote-to-sqlite/actions?query=workflow%3ATest)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/dogsheep/evernote-to-sqlite/blob/master/LICENSE)

  Tools for converting Evernote content to SQLite. See [Building an Evernote to SQLite exporter](https://simonwillison.net/2020/Oct/16/building-evernote-sqlite-exporter/) for background on this project.

  ## Installation

  Install this tool using `pip`:

      $ pip install evernote-to-sqlite

  ## Usage

  Currently the only available command is `evernote-to-sqlite enex`, which converts Evernote's ENEX export files into a SQLite database.

  You can create [an ENEX export](https://help.evernote.com/hc/en-us/articles/209005557-Export-notes-and-notebooks-as-ENEX-or-HTML) in the Evernote desktop application by selecting some notes (or all of your notes) and using the `File -> Export Notes...` menu option.

  This used to be able to export everything in one go, but it looks like more recent Evernote versions only allow exporting up to fifty notes at a time, or let you export an entire notebook by right-clicking on the notebook and selecting "Export notebook...".

  You can convert that file to SQLite like so:

      $ evernote-to-sqlite enex evernote.db MyNotes.enex

  This will display a progress bar and create a SQLite database file called `evernote.db`.

  ### Limitations

  Unfortunately the ENEX export format does not include a unique identifier for each note. This means you cannot use this tool to re-import notes after they have been updated - you should consider this tool to be a one-time transformation of an ENEX file into an equivalent SQLite database.

  ENEX exports also do not include details of which notebook a note belongs to.

  ## Development

  To contribute to this tool, first checkout the code. Then create a new virtual environment:

      cd evernote-to-sqlite
      python -mvenv venv
      source venv/bin/activate

  Or if you are using `pipenv`:

      pipenv shell

  Now install the dependencies and tests:

      pip install -e '.[test]'

  To run the tests:

      pytest

readme_html:
  <div id="readme" class="md" data-path="README.md"><article class="markdown-body entry-content container-lg" itemprop="text"><div class="markdown-heading" dir="auto"><h1 class="heading-element" dir="auto">evernote-to-sqlite</h1><a id="user-content-evernote-to-sqlite" class="anchor" aria-label="Permalink: evernote-to-sqlite" href="#evernote-to-sqlite"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto"><a href="https://pypi.org/project/evernote-to-sqlite/" rel="nofollow"><img src="https://img.shields.io/pypi/v/evernote-to-sqlite.svg" alt="PyPI" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/e7d3c6a4c7deef5a4c2b73b7740c0bb7f893b76c52d4e2139c03d381c0d02f4b/68747470733a2f2f696d672e736869656c64732e696f2f707970692f762f657665726e6f74652d746f2d73716c6974652e737667"></a>
  <a href="https://github.com/dogsheep/evernote-to-sqlite/releases"><img src="https://img.shields.io/github/v/release/dogsheep/evernote-to-sqlite?include_prereleases&label=changelog" alt="Changelog" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/42736b299ae7fbd782c394480d05cdf728643fea10d302a7b20b4d845f29c459/68747470733a2f2f696d672e736869656c64732e696f2f6769746875622f762f72656c656173652f646f6773686565702f657665726e6f74652d746f2d73716c6974653f696e636c7564655f70726572656c6561736573266c6162656c3d6368616e67656c6f67"></a>
  <a href="https://github.com/dogsheep/evernote-to-sqlite/actions?query=workflow%3ATest"><img src="https://github.com/dogsheep/evernote-to-sqlite/workflows/Test/badge.svg" alt="Tests" style="max-width: 100%;"></a>
  <a href="https://github.com/dogsheep/evernote-to-sqlite/blob/master/LICENSE"><img src="https://img.shields.io/badge/license-Apache%202.0-blue.svg" alt="License" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/c355f200ea90fddaa407b6eaab303663a669248ea3ca7b1fcf77dbe04ff5f48c/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6c6963656e73652d417061636865253230322e302d626c75652e737667"></a></p>
  <p dir="auto">Tools for converting Evernote content to SQLite. See <a href="https://simonwillison.net/2020/Oct/16/building-evernote-sqlite-exporter/" rel="nofollow">Building an Evernote to SQLite exporter</a> for background on this project.</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Installation</h2><a id="user-content-installation" class="anchor" aria-label="Permalink: Installation" href="#installation"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">Install this tool using <code>pip</code>:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ pip install evernote-to-sqlite"><pre class="notranslate"><code>$ pip install evernote-to-sqlite
  </code></pre></div>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Usage</h2><a id="user-content-usage" class="anchor" aria-label="Permalink: Usage" href="#usage"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">Currently the only available command is <code>evernote-to-sqlite enex</code>, which converts Evernote's ENEX export files into a SQLite database.</p>
  <p dir="auto">You can create <a href="https://help.evernote.com/hc/en-us/articles/209005557-Export-notes-and-notebooks-as-ENEX-or-HTML" rel="nofollow">an ENEX export</a> in the Evernote desktop application by selecting some notes (or all of your notes) and using the <code>File -> Export Notes...</code> menu option.</p>
  <p dir="auto">This used to be able to export everything in one go, but it looks like more recent Evernote versions only allow exporting up to fifty notes at a time, or let you export an entire notebook by right-clicking on the notebook and selecting "Export notebook...".</p>
  <p dir="auto">You can convert that file to SQLite like so:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ evernote-to-sqlite enex evernote.db MyNotes.enex"><pre class="notranslate"><code>$ evernote-to-sqlite enex evernote.db MyNotes.enex
  </code></pre></div>
  <p dir="auto">This will display a progress bar and create a SQLite database file called <code>evernote.db</code>.</p>
  <div class="markdown-heading" dir="auto"><h3 class="heading-element" dir="auto">Limitations</h3><a id="user-content-limitations" class="anchor" aria-label="Permalink: Limitations" href="#limitations"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">Unfortunately the ENEX export format does not include a unique identifier for each note. This means you cannot use this tool to re-import notes after they have been updated - you should consider this tool to be a one-time transformation of an ENEX file into an equivalent SQLite database.</p>
  <p dir="auto">ENEX exports also do not include details of which notebook a note belongs to.</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Development</h2><a id="user-content-development" class="anchor" aria-label="Permalink: Development" href="#development"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">To contribute to this tool, first checkout the code. Then create a new virtual environment:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="cd evernote-to-sqlite
  python -mvenv venv
  source venv/bin/activate"><pre class="notranslate"><code>cd evernote-to-sqlite
  python -mvenv venv
  source venv/bin/activate
  </code></pre></div>
  <p dir="auto">Or if you are using <code>pipenv</code>:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="pipenv shell"><pre class="notranslate"><code>pipenv shell
  </code></pre></div>
  <p dir="auto">Now install the dependencies and tests:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="pip install -e '.[test]'"><pre class="notranslate"><code>pip install -e '.[test]'
  </code></pre></div>
  <p dir="auto">To run the tests:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="pytest"><pre class="notranslate"><code>pytest
  </code></pre></div>
  </article></div>


node_id: MDEwOlJlcG9zaXRvcnkzMDUxOTk2NjE=
name: sphinx-to-sqlite
full_name: simonw/sphinx-to-sqlite
html_url: https://github.com/simonw/sphinx-to-sqlite
description: Create a SQLite database from Sphinx documentation
created_at: 2020-10-18T21:26:55Z
updated_at: 2023-07-14T19:47:31Z
pushed_at: 2020-10-22T04:55:45Z
language: Python
license: apache-2.0
topics: ["datasette-io", "datasette-tool", "sphinx", "sqlite"]
visibility: public
default_branch: main
permissions: {"admin": false, "maintain": false, "push": false, "triage": false, "pull": true}
readme:
  # sphinx-to-sqlite

  [![PyPI](https://img.shields.io/pypi/v/sphinx-to-sqlite.svg)](https://pypi.org/project/sphinx-to-sqlite/)
  [![Changelog](https://img.shields.io/github/v/release/simonw/sphinx-to-sqlite?include_prereleases&label=changelog)](https://github.com/simonw/sphinx-to-sqlite/releases)
  [![Tests](https://github.com/simonw/sphinx-to-sqlite/workflows/Test/badge.svg)](https://github.com/simonw/sphinx-to-sqlite/actions?query=workflow%3ATest)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/sphinx-to-sqlite/blob/master/LICENSE)

  Create a SQLite database from Sphinx documentation.

  ## Demo

  You can see the results of running this tool against the [Datasette documentation](https://docs.datasette.io/) at https://latest-docs.datasette.io/docs/sections

  ## Installation

  Install this tool using `pip`:

      $ pip install sphinx-to-sqlite

  ## Usage

  First run `sphinx-build` with the `-b xml` option to create XML files in your `_build/` directory.

  Then run:

      $ sphinx-to-sqlite docs.db path/to/_build

  To build the SQLite database.

  ## Development

  To contribute to this tool, first checkout the code. Then create a new virtual environment:

      cd sphinx-to-sqlite
      python -mvenv venv
      source venv/bin/activate

  Or if you are using `pipenv`:

      pipenv shell

  Now install the dependencies and tests:

      pip install -e '.[test]'

  To run the tests:

      pytest

readme_html:
  <div id="readme" class="md" data-path="README.md"><article class="markdown-body entry-content container-lg" itemprop="text"><div class="markdown-heading" dir="auto"><h1 class="heading-element" dir="auto">sphinx-to-sqlite</h1><a id="user-content-sphinx-to-sqlite" class="anchor" aria-label="Permalink: sphinx-to-sqlite" href="#sphinx-to-sqlite"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto"><a href="https://pypi.org/project/sphinx-to-sqlite/" rel="nofollow"><img src="https://img.shields.io/pypi/v/sphinx-to-sqlite.svg" alt="PyPI" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/7a020f06c46b3fa4ee46b7d576306ea368f5d977a6e23c566a394fc0b2abc2a4/68747470733a2f2f696d672e736869656c64732e696f2f707970692f762f737068696e782d746f2d73716c6974652e737667"></a>
  <a href="https://github.com/simonw/sphinx-to-sqlite/releases"><img src="https://img.shields.io/github/v/release/simonw/sphinx-to-sqlite?include_prereleases&label=changelog" alt="Changelog" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/2690c713087af1a22275b3dadb590fc77d6e76362528c89678d3a5401f84814e/68747470733a2f2f696d672e736869656c64732e696f2f6769746875622f762f72656c656173652f73696d6f6e772f737068696e782d746f2d73716c6974653f696e636c7564655f70726572656c6561736573266c6162656c3d6368616e67656c6f67"></a>
  <a href="https://github.com/simonw/sphinx-to-sqlite/actions?query=workflow%3ATest"><img src="https://github.com/simonw/sphinx-to-sqlite/workflows/Test/badge.svg" alt="Tests" style="max-width: 100%;"></a>
  <a href="https://github.com/simonw/sphinx-to-sqlite/blob/master/LICENSE"><img src="https://img.shields.io/badge/license-Apache%202.0-blue.svg" alt="License" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/c355f200ea90fddaa407b6eaab303663a669248ea3ca7b1fcf77dbe04ff5f48c/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6c6963656e73652d417061636865253230322e302d626c75652e737667"></a></p>
  <p dir="auto">Create a SQLite database from Sphinx documentation.</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Demo</h2><a id="user-content-demo" class="anchor" aria-label="Permalink: Demo" href="#demo"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">You can see the results of running this tool against the <a href="https://docs.datasette.io/" rel="nofollow">Datasette documentation</a> at <a href="https://latest-docs.datasette.io/docs/sections" rel="nofollow">https://latest-docs.datasette.io/docs/sections</a></p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Installation</h2><a id="user-content-installation" class="anchor" aria-label="Permalink: Installation" href="#installation"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">Install this tool using <code>pip</code>:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ pip install sphinx-to-sqlite"><pre class="notranslate"><code>$ pip install sphinx-to-sqlite
  </code></pre></div>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Usage</h2><a id="user-content-usage" class="anchor" aria-label="Permalink: Usage" href="#usage"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">First run <code>sphinx-build</code> with the <code>-b xml</code> option to create XML files in your <code>_build/</code> directory.</p>
  <p dir="auto">Then run:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ sphinx-to-sqlite docs.db path/to/_build"><pre class="notranslate"><code>$ sphinx-to-sqlite docs.db path/to/_build
  </code></pre></div>
  <p dir="auto">To build the SQLite database.</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Development</h2><a id="user-content-development" class="anchor" aria-label="Permalink: Development" href="#development"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">To contribute to this tool, first checkout the code. Then create a new virtual environment:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="cd sphinx-to-sqlite
  python -mvenv venv
  source venv/bin/activate"><pre class="notranslate"><code>cd sphinx-to-sqlite
  python -mvenv venv
  source venv/bin/activate
  </code></pre></div>
  <p dir="auto">Or if you are using <code>pipenv</code>:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="pipenv shell"><pre class="notranslate"><code>pipenv shell
  </code></pre></div>
  <p dir="auto">Now install the dependencies and tests:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="pip install -e '.[test]'"><pre class="notranslate"><code>pip install -e '.[test]'
  </code></pre></div>
  <p dir="auto">To run the tests:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="pytest"><pre class="notranslate"><code>pytest
  </code></pre></div>
  </article></div>


node_id: MDEwOlJlcG9zaXRvcnkzMDY5NzY1MzA=
name: datasette-reconcile
full_name: drkane/datasette-reconcile
html_url: https://github.com/drkane/datasette-reconcile
created_at: 2020-10-24T21:41:06Z
updated_at: 2025-03-24T13:58:17Z
pushed_at: 2024-02-02T17:16:05Z
language: Python
license: mit
topics: ["datasette", "datasette-plugin", "python", "reconciliation-api", "reconciliation-service"]
visibility: public
default_branch: main
permissions: {"admin": false, "maintain": false, "push": false, "triage": false, "pull": true}
description:
  Adds a reconciliation API endpoint to Datasette, based on the Reconciliation Service API specification.

readme:
  # datasette-reconcile

  [![PyPI - Version](https://img.shields.io/pypi/v/datasette-reconcile.svg)](https://pypi.org/project/datasette-reconcile)
  [![PyPI - Python Version](https://img.shields.io/pypi/pyversions/datasette-reconcile.svg)](https://pypi.org/project/datasette-reconcile)
  [![Changelog](https://img.shields.io/github/v/release/drkane/datasette-reconcile?include_prereleases&label=changelog)](https://github.com/drkane/datasette-reconcile/releases)
  [![Tests](https://github.com/drkane/datasette-reconcile/workflows/Test/badge.svg)](https://github.com/drkane/datasette-reconcile/actions?query=workflow%3ATest)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/drkane/datasette-reconcile/blob/main/LICENSE)

  Adds a reconciliation API endpoint to [Datasette](https://docs.datasette.io/en/stable/), based on the [Reconciliation Service API](https://reconciliation-api.github.io/specs/latest/) specification.

  The reconciliation API is used to match a set of strings to their correct identifiers, to help with disambiguation and consistency in large datasets. For example, the strings "United Kingdom", "United Kingdom of Great Britain and Northern Ireland" and "UK" could all be used to identify the country which has the ISO country code `GB`. It is particularly implemented in [OpenRefine](https://openrefine.org/).

  The plugin adds a `/-/reconcile` endpoint to a table served by datasette, which responds based on the Reconciliation Service API specification. In order to activate this endpoint you need to configure the reconciliation service, as dscribed in the [usage](#usage) section.

  ## Installation

  Install this plugin in the same environment as Datasette.

      $ datasette install datasette-reconcile

  ## Usage

  ### Plugin configuration

  The plugin should be configured using Datasette's [`metadata.json`](https://docs.datasette.io/en/stable/metadata.html) file. The configuration can be put at the root, database or table layer of `metadata.json`, for most use cases it will make most sense to configure at the table level.

  Add a `datasette-reconcile` object under `plugins` in `metadata.json`. This should look something like:

  ```json
  {
    "databases": {
      "sf-trees": {
        "tables": {
          "Street_Tree_List": {
            "plugins": {
              "datasette-reconcile": {
                "id_field": "id",
                "name_field": "name",
                "type_field": "type",
                "type_default": [
                  {
                    "id": "tree",
                    "name": "Tree"
                  }
                ],
                "max_limit": 5,
                "service_name": "Tree reconciliation",
                "view_url": "https://example.com/trees/{{id}}"
              }
            }
          }
        }
      }
    }
  }
  ```

  The only required item in the configuration is `name_field`. This refers to the field in the table which will be searched to match the query text.

  The rest of the configuration items are optional, and are as follows:

  - `id_field`: The field containing the identifier for this entity. If not provided, and there is a primary key set, then the primary key will be used. A primary key of more than one field will give an error.
  - `type_field`: If provided, this field will be used to determine the type of the entity. If not provided, then the `type_default` setting will be used instead.
  - `type_default`: If provided, this value will be used as the type of every entity returned. If not provided the default of `Object` will be used for every entity.
  - `max_limit`: The maximum number of records that a query can request to return. This is 5 by default. A individual query can request fewer results than this, but it cannot request more.
  - `service_name`: The name of the reconciliation service that will appear in the service manifest. If not provided it will take the form `<database name> <table name> reconciliation`.
  - `identifierSpace`: [Identifier space](https://reconciliation-api.github.io/specs/latest/#identifier-and-schema-spaces) given in the service manifest. If not provided a default of `http://rdf.freebase.com/ns/type.object.id` is used.
  - `schemaSpace`: [Schema space](https://reconciliation-api.github.io/specs/latest/#identifier-and-schema-spaces) given in the service manifest. If not provided a default of `http://rdf.freebase.com/ns/type.object.id` is used.
  - `view_url`: [URL for a view of an individual entity](https://reconciliation-api.github.io/specs/latest/#dfn-view-template). It must contain the string `{{id}}` which will be replaced with the ID of the entity. If not provided it will use the default datasette view for the entity record (something like `/<db_name>/<table>/{{id}}`).

  ### Using the endpoint

  Once the plugin is configured for a particular database or table, you can access the reconciliation endpoint using the url `/<db_name>/<table>/-/reconcile`.

  A simple GET request to `/<db_name>/<table>/-/reconcile` will return the [Service Manifest](https://reconciliation-api.github.io/specs/latest/#service-manifest) as JSON which reconciliation clients can use to determine how the service is set up.

  A POST request to the same url with the `queries` argument set will trigger the reconciliation process. The `queries` parameter should be a json object in the format described in [the specification](https://reconciliation-api.github.io/specs/latest/#reconciliation-queries). An example set of two queries would look like:

  ```json
  {
    "q1": {
      "query": "Hans-Eberhard Urbaniak"
    },
    "q2": {
      "query": "Ernst Schwanhold"
    }
  }
  ```

  The query can optionally be encoded as a `queries` parameter in a GET request. For example:

  ```
  /<db_name>/<table>/-/reconcile?queries={"q1":{"query":"Hans-Eberhard Urbaniak"},"q2":{"query": "Ernst Schwanhold"}}
  ```

  Various options are available in the query object. Current the only ones implemented in datasette-reconcile are the mandatory `query` string, and the `limit` option, which must be less than or equal to the value in the `max_limit` configration option.

  All endpoints that start with `/<db_name>/<table>/-/reconcile` are configured to send an `Access-Control-Allow-Origin: *` CORS header to allow access [as described in the specification](https://reconciliation-api.github.io/specs/latest/#cross-origin-access).

  JSONP output is not yet supported.

  ### Returned value

  The result of the GET or POST `queries` requests described above is a json object describing potential [reconciliation candidates](https://reconciliation-api.github.io/specs/latest/#reconciliation-query-responses) for each of the queries specified. The result will look something like:

  ```json
  {
    "q1": {
      "result": [
        {
          "id": "120333937",
          "name": "Urbaniak, Regina",
          "score": 53.015232,
          "match": false,
          "type": [
            {
              "id": "person",
              "name": "Person"
            }
          ]
        },
        {
          "id": "1127147390",
          "name": "Urbaniak, Jan",
          "score": 52.357353,
          "match": false,
          "type": [
            {
              "id": "person",
              "name": "Person"
            }
          ]
        }
      ]
    },
    "q2": {
      "result": [
        {
          "id": "123064325",
          "name": "Schwanhold, Ernst",
          "score": 86.43497,
          "match": true,
          "type": [
            {
              "id": "person",
              "name": "Person"
            }
          ]
        },
        {
          "id": "116362988X",
          "name": "Schwanhold, Nadine",
          "score": 62.04763,
          "match": false,
          "type": [
            {
              "id": "person",
              "name": "Person"
            }
          ]
        }
      ]
    }
  }
  ```

  ### Behind the scenes

  The reconcile engine works by performing an SQL query against the `name_field` within the specified database table. Where that table has a full text search index implemented, the search will be performed against that index.

  When a full text search index is present on the table, the SQL query takes the form (based on the search query `test`, note that double quotes are added to facilitate searching - these are not present in the original query):

  ```sql
  select <id_field>, <name_field>
  from <table>
    inner join (
      select "rowid", "rank"
      from <fts_table>
      where <fts_table> MATCH '"test"'
    ) as "a" on <table>."rowid" = a."rowid"
  order by a.rank
  limit 5
  ```

  If a full text search index is not present, the query looks like this (note that the wildcard `%` is added to either side of the query - these are not present in the original query):

  ```sql
  select <id_field>, <name_field>
  from <table>
  where <name_field> like '%test%'
  limit 5
  ```

  ### Extend endpoint

  You can also use the reconciliation API [Data extension service](https://www.w3.org/community/reports/reconciliation/CG-FINAL-specs-0.2-20230410/#data-extension-service) to find additional properties for a set of entities, given an ID.

  Send a GET request to the `/<db_name>/<table>/-/reconcile/extend/propose` endpoint to find a list of the possible properties you can select. The properties are all the columns in the table (excluding any that have been hidden). An example response would look like:

  ```json
  {
    "limit": 5,
    "type": "Person",
    "properties": [
      {
        "id": "preferredName",
        "name": "preferredName"
      },
      {
        "id": "professionOrOccupation",
        "name": "professionOrOccupation"
      },
      {
        "id": "wikidataId",
        "name": "wikidataId"
      }
    ]
  }
  ```

  Then send a POST request to the `/<db_name>/<table>/-/reconcile` endpoint with an `extend` argument. The `extend` argument should be a JSON object with a set of `ids` to lookup and `properties` to return. For example:

  ```json
  {
    "ids": ["10662041X", "1064905412"],
    "properties": [
      {
        "id": "professionOrOccupation"
      },
      {
        "id": "wikidataId"
      }
    ]
  }
  ```

  The endpoint will return a result that looks like:

  ```json
  {
    "meta": [
      {
        "id": "professionOrOccupation",
        "name": "professionOrOccupation"
      },
      {
        "id": "wikidataId",
        "name": "wikidataId"
      }
    ],
    "rows": {
      "10662041X": {
        "professionOrOccupation": [
          {
            "str": "Doctor"
          }
        ],
        "wikidataId": [
          {
            "str": "Q3874347"
          }
        ]
      },
      "1064905412": {
        "professionOrOccupation": [
          {
            "str": "Architect"
          }
        ],
        "wikidataId": [
          {
            "str": "Q3874347"
          }
        ]
      }
    }
  }
  ```

  ### Suggest endpoints

  You can also use the [suggest endpoints](https://www.w3.org/community/reports/reconciliation/CG-FINAL-specs-0.2-20230410/#suggest-services) to get quick suggestions, for example for an auto-complete dropdown menu. The following endpoints are available:

  - `/<db_name>/<table>/-/reconcile/suggest/property` - looks up in a list of table columns
  - `/<db_name>/<table>/-/reconcile/suggest/entity` - looks up in a list of table rows
  - `/<db_name>/<table>/-/reconcile/suggest/type` - not currently implemented

  Each endpoint takes a `prefix` argument which can be used in a GET request. For example, the GET request `/<db_name>/<table>/-/reconcile/suggest/entity?prefix=abc` will produce a response such as:

  ```json
  {
    "result": [
      {
        "name": "abc company limited",
        "id": "Q123456"
      },
      {
        "name": "abc other company limited",
        "id": "Q123457"
      }
    ]
  }
  ```

  ## Development

  This plugin uses hatch for build and testing. To set up this plugin locally, first checkout the code.

  You'll need to fetch the git submodules for the tests too:

      git submodule init
      git submodule update

  To run the tests:

      hatch run test

  Run tests then report on coverage

      hatch run cov

  Run tests then run a server showing where coverage is missing

      hatch run cov-html

  ### Linting/formatting

  Black and ruff should be run before committing any changes.

  To check for any changes needed:

      hatch run lint:style

  To run any autoformatting possible:

      hatch run lint:fmt

  ### Publish to pypi

      hatch build
      hatch publish
      git tag v<VERSION_NUMBER>
      git push origin v<VERSION_NUMBER>

  ## Acknowledgements

  Thanks for [@simonw](https://github.com/simonw/) for developing datasette and the datasette ecosystem.

  Other contributions from:

  - [@JBPressac](https://github.com/JBPressac/)
  - [@nicokant](https://github.com/nicokant/) - implementation of extend service

readme_html:
  <div id="readme" class="md" data-path="README.md"><article class="markdown-body entry-content container-lg" itemprop="text"><div class="markdown-heading" dir="auto"><h1 class="heading-element" dir="auto">datasette-reconcile</h1><a id="user-content-datasette-reconcile" class="anchor" aria-label="Permalink: datasette-reconcile" href="#datasette-reconcile"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto"><a href="https://pypi.org/project/datasette-reconcile" rel="nofollow"><img src="https://img.shields.io/pypi/v/datasette-reconcile.svg" alt="PyPI - Version" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/2d5672b63cce54caab1d6178d996f685c700f1b0e815a1208c2f00093e4e9040/68747470733a2f2f696d672e736869656c64732e696f2f707970692f762f6461746173657474652d7265636f6e63696c652e737667"></a>
  <a href="https://pypi.org/project/datasette-reconcile" rel="nofollow"><img src="https://img.shields.io/pypi/pyversions/datasette-reconcile.svg" alt="PyPI - Python Version" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/ae1cab5a3d030380789f49e8c76c0cc82f5855063bf3d3ab4b8415b7b6d262b5/68747470733a2f2f696d672e736869656c64732e696f2f707970692f707976657273696f6e732f6461746173657474652d7265636f6e63696c652e737667"></a>
  <a href="https://github.com/drkane/datasette-reconcile/releases"><img src="https://img.shields.io/github/v/release/drkane/datasette-reconcile?include_prereleases&label=changelog" alt="Changelog" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/be2acb7981700fcd95db7c48ee6063d75e7cb6eaa5043981bc19a696f6d9da01/68747470733a2f2f696d672e736869656c64732e696f2f6769746875622f762f72656c656173652f64726b616e652f6461746173657474652d7265636f6e63696c653f696e636c7564655f70726572656c6561736573266c6162656c3d6368616e67656c6f67"></a>
  <a href="https://github.com/drkane/datasette-reconcile/actions?query=workflow%3ATest"><img src="https://github.com/drkane/datasette-reconcile/workflows/Test/badge.svg" alt="Tests" style="max-width: 100%;"></a>
  <a href="https://github.com/drkane/datasette-reconcile/blob/main/LICENSE"><img src="https://img.shields.io/badge/license-Apache%202.0-blue.svg" alt="License" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/c355f200ea90fddaa407b6eaab303663a669248ea3ca7b1fcf77dbe04ff5f48c/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6c6963656e73652d417061636865253230322e302d626c75652e737667"></a></p>
  <p dir="auto">Adds a reconciliation API endpoint to <a href="https://docs.datasette.io/en/stable/" rel="nofollow">Datasette</a>, based on the <a href="https://reconciliation-api.github.io/specs/latest/" rel="nofollow">Reconciliation Service API</a> specification.</p>
  <p dir="auto">The reconciliation API is used to match a set of strings to their correct identifiers, to help with disambiguation and consistency in large datasets. For example, the strings "United Kingdom", "United Kingdom of Great Britain and Northern Ireland" and "UK" could all be used to identify the country which has the ISO country code <code>GB</code>. It is particularly implemented in <a href="https://openrefine.org/" rel="nofollow">OpenRefine</a>.</p>
  <p dir="auto">The plugin adds a <code>/-/reconcile</code> endpoint to a table served by datasette, which responds based on the Reconciliation Service API specification. In order to activate this endpoint you need to configure the reconciliation service, as dscribed in the <a href="#usage">usage</a> section.</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Installation</h2><a id="user-content-installation" class="anchor" aria-label="Permalink: Installation" href="#installation"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">Install this plugin in the same environment as Datasette.</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ datasette install datasette-reconcile"><pre class="notranslate"><code>$ datasette install datasette-reconcile
  </code></pre></div>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Usage</h2><a id="user-content-usage" class="anchor" aria-label="Permalink: Usage" href="#usage"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <div class="markdown-heading" dir="auto"><h3 class="heading-element" dir="auto">Plugin configuration</h3><a id="user-content-plugin-configuration" class="anchor" aria-label="Permalink: Plugin configuration" href="#plugin-configuration"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">The plugin should be configured using Datasette's <a href="https://docs.datasette.io/en/stable/metadata.html" rel="nofollow"><code>metadata.json</code></a> file. The configuration can be put at the root, database or table layer of <code>metadata.json</code>, for most use cases it will make most sense to configure at the table level.</p>
  <p dir="auto">Add a <code>datasette-reconcile</code> object under <code>plugins</code> in <code>metadata.json</code>. This should look something like:</p>
  <div class="highlight highlight-source-json notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="{
    "databases": {
      "sf-trees": {
        "tables": {
          "Street_Tree_List": {
            "plugins": {
              "datasette-reconcile": {
                "id_field": "id",
                "name_field": "name",
                "type_field": "type",
                "type_default": [
                  {
                    "id": "tree",
                    "name": "Tree"
                  }
                ],
                "max_limit": 5,
                "service_name": "Tree reconciliation",
                "view_url": "https://example.com/trees/{{id}}"
              }
            }
          }
        }
      }
    }
  }"><pre>{
    <span class="pl-ent">"databases"</span>: {
      <span class="pl-ent">"sf-trees"</span>: {
        <span class="pl-ent">"tables"</span>: {
          <span class="pl-ent">"Street_Tree_List"</span>: {
            <span class="pl-ent">"plugins"</span>: {
              <span class="pl-ent">"datasette-reconcile"</span>: {
                <span class="pl-ent">"id_field"</span>: <span class="pl-s"><span class="pl-pds">"</span>id<span class="pl-pds">"</span></span>,
                <span class="pl-ent">"name_field"</span>: <span class="pl-s"><span class="pl-pds">"</span>name<span class="pl-pds">"</span></span>,
                <span class="pl-ent">"type_field"</span>: <span class="pl-s"><span class="pl-pds">"</span>type<span class="pl-pds">"</span></span>,
                <span class="pl-ent">"type_default"</span>: [
                  {
                    <span class="pl-ent">"id"</span>: <span class="pl-s"><span class="pl-pds">"</span>tree<span class="pl-pds">"</span></span>,
                    <span class="pl-ent">"name"</span>: <span class="pl-s"><span class="pl-pds">"</span>Tree<span class="pl-pds">"</span></span>
                  }
                ],
                <span class="pl-ent">"max_limit"</span>: <span class="pl-c1">5</span>,
                <span class="pl-ent">"service_name"</span>: <span class="pl-s"><span class="pl-pds">"</span>Tree reconciliation<span class="pl-pds">"</span></span>,
                <span class="pl-ent">"view_url"</span>: <span class="pl-s"><span class="pl-pds">"</span>https://example.com/trees/{{id}}<span class="pl-pds">"</span></span>
              }
            }
          }
        }
      }
    }
  }</pre></div>
  <p dir="auto">The only required item in the configuration is <code>name_field</code>. This refers to the field in the table which will be searched to match the query text.</p>
  <p dir="auto">The rest of the configuration items are optional, and are as follows:</p>
  <ul dir="auto">
  <li><code>id_field</code>: The field containing the identifier for this entity. If not provided, and there is a primary key set, then the primary key will be used. A primary key of more than one field will give an error.</li>
  <li><code>type_field</code>: If provided, this field will be used to determine the type of the entity. If not provided, then the <code>type_default</code> setting will be used instead.</li>
  <li><code>type_default</code>: If provided, this value will be used as the type of every entity returned. If not provided the default of <code>Object</code> will be used for every entity.</li>
  <li><code>max_limit</code>: The maximum number of records that a query can request to return. This is 5 by default. A individual query can request fewer results than this, but it cannot request more.</li>
  <li><code>service_name</code>: The name of the reconciliation service that will appear in the service manifest. If not provided it will take the form <code><database name> <table name> reconciliation</code>.</li>
  <li><code>identifierSpace</code>: <a href="https://reconciliation-api.github.io/specs/latest/#identifier-and-schema-spaces" rel="nofollow">Identifier space</a> given in the service manifest. If not provided a default of <code>http://rdf.freebase.com/ns/type.object.id</code> is used.</li>
  <li><code>schemaSpace</code>: <a href="https://reconciliation-api.github.io/specs/latest/#identifier-and-schema-spaces" rel="nofollow">Schema space</a> given in the service manifest. If not provided a default of <code>http://rdf.freebase.com/ns/type.object.id</code> is used.</li>
  <li><code>view_url</code>: <a href="https://reconciliation-api.github.io/specs/latest/#dfn-view-template" rel="nofollow">URL for a view of an individual entity</a>. It must contain the string <code>{{id}}</code> which will be replaced with the ID of the entity. If not provided it will use the default datasette view for the entity record (something like <code>/<db_name>/<table>/{{id}}</code>).</li>
  </ul>
  <div class="markdown-heading" dir="auto"><h3 class="heading-element" dir="auto">Using the endpoint</h3><a id="user-content-using-the-endpoint" class="anchor" aria-label="Permalink: Using the endpoint" href="#using-the-endpoint"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">Once the plugin is configured for a particular database or table, you can access the reconciliation endpoint using the url <code>/<db_name>/<table>/-/reconcile</code>.</p>
  <p dir="auto">A simple GET request to <code>/<db_name>/<table>/-/reconcile</code> will return the <a href="https://reconciliation-api.github.io/specs/latest/#service-manifest" rel="nofollow">Service Manifest</a> as JSON which reconciliation clients can use to determine how the service is set up.</p>
  <p dir="auto">A POST request to the same url with the <code>queries</code> argument set will trigger the reconciliation process. The <code>queries</code> parameter should be a json object in the format described in <a href="https://reconciliation-api.github.io/specs/latest/#reconciliation-queries" rel="nofollow">the specification</a>. An example set of two queries would look like:</p>
  <div class="highlight highlight-source-json notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="{
    "q1": {
      "query": "Hans-Eberhard Urbaniak"
    },
    "q2": {
      "query": "Ernst Schwanhold"
    }
  }"><pre>{
    <span class="pl-ent">"q1"</span>: {
      <span class="pl-ent">"query"</span>: <span class="pl-s"><span class="pl-pds">"</span>Hans-Eberhard Urbaniak<span class="pl-pds">"</span></span>
    },
    <span class="pl-ent">"q2"</span>: {
      <span class="pl-ent">"query"</span>: <span class="pl-s"><span class="pl-pds">"</span>Ernst Schwanhold<span class="pl-pds">"</span></span>
    }
  }</pre></div>
  <p dir="auto">The query can optionally be encoded as a <code>queries</code> parameter in a GET request. For example:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="/<db_name>/<table>/-/reconcile?queries={"q1":{"query":"Hans-Eberhard Urbaniak"},"q2":{"query": "Ernst Schwanhold"}}"><pre class="notranslate"><code>/<db_name>/<table>/-/reconcile?queries={"q1":{"query":"Hans-Eberhard Urbaniak"},"q2":{"query": "Ernst Schwanhold"}}
  </code></pre></div>
  <p dir="auto">Various options are available in the query object. Current the only ones implemented in datasette-reconcile are the mandatory <code>query</code> string, and the <code>limit</code> option, which must be less than or equal to the value in the <code>max_limit</code> configration option.</p>
  <p dir="auto">All endpoints that start with <code>/<db_name>/<table>/-/reconcile</code> are configured to send an <code>Access-Control-Allow-Origin: *</code> CORS header to allow access <a href="https://reconciliation-api.github.io/specs/latest/#cross-origin-access" rel="nofollow">as described in the specification</a>.</p>
  <p dir="auto">JSONP output is not yet supported.</p>
  <div class="markdown-heading" dir="auto"><h3 class="heading-element" dir="auto">Returned value</h3><a id="user-content-returned-value" class="anchor" aria-label="Permalink: Returned value" href="#returned-value"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">The result of the GET or POST <code>queries</code> requests described above is a json object describing potential <a href="https://reconciliation-api.github.io/specs/latest/#reconciliation-query-responses" rel="nofollow">reconciliation candidates</a> for each of the queries specified. The result will look something like:</p>
  <div class="highlight highlight-source-json notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="{
    "q1": {
      "result": [
        {
          "id": "120333937",
          "name": "Urbaniak, Regina",
          "score": 53.015232,
          "match": false,
          "type": [
            {
              "id": "person",
              "name": "Person"
            }
          ]
        },
        {
          "id": "1127147390",
          "name": "Urbaniak, Jan",
          "score": 52.357353,
          "match": false,
          "type": [
            {
              "id": "person",
              "name": "Person"
            }
          ]
        }
      ]
    },
    "q2": {
      "result": [
        {
          "id": "123064325",
          "name": "Schwanhold, Ernst",
          "score": 86.43497,
          "match": true,
          "type": [
            {
              "id": "person",
              "name": "Person"
            }
          ]
        },
        {
          "id": "116362988X",
          "name": "Schwanhold, Nadine",
          "score": 62.04763,
          "match": false,
          "type": [
            {
              "id": "person",
              "name": "Person"
            }
          ]
        }
      ]
    }
  }"><pre>{
    <span class="pl-ent">"q1"</span>: {
      <span class="pl-ent">"result"</span>: [
        {
          <span class="pl-ent">"id"</span>: <span class="pl-s"><span class="pl-pds">"</span>120333937<span class="pl-pds">"</span></span>,
          <span class="pl-ent">"name"</span>: <span class="pl-s"><span class="pl-pds">"</span>Urbaniak, Regina<span class="pl-pds">"</span></span>,
          <span class="pl-ent">"score"</span>: <span class="pl-c1">53.015232</span>,
          <span class="pl-ent">"match"</span>: <span class="pl-c1">false</span>,
          <span class="pl-ent">"type"</span>: [
            {
              <span class="pl-ent">"id"</span>: <span class="pl-s"><span class="pl-pds">"</span>person<span class="pl-pds">"</span></span>,
              <span class="pl-ent">"name"</span>: <span class="pl-s"><span class="pl-pds">"</span>Person<span class="pl-pds">"</span></span>
            }
          ]
        },
        {
          <span class="pl-ent">"id"</span>: <span class="pl-s"><span class="pl-pds">"</span>1127147390<span class="pl-pds">"</span></span>,
          <span class="pl-ent">"name"</span>: <span class="pl-s"><span class="pl-pds">"</span>Urbaniak, Jan<span class="pl-pds">"</span></span>,
          <span class="pl-ent">"score"</span>: <span class="pl-c1">52.357353</span>,
          <span class="pl-ent">"match"</span>: <span class="pl-c1">false</span>,
          <span class="pl-ent">"type"</span>: [
            {
              <span class="pl-ent">"id"</span>: <span class="pl-s"><span class="pl-pds">"</span>person<span class="pl-pds">"</span></span>,
              <span class="pl-ent">"name"</span>: <span class="pl-s"><span class="pl-pds">"</span>Person<span class="pl-pds">"</span></span>
            }
          ]
        }
      ]
    },
    <span class="pl-ent">"q2"</span>: {
      <span class="pl-ent">"result"</span>: [
        {
          <span class="pl-ent">"id"</span>: <span class="pl-s"><span class="pl-pds">"</span>123064325<span class="pl-pds">"</span></span>,
          <span class="pl-ent">"name"</span>: <span class="pl-s"><span class="pl-pds">"</span>Schwanhold, Ernst<span class="pl-pds">"</span></span>,
          <span class="pl-ent">"score"</span>: <span class="pl-c1">86.43497</span>,
          <span class="pl-ent">"match"</span>: <span class="pl-c1">true</span>,
          <span class="pl-ent">"type"</span>: [
            {
              <span class="pl-ent">"id"</span>: <span class="pl-s"><span class="pl-pds">"</span>person<span class="pl-pds">"</span></span>,
              <span class="pl-ent">"name"</span>: <span class="pl-s"><span class="pl-pds">"</span>Person<span class="pl-pds">"</span></span>
            }
          ]
        },
        {
          <span class="pl-ent">"id"</span>: <span class="pl-s"><span class="pl-pds">"</span>116362988X<span class="pl-pds">"</span></span>,
          <span class="pl-ent">"name"</span>: <span class="pl-s"><span class="pl-pds">"</span>Schwanhold, Nadine<span class="pl-pds">"</span></span>,
          <span class="pl-ent">"score"</span>: <span class="pl-c1">62.04763</span>,
          <span class="pl-ent">"match"</span>: <span class="pl-c1">false</span>,
          <span class="pl-ent">"type"</span>: [
            {
              <span class="pl-ent">"id"</span>: <span class="pl-s"><span class="pl-pds">"</span>person<span class="pl-pds">"</span></span>,
              <span class="pl-ent">"name"</span>: <span class="pl-s"><span class="pl-pds">"</span>Person<span class="pl-pds">"</span></span>
            }
          ]
        }
      ]
    }
  }</pre></div>
  <div class="markdown-heading" dir="auto"><h3 class="heading-element" dir="auto">Behind the scenes</h3><a id="user-content-behind-the-scenes" class="anchor" aria-label="Permalink: Behind the scenes" href="#behind-the-scenes"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">The reconcile engine works by performing an SQL query against the <code>name_field</code> within the specified database table. Where that table has a full text search index implemented, the search will be performed against that index.</p>
  <p dir="auto">When a full text search index is present on the table, the SQL query takes the form (based on the search query <code>test</code>, note that double quotes are added to facilitate searching - these are not present in the original query):</p>
  <div class="highlight highlight-source-sql notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="select <id_field>, <name_field>
  from <table>
    inner join (
      select "rowid", "rank"
      from <fts_table>
      where <fts_table> MATCH '"test"'
    ) as "a" on <table>."rowid" = a."rowid"
  order by a.rank
  limit 5"><pre><span class="pl-k">select</span> <span class="pl-k"><</span>id_field<span class="pl-k">></span>, <span class="pl-k"><</span>name_field<span class="pl-k">></span>
  <span class="pl-k">from</span> <span class="pl-k"><</span>table<span class="pl-k">></span>
    <span class="pl-k">inner join</span> (
      <span class="pl-k">select</span> <span class="pl-s"><span class="pl-pds">"</span>rowid<span class="pl-pds">"</span></span>, <span class="pl-s"><span class="pl-pds">"</span>rank<span class="pl-pds">"</span></span>
      <span class="pl-k">from</span> <span class="pl-k"><</span>fts_table<span class="pl-k">></span>
      <span class="pl-k">where</span> <span class="pl-k"><</span>fts_table<span class="pl-k">></span> MATCH <span class="pl-s"><span class="pl-pds">'</span>"test"<span class="pl-pds">'</span></span>
    ) <span class="pl-k">as</span> <span class="pl-s"><span class="pl-pds">"</span>a<span class="pl-pds">"</span></span> <span class="pl-k">on</span> <span class="pl-k"><</span>table<span class="pl-k">></span>.<span class="pl-s"><span class="pl-pds">"</span>rowid<span class="pl-pds">"</span></span> <span class="pl-k">=</span> a.<span class="pl-s"><span class="pl-pds">"</span>rowid<span class="pl-pds">"</span></span>
  <span class="pl-k">order by</span> <span class="pl-c1">a</span>.<span class="pl-c1">rank</span>
  <span class="pl-k">limit</span> <span class="pl-c1">5</span></pre></div>
  <p dir="auto">If a full text search index is not present, the query looks like this (note that the wildcard <code>%</code> is added to either side of the query - these are not present in the original query):</p>
  <div class="highlight highlight-source-sql notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="select <id_field>, <name_field>
  from <table>
  where <name_field> like '%test%'
  limit 5"><pre><span class="pl-k">select</span> <span class="pl-k"><</span>id_field<span class="pl-k">></span>, <span class="pl-k"><</span>name_field<span class="pl-k">></span>
  <span class="pl-k">from</span> <span class="pl-k"><</span>table<span class="pl-k">></span>
  <span class="pl-k">where</span> <span class="pl-k"><</span>name_field<span class="pl-k">></span> <span class="pl-k">like</span> <span class="pl-s"><span class="pl-pds">'</span>%test%<span class="pl-pds">'</span></span>
  <span class="pl-k">limit</span> <span class="pl-c1">5</span></pre></div>
  <div class="markdown-heading" dir="auto"><h3 class="heading-element" dir="auto">Extend endpoint</h3><a id="user-content-extend-endpoint" class="anchor" aria-label="Permalink: Extend endpoint" href="#extend-endpoint"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">You can also use the reconciliation API <a href="https://www.w3.org/community/reports/reconciliation/CG-FINAL-specs-0.2-20230410/#data-extension-service" rel="nofollow">Data extension service</a> to find additional properties for a set of entities, given an ID.</p>
  <p dir="auto">Send a GET request to the <code>/<db_name>/<table>/-/reconcile/extend/propose</code> endpoint to find a list of the possible properties you can select. The properties are all the columns in the table (excluding any that have been hidden). An example response would look like:</p>
  <div class="highlight highlight-source-json notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="{
    "limit": 5,
    "type": "Person",
    "properties": [
      {
        "id": "preferredName",
        "name": "preferredName"
      },
      {
        "id": "professionOrOccupation",
        "name": "professionOrOccupation"
      },
      {
        "id": "wikidataId",
        "name": "wikidataId"
      }
    ]
  }"><pre>{
    <span class="pl-ent">"limit"</span>: <span class="pl-c1">5</span>,
    <span class="pl-ent">"type"</span>: <span class="pl-s"><span class="pl-pds">"</span>Person<span class="pl-pds">"</span></span>,
    <span class="pl-ent">"properties"</span>: [
      {
        <span class="pl-ent">"id"</span>: <span class="pl-s"><span class="pl-pds">"</span>preferredName<span class="pl-pds">"</span></span>,
        <span class="pl-ent">"name"</span>: <span class="pl-s"><span class="pl-pds">"</span>preferredName<span class="pl-pds">"</span></span>
      },
      {
        <span class="pl-ent">"id"</span>: <span class="pl-s"><span class="pl-pds">"</span>professionOrOccupation<span class="pl-pds">"</span></span>,
        <span class="pl-ent">"name"</span>: <span class="pl-s"><span class="pl-pds">"</span>professionOrOccupation<span class="pl-pds">"</span></span>
      },
      {
        <span class="pl-ent">"id"</span>: <span class="pl-s"><span class="pl-pds">"</span>wikidataId<span class="pl-pds">"</span></span>,
        <span class="pl-ent">"name"</span>: <span class="pl-s"><span class="pl-pds">"</span>wikidataId<span class="pl-pds">"</span></span>
      }
    ]
  }</pre></div>
  <p dir="auto">Then send a POST request to the <code>/<db_name>/<table>/-/reconcile</code> endpoint with an <code>extend</code> argument. The <code>extend</code> argument should be a JSON object with a set of <code>ids</code> to lookup and <code>properties</code> to return. For example:</p>
  <div class="highlight highlight-source-json notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="{
    "ids": ["10662041X", "1064905412"],
    "properties": [
      {
        "id": "professionOrOccupation"
      },
      {
        "id": "wikidataId"
      }
    ]
  }"><pre>{
    <span class="pl-ent">"ids"</span>: [<span class="pl-s"><span class="pl-pds">"</span>10662041X<span class="pl-pds">"</span></span>, <span class="pl-s"><span class="pl-pds">"</span>1064905412<span class="pl-pds">"</span></span>],
    <span class="pl-ent">"properties"</span>: [
      {
        <span class="pl-ent">"id"</span>: <span class="pl-s"><span class="pl-pds">"</span>professionOrOccupation<span class="pl-pds">"</span></span>
      },
      {
        <span class="pl-ent">"id"</span>: <span class="pl-s"><span class="pl-pds">"</span>wikidataId<span class="pl-pds">"</span></span>
      }
    ]
  }</pre></div>
  <p dir="auto">The endpoint will return a result that looks like:</p>
  <div class="highlight highlight-source-json notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="{
    "meta": [
      {
        "id": "professionOrOccupation",
        "name": "professionOrOccupation"
      },
      {
        "id": "wikidataId",
        "name": "wikidataId"
      }
    ],
    "rows": {
      "10662041X": {
        "professionOrOccupation": [
          {
            "str": "Doctor"
          }
        ],
        "wikidataId": [
          {
            "str": "Q3874347"
          }
        ]
      },
      "1064905412": {
        "professionOrOccupation": [
          {
            "str": "Architect"
          }
        ],
        "wikidataId": [
          {
            "str": "Q3874347"
          }
        ]
      }
    }
  }"><pre>{
    <span class="pl-ent">"meta"</span>: [
      {
        <span class="pl-ent">"id"</span>: <span class="pl-s"><span class="pl-pds">"</span>professionOrOccupation<span class="pl-pds">"</span></span>,
        <span class="pl-ent">"name"</span>: <span class="pl-s"><span class="pl-pds">"</span>professionOrOccupation<span class="pl-pds">"</span></span>
      },
      {
        <span class="pl-ent">"id"</span>: <span class="pl-s"><span class="pl-pds">"</span>wikidataId<span class="pl-pds">"</span></span>,
        <span class="pl-ent">"name"</span>: <span class="pl-s"><span class="pl-pds">"</span>wikidataId<span class="pl-pds">"</span></span>
      }
    ],
    <span class="pl-ent">"rows"</span>: {
      <span class="pl-ent">"10662041X"</span>: {
        <span class="pl-ent">"professionOrOccupation"</span>: [
          {
            <span class="pl-ent">"str"</span>: <span class="pl-s"><span class="pl-pds">"</span>Doctor<span class="pl-pds">"</span></span>
          }
        ],
        <span class="pl-ent">"wikidataId"</span>: [
          {
            <span class="pl-ent">"str"</span>: <span class="pl-s"><span class="pl-pds">"</span>Q3874347<span class="pl-pds">"</span></span>
          }
        ]
      },
      <span class="pl-ent">"1064905412"</span>: {
        <span class="pl-ent">"professionOrOccupation"</span>: [
          {
            <span class="pl-ent">"str"</span>: <span class="pl-s"><span class="pl-pds">"</span>Architect<span class="pl-pds">"</span></span>
          }
        ],
        <span class="pl-ent">"wikidataId"</span>: [
          {
            <span class="pl-ent">"str"</span>: <span class="pl-s"><span class="pl-pds">"</span>Q3874347<span class="pl-pds">"</span></span>
          }
        ]
      }
    }
  }</pre></div>
  <div class="markdown-heading" dir="auto"><h3 class="heading-element" dir="auto">Suggest endpoints</h3><a id="user-content-suggest-endpoints" class="anchor" aria-label="Permalink: Suggest endpoints" href="#suggest-endpoints"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">You can also use the <a href="https://www.w3.org/community/reports/reconciliation/CG-FINAL-specs-0.2-20230410/#suggest-services" rel="nofollow">suggest endpoints</a> to get quick suggestions, for example for an auto-complete dropdown menu. The following endpoints are available:</p>
  <ul dir="auto">
  <li><code>/<db_name>/<table>/-/reconcile/suggest/property</code> - looks up in a list of table columns</li>
  <li><code>/<db_name>/<table>/-/reconcile/suggest/entity</code> - looks up in a list of table rows</li>
  <li><code>/<db_name>/<table>/-/reconcile/suggest/type</code> - not currently implemented</li>
  </ul>
  <p dir="auto">Each endpoint takes a <code>prefix</code> argument which can be used in a GET request. For example, the GET request <code>/<db_name>/<table>/-/reconcile/suggest/entity?prefix=abc</code> will produce a response such as:</p>
  <div class="highlight highlight-source-json notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="{
    "result": [
      {
        "name": "abc company limited",
        "id": "Q123456"
      },
      {
        "name": "abc other company limited",
        "id": "Q123457"
      }
    ]
  }"><pre>{
    <span class="pl-ent">"result"</span>: [
      {
        <span class="pl-ent">"name"</span>: <span class="pl-s"><span class="pl-pds">"</span>abc company limited<span class="pl-pds">"</span></span>,
        <span class="pl-ent">"id"</span>: <span class="pl-s"><span class="pl-pds">"</span>Q123456<span class="pl-pds">"</span></span>
      },
      {
        <span class="pl-ent">"name"</span>: <span class="pl-s"><span class="pl-pds">"</span>abc other company limited<span class="pl-pds">"</span></span>,
        <span class="pl-ent">"id"</span>: <span class="pl-s"><span class="pl-pds">"</span>Q123457<span class="pl-pds">"</span></span>
      }
    ]
  }</pre></div>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Development</h2><a id="user-content-development" class="anchor" aria-label="Permalink: Development" href="#development"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">This plugin uses hatch for build and testing. To set up this plugin locally, first checkout the code.</p>
  <p dir="auto">You'll need to fetch the git submodules for the tests too:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="git submodule init
  git submodule update"><pre class="notranslate"><code>git submodule init
  git submodule update
  </code></pre></div>
  <p dir="auto">To run the tests:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="hatch run test"><pre class="notranslate"><code>hatch run test
  </code></pre></div>
  <p dir="auto">Run tests then report on coverage</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="hatch run cov"><pre class="notranslate"><code>hatch run cov
  </code></pre></div>
  <p dir="auto">Run tests then run a server showing where coverage is missing</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="hatch run cov-html"><pre class="notranslate"><code>hatch run cov-html
  </code></pre></div>
  <div class="markdown-heading" dir="auto"><h3 class="heading-element" dir="auto">Linting/formatting</h3><a id="user-content-lintingformatting" class="anchor" aria-label="Permalink: Linting/formatting" href="#lintingformatting"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">Black and ruff should be run before committing any changes.</p>
  <p dir="auto">To check for any changes needed:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="hatch run lint:style"><pre class="notranslate"><code>hatch run lint:style
  </code></pre></div>
  <p dir="auto">To run any autoformatting possible:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="hatch run lint:fmt"><pre class="notranslate"><code>hatch run lint:fmt
  </code></pre></div>
  <div class="markdown-heading" dir="auto"><h3 class="heading-element" dir="auto">Publish to pypi</h3><a id="user-content-publish-to-pypi" class="anchor" aria-label="Permalink: Publish to pypi" href="#publish-to-pypi"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="hatch build
  hatch publish
  git tag v<VERSION_NUMBER>
  git push origin v<VERSION_NUMBER>"><pre class="notranslate"><code>hatch build
  hatch publish
  git tag v<VERSION_NUMBER>
  git push origin v<VERSION_NUMBER>
  </code></pre></div>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Acknowledgements</h2><a id="user-content-acknowledgements" class="anchor" aria-label="Permalink: Acknowledgements" href="#acknowledgements"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">Thanks for <a href="https://github.com/simonw/">@simonw</a> for developing datasette and the datasette ecosystem.</p>
  <p dir="auto">Other contributions from:</p>
  <ul dir="auto">
  <li><a href="https://github.com/JBPressac/">@JBPressac</a></li>
  <li><a href="https://github.com/nicokant/">@nicokant</a> - implementation of extend service</li>
  </ul>
  </article></div>


