# repos

node_id: MDEwOlJlcG9zaXRvcnkyMzYxMTA3NTk=
name: datasette-auth-existing-cookies
full_name: simonw/datasette-auth-existing-cookies
html_url: https://github.com/simonw/datasette-auth-existing-cookies
description: Datasette plugin that authenticates users based on existing domain cookies
created_at: 2020-01-25T01:20:31Z
updated_at: 2025-03-28T05:21:01Z
pushed_at: 2025-03-28T05:21:36Z
language: Python
license: apache-2.0
topics: ["datasette", "datasette-io", "datasette-plugin"]
visibility: public
default_branch: main
permissions: {"admin": false, "maintain": false, "push": false, "triage": false, "pull": true}
readme:
  # datasette-auth-existing-cookies

  [![PyPI](https://img.shields.io/pypi/v/datasette-auth-existing-cookies.svg)](https://pypi.org/project/datasette-auth-existing-cookies/)
  [![Changelog](https://img.shields.io/github/v/release/simonw/datasette-auth-existing-cookies?include_prereleases&label=changelog)](https://github.com/simonw/datasette-auth-existing-cookies/releases)
  [![Tests](https://github.com/simonw/datasette-auth-existing-cookies/workflows/Test/badge.svg)](https://github.com/simonw/datasette-auth-existing-cookies/actions?query=workflow%3ATest)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/datasette-auth-existing-cookies/blob/master/LICENSE)

  Datasette plugin that authenticates users based on existing domain cookies.

  ## When to use this

  This plugin allows you to build custom authentication for Datasette when you are hosting a Datasette instance on the same domain as another, authenticated website.

  Consider a website on `www.example.com` which supports user authentication.

  You could run Datasette on `data.example.com` in a way that lets it see cookies that were set for the `.example.com` domain.

  Using this plugin, you could build an API endpoint at `www.example.com/user-for-cookies` which returns a JSON object representing the currently signed-in user, based on their cookies.

  The plugin running on `data.example.com` will then make the `actor` available to the rest of Datasette based on the response from that API.

  Read about [Datasette's authentication and permissions system](https://docs.datasette.io/en/stable/authentication.html) for more on how actors and permissions work.

  ## Configuration

  This plugin requires some configuration in the Datasette [metadata.json file](https://datasette.readthedocs.io/en/stable/plugins.html#plugin-configuration).

  The following configuration options are supported:

  - `api_url`: this is the API endpoint that Datasette should call with the user's cookies in order to identify the logged in user.
  - `cookies`: optional. A list of cookie names that should be passed through to the API endpoint - if left blank, the default is to send all cookies.
  - `ttl`: optional. By default Datasette will make a request to the API endpoint for every HTTP request recieved by Datasette itself. A `ttl` value of 5 will cause Datasette to cache the actor associated with the user's cookies for 5 seconds, reducing that API traffic.
  - `headers`: an optional list of other headers to forward to the API endpoint as query string parameters.

  Here is an example that uses all four of these settings:

  ```json
  {
      "plugins": {
          "datasette-auth-existing-cookies": {
              "api_url": "http://www.example.com/user-from-cookies",
              "cookies": ["sessionid"],
              "headers": ["host"],
              "ttl": 10
          }
      }
  }
  ```
  With this configuration any hit to a Datasette hosted at `data.example.com` will result in the following request being made to the `http://www.example.com/user-from-cookies` API endpoint:
  ```
  GET http://www.example.com/user-from-cookies?host=data.example.com
  Cookie: sessionid=abc123
  ```
  That API is expected to return a JSON object representing the current user:

  ```json
  {
      "id": 1,
      "name": "Barry"
  }
  ```
  Since `ttl` is set to 10 that actor will be cached for ten seconds against that exact combination of cookies and headers. When that cache expires another hit will be made to the API.

  When deciding on a TTL value, take into account that users who lose access to the core site - maybe because their session expires, or their account is disabled - will still be able to access the Datasette instance until that cache expires.

readme_html:
  <div id="readme" class="md" data-path="README.md"><article class="markdown-body entry-content container-lg" itemprop="text"><div class="markdown-heading" dir="auto"><h1 class="heading-element" dir="auto">datasette-auth-existing-cookies</h1><a id="user-content-datasette-auth-existing-cookies" class="anchor" aria-label="Permalink: datasette-auth-existing-cookies" href="#datasette-auth-existing-cookies"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto"><a href="https://pypi.org/project/datasette-auth-existing-cookies/" rel="nofollow"><img src="https://img.shields.io/pypi/v/datasette-auth-existing-cookies.svg" alt="PyPI" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/f7d653386c3caaf112dfbf15a6be7783cd3032babb5f3348eada8086b00bd0dd/68747470733a2f2f696d672e736869656c64732e696f2f707970692f762f6461746173657474652d617574682d6578697374696e672d636f6f6b6965732e737667"></a>
  <a href="https://github.com/simonw/datasette-auth-existing-cookies/releases"><img src="https://img.shields.io/github/v/release/simonw/datasette-auth-existing-cookies?include_prereleases&label=changelog" alt="Changelog" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/34ecbc5806e61da22dfa96fbe8c71dd3622751ecb609f527a79fa6f7e981194f/68747470733a2f2f696d672e736869656c64732e696f2f6769746875622f762f72656c656173652f73696d6f6e772f6461746173657474652d617574682d6578697374696e672d636f6f6b6965733f696e636c7564655f70726572656c6561736573266c6162656c3d6368616e67656c6f67"></a>
  <a href="https://github.com/simonw/datasette-auth-existing-cookies/actions?query=workflow%3ATest"><img src="https://github.com/simonw/datasette-auth-existing-cookies/workflows/Test/badge.svg" alt="Tests" style="max-width: 100%;"></a>
  <a href="https://github.com/simonw/datasette-auth-existing-cookies/blob/master/LICENSE"><img src="https://img.shields.io/badge/license-Apache%202.0-blue.svg" alt="License" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/c355f200ea90fddaa407b6eaab303663a669248ea3ca7b1fcf77dbe04ff5f48c/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6c6963656e73652d417061636865253230322e302d626c75652e737667"></a></p>
  <p dir="auto">Datasette plugin that authenticates users based on existing domain cookies.</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">When to use this</h2><a id="user-content-when-to-use-this" class="anchor" aria-label="Permalink: When to use this" href="#when-to-use-this"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">This plugin allows you to build custom authentication for Datasette when you are hosting a Datasette instance on the same domain as another, authenticated website.</p>
  <p dir="auto">Consider a website on <code>www.example.com</code> which supports user authentication.</p>
  <p dir="auto">You could run Datasette on <code>data.example.com</code> in a way that lets it see cookies that were set for the <code>.example.com</code> domain.</p>
  <p dir="auto">Using this plugin, you could build an API endpoint at <code>www.example.com/user-for-cookies</code> which returns a JSON object representing the currently signed-in user, based on their cookies.</p>
  <p dir="auto">The plugin running on <code>data.example.com</code> will then make the <code>actor</code> available to the rest of Datasette based on the response from that API.</p>
  <p dir="auto">Read about <a href="https://docs.datasette.io/en/stable/authentication.html" rel="nofollow">Datasette's authentication and permissions system</a> for more on how actors and permissions work.</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Configuration</h2><a id="user-content-configuration" class="anchor" aria-label="Permalink: Configuration" href="#configuration"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">This plugin requires some configuration in the Datasette <a href="https://datasette.readthedocs.io/en/stable/plugins.html#plugin-configuration" rel="nofollow">metadata.json file</a>.</p>
  <p dir="auto">The following configuration options are supported:</p>
  <ul dir="auto">
  <li><code>api_url</code>: this is the API endpoint that Datasette should call with the user's cookies in order to identify the logged in user.</li>
  <li><code>cookies</code>: optional. A list of cookie names that should be passed through to the API endpoint - if left blank, the default is to send all cookies.</li>
  <li><code>ttl</code>: optional. By default Datasette will make a request to the API endpoint for every HTTP request recieved by Datasette itself. A <code>ttl</code> value of 5 will cause Datasette to cache the actor associated with the user's cookies for 5 seconds, reducing that API traffic.</li>
  <li><code>headers</code>: an optional list of other headers to forward to the API endpoint as query string parameters.</li>
  </ul>
  <p dir="auto">Here is an example that uses all four of these settings:</p>
  <div class="highlight highlight-source-json notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="{
      "plugins": {
          "datasette-auth-existing-cookies": {
              "api_url": "http://www.example.com/user-from-cookies",
              "cookies": ["sessionid"],
              "headers": ["host"],
              "ttl": 10
          }
      }
  }"><pre>{
      <span class="pl-ent">"plugins"</span>: {
          <span class="pl-ent">"datasette-auth-existing-cookies"</span>: {
              <span class="pl-ent">"api_url"</span>: <span class="pl-s"><span class="pl-pds">"</span>http://www.example.com/user-from-cookies<span class="pl-pds">"</span></span>,
              <span class="pl-ent">"cookies"</span>: [<span class="pl-s"><span class="pl-pds">"</span>sessionid<span class="pl-pds">"</span></span>],
              <span class="pl-ent">"headers"</span>: [<span class="pl-s"><span class="pl-pds">"</span>host<span class="pl-pds">"</span></span>],
              <span class="pl-ent">"ttl"</span>: <span class="pl-c1">10</span>
          }
      }
  }</pre></div>
  <p dir="auto">With this configuration any hit to a Datasette hosted at <code>data.example.com</code> will result in the following request being made to the <code>http://www.example.com/user-from-cookies</code> API endpoint:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="GET http://www.example.com/user-from-cookies?host=data.example.com
  Cookie: sessionid=abc123"><pre class="notranslate"><code>GET http://www.example.com/user-from-cookies?host=data.example.com
  Cookie: sessionid=abc123
  </code></pre></div>
  <p dir="auto">That API is expected to return a JSON object representing the current user:</p>
  <div class="highlight highlight-source-json notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="{
      "id": 1,
      "name": "Barry"
  }"><pre>{
      <span class="pl-ent">"id"</span>: <span class="pl-c1">1</span>,
      <span class="pl-ent">"name"</span>: <span class="pl-s"><span class="pl-pds">"</span>Barry<span class="pl-pds">"</span></span>
  }</pre></div>
  <p dir="auto">Since <code>ttl</code> is set to 10 that actor will be cached for ten seconds against that exact combination of cookies and headers. When that cache expires another hit will be made to the API.</p>
  <p dir="auto">When deciding on a TTL value, take into account that users who lose access to the core site - maybe because their session expires, or their account is disabled - will still be able to access the Datasette instance until that cache expires.</p>
  </article></div>


node_id: MDEwOlJlcG9zaXRvcnkyMzY4NjcwMjc=
name: datasette-sentry
full_name: simonw/datasette-sentry
html_url: https://github.com/simonw/datasette-sentry
description: Datasette plugin for configuring Sentry
created_at: 2020-01-28T23:41:27Z
updated_at: 2023-11-22T10:53:15Z
pushed_at: 2023-11-21T19:22:42Z
language: Python
license: apache-2.0
topics: ["datasette", "datasette-io", "datasette-plugin", "sentry"]
visibility: public
default_branch: main
permissions: {"admin": false, "maintain": false, "push": false, "triage": false, "pull": true}
readme:
  # datasette-sentry

  [![PyPI](https://img.shields.io/pypi/v/datasette-sentry.svg)](https://pypi.org/project/datasette-sentry/)
  [![Changelog](https://img.shields.io/github/v/release/simonw/datasette-sentry?include_prereleases&label=changelog)](https://github.com/simonw/datasette-sentry/releases)
  [![Tests](https://github.com/simonw/datasette-sentry/workflows/Test/badge.svg)](https://github.com/simonw/datasette-sentry/actions?query=workflow%3ATest)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/datasette-sentry/blob/main/LICENSE)

  Datasette plugin for configuring Sentry for error reporting
   
  ## Installation
  ```bash
  pip install datasette-sentry
  ```
  ## Usage

  This plugin only takes effect if your `metadata.json` file contains relevant top-level plugin configuration in a `"datasette-sentry"` configuration key.

  You will need a Sentry DSN - see their [Getting Started instructions](https://docs.sentry.io/error-reporting/quickstart/?platform=python).

  Add it to `metadata.json` like this:

  ```json
  {
      "plugins": {
          "datasette-sentry": {
              "dsn": "https://KEY@sentry.io/PROJECTID"
          }
      }
  }
  ```
  Settings in `metadata.json` are visible to anyone who visits the `/-/metadata` URL so this is a good place to take advantage of Datasette's [secret configuration values](https://datasette.readthedocs.io/en/stable/plugins.html#secret-configuration-values), in which case your configuration will look more like this:
  ```json
  {
      "plugins": {
          "datasette-sentry": {
              "dsn": {
                  "$env": "SENTRY_DSN"
              }
          }
      }
  }
  ```
  Then make a `SENTRY_DSN` environment variable available to Datasette.

  ## Configuration

  In addition to the `dsn` setting, you can also configure the Sentry [sample rate](https://docs.sentry.io/platforms/python/configuration/sampling/) by setting  `sample_rate` to a floating point number between 0 and 1.

  For example, to capture 25% of errors you would do this:

  ```json
  {
      "plugins": {
          "datasette-sentry": {
              "dsn": {
                  "$env": "SENTRY_DSN"
              },
              "sample_rate": 0.25
          }
      }
  }
  ```

  ### Performance monitoring

  Sentry [Performance Monitoring](https://docs.sentry.io/product/performance/) records full traces of page for further analysis, in addition to tracking errors.

  You can enable that by adding "enable_tracing" to your plugin configuration:

  ```json
  {
      "plugins": {
          "datasette-sentry": {
              "dsn": {
                  "$env": "SENTRY_DSN"
              },
              "enable_tracing": true
          }
      }
  }
  ```
  The default sample rate if you do this will be `1.0`, meaning every response will be traced. This can get expensive - you can adjust the tracing rate using `traces_sample_rate`. Set that to `0.1` to sample 10% of requests, for example:

  ```json
  {
      "plugins": {
          "datasette-sentry": {
              "dsn": {
                  "$env": "SENTRY_DSN"
              },
              "enable_tracing": true,
              "traces_sample_rate": 0.1
          }
      }
  }
  ```

readme_html:
  <div id="readme" class="md" data-path="README.md"><article class="markdown-body entry-content container-lg" itemprop="text"><div class="markdown-heading" dir="auto"><h1 class="heading-element" dir="auto">datasette-sentry</h1><a id="user-content-datasette-sentry" class="anchor" aria-label="Permalink: datasette-sentry" href="#datasette-sentry"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto"><a href="https://pypi.org/project/datasette-sentry/" rel="nofollow"><img src="https://img.shields.io/pypi/v/datasette-sentry.svg" alt="PyPI" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/85dd6e3f2ddd76e2a2c7e5780026e52ef9783e72c30799a37dc22c926f47c0a6/68747470733a2f2f696d672e736869656c64732e696f2f707970692f762f6461746173657474652d73656e7472792e737667"></a>
  <a href="https://github.com/simonw/datasette-sentry/releases"><img src="https://img.shields.io/github/v/release/simonw/datasette-sentry?include_prereleases&label=changelog" alt="Changelog" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/256fbc187de6b943215d4bb491687fa4b246c64640352ddc5a47332accf16e8a/68747470733a2f2f696d672e736869656c64732e696f2f6769746875622f762f72656c656173652f73696d6f6e772f6461746173657474652d73656e7472793f696e636c7564655f70726572656c6561736573266c6162656c3d6368616e67656c6f67"></a>
  <a href="https://github.com/simonw/datasette-sentry/actions?query=workflow%3ATest"><img src="https://github.com/simonw/datasette-sentry/workflows/Test/badge.svg" alt="Tests" style="max-width: 100%;"></a>
  <a href="https://github.com/simonw/datasette-sentry/blob/main/LICENSE"><img src="https://img.shields.io/badge/license-Apache%202.0-blue.svg" alt="License" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/c355f200ea90fddaa407b6eaab303663a669248ea3ca7b1fcf77dbe04ff5f48c/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6c6963656e73652d417061636865253230322e302d626c75652e737667"></a></p>
  <p dir="auto">Datasette plugin for configuring Sentry for error reporting</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Installation</h2><a id="user-content-installation" class="anchor" aria-label="Permalink: Installation" href="#installation"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="pip install datasette-sentry"><pre>pip install datasette-sentry</pre></div>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Usage</h2><a id="user-content-usage" class="anchor" aria-label="Permalink: Usage" href="#usage"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">This plugin only takes effect if your <code>metadata.json</code> file contains relevant top-level plugin configuration in a <code>"datasette-sentry"</code> configuration key.</p>
  <p dir="auto">You will need a Sentry DSN - see their <a href="https://docs.sentry.io/error-reporting/quickstart/?platform=python" rel="nofollow">Getting Started instructions</a>.</p>
  <p dir="auto">Add it to <code>metadata.json</code> like this:</p>
  <div class="highlight highlight-source-json notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="{
      "plugins": {
          "datasette-sentry": {
              "dsn": "https://KEY@sentry.io/PROJECTID"
          }
      }
  }"><pre>{
      <span class="pl-ent">"plugins"</span>: {
          <span class="pl-ent">"datasette-sentry"</span>: {
              <span class="pl-ent">"dsn"</span>: <span class="pl-s"><span class="pl-pds">"</span>https://KEY@sentry.io/PROJECTID<span class="pl-pds">"</span></span>
          }
      }
  }</pre></div>
  <p dir="auto">Settings in <code>metadata.json</code> are visible to anyone who visits the <code>/-/metadata</code> URL so this is a good place to take advantage of Datasette's <a href="https://datasette.readthedocs.io/en/stable/plugins.html#secret-configuration-values" rel="nofollow">secret configuration values</a>, in which case your configuration will look more like this:</p>
  <div class="highlight highlight-source-json notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="{
      "plugins": {
          "datasette-sentry": {
              "dsn": {
                  "$env": "SENTRY_DSN"
              }
          }
      }
  }"><pre>{
      <span class="pl-ent">"plugins"</span>: {
          <span class="pl-ent">"datasette-sentry"</span>: {
              <span class="pl-ent">"dsn"</span>: {
                  <span class="pl-ent">"$env"</span>: <span class="pl-s"><span class="pl-pds">"</span>SENTRY_DSN<span class="pl-pds">"</span></span>
              }
          }
      }
  }</pre></div>
  <p dir="auto">Then make a <code>SENTRY_DSN</code> environment variable available to Datasette.</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Configuration</h2><a id="user-content-configuration" class="anchor" aria-label="Permalink: Configuration" href="#configuration"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">In addition to the <code>dsn</code> setting, you can also configure the Sentry <a href="https://docs.sentry.io/platforms/python/configuration/sampling/" rel="nofollow">sample rate</a> by setting  <code>sample_rate</code> to a floating point number between 0 and 1.</p>
  <p dir="auto">For example, to capture 25% of errors you would do this:</p>
  <div class="highlight highlight-source-json notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="{
      "plugins": {
          "datasette-sentry": {
              "dsn": {
                  "$env": "SENTRY_DSN"
              },
              "sample_rate": 0.25
          }
      }
  }"><pre>{
      <span class="pl-ent">"plugins"</span>: {
          <span class="pl-ent">"datasette-sentry"</span>: {
              <span class="pl-ent">"dsn"</span>: {
                  <span class="pl-ent">"$env"</span>: <span class="pl-s"><span class="pl-pds">"</span>SENTRY_DSN<span class="pl-pds">"</span></span>
              },
              <span class="pl-ent">"sample_rate"</span>: <span class="pl-c1">0.25</span>
          }
      }
  }</pre></div>
  <div class="markdown-heading" dir="auto"><h3 class="heading-element" dir="auto">Performance monitoring</h3><a id="user-content-performance-monitoring" class="anchor" aria-label="Permalink: Performance monitoring" href="#performance-monitoring"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">Sentry <a href="https://docs.sentry.io/product/performance/" rel="nofollow">Performance Monitoring</a> records full traces of page for further analysis, in addition to tracking errors.</p>
  <p dir="auto">You can enable that by adding "enable_tracing" to your plugin configuration:</p>
  <div class="highlight highlight-source-json notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="{
      "plugins": {
          "datasette-sentry": {
              "dsn": {
                  "$env": "SENTRY_DSN"
              },
              "enable_tracing": true
          }
      }
  }"><pre>{
      <span class="pl-ent">"plugins"</span>: {
          <span class="pl-ent">"datasette-sentry"</span>: {
              <span class="pl-ent">"dsn"</span>: {
                  <span class="pl-ent">"$env"</span>: <span class="pl-s"><span class="pl-pds">"</span>SENTRY_DSN<span class="pl-pds">"</span></span>
              },
              <span class="pl-ent">"enable_tracing"</span>: <span class="pl-c1">true</span>
          }
      }
  }</pre></div>
  <p dir="auto">The default sample rate if you do this will be <code>1.0</code>, meaning every response will be traced. This can get expensive - you can adjust the tracing rate using <code>traces_sample_rate</code>. Set that to <code>0.1</code> to sample 10% of requests, for example:</p>
  <div class="highlight highlight-source-json notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="{
      "plugins": {
          "datasette-sentry": {
              "dsn": {
                  "$env": "SENTRY_DSN"
              },
              "enable_tracing": true,
              "traces_sample_rate": 0.1
          }
      }
  }"><pre>{
      <span class="pl-ent">"plugins"</span>: {
          <span class="pl-ent">"datasette-sentry"</span>: {
              <span class="pl-ent">"dsn"</span>: {
                  <span class="pl-ent">"$env"</span>: <span class="pl-s"><span class="pl-pds">"</span>SENTRY_DSN<span class="pl-pds">"</span></span>
              },
              <span class="pl-ent">"enable_tracing"</span>: <span class="pl-c1">true</span>,
              <span class="pl-ent">"traces_sample_rate"</span>: <span class="pl-c1">0.1</span>
          }
      }
  }</pre></div>
  </article></div>


node_id: MDEwOlJlcG9zaXRvcnkyMzczMjEyNjc=
name: geojson-to-sqlite
full_name: simonw/geojson-to-sqlite
html_url: https://github.com/simonw/geojson-to-sqlite
description: CLI tool for converting GeoJSON files to SQLite (with SpatiaLite)
created_at: 2020-01-30T22:51:05Z
updated_at: 2025-04-02T21:34:36Z
pushed_at: 2022-04-13T23:39:25Z
language: Python
license: apache-2.0
topics: ["datasette-io", "datasette-tool", "geojson", "gis", "sqlite"]
visibility: public
default_branch: main
permissions: {"admin": false, "maintain": false, "push": false, "triage": false, "pull": true}
readme:
  # geojson-to-sqlite

  [![PyPI](https://img.shields.io/pypi/v/geojson-to-sqlite.svg)](https://pypi.org/project/geojson-to-sqlite/)
  [![Changelog](https://img.shields.io/github/v/release/simonw/geojson-to-sqlite?include_prereleases&label=changelog)](https://github.com/simonw/geojson-to-sqlite/releases)
  [![Tests](https://github.com/simonw/geojson-to-sqlite/workflows/Test/badge.svg)](https://github.com/simonw/geojson-to-sqlite/actions?query=workflow%3ATest)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/geojson-to-sqlite/blob/main/LICENSE)

  CLI tool for converting GeoJSON to SQLite (optionally with SpatiaLite)

  [RFC 7946: The GeoJSON Format](https://tools.ietf.org/html/rfc7946)

  ## How to install

      $ pip install geojson-to-sqlite

  ## How to use

  You can run this tool against a GeoJSON file like so:

      $ geojson-to-sqlite my.db features features.geojson

  This will load all of the features from the `features.geojson` file into a table called `features`.

  Each row will have a `geometry` column containing the feature geometry, and columns for each of the keys found in any `properties` attached to those features. (To bundle all properties into a single JSON object, use the `--properties` flag.)

  The table will be created the first time you run the command.

  On subsequent runs you can use the `--alter` option to add any new columns that are missing from the table.

  You can pass more than one GeoJSON file, in which case the contents of all of the files will be inserted into the same table.

  If your features have an `"id"` property it will be used as the primary key for the table. You can also use `--pk=PROPERTY` with the name of a different property to use that as the primary key instead. If you don't want to use the `"id"` as the primary key (maybe it contains duplicate values) you can use `--pk ''` to specify no primary key.

  Specifying a primary key also will allow you to upsert data into the rows instead of insert data into new rows.

  If no primary key is specified, a SQLite `rowid` column will be used.

  You can use `-` as the filename to import from standard input. For example:

      $ curl https://eric.clst.org/assets/wiki/uploads/Stuff/gz_2010_us_040_00_20m.json \
          | geojson-to-sqlite my.db states - --pk GEO_ID

  ## Using with SpatiaLite

  By default, the `geometry` column will contain JSON.

  If you have installed the [SpatiaLite](https://www.gaia-gis.it/fossil/libspatialite/index) module for SQLite you can instead import the geometry into a geospatially indexed column.

  You can do this using the `--spatialite` option, like so:

      $ geojson-to-sqlite my.db features features.geojson --spatialite

  The tool will search for the SpatiaLite module in the following locations:

  - `/usr/lib/x86_64-linux-gnu/mod_spatialite.so`
  - `/usr/local/lib/mod_spatialite.dylib`

  If you have installed the module in another location, you can use the `--spatialite_mod=xxx` option to specify where:

      $ geojson-to-sqlite my.db features features.geojson \
          --spatialite_mod=/usr/lib/mod_spatialite.dylib

  You can create a SpatiaLite spatial index on the `geometry` column using the `--spatial-index` option:

      $ geojson-to-sqlite my.db features features.geojson --spatial-index

  Using this option implies `--spatialite` so you do not need to add that.

  ## Streaming large datasets

  For large datasets, consider using newline-delimited JSON to stream features into the database without loading the entire feature collection into memory.

  For example, to load a day of earthquake reports from USGS:

      $ geojson-to-sqlite quakes.db quakes tests/quakes.ndjson \
        --nl --pk=id --spatialite

  When using newline-delimited JSON, tables will also be created from the first feature, instead of guessing types based on the first 100 features.

  If you want to use a larger subset of your data to guess column types (for example, if some fields are inconsistent) you can use [fiona](https://fiona.readthedocs.io/en/latest/cli.html) to collect features into a single collection.

      $ head tests/quakes.ndjson | fio collect | \
        geojson-to-sqlite quakes.db quakes - --spatialite

  This will take the first 10 lines from `tests/quakes.ndjson`, pass them to `fio collect`, which turns them into a single feature collection, and pass that, in turn, to `geojson-to-sqlite`.

  ## Using this with Datasette

  Databases created using this tool can be explored and published using [Datasette](https://datasette.readthedocs.io/).

  The Datasette documentation includes a section on [how to use it to browse SpatiaLite databases](https://datasette.readthedocs.io/en/stable/spatialite.html).

  The [datasette-leaflet-geojson](https://datasette.io/plugins/datasette-leaflet-geojson) plugin can be used to visualize columns containing GeoJSON geometries on a [Leaflet](https://leafletjs.com/) map.

  If you are using SpatiaLite you will need to output the geometry as GeoJSON in order for that plugin to work. You can do that using the SpaitaLite `AsGeoJSON()` function - something like this:

  ```sql
  select rowid, AsGeoJSON(geometry) from mytable limit 10
  ```

  The [datasette-geojson-map](https://datasette.io/plugins/datasette-geojson-map) is an alternative plugin which will automatically render SpatiaLite geometries as a Leaflet map on the corresponding table page, without needing you to call `AsGeoJSON(geometry)`.

readme_html:
  <div id="readme" class="md" data-path="README.md"><article class="markdown-body entry-content container-lg" itemprop="text"><div class="markdown-heading" dir="auto"><h1 class="heading-element" dir="auto">geojson-to-sqlite</h1><a id="user-content-geojson-to-sqlite" class="anchor" aria-label="Permalink: geojson-to-sqlite" href="#geojson-to-sqlite"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto"><a href="https://pypi.org/project/geojson-to-sqlite/" rel="nofollow"><img src="https://img.shields.io/pypi/v/geojson-to-sqlite.svg" alt="PyPI" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/19bd0d1226fa7f8bed32734c5b88d1ef6464b668145ac03372846c09ce02482d/68747470733a2f2f696d672e736869656c64732e696f2f707970692f762f67656f6a736f6e2d746f2d73716c6974652e737667"></a>
  <a href="https://github.com/simonw/geojson-to-sqlite/releases"><img src="https://img.shields.io/github/v/release/simonw/geojson-to-sqlite?include_prereleases&label=changelog" alt="Changelog" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/e3a454d9c1b4a5b23b3bc6ebeb2e471a88291e77397d18f51fb50a15af735cc4/68747470733a2f2f696d672e736869656c64732e696f2f6769746875622f762f72656c656173652f73696d6f6e772f67656f6a736f6e2d746f2d73716c6974653f696e636c7564655f70726572656c6561736573266c6162656c3d6368616e67656c6f67"></a>
  <a href="https://github.com/simonw/geojson-to-sqlite/actions?query=workflow%3ATest"><img src="https://github.com/simonw/geojson-to-sqlite/workflows/Test/badge.svg" alt="Tests" style="max-width: 100%;"></a>
  <a href="https://github.com/simonw/geojson-to-sqlite/blob/main/LICENSE"><img src="https://img.shields.io/badge/license-Apache%202.0-blue.svg" alt="License" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/c355f200ea90fddaa407b6eaab303663a669248ea3ca7b1fcf77dbe04ff5f48c/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6c6963656e73652d417061636865253230322e302d626c75652e737667"></a></p>
  <p dir="auto">CLI tool for converting GeoJSON to SQLite (optionally with SpatiaLite)</p>
  <p dir="auto"><a href="https://tools.ietf.org/html/rfc7946" rel="nofollow">RFC 7946: The GeoJSON Format</a></p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">How to install</h2><a id="user-content-how-to-install" class="anchor" aria-label="Permalink: How to install" href="#how-to-install"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ pip install geojson-to-sqlite"><pre class="notranslate"><code>$ pip install geojson-to-sqlite
  </code></pre></div>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">How to use</h2><a id="user-content-how-to-use" class="anchor" aria-label="Permalink: How to use" href="#how-to-use"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">You can run this tool against a GeoJSON file like so:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ geojson-to-sqlite my.db features features.geojson"><pre class="notranslate"><code>$ geojson-to-sqlite my.db features features.geojson
  </code></pre></div>
  <p dir="auto">This will load all of the features from the <code>features.geojson</code> file into a table called <code>features</code>.</p>
  <p dir="auto">Each row will have a <code>geometry</code> column containing the feature geometry, and columns for each of the keys found in any <code>properties</code> attached to those features. (To bundle all properties into a single JSON object, use the <code>--properties</code> flag.)</p>
  <p dir="auto">The table will be created the first time you run the command.</p>
  <p dir="auto">On subsequent runs you can use the <code>--alter</code> option to add any new columns that are missing from the table.</p>
  <p dir="auto">You can pass more than one GeoJSON file, in which case the contents of all of the files will be inserted into the same table.</p>
  <p dir="auto">If your features have an <code>"id"</code> property it will be used as the primary key for the table. You can also use <code>--pk=PROPERTY</code> with the name of a different property to use that as the primary key instead. If you don't want to use the <code>"id"</code> as the primary key (maybe it contains duplicate values) you can use <code>--pk ''</code> to specify no primary key.</p>
  <p dir="auto">Specifying a primary key also will allow you to upsert data into the rows instead of insert data into new rows.</p>
  <p dir="auto">If no primary key is specified, a SQLite <code>rowid</code> column will be used.</p>
  <p dir="auto">You can use <code>-</code> as the filename to import from standard input. For example:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ curl https://eric.clst.org/assets/wiki/uploads/Stuff/gz_2010_us_040_00_20m.json \
      | geojson-to-sqlite my.db states - --pk GEO_ID"><pre class="notranslate"><code>$ curl https://eric.clst.org/assets/wiki/uploads/Stuff/gz_2010_us_040_00_20m.json \
      | geojson-to-sqlite my.db states - --pk GEO_ID
  </code></pre></div>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Using with SpatiaLite</h2><a id="user-content-using-with-spatialite" class="anchor" aria-label="Permalink: Using with SpatiaLite" href="#using-with-spatialite"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">By default, the <code>geometry</code> column will contain JSON.</p>
  <p dir="auto">If you have installed the <a href="https://www.gaia-gis.it/fossil/libspatialite/index" rel="nofollow">SpatiaLite</a> module for SQLite you can instead import the geometry into a geospatially indexed column.</p>
  <p dir="auto">You can do this using the <code>--spatialite</code> option, like so:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ geojson-to-sqlite my.db features features.geojson --spatialite"><pre class="notranslate"><code>$ geojson-to-sqlite my.db features features.geojson --spatialite
  </code></pre></div>
  <p dir="auto">The tool will search for the SpatiaLite module in the following locations:</p>
  <ul dir="auto">
  <li><code>/usr/lib/x86_64-linux-gnu/mod_spatialite.so</code></li>
  <li><code>/usr/local/lib/mod_spatialite.dylib</code></li>
  </ul>
  <p dir="auto">If you have installed the module in another location, you can use the <code>--spatialite_mod=xxx</code> option to specify where:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ geojson-to-sqlite my.db features features.geojson \
      --spatialite_mod=/usr/lib/mod_spatialite.dylib"><pre class="notranslate"><code>$ geojson-to-sqlite my.db features features.geojson \
      --spatialite_mod=/usr/lib/mod_spatialite.dylib
  </code></pre></div>
  <p dir="auto">You can create a SpatiaLite spatial index on the <code>geometry</code> column using the <code>--spatial-index</code> option:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ geojson-to-sqlite my.db features features.geojson --spatial-index"><pre class="notranslate"><code>$ geojson-to-sqlite my.db features features.geojson --spatial-index
  </code></pre></div>
  <p dir="auto">Using this option implies <code>--spatialite</code> so you do not need to add that.</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Streaming large datasets</h2><a id="user-content-streaming-large-datasets" class="anchor" aria-label="Permalink: Streaming large datasets" href="#streaming-large-datasets"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">For large datasets, consider using newline-delimited JSON to stream features into the database without loading the entire feature collection into memory.</p>
  <p dir="auto">For example, to load a day of earthquake reports from USGS:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ geojson-to-sqlite quakes.db quakes tests/quakes.ndjson \
    --nl --pk=id --spatialite"><pre class="notranslate"><code>$ geojson-to-sqlite quakes.db quakes tests/quakes.ndjson \
    --nl --pk=id --spatialite
  </code></pre></div>
  <p dir="auto">When using newline-delimited JSON, tables will also be created from the first feature, instead of guessing types based on the first 100 features.</p>
  <p dir="auto">If you want to use a larger subset of your data to guess column types (for example, if some fields are inconsistent) you can use <a href="https://fiona.readthedocs.io/en/latest/cli.html" rel="nofollow">fiona</a> to collect features into a single collection.</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ head tests/quakes.ndjson | fio collect | \
    geojson-to-sqlite quakes.db quakes - --spatialite"><pre class="notranslate"><code>$ head tests/quakes.ndjson | fio collect | \
    geojson-to-sqlite quakes.db quakes - --spatialite
  </code></pre></div>
  <p dir="auto">This will take the first 10 lines from <code>tests/quakes.ndjson</code>, pass them to <code>fio collect</code>, which turns them into a single feature collection, and pass that, in turn, to <code>geojson-to-sqlite</code>.</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Using this with Datasette</h2><a id="user-content-using-this-with-datasette" class="anchor" aria-label="Permalink: Using this with Datasette" href="#using-this-with-datasette"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">Databases created using this tool can be explored and published using <a href="https://datasette.readthedocs.io/" rel="nofollow">Datasette</a>.</p>
  <p dir="auto">The Datasette documentation includes a section on <a href="https://datasette.readthedocs.io/en/stable/spatialite.html" rel="nofollow">how to use it to browse SpatiaLite databases</a>.</p>
  <p dir="auto">The <a href="https://datasette.io/plugins/datasette-leaflet-geojson" rel="nofollow">datasette-leaflet-geojson</a> plugin can be used to visualize columns containing GeoJSON geometries on a <a href="https://leafletjs.com/" rel="nofollow">Leaflet</a> map.</p>
  <p dir="auto">If you are using SpatiaLite you will need to output the geometry as GeoJSON in order for that plugin to work. You can do that using the SpaitaLite <code>AsGeoJSON()</code> function - something like this:</p>
  <div class="highlight highlight-source-sql notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="select rowid, AsGeoJSON(geometry) from mytable limit 10"><pre><span class="pl-k">select</span> rowid, AsGeoJSON(geometry) <span class="pl-k">from</span> mytable <span class="pl-k">limit</span> <span class="pl-c1">10</span></pre></div>
  <p dir="auto">The <a href="https://datasette.io/plugins/datasette-geojson-map" rel="nofollow">datasette-geojson-map</a> is an alternative plugin which will automatically render SpatiaLite geometries as a Leaflet map on the corresponding table page, without needing you to call <code>AsGeoJSON(geometry)</code>.</p>
  </article></div>


node_id: MDEwOlJlcG9zaXRvcnkyMzgzMzk0MTI=
name: datasette-debug-asgi
full_name: simonw/datasette-debug-asgi
html_url: https://github.com/simonw/datasette-debug-asgi
description: Datasette plugin for dumping out the ASGI scope
created_at: 2020-02-05T00:57:09Z
updated_at: 2022-04-27T15:06:27Z
pushed_at: 2021-08-17T23:41:03Z
homepage: https://datasette.io/plugins/datasette-debug-asgi
language: Python
license: apache-2.0
topics: ["asgi", "datasette-io", "datasette-plugin"]
visibility: public
default_branch: main
permissions: {"admin": false, "maintain": false, "push": false, "triage": false, "pull": true}
readme:
  # datasette-debug-asgi

  [![PyPI](https://img.shields.io/pypi/v/datasette-debug-asgi.svg)](https://pypi.org/project/datasette-debug-asgi/)
  [![Changelog](https://img.shields.io/github/v/release/simonw/datasette-debug-asgi?include_prereleases&label=changelog)](https://github.com/simonw/datasette-debug-asgi/releases)
  [![Tests](https://github.com/simonw/datasette-debug-asgi/workflows/Test/badge.svg)](https://github.com/simonw/datasette-debug-asgi/actions?query=workflow%3ATest)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/datasette-debug-asgi/blob/main/LICENSE)

  Datasette plugin for dumping out the ASGI scope.

  Adds a new URL at `/-/asgi-scope` which shows the current ASGI scope. Demo here: https://datasette.io/-/asgi-scope

  ## Installation

      pip install datasette-debug-asgi

  ## Usage

  Visit `/-/asgi-scope` to see debug output showing the ASGI scope.

  You can add query string parameters such as `/-/asgi-scope?q=hello`.

  You can also add extra path components such as `/-/asgi-scope/more/path/here`.

readme_html:
  <div id="readme" class="md" data-path="README.md"><article class="markdown-body entry-content container-lg" itemprop="text"><div class="markdown-heading" dir="auto"><h1 class="heading-element" dir="auto">datasette-debug-asgi</h1><a id="user-content-datasette-debug-asgi" class="anchor" aria-label="Permalink: datasette-debug-asgi" href="#datasette-debug-asgi"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto"><a href="https://pypi.org/project/datasette-debug-asgi/" rel="nofollow"><img src="https://img.shields.io/pypi/v/datasette-debug-asgi.svg" alt="PyPI" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/1da864869b3be4a8fce3e6a396ef5347d64ac447346fd497f8f63cd20515ad68/68747470733a2f2f696d672e736869656c64732e696f2f707970692f762f6461746173657474652d64656275672d617367692e737667"></a>
  <a href="https://github.com/simonw/datasette-debug-asgi/releases"><img src="https://img.shields.io/github/v/release/simonw/datasette-debug-asgi?include_prereleases&label=changelog" alt="Changelog" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/8acfa0032740fda35d5c56af2b9a91dd34b0c27abf5b70990dbf2584e2682220/68747470733a2f2f696d672e736869656c64732e696f2f6769746875622f762f72656c656173652f73696d6f6e772f6461746173657474652d64656275672d617367693f696e636c7564655f70726572656c6561736573266c6162656c3d6368616e67656c6f67"></a>
  <a href="https://github.com/simonw/datasette-debug-asgi/actions?query=workflow%3ATest"><img src="https://github.com/simonw/datasette-debug-asgi/workflows/Test/badge.svg" alt="Tests" style="max-width: 100%;"></a>
  <a href="https://github.com/simonw/datasette-debug-asgi/blob/main/LICENSE"><img src="https://img.shields.io/badge/license-Apache%202.0-blue.svg" alt="License" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/c355f200ea90fddaa407b6eaab303663a669248ea3ca7b1fcf77dbe04ff5f48c/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6c6963656e73652d417061636865253230322e302d626c75652e737667"></a></p>
  <p dir="auto">Datasette plugin for dumping out the ASGI scope.</p>
  <p dir="auto">Adds a new URL at <code>/-/asgi-scope</code> which shows the current ASGI scope. Demo here: <a href="https://datasette.io/-/asgi-scope" rel="nofollow">https://datasette.io/-/asgi-scope</a></p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Installation</h2><a id="user-content-installation" class="anchor" aria-label="Permalink: Installation" href="#installation"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="pip install datasette-debug-asgi"><pre class="notranslate"><code>pip install datasette-debug-asgi
  </code></pre></div>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Usage</h2><a id="user-content-usage" class="anchor" aria-label="Permalink: Usage" href="#usage"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">Visit <code>/-/asgi-scope</code> to see debug output showing the ASGI scope.</p>
  <p dir="auto">You can add query string parameters such as <code>/-/asgi-scope?q=hello</code>.</p>
  <p dir="auto">You can also add extra path components such as <code>/-/asgi-scope/more/path/here</code>.</p>
  </article></div>


node_id: MDEwOlJlcG9zaXRvcnkyNDA4MTU5Mzg=
name: shapefile-to-sqlite
full_name: simonw/shapefile-to-sqlite
html_url: https://github.com/simonw/shapefile-to-sqlite
description: Load shapefiles into a SQLite (optionally SpatiaLite) database
created_at: 2020-02-16T01:55:29Z
updated_at: 2025-02-28T21:20:17Z
pushed_at: 2023-01-08T20:24:46Z
language: Python
license: apache-2.0
topics: ["datasette", "datasette-io", "datasette-tool", "gis", "shapefiles", "spatialite", "sqlite"]
visibility: public
default_branch: main
permissions: {"admin": false, "maintain": false, "push": false, "triage": false, "pull": true}
readme:
  # shapefile-to-sqlite

  [![PyPI](https://img.shields.io/pypi/v/shapefile-to-sqlite.svg)](https://pypi.org/project/shapefile-to-sqlite/)
  [![Changelog](https://img.shields.io/github/v/release/simonw/shapefile-to-sqlite?include_prereleases&label=changelog)](https://github.com/simonw/shapefile-to-sqlite/releases)
  [![Tests](https://github.com/simonw/shapefile-to-sqlite/workflows/Test/badge.svg)](https://github.com/simonw/shapefile-to-sqlite/actions?query=workflow%3ATest)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/shapefile-to-sqlite/blob/main/LICENSE)

  Load shapefiles into a SQLite (optionally SpatiaLite) database.

  Project background: [Things I learned about shapefiles building shapefile-to-sqlite](https://simonwillison.net/2020/Feb/19/shapefile-to-sqlite/)

  ## How to install

      $ pip install shapefile-to-sqlite

  ## How to use

  You can run this tool against a shapefile file like so:

      $ shapefile-to-sqlite my.db features.shp

  This will load the geometries as GeoJSON in a text column.

  ## Using with SpatiaLite

  If you have [SpatiaLite](https://www.gaia-gis.it/fossil/libspatialite/index) available you can load them as SpatiaLite geometries like this:

      $ shapefile-to-sqlite my.db features.shp --spatialite

  The data will be loaded into a table called `features` - based on the name of the shapefile. You can specify an alternative table name using `--table`:

      $ shapefile-to-sqlite my.db features.shp --table=places --spatialite

  The tool will search for the SpatiaLite module in the following locations:

  - `/usr/lib/x86_64-linux-gnu/mod_spatialite.so`
  - `/usr/local/lib/mod_spatialite.dylib`

  If you have installed the module in another location, you can use the `--spatialite_mod=xxx` option to specify where:

      $ shapefile-to-sqlite my.db features.shp \
          --spatialite_mod=/usr/lib/mod_spatialite.dylib

  You can use the `--spatial-index` option to create a spatial index on the `geometry` column:

      $ shapefile-to-sqlite my.db features.shp --spatial-index

  You can omit `--spatialite` if you use either `--spatialite-mod` or `--spatial-index`.

  ## Projections

  By default, this tool will attempt to convert geometries in the shapefile to the WGS 84 projection, for best conformance with the [GeoJSON specification](https://tools.ietf.org/html/rfc7946).

  If you want it to leave the data in whatever projection was used by the shapefile, use the `--crs=keep` option.

  You can convert the data to another output projection by passing it to the `--crs` option. For example, to convert to [EPSG:2227](https://epsg.io/2227) (California zone 3) use `--crs=espg:2227`.

  The full list of formats accepted by the `--crs` option is [documented here](https://pyproj4.github.io/pyproj/stable/api/crs.html#pyproj.crs.CRS.__init__).

  ## Extracting columns

  If your data contains columns with a small number of heavily duplicated values - the names of specific agencies responsible for parcels of land for example - you can extract those columns into separate lookup tables referenced by foreign keys using the `-c` option:

      $ shapefile-to-sqlite my.db features.shp -c agency

  This will create a `agency` table with `id` and `name` columns, and will create the `agency` column in your main table as an integer foreign key reference to that table.

  The `-c` option can be used multiple times.

  [CPAD_2020a_Units](https://calands.datasettes.com/calands/CPAD_2020a_Units) is an example of a table created using the `-c` option.

readme_html:
  <div id="readme" class="md" data-path="README.md"><article class="markdown-body entry-content container-lg" itemprop="text"><div class="markdown-heading" dir="auto"><h1 class="heading-element" dir="auto">shapefile-to-sqlite</h1><a id="user-content-shapefile-to-sqlite" class="anchor" aria-label="Permalink: shapefile-to-sqlite" href="#shapefile-to-sqlite"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto"><a href="https://pypi.org/project/shapefile-to-sqlite/" rel="nofollow"><img src="https://img.shields.io/pypi/v/shapefile-to-sqlite.svg" alt="PyPI" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/09d75bf32f7b6b4efa0e45a226e06f166c1c6a287e8d83220615f56a3a0dc9d8/68747470733a2f2f696d672e736869656c64732e696f2f707970692f762f736861706566696c652d746f2d73716c6974652e737667"></a>
  <a href="https://github.com/simonw/shapefile-to-sqlite/releases"><img src="https://img.shields.io/github/v/release/simonw/shapefile-to-sqlite?include_prereleases&label=changelog" alt="Changelog" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/8cf7c090fac3efc02980d6c5e2e2c8458a7a34ba43672ab8a880d9408b59c759/68747470733a2f2f696d672e736869656c64732e696f2f6769746875622f762f72656c656173652f73696d6f6e772f736861706566696c652d746f2d73716c6974653f696e636c7564655f70726572656c6561736573266c6162656c3d6368616e67656c6f67"></a>
  <a href="https://github.com/simonw/shapefile-to-sqlite/actions?query=workflow%3ATest"><img src="https://github.com/simonw/shapefile-to-sqlite/workflows/Test/badge.svg" alt="Tests" style="max-width: 100%;"></a>
  <a href="https://github.com/simonw/shapefile-to-sqlite/blob/main/LICENSE"><img src="https://img.shields.io/badge/license-Apache%202.0-blue.svg" alt="License" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/c355f200ea90fddaa407b6eaab303663a669248ea3ca7b1fcf77dbe04ff5f48c/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6c6963656e73652d417061636865253230322e302d626c75652e737667"></a></p>
  <p dir="auto">Load shapefiles into a SQLite (optionally SpatiaLite) database.</p>
  <p dir="auto">Project background: <a href="https://simonwillison.net/2020/Feb/19/shapefile-to-sqlite/" rel="nofollow">Things I learned about shapefiles building shapefile-to-sqlite</a></p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">How to install</h2><a id="user-content-how-to-install" class="anchor" aria-label="Permalink: How to install" href="#how-to-install"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ pip install shapefile-to-sqlite"><pre class="notranslate"><code>$ pip install shapefile-to-sqlite
  </code></pre></div>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">How to use</h2><a id="user-content-how-to-use" class="anchor" aria-label="Permalink: How to use" href="#how-to-use"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">You can run this tool against a shapefile file like so:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ shapefile-to-sqlite my.db features.shp"><pre class="notranslate"><code>$ shapefile-to-sqlite my.db features.shp
  </code></pre></div>
  <p dir="auto">This will load the geometries as GeoJSON in a text column.</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Using with SpatiaLite</h2><a id="user-content-using-with-spatialite" class="anchor" aria-label="Permalink: Using with SpatiaLite" href="#using-with-spatialite"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">If you have <a href="https://www.gaia-gis.it/fossil/libspatialite/index" rel="nofollow">SpatiaLite</a> available you can load them as SpatiaLite geometries like this:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ shapefile-to-sqlite my.db features.shp --spatialite"><pre class="notranslate"><code>$ shapefile-to-sqlite my.db features.shp --spatialite
  </code></pre></div>
  <p dir="auto">The data will be loaded into a table called <code>features</code> - based on the name of the shapefile. You can specify an alternative table name using <code>--table</code>:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ shapefile-to-sqlite my.db features.shp --table=places --spatialite"><pre class="notranslate"><code>$ shapefile-to-sqlite my.db features.shp --table=places --spatialite
  </code></pre></div>
  <p dir="auto">The tool will search for the SpatiaLite module in the following locations:</p>
  <ul dir="auto">
  <li><code>/usr/lib/x86_64-linux-gnu/mod_spatialite.so</code></li>
  <li><code>/usr/local/lib/mod_spatialite.dylib</code></li>
  </ul>
  <p dir="auto">If you have installed the module in another location, you can use the <code>--spatialite_mod=xxx</code> option to specify where:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ shapefile-to-sqlite my.db features.shp \
      --spatialite_mod=/usr/lib/mod_spatialite.dylib"><pre class="notranslate"><code>$ shapefile-to-sqlite my.db features.shp \
      --spatialite_mod=/usr/lib/mod_spatialite.dylib
  </code></pre></div>
  <p dir="auto">You can use the <code>--spatial-index</code> option to create a spatial index on the <code>geometry</code> column:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ shapefile-to-sqlite my.db features.shp --spatial-index"><pre class="notranslate"><code>$ shapefile-to-sqlite my.db features.shp --spatial-index
  </code></pre></div>
  <p dir="auto">You can omit <code>--spatialite</code> if you use either <code>--spatialite-mod</code> or <code>--spatial-index</code>.</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Projections</h2><a id="user-content-projections" class="anchor" aria-label="Permalink: Projections" href="#projections"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">By default, this tool will attempt to convert geometries in the shapefile to the WGS 84 projection, for best conformance with the <a href="https://tools.ietf.org/html/rfc7946" rel="nofollow">GeoJSON specification</a>.</p>
  <p dir="auto">If you want it to leave the data in whatever projection was used by the shapefile, use the <code>--crs=keep</code> option.</p>
  <p dir="auto">You can convert the data to another output projection by passing it to the <code>--crs</code> option. For example, to convert to <a href="https://epsg.io/2227" rel="nofollow">EPSG:2227</a> (California zone 3) use <code>--crs=espg:2227</code>.</p>
  <p dir="auto">The full list of formats accepted by the <code>--crs</code> option is <a href="https://pyproj4.github.io/pyproj/stable/api/crs.html#pyproj.crs.CRS.__init__" rel="nofollow">documented here</a>.</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Extracting columns</h2><a id="user-content-extracting-columns" class="anchor" aria-label="Permalink: Extracting columns" href="#extracting-columns"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">If your data contains columns with a small number of heavily duplicated values - the names of specific agencies responsible for parcels of land for example - you can extract those columns into separate lookup tables referenced by foreign keys using the <code>-c</code> option:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ shapefile-to-sqlite my.db features.shp -c agency"><pre class="notranslate"><code>$ shapefile-to-sqlite my.db features.shp -c agency
  </code></pre></div>
  <p dir="auto">This will create a <code>agency</code> table with <code>id</code> and <code>name</code> columns, and will create the <code>agency</code> column in your main table as an integer foreign key reference to that table.</p>
  <p dir="auto">The <code>-c</code> option can be used multiple times.</p>
  <p dir="auto"><a href="https://calands.datasettes.com/calands/CPAD_2020a_Units" rel="nofollow">CPAD_2020a_Units</a> is an example of a table created using the <code>-c</code> option.</p>
  </article></div>


node_id: MDEwOlJlcG9zaXRvcnkyNDIyNjA1ODM=
name: datasette-mask-columns
full_name: simonw/datasette-mask-columns
html_url: https://github.com/simonw/datasette-mask-columns
description: Datasette plugin that masks specified database columns
created_at: 2020-02-22T01:29:16Z
updated_at: 2023-11-17T20:10:32Z
pushed_at: 2024-01-25T05:08:09Z
homepage: https://datasette.io/plugins/datasette-mask-columns
language: Python
license: apache-2.0
topics: ["datasette", "datasette-io", "datasette-plugin"]
visibility: public
default_branch: main
permissions: {"admin": false, "maintain": false, "push": false, "triage": false, "pull": true}
readme:
  # datasette-mask-columns

  [![PyPI](https://img.shields.io/pypi/v/datasette-mask-columns.svg)](https://pypi.org/project/datasette-mask-columns/)
  [![Changelog](https://img.shields.io/github/v/release/simonw/datasette-mask-columns?include_prereleases&label=changelog)](https://github.com/simonw/datasette-mask-columns/releases)
  [![Tests](https://github.com/simonw/datasette-mask-columns/workflows/Test/badge.svg)](https://github.com/simonw/datasette-mask-columns/actions?query=workflow%3ATest)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/datasette-mask-columns/blob/main/LICENSE)

  Datasette plugin that masks specified database columns

  ## Installation

      pip install datasette-mask-columns

  This depends on plugin hook changes in a not-yet released branch of Datasette. See [issue #678](https://github.com/simonw/datasette/issues/678) for details.

  ## Usage

  In your `metadata.json` file add a section like this describing the database and table in which you wish to mask columns:

  ```json
  {
      "databases": {
          "my-database": {
              "plugins": {
                  "datasette-mask-columns": {
                      "users": ["password"]
                  }
              }
          }
      }
  }
  ```
  All SQL queries against the `users` table in `my-database.db` will now return `null` for the `password` column, no matter what value that column actually holds.

  The table page for `users` will display the text `REDACTED` in the masked column. This visual hint will only be available on the table page; it will not display his text for arbitrary queries against the table.

readme_html:
  <div id="readme" class="md" data-path="README.md"><article class="markdown-body entry-content container-lg" itemprop="text"><div class="markdown-heading" dir="auto"><h1 class="heading-element" dir="auto">datasette-mask-columns</h1><a id="user-content-datasette-mask-columns" class="anchor" aria-label="Permalink: datasette-mask-columns" href="#datasette-mask-columns"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto"><a href="https://pypi.org/project/datasette-mask-columns/" rel="nofollow"><img src="https://img.shields.io/pypi/v/datasette-mask-columns.svg" alt="PyPI" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/1f90ee7a4fba91e58b353469cef20f5356b1ba2031b109b3c8bc7e7ca8353ef5/68747470733a2f2f696d672e736869656c64732e696f2f707970692f762f6461746173657474652d6d61736b2d636f6c756d6e732e737667"></a>
  <a href="https://github.com/simonw/datasette-mask-columns/releases"><img src="https://img.shields.io/github/v/release/simonw/datasette-mask-columns?include_prereleases&label=changelog" alt="Changelog" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/e4bf7366246ead76ce2557ff24544b05382210ad0c9259527f1ead1eb06dc087/68747470733a2f2f696d672e736869656c64732e696f2f6769746875622f762f72656c656173652f73696d6f6e772f6461746173657474652d6d61736b2d636f6c756d6e733f696e636c7564655f70726572656c6561736573266c6162656c3d6368616e67656c6f67"></a>
  <a href="https://github.com/simonw/datasette-mask-columns/actions?query=workflow%3ATest"><img src="https://github.com/simonw/datasette-mask-columns/workflows/Test/badge.svg" alt="Tests" style="max-width: 100%;"></a>
  <a href="https://github.com/simonw/datasette-mask-columns/blob/main/LICENSE"><img src="https://img.shields.io/badge/license-Apache%202.0-blue.svg" alt="License" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/c355f200ea90fddaa407b6eaab303663a669248ea3ca7b1fcf77dbe04ff5f48c/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6c6963656e73652d417061636865253230322e302d626c75652e737667"></a></p>
  <p dir="auto">Datasette plugin that masks specified database columns</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Installation</h2><a id="user-content-installation" class="anchor" aria-label="Permalink: Installation" href="#installation"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="pip install datasette-mask-columns"><pre class="notranslate"><code>pip install datasette-mask-columns
  </code></pre></div>
  <p dir="auto">This depends on plugin hook changes in a not-yet released branch of Datasette. See <a href="https://github.com/simonw/datasette/issues/678" data-hovercard-type="issue" data-hovercard-url="/simonw/datasette/issues/678/hovercard">issue #678</a> for details.</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Usage</h2><a id="user-content-usage" class="anchor" aria-label="Permalink: Usage" href="#usage"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">In your <code>metadata.json</code> file add a section like this describing the database and table in which you wish to mask columns:</p>
  <div class="highlight highlight-source-json notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="{
      "databases": {
          "my-database": {
              "plugins": {
                  "datasette-mask-columns": {
                      "users": ["password"]
                  }
              }
          }
      }
  }"><pre>{
      <span class="pl-ent">"databases"</span>: {
          <span class="pl-ent">"my-database"</span>: {
              <span class="pl-ent">"plugins"</span>: {
                  <span class="pl-ent">"datasette-mask-columns"</span>: {
                      <span class="pl-ent">"users"</span>: [<span class="pl-s"><span class="pl-pds">"</span>password<span class="pl-pds">"</span></span>]
                  }
              }
          }
      }
  }</pre></div>
  <p dir="auto">All SQL queries against the <code>users</code> table in <code>my-database.db</code> will now return <code>null</code> for the <code>password</code> column, no matter what value that column actually holds.</p>
  <p dir="auto">The table page for <code>users</code> will display the text <code>REDACTED</code> in the masked column. This visual hint will only be available on the table page; it will not display his text for arbitrary queries against the table.</p>
  </article></div>


node_id: MDEwOlJlcG9zaXRvcnkyNDM3MTA3MzM=
name: datasette-ics
full_name: simonw/datasette-ics
html_url: https://github.com/simonw/datasette-ics
description: Datasette plugin for outputting iCalendar files
created_at: 2020-02-28T08:11:01Z
updated_at: 2025-03-18T21:45:34Z
pushed_at: 2022-07-12T02:08:10Z
homepage: https://datasette.io/plugins/datasette-ics
language: Python
license: apache-2.0
topics: ["datasette", "datasette-io", "datasette-plugin", "icalendar", "ics"]
visibility: public
default_branch: main
permissions: {"admin": false, "maintain": false, "push": false, "triage": false, "pull": true}
readme:
  # datasette-ics

  [![PyPI](https://img.shields.io/pypi/v/datasette-ics.svg)](https://pypi.org/project/datasette-ics/)
  [![Changelog](https://img.shields.io/github/v/release/simonw/datasette-ics?include_prereleases&label=changelog)](https://github.com/simonw/datasette-ics/releases)
  [![Tests](https://github.com/simonw/datasette-ics/workflows/Test/badge.svg)](https://github.com/simonw/datasette-ics/actions?query=workflow%3ATest)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/datasette-ics/blob/main/LICENSE)

  Datasette plugin that adds support for generating [iCalendar .ics files](https://tools.ietf.org/html/rfc5545) with the results of a SQL query.

  ## Installation

  Install this plugin in the same environment as Datasette to enable the `.ics` output extension.

      $ pip install datasette-ics

  ## Usage

  To create an iCalendar file you need to define a custom SQL query that returns a required set of columns:

  * `event_name` - the short name for the event
  * `event_dtstart` - when the event starts

  The following columns are optional:

  * `event_dtend` - when the event ends
  * `event_duration` - the duration of the event (use instead of `dtend`)
  * `event_description` - a longer description of the event
  * `event_uid` - a globally unique identifier for this event
  * `event_tzid` - the timezone for the event, e.g. `America/Chicago`

  A query that returns these columns can then be returned as an ics feed by adding the `.ics` extension.

  ## Demo

  [This SQL query]([https://www.rockybeaches.com/data?sql=with+inner+as+(%0D%0A++select%0D%0A++++datetime%2C%0D%0A++++substr(datetime%2C+0%2C+11)+as+date%2C%0D%0A++++mllw_feet%2C%0D%0A++++lag(mllw_feet)+over+win+as+previous_mllw_feet%2C%0D%0A++++lead(mllw_feet)+over+win+as+next_mllw_feet%0D%0A++from%0D%0A++++tide_predictions%0D%0A++where%0D%0A++++station_id+%3D+%3Astation_id%0D%0A++++and+datetime+%3E%3D+date()%0D%0A++++window+win+as+(%0D%0A++++++order+by%0D%0A++++++++datetime%0D%0A++++)%0D%0A++order+by%0D%0A++++datetime%0D%0A)%2C%0D%0Alowest_tide_per_day+as+(%0D%0A++select%0D%0A++++date%2C%0D%0A++++datetime%2C%0D%0A++++mllw_feet%0D%0A++from%0D%0A++++inner%0D%0A++where%0D%0A++++mllw_feet+%3C%3D+previous_mllw_feet%0D%0A++++and+mllw_feet+%3C%3D+next_mllw_feet%0D%0A)%0D%0Aselect%0D%0A++min(datetime)+as+event_dtstart%2C%0D%0A++%27Low+tide%3A+%27+||+mllw_feet+||+%27+feet%27+as+event_name%2C%0D%0A++%27America%2FLos_Angeles%27+as+event_tzid%0D%0Afrom%0D%0A++lowest_tide_per_day%0D%0Agroup+by%0D%0A++date%0D%0Aorder+by%0D%0A++date&station_id=9414131) calculates the lowest tide per day at Pillar Point in Half Moon Bay, California.

  Since the query returns `event_name`, `event_dtstart` and `event_tzid` columns it produces [this ICS feed](https://www.rockybeaches.com/data.ics?sql=with+inner+as+(%0D%0A++select%0D%0A++++datetime%2C%0D%0A++++substr(datetime%2C+0%2C+11)+as+date%2C%0D%0A++++mllw_feet%2C%0D%0A++++lag(mllw_feet)+over+win+as+previous_mllw_feet%2C%0D%0A++++lead(mllw_feet)+over+win+as+next_mllw_feet%0D%0A++from%0D%0A++++tide_predictions%0D%0A++where%0D%0A++++station_id+%3D+%3Astation_id%0D%0A++++and+datetime+%3E%3D+date()%0D%0A++++window+win+as+(%0D%0A++++++order+by%0D%0A++++++++datetime%0D%0A++++)%0D%0A++order+by%0D%0A++++datetime%0D%0A)%2C%0D%0Alowest_tide_per_day+as+(%0D%0A++select%0D%0A++++date%2C%0D%0A++++datetime%2C%0D%0A++++mllw_feet%0D%0A++from%0D%0A++++inner%0D%0A++where%0D%0A++++mllw_feet+%3C%3D+previous_mllw_feet%0D%0A++++and+mllw_feet+%3C%3D+next_mllw_feet%0D%0A)%0D%0Aselect%0D%0A++min(datetime)+as+event_dtstart%2C%0D%0A++%27Low+tide%3A+%27+||+mllw_feet+||+%27+feet%27+as+event_name%2C%0D%0A++%27America%2FLos_Angeles%27+as+event_tzid%0D%0Afrom%0D%0A++lowest_tide_per_day%0D%0Agroup+by%0D%0A++date%0D%0Aorder+by%0D%0A++date&station_id=9414131). If you subscribe to that in a calendar application such as Apple Calendar you get something that looks like this:

  ![Apple Calendar showing low tides at Pillar Point during a week](https://user-images.githubusercontent.com/9599/173158984-e5ec6bd0-33fc-4fc0-ba9d-17ae674f310a.jpg)

  ## Using a canned query

  Datasette's [canned query mechanism](https://datasette.readthedocs.io/en/stable/sql_queries.html#canned-queries) can be used to configure calendars. If a canned query definition has a `title` that will be used as the title of the calendar.

  Here's an example, defined using a `metadata.yaml` file:

  ```yaml
  databases:
    mydatabase:
      queries:
        calendar:
          title: My Calendar
          sql: |-
            select
              title as event_name,
              start as event_dtstart,
              description as event_description
            from
              events
            order by
              start
            limit
              100
  ```
  This will result in a calendar feed at `http://localhost:8001/mydatabase/calendar.ics`

readme_html:
  <div id="readme" class="md" data-path="README.md"><article class="markdown-body entry-content container-lg" itemprop="text"><div class="markdown-heading" dir="auto"><h1 class="heading-element" dir="auto">datasette-ics</h1><a id="user-content-datasette-ics" class="anchor" aria-label="Permalink: datasette-ics" href="#datasette-ics"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto"><a href="https://pypi.org/project/datasette-ics/" rel="nofollow"><img src="https://img.shields.io/pypi/v/datasette-ics.svg" alt="PyPI" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/96aeb3e52c77d234c30ca1642a8641f59511bd3df71ddeae57f4fe8c95425c20/68747470733a2f2f696d672e736869656c64732e696f2f707970692f762f6461746173657474652d6963732e737667"></a>
  <a href="https://github.com/simonw/datasette-ics/releases"><img src="https://img.shields.io/github/v/release/simonw/datasette-ics?include_prereleases&label=changelog" alt="Changelog" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/2048a81ed637754656ae30f9422e580cedea5a32d960b060760c68233ac448bd/68747470733a2f2f696d672e736869656c64732e696f2f6769746875622f762f72656c656173652f73696d6f6e772f6461746173657474652d6963733f696e636c7564655f70726572656c6561736573266c6162656c3d6368616e67656c6f67"></a>
  <a href="https://github.com/simonw/datasette-ics/actions?query=workflow%3ATest"><img src="https://github.com/simonw/datasette-ics/workflows/Test/badge.svg" alt="Tests" style="max-width: 100%;"></a>
  <a href="https://github.com/simonw/datasette-ics/blob/main/LICENSE"><img src="https://img.shields.io/badge/license-Apache%202.0-blue.svg" alt="License" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/c355f200ea90fddaa407b6eaab303663a669248ea3ca7b1fcf77dbe04ff5f48c/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6c6963656e73652d417061636865253230322e302d626c75652e737667"></a></p>
  <p dir="auto">Datasette plugin that adds support for generating <a href="https://tools.ietf.org/html/rfc5545" rel="nofollow">iCalendar .ics files</a> with the results of a SQL query.</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Installation</h2><a id="user-content-installation" class="anchor" aria-label="Permalink: Installation" href="#installation"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">Install this plugin in the same environment as Datasette to enable the <code>.ics</code> output extension.</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ pip install datasette-ics"><pre class="notranslate"><code>$ pip install datasette-ics
  </code></pre></div>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Usage</h2><a id="user-content-usage" class="anchor" aria-label="Permalink: Usage" href="#usage"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">To create an iCalendar file you need to define a custom SQL query that returns a required set of columns:</p>
  <ul dir="auto">
  <li><code>event_name</code> - the short name for the event</li>
  <li><code>event_dtstart</code> - when the event starts</li>
  </ul>
  <p dir="auto">The following columns are optional:</p>
  <ul dir="auto">
  <li><code>event_dtend</code> - when the event ends</li>
  <li><code>event_duration</code> - the duration of the event (use instead of <code>dtend</code>)</li>
  <li><code>event_description</code> - a longer description of the event</li>
  <li><code>event_uid</code> - a globally unique identifier for this event</li>
  <li><code>event_tzid</code> - the timezone for the event, e.g. <code>America/Chicago</code></li>
  </ul>
  <p dir="auto">A query that returns these columns can then be returned as an ics feed by adding the <code>.ics</code> extension.</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Demo</h2><a id="user-content-demo" class="anchor" aria-label="Permalink: Demo" href="#demo"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">This SQL query calculates the lowest tide per day at Pillar Point in Half Moon Bay, California.</p>
  <p dir="auto">Since the query returns <code>event_name</code>, <code>event_dtstart</code> and <code>event_tzid</code> columns it produces <a href="https://www.rockybeaches.com/data.ics?sql=with+inner+as+(%0D%0A++select%0D%0A++++datetime%2C%0D%0A++++substr(datetime%2C+0%2C+11)+as+date%2C%0D%0A++++mllw_feet%2C%0D%0A++++lag(mllw_feet)+over+win+as+previous_mllw_feet%2C%0D%0A++++lead(mllw_feet)+over+win+as+next_mllw_feet%0D%0A++from%0D%0A++++tide_predictions%0D%0A++where%0D%0A++++station_id+%3D+%3Astation_id%0D%0A++++and+datetime+%3E%3D+date()%0D%0A++++window+win+as+(%0D%0A++++++order+by%0D%0A++++++++datetime%0D%0A++++)%0D%0A++order+by%0D%0A++++datetime%0D%0A)%2C%0D%0Alowest_tide_per_day+as+(%0D%0A++select%0D%0A++++date%2C%0D%0A++++datetime%2C%0D%0A++++mllw_feet%0D%0A++from%0D%0A++++inner%0D%0A++where%0D%0A++++mllw_feet+%3C%3D+previous_mllw_feet%0D%0A++++and+mllw_feet+%3C%3D+next_mllw_feet%0D%0A)%0D%0Aselect%0D%0A++min(datetime)+as+event_dtstart%2C%0D%0A++%27Low+tide%3A+%27+%7C%7C+mllw_feet+%7C%7C+%27+feet%27+as+event_name%2C%0D%0A++%27America%2FLos_Angeles%27+as+event_tzid%0D%0Afrom%0D%0A++lowest_tide_per_day%0D%0Agroup+by%0D%0A++date%0D%0Aorder+by%0D%0A++date&station_id=9414131" rel="nofollow">this ICS feed</a>. If you subscribe to that in a calendar application such as Apple Calendar you get something that looks like this:</p>
  <p dir="auto"><a target="_blank" rel="noopener noreferrer nofollow" href="https://user-images.githubusercontent.com/9599/173158984-e5ec6bd0-33fc-4fc0-ba9d-17ae674f310a.jpg"><img src="https://user-images.githubusercontent.com/9599/173158984-e5ec6bd0-33fc-4fc0-ba9d-17ae674f310a.jpg" alt="Apple Calendar showing low tides at Pillar Point during a week" style="max-width: 100%;"></a></p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Using a canned query</h2><a id="user-content-using-a-canned-query" class="anchor" aria-label="Permalink: Using a canned query" href="#using-a-canned-query"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">Datasette's <a href="https://datasette.readthedocs.io/en/stable/sql_queries.html#canned-queries" rel="nofollow">canned query mechanism</a> can be used to configure calendars. If a canned query definition has a <code>title</code> that will be used as the title of the calendar.</p>
  <p dir="auto">Here's an example, defined using a <code>metadata.yaml</code> file:</p>
  <div class="highlight highlight-source-yaml notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="databases:
    mydatabase:
      queries:
        calendar:
          title: My Calendar
          sql: |-
            select
              title as event_name,
              start as event_dtstart,
              description as event_description
            from
              events
            order by
              start
            limit
              100"><pre><span class="pl-ent">databases</span>:
    <span class="pl-ent">mydatabase</span>:
      <span class="pl-ent">queries</span>:
        <span class="pl-ent">calendar</span>:
          <span class="pl-ent">title</span>: <span class="pl-s">My Calendar</span>
          <span class="pl-ent">sql</span>: <span class="pl-s">|-</span>
  <span class="pl-s">          select</span>
  <span class="pl-s">            title as event_name,</span>
  <span class="pl-s">            start as event_dtstart,</span>
  <span class="pl-s">            description as event_description</span>
  <span class="pl-s">          from</span>
  <span class="pl-s">            events</span>
  <span class="pl-s">          order by</span>
  <span class="pl-s">            start</span>
  <span class="pl-s">          limit</span>
  <span class="pl-s">            100</span></pre></div>
  <p dir="auto">This will result in a calendar feed at <code>http://localhost:8001/mydatabase/calendar.ics</code></p>
  </article></div>


node_id: MDEwOlJlcG9zaXRvcnkyNDM4ODcwMzY=
name: datasette-configure-fts
full_name: simonw/datasette-configure-fts
html_url: https://github.com/simonw/datasette-configure-fts
description: Datasette plugin for enabling full-text search against selected table columns
created_at: 2020-02-29T01:50:57Z
updated_at: 2025-12-23T02:21:01Z
pushed_at: 2025-11-10T19:38:47Z
language: Python
license: apache-2.0
topics: ["datasette", "datasette-io", "datasette-plugin"]
visibility: public
default_branch: main
permissions: {"admin": false, "maintain": false, "push": false, "triage": false, "pull": false}
pull_request_creation_policy: all
readme:
  # datasette-configure-fts

  [![PyPI](https://img.shields.io/pypi/v/datasette-configure-fts.svg)](https://pypi.org/project/datasette-configure-fts/)
  [![Changelog](https://img.shields.io/github/v/release/simonw/datasette-configure-fts?include_prereleases&label=changelog)](https://github.com/simonw/datasette-configure-fts/releases)
  [![Tests](https://github.com/simonw/datasette-configure-fts/workflows/Test/badge.svg)](https://github.com/simonw/datasette-configure-fts/actions?query=workflow%3ATest)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/datasette-configure-fts/blob/main/LICENSE)

  Datasette plugin for enabling full-text search against selected table columns

  ## Installation

  Install this plugin in the same environment as Datasette.

      $ datasette install datasette-configure-fts

  ## Usage

  Having installed the plugin, visit `/-/configure-fts` on your Datasette instance to configure FTS for tables on attached writable databases.

  Any time you have permission to configure FTS for a table a menu item will appear in the table actions menu on the table page.

  By default only [the root actor](https://datasette.readthedocs.io/en/stable/authentication.html#using-the-root-actor) can access the page - so you'll need to run Datasette with the `--root` option and click on the link shown in the terminal to sign in and access the page.

  The `configure-fts` permission governs access. You can use permission plugins such as [datasette-permissions-sql](https://github.com/simonw/datasette-permissions-sql) to grant additional access to the write interface.

readme_html:
  <div id="readme" class="md" data-path="README.md"><article class="markdown-body entry-content container-lg" itemprop="text"><div class="markdown-heading" dir="auto"><h1 class="heading-element" dir="auto">datasette-configure-fts</h1><a id="user-content-datasette-configure-fts" class="anchor" aria-label="Permalink: datasette-configure-fts" href="#datasette-configure-fts"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto"><a href="https://pypi.org/project/datasette-configure-fts/" rel="nofollow"><img src="https://img.shields.io/pypi/v/datasette-configure-fts.svg" alt="PyPI" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/e88ada7a8d31f9f07ac5895d3418a5f547458a4a8385b7d2e0497574b17cf6fc/68747470733a2f2f696d672e736869656c64732e696f2f707970692f762f6461746173657474652d636f6e6669677572652d6674732e737667"></a>
  <a href="https://github.com/simonw/datasette-configure-fts/releases"><img src="https://img.shields.io/github/v/release/simonw/datasette-configure-fts?include_prereleases&label=changelog" alt="Changelog" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/705047c55c40cd9ce6724ffe1364234055f21d14aaa407a7446b557f0677270f/68747470733a2f2f696d672e736869656c64732e696f2f6769746875622f762f72656c656173652f73696d6f6e772f6461746173657474652d636f6e6669677572652d6674733f696e636c7564655f70726572656c6561736573266c6162656c3d6368616e67656c6f67"></a>
  <a href="https://github.com/simonw/datasette-configure-fts/actions?query=workflow%3ATest"><img src="https://github.com/simonw/datasette-configure-fts/workflows/Test/badge.svg" alt="Tests" style="max-width: 100%;"></a>
  <a href="https://github.com/simonw/datasette-configure-fts/blob/main/LICENSE"><img src="https://img.shields.io/badge/license-Apache%202.0-blue.svg" alt="License" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/c355f200ea90fddaa407b6eaab303663a669248ea3ca7b1fcf77dbe04ff5f48c/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6c6963656e73652d417061636865253230322e302d626c75652e737667"></a></p>
  <p dir="auto">Datasette plugin for enabling full-text search against selected table columns</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Installation</h2><a id="user-content-installation" class="anchor" aria-label="Permalink: Installation" href="#installation"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">Install this plugin in the same environment as Datasette.</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ datasette install datasette-configure-fts"><pre class="notranslate"><code>$ datasette install datasette-configure-fts
  </code></pre></div>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Usage</h2><a id="user-content-usage" class="anchor" aria-label="Permalink: Usage" href="#usage"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">Having installed the plugin, visit <code>/-/configure-fts</code> on your Datasette instance to configure FTS for tables on attached writable databases.</p>
  <p dir="auto">Any time you have permission to configure FTS for a table a menu item will appear in the table actions menu on the table page.</p>
  <p dir="auto">By default only <a href="https://datasette.readthedocs.io/en/stable/authentication.html#using-the-root-actor" rel="nofollow">the root actor</a> can access the page - so you'll need to run Datasette with the <code>--root</code> option and click on the link shown in the terminal to sign in and access the page.</p>
  <p dir="auto">The <code>configure-fts</code> permission governs access. You can use permission plugins such as <a href="https://github.com/simonw/datasette-permissions-sql">datasette-permissions-sql</a> to grant additional access to the write interface.</p>
  </article></div>


node_id: MDEwOlJlcG9zaXRvcnkyNDU2NzA2NzA=
name: fec-to-sqlite
full_name: simonw/fec-to-sqlite
html_url: https://github.com/simonw/fec-to-sqlite
description: Save FEC campaign finance data to a SQLite database
created_at: 2020-03-07T16:52:49Z
updated_at: 2025-04-02T23:01:15Z
pushed_at: 2023-08-10T17:00:47Z
language: Python
license: apache-2.0
topics: ["datasette", "datasette-io", "datasette-tool", "fec", "sqlite"]
visibility: public
default_branch: master
permissions: {"admin": false, "maintain": false, "push": false, "triage": false, "pull": true}
readme:
  # fec-to-sqlite

  [![PyPI](https://img.shields.io/pypi/v/fec-to-sqlite.svg)](https://pypi.org/project/fec-to-sqlite/)
  [![CircleCI](https://circleci.com/gh/simonw/fec-to-sqlite.svg?style=svg)](https://circleci.com/gh/simonw/fec-to-sqlite)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/fec-to-sqlite/blob/master/LICENSE)

  Create a SQLite database using FEC campaign contributions data.

  This tool builds on [fecfile](https://github.com/esonderegger/fecfile) by Evan Sonderegger.

  ## How to install

      $ pip install fec-to-sqlite

  ## Usage

      $ fec-to-sqlite filings filings.db 1146148

  This fetches the filing with ID `1146148` and stores it in tables in a SQLite database called `filings.db`. It will create any tables it needs.

  You can pass more than one filing ID, separated by spaces.

readme_html:
  <div id="readme" class="md" data-path="README.md"><article class="markdown-body entry-content container-lg" itemprop="text"><div class="markdown-heading" dir="auto"><h1 class="heading-element" dir="auto">fec-to-sqlite</h1><a id="user-content-fec-to-sqlite" class="anchor" aria-label="Permalink: fec-to-sqlite" href="#fec-to-sqlite"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto"><a href="https://pypi.org/project/fec-to-sqlite/" rel="nofollow"><img src="https://img.shields.io/pypi/v/fec-to-sqlite.svg" alt="PyPI" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/7390fb646c61e4a5f1b05ea9a2873b8e326f3fa82a5c951d422db8d3a4493de6/68747470733a2f2f696d672e736869656c64732e696f2f707970692f762f6665632d746f2d73716c6974652e737667"></a>
  <a href="https://circleci.com/gh/simonw/fec-to-sqlite" rel="nofollow"><img src="https://circleci.com/gh/simonw/fec-to-sqlite.svg?style=svg" alt="CircleCI" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/d1cc59cf4d5ff64e3942f4857729f9575ab5ec6787182b45ea0bb439e03a32cb/68747470733a2f2f636972636c6563692e636f6d2f67682f73696d6f6e772f6665632d746f2d73716c6974652e7376673f7374796c653d737667"></a>
  <a href="https://github.com/simonw/fec-to-sqlite/blob/master/LICENSE"><img src="https://img.shields.io/badge/license-Apache%202.0-blue.svg" alt="License" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/c355f200ea90fddaa407b6eaab303663a669248ea3ca7b1fcf77dbe04ff5f48c/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6c6963656e73652d417061636865253230322e302d626c75652e737667"></a></p>
  <p dir="auto">Create a SQLite database using FEC campaign contributions data.</p>
  <p dir="auto">This tool builds on <a href="https://github.com/esonderegger/fecfile">fecfile</a> by Evan Sonderegger.</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">How to install</h2><a id="user-content-how-to-install" class="anchor" aria-label="Permalink: How to install" href="#how-to-install"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ pip install fec-to-sqlite"><pre class="notranslate"><code>$ pip install fec-to-sqlite
  </code></pre></div>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Usage</h2><a id="user-content-usage" class="anchor" aria-label="Permalink: Usage" href="#usage"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ fec-to-sqlite filings filings.db 1146148"><pre class="notranslate"><code>$ fec-to-sqlite filings filings.db 1146148
  </code></pre></div>
  <p dir="auto">This fetches the filing with ID <code>1146148</code> and stores it in tables in a SQLite database called <code>filings.db</code>. It will create any tables it needs.</p>
  <p dir="auto">You can pass more than one filing ID, separated by spaces.</p>
  </article></div>


node_id: MDEwOlJlcG9zaXRvcnkyNDU4NTY3MzE=
name: datasette-search-all
full_name: simonw/datasette-search-all
html_url: https://github.com/simonw/datasette-search-all
description: Datasette plugin for searching all searchable tables at once
created_at: 2020-03-08T17:21:54Z
updated_at: 2025-11-04T08:26:00Z
pushed_at: 2025-11-03T23:47:28Z
language: Python
license: apache-2.0
topics: ["datasette", "datasette-io", "datasette-plugin", "search"]
visibility: public
default_branch: main
permissions: {"admin": false, "maintain": false, "push": false, "triage": false, "pull": false}
readme:
  # datasette-search-all

  [![PyPI](https://img.shields.io/pypi/v/datasette-search-all.svg)](https://pypi.org/project/datasette-search-all/)
  [![Changelog](https://img.shields.io/github/v/release/simonw/datasette-search-all?include_prereleases&label=changelog)](https://github.com/simonw/datasette-search-all/releases)
  [![Tests](https://github.com/simonw/datasette-search-all/workflows/Test/badge.svg)](https://github.com/simonw/datasette-search-all/actions?query=workflow%3ATest)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/datasette-search-all/blob/main/LICENSE)

  Datasette plugin for searching all searchable tables at once.

  ## Installation

  Install the plugin in the same Python environment as Datasette:

      pip install datasette-search-all

  ## Background

  See [datasette-search-all: a new plugin for searching multiple Datasette tables at once](https://simonwillison.net/2020/Mar/9/datasette-search-all/) for background on this project. You can try the plugin out at https://fara.datasettes.com/

  ## Usage

  This plugin only works if at least one of the tables connected to your Datasette instance has been configured for SQLite's full-text search.

  The [Datasette search documentation](https://docs.datasette.io/en/stable/full_text_search.html) includes details on how to enable full-text search for a table.

  You can also use the following tools:

  * [sqlite-utils](https://sqlite-utils.datasette.io/en/stable/cli.html#configuring-full-text-search) includes a command-line tool for enabling full-text search.
  * [datasette-enable-fts](https://github.com/simonw/datasette-enable-fts) is a Datasette plugin that adds a web interface for enabling search for specific columns.

  If the plugin detects at least one searchable table it will add a search form to the homepage.

  You can also navigate to `/-/search` on your Datasette instance to use the search interface directly.

  ## Screenshot

  ![Animated screenshot showing the plugin in action](https://raw.githubusercontent.com/simonw/datasette-search-all/main/animated-screenshot.gif)

  ## Development

  To set up this plugin locally, first checkout the code. Then create a new virtual environment:
  ```bash
  cd datasette-search-all
  python -m venv venv
  source venv/bin/activate
  ```
  Or if you are using `pipenv`:
  ```bash
  pipenv shell
  ```
  Now install the dependencies and tests:
  ```bash
  pip install -e '.[test]'
  ```
  To run the tests:
  ```bash
  pytest
  ```
  To run the browser automation tests:
  ```bash
  pip install -e '.[test,playwright]'
  pytest
  ```

readme_html:
  <div id="readme" class="md" data-path="README.md"><article class="markdown-body entry-content container-lg" itemprop="text"><div class="markdown-heading" dir="auto"><h1 class="heading-element" dir="auto">datasette-search-all</h1><a id="user-content-datasette-search-all" class="anchor" aria-label="Permalink: datasette-search-all" href="#datasette-search-all"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto"><a href="https://pypi.org/project/datasette-search-all/" rel="nofollow"><img src="https://img.shields.io/pypi/v/datasette-search-all.svg" alt="PyPI" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/ae4863b77edbc6822c9276569dcdfe32812f33e0c028e4727e7a4407f984fa10/68747470733a2f2f696d672e736869656c64732e696f2f707970692f762f6461746173657474652d7365617263682d616c6c2e737667"></a>
  <a href="https://github.com/simonw/datasette-search-all/releases"><img src="https://img.shields.io/github/v/release/simonw/datasette-search-all?include_prereleases&label=changelog" alt="Changelog" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/76900da854564263a5298d6e449e5434835be9e323938c45b3d37d318bfe97f3/68747470733a2f2f696d672e736869656c64732e696f2f6769746875622f762f72656c656173652f73696d6f6e772f6461746173657474652d7365617263682d616c6c3f696e636c7564655f70726572656c6561736573266c6162656c3d6368616e67656c6f67"></a>
  <a href="https://github.com/simonw/datasette-search-all/actions?query=workflow%3ATest"><img src="https://github.com/simonw/datasette-search-all/workflows/Test/badge.svg" alt="Tests" style="max-width: 100%;"></a>
  <a href="https://github.com/simonw/datasette-search-all/blob/main/LICENSE"><img src="https://img.shields.io/badge/license-Apache%202.0-blue.svg" alt="License" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/c355f200ea90fddaa407b6eaab303663a669248ea3ca7b1fcf77dbe04ff5f48c/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6c6963656e73652d417061636865253230322e302d626c75652e737667"></a></p>
  <p dir="auto">Datasette plugin for searching all searchable tables at once.</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Installation</h2><a id="user-content-installation" class="anchor" aria-label="Permalink: Installation" href="#installation"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">Install the plugin in the same Python environment as Datasette:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="pip install datasette-search-all"><pre class="notranslate"><code>pip install datasette-search-all
  </code></pre></div>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Background</h2><a id="user-content-background" class="anchor" aria-label="Permalink: Background" href="#background"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">See <a href="https://simonwillison.net/2020/Mar/9/datasette-search-all/" rel="nofollow">datasette-search-all: a new plugin for searching multiple Datasette tables at once</a> for background on this project. You can try the plugin out at <a href="https://fara.datasettes.com/" rel="nofollow">https://fara.datasettes.com/</a></p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Usage</h2><a id="user-content-usage" class="anchor" aria-label="Permalink: Usage" href="#usage"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">This plugin only works if at least one of the tables connected to your Datasette instance has been configured for SQLite's full-text search.</p>
  <p dir="auto">The <a href="https://docs.datasette.io/en/stable/full_text_search.html" rel="nofollow">Datasette search documentation</a> includes details on how to enable full-text search for a table.</p>
  <p dir="auto">You can also use the following tools:</p>
  <ul dir="auto">
  <li><a href="https://sqlite-utils.datasette.io/en/stable/cli.html#configuring-full-text-search" rel="nofollow">sqlite-utils</a> includes a command-line tool for enabling full-text search.</li>
  <li><a href="https://github.com/simonw/datasette-enable-fts">datasette-enable-fts</a> is a Datasette plugin that adds a web interface for enabling search for specific columns.</li>
  </ul>
  <p dir="auto">If the plugin detects at least one searchable table it will add a search form to the homepage.</p>
  <p dir="auto">You can also navigate to <code>/-/search</code> on your Datasette instance to use the search interface directly.</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Screenshot</h2><a id="user-content-screenshot" class="anchor" aria-label="Permalink: Screenshot" href="#screenshot"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto"><a target="_blank" rel="noopener noreferrer nofollow" href="https://raw.githubusercontent.com/simonw/datasette-search-all/main/animated-screenshot.gif"><img src="https://raw.githubusercontent.com/simonw/datasette-search-all/main/animated-screenshot.gif" alt="Animated screenshot showing the plugin in action" data-animated-image="" style="max-width: 100%;"></a></p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Development</h2><a id="user-content-development" class="anchor" aria-label="Permalink: Development" href="#development"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">To set up this plugin locally, first checkout the code. Then create a new virtual environment:</p>
  <div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="cd datasette-search-all
  python -m venv venv
  source venv/bin/activate"><pre><span class="pl-c1">cd</span> datasette-search-all
  python -m venv venv
  <span class="pl-c1">source</span> venv/bin/activate</pre></div>
  <p dir="auto">Or if you are using <code>pipenv</code>:</p>
  <div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="pipenv shell"><pre>pipenv shell</pre></div>
  <p dir="auto">Now install the dependencies and tests:</p>
  <div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="pip install -e '.[test]'"><pre>pip install -e <span class="pl-s"><span class="pl-pds">'</span>.[test]<span class="pl-pds">'</span></span></pre></div>
  <p dir="auto">To run the tests:</p>
  <div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="pytest"><pre>pytest</pre></div>
  <p dir="auto">To run the browser automation tests:</p>
  <div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="pip install -e '.[test,playwright]'
  pytest"><pre>pip install -e <span class="pl-s"><span class="pl-pds">'</span>.[test,playwright]<span class="pl-pds">'</span></span>
  pytest</pre></div>
  </article></div>


node_id: MDEwOlJlcG9zaXRvcnkyNDYxMDg1NjE=
name: datasette-column-inspect
full_name: simonw/datasette-column-inspect
html_url: https://github.com/simonw/datasette-column-inspect
description: Experimental plugin that adds a column inspector
created_at: 2020-03-09T18:11:00Z
updated_at: 2023-11-16T19:53:10Z
pushed_at: 2020-12-09T21:47:38Z
language: HTML
license: apache-2.0
topics: ["datasette", "datasette-io", "datasette-plugin"]
visibility: public
default_branch: main
permissions: {"admin": false, "maintain": false, "push": false, "triage": false, "pull": true}
readme:
  # datasette-column-inspect

  [![PyPI](https://img.shields.io/pypi/v/datasette-column-inspect.svg)](https://pypi.org/project/datasette-column-inspect/)
  [![Changelog](https://img.shields.io/github/v/release/simonw/datasette-column-inspect?include_prereleases&label=changelog)](https://github.com/simonw/datasette-column-inspect/releases)
  [![Tests](https://github.com/simonw/datasette-column-inspect/workflows/Test/badge.svg)](https://github.com/simonw/datasette-column-inspect/actions?query=workflow%3ATest)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/datasette-column-inspect/blob/main/LICENSE)

  Highly experimental Datasette plugin for inspecting columns.

  ## Installation

  Install this plugin in the same environment as Datasette.

      $ pip install datasette-column-inspect

  ## Usage

  This plugin adds an icon to each column on the table page which opens an inspection side panel.

readme_html:
  <div id="readme" class="md" data-path="README.md"><article class="markdown-body entry-content container-lg" itemprop="text"><div class="markdown-heading" dir="auto"><h1 class="heading-element" dir="auto">datasette-column-inspect</h1><a id="user-content-datasette-column-inspect" class="anchor" aria-label="Permalink: datasette-column-inspect" href="#datasette-column-inspect"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto"><a href="https://pypi.org/project/datasette-column-inspect/" rel="nofollow"><img src="https://img.shields.io/pypi/v/datasette-column-inspect.svg" alt="PyPI" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/019a904bbd8ad78253d6bbfba5da9b173c6260da48ee8ca1d005e7f7d16a16fa/68747470733a2f2f696d672e736869656c64732e696f2f707970692f762f6461746173657474652d636f6c756d6e2d696e73706563742e737667"></a>
  <a href="https://github.com/simonw/datasette-column-inspect/releases"><img src="https://img.shields.io/github/v/release/simonw/datasette-column-inspect?include_prereleases&label=changelog" alt="Changelog" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/9cc04aa2f094f1f05f3a7575ccdeacce783fe0a9e4ca90d10c5e4421a7e2aa6c/68747470733a2f2f696d672e736869656c64732e696f2f6769746875622f762f72656c656173652f73696d6f6e772f6461746173657474652d636f6c756d6e2d696e73706563743f696e636c7564655f70726572656c6561736573266c6162656c3d6368616e67656c6f67"></a>
  <a href="https://github.com/simonw/datasette-column-inspect/actions?query=workflow%3ATest"><img src="https://github.com/simonw/datasette-column-inspect/workflows/Test/badge.svg" alt="Tests" style="max-width: 100%;"></a>
  <a href="https://github.com/simonw/datasette-column-inspect/blob/main/LICENSE"><img src="https://img.shields.io/badge/license-Apache%202.0-blue.svg" alt="License" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/c355f200ea90fddaa407b6eaab303663a669248ea3ca7b1fcf77dbe04ff5f48c/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6c6963656e73652d417061636865253230322e302d626c75652e737667"></a></p>
  <p dir="auto">Highly experimental Datasette plugin for inspecting columns.</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Installation</h2><a id="user-content-installation" class="anchor" aria-label="Permalink: Installation" href="#installation"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">Install this plugin in the same environment as Datasette.</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ pip install datasette-column-inspect"><pre class="notranslate"><code>$ pip install datasette-column-inspect
  </code></pre></div>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Usage</h2><a id="user-content-usage" class="anchor" aria-label="Permalink: Usage" href="#usage"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">This plugin adds an icon to each column on the table page which opens an inspection side panel.</p>
  </article></div>


node_id: MDEwOlJlcG9zaXRvcnkyNDc1Mjc0Mzg=
name: datasette-edit-schema
full_name: simonw/datasette-edit-schema
html_url: https://github.com/simonw/datasette-edit-schema
description: Datasette plugin for modifying table schemas
created_at: 2020-03-15T18:34:06Z
updated_at: 2025-11-10T22:03:49Z
pushed_at: 2025-11-10T22:03:50Z
language: JavaScript
license: apache-2.0
topics: ["datasette", "datasette-io", "datasette-plugin"]
visibility: public
default_branch: main
permissions: {"admin": false, "maintain": false, "push": false, "triage": false, "pull": false}
pull_request_creation_policy: all
readme:
  # datasette-edit-schema

  [![PyPI](https://img.shields.io/pypi/v/datasette-edit-schema.svg)](https://pypi.org/project/datasette-edit-schema/)
  [![Changelog](https://img.shields.io/github/v/release/simonw/datasette-edit-schema?include_prereleases&label=changelog)](https://github.com/simonw/datasette-edit-schema/releases)
  [![Tests](https://github.com/simonw/datasette-edit-schema/workflows/Test/badge.svg)](https://github.com/simonw/datasette-edit-schema/actions?query=workflow%3ATest)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/datasette-edit-schema/blob/master/LICENSE)

  Datasette plugin for modifying table schemas

  > :warning: The latest alpha release depends on Datasette 1.09a. Use [version 0.7.1](https://github.com/simonw/datasette-edit-schema/blob/0.7.1/README.md) with older releases of Datasette.

  ## Features

  * Add new columns to a table
  * Rename columns in a table
  * Modify the type of columns in a table
  * Re-order the columns in a table
  * Rename a table
  * Delete a table
  * Change the primary key of a table to another column containing unique values
  * Update the foreign key constraints on a table
  * Add an index (or unique index) to a column on a table
  * Drop an index from a table

  ## Installation

  Install this plugin in the same environment as Datasette.
  ```bash
  pip install datasette-edit-schema
  ```
  ## Usage

  Navigate to `/-/edit-schema/dbname/tablename` on your Datasette instance to edit a specific table.

  Use `/-/edit-schema/dbname` to create a new table in a specific database.

  By default only [the root actor](https://datasette.readthedocs.io/en/stable/authentication.html#using-the-root-actor) can access the page - so you'll need to run Datasette with the `--root` option and click on the link shown in the terminal to sign in and access the page.

  ## Permissions

  The `edit-schema` permission provides access to all functionality.

  You can use permission plugins such as [datasette-permissions-sql](https://github.com/simonw/datasette-permissions-sql) to grant additional access to the write interface.

  These permission checks will call the `permission_allowed()` plugin hook with three arguments:

  - `action` will be the string `"edit-schema"`
  - `actor` will be the currently authenticated actor - usually a dictionary
  - `resource` will be the string name of the database

  You can instead use more finely-grained permissions from the default Datasette permissions collection:

  - `create-table` allows users to create a new table. The `resource` will be the name of the database.
  - `drop-table` allows users to drop a table. The `resource` will be a tuple of `(database_name, table_name)`.
  - `alter-table` allows users to alter a table. The `resource` will be a tuple of `(database_name, table_name)`.

  To rename a table a user must have both `drop-table` permission for that table and `create-table` permission for that database.

  For example, to configure Datasette to allow the user with ID `pelican` to create, alter and drop tables in the `marketing` database and to alter just the `notes` table in the `sales` database, you could use the following configuration:

  ```yaml
  databases:
    marketing:
      permissions:
        create-table:
          id: pelican
        drop-table:
          id: pelican
        alter-table:
          id: pelican
    sales:
      tables:
        notes:
          permissions:
            alter-table:
              id: pelican
  ```

  ## Events

  This plugin fires `create-table`, `alter-table` and `drop-table` events when tables are modified, using the [Datasette Events](https://docs.datasette.io/en/latest/events.html) system introduced in [Datasette 1.0a8](https://docs.datasette.io/en/latest/changelog.html#a8-2024-02-07).

  ## Screenshot

  ![datasette-edit-schema interface](https://raw.githubusercontent.com/simonw/datasette-edit-schema/main/datasette-edit-schema.png)

  ## Development

  To set up this plugin locally, first checkout the code. Then create a new virtual environment:
  ```bash
  cd datasette-edit-schema
  python3 -mvenv venv
  source venv/bin/activate
  ```
  Or if you are using `pipenv`:
  ```bash
  pipenv shell
  ```
  Now install the dependencies and tests:
  ```bash
  pip install -e '.[test]'
  ```
  To run the tests:
  ```bash
  pytest
  ```

readme_html:
  <div id="readme" class="md" data-path="README.md"><article class="markdown-body entry-content container-lg" itemprop="text"><div class="markdown-heading" dir="auto"><h1 class="heading-element" dir="auto">datasette-edit-schema</h1><a id="user-content-datasette-edit-schema" class="anchor" aria-label="Permalink: datasette-edit-schema" href="#datasette-edit-schema"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto"><a href="https://pypi.org/project/datasette-edit-schema/" rel="nofollow"><img src="https://img.shields.io/pypi/v/datasette-edit-schema.svg" alt="PyPI" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/de433b01438e7dcbd1c68f940240c10c96c2dbad077e104e85fe52a703814187/68747470733a2f2f696d672e736869656c64732e696f2f707970692f762f6461746173657474652d656469742d736368656d612e737667"></a>
  <a href="https://github.com/simonw/datasette-edit-schema/releases"><img src="https://img.shields.io/github/v/release/simonw/datasette-edit-schema?include_prereleases&label=changelog" alt="Changelog" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/cb596328e027aa3624ee47fb0967278fca40a271150b067056f0cdf43aa24631/68747470733a2f2f696d672e736869656c64732e696f2f6769746875622f762f72656c656173652f73696d6f6e772f6461746173657474652d656469742d736368656d613f696e636c7564655f70726572656c6561736573266c6162656c3d6368616e67656c6f67"></a>
  <a href="https://github.com/simonw/datasette-edit-schema/actions?query=workflow%3ATest"><img src="https://github.com/simonw/datasette-edit-schema/workflows/Test/badge.svg" alt="Tests" style="max-width: 100%;"></a>
  <a href="https://github.com/simonw/datasette-edit-schema/blob/master/LICENSE"><img src="https://img.shields.io/badge/license-Apache%202.0-blue.svg" alt="License" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/c355f200ea90fddaa407b6eaab303663a669248ea3ca7b1fcf77dbe04ff5f48c/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6c6963656e73652d417061636865253230322e302d626c75652e737667"></a></p>
  <p dir="auto">Datasette plugin for modifying table schemas</p>
  <blockquote>
  <p dir="auto"><g-emoji class="g-emoji" alias="warning">⚠️</g-emoji> The latest alpha release depends on Datasette 1.09a. Use <a href="https://github.com/simonw/datasette-edit-schema/blob/0.7.1/README.md">version 0.7.1</a> with older releases of Datasette.</p>
  </blockquote>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Features</h2><a id="user-content-features" class="anchor" aria-label="Permalink: Features" href="#features"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <ul dir="auto">
  <li>Add new columns to a table</li>
  <li>Rename columns in a table</li>
  <li>Modify the type of columns in a table</li>
  <li>Re-order the columns in a table</li>
  <li>Rename a table</li>
  <li>Delete a table</li>
  <li>Change the primary key of a table to another column containing unique values</li>
  <li>Update the foreign key constraints on a table</li>
  <li>Add an index (or unique index) to a column on a table</li>
  <li>Drop an index from a table</li>
  </ul>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Installation</h2><a id="user-content-installation" class="anchor" aria-label="Permalink: Installation" href="#installation"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">Install this plugin in the same environment as Datasette.</p>
  <div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="pip install datasette-edit-schema"><pre>pip install datasette-edit-schema</pre></div>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Usage</h2><a id="user-content-usage" class="anchor" aria-label="Permalink: Usage" href="#usage"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">Navigate to <code>/-/edit-schema/dbname/tablename</code> on your Datasette instance to edit a specific table.</p>
  <p dir="auto">Use <code>/-/edit-schema/dbname</code> to create a new table in a specific database.</p>
  <p dir="auto">By default only <a href="https://datasette.readthedocs.io/en/stable/authentication.html#using-the-root-actor" rel="nofollow">the root actor</a> can access the page - so you'll need to run Datasette with the <code>--root</code> option and click on the link shown in the terminal to sign in and access the page.</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Permissions</h2><a id="user-content-permissions" class="anchor" aria-label="Permalink: Permissions" href="#permissions"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">The <code>edit-schema</code> permission provides access to all functionality.</p>
  <p dir="auto">You can use permission plugins such as <a href="https://github.com/simonw/datasette-permissions-sql">datasette-permissions-sql</a> to grant additional access to the write interface.</p>
  <p dir="auto">These permission checks will call the <code>permission_allowed()</code> plugin hook with three arguments:</p>
  <ul dir="auto">
  <li><code>action</code> will be the string <code>"edit-schema"</code></li>
  <li><code>actor</code> will be the currently authenticated actor - usually a dictionary</li>
  <li><code>resource</code> will be the string name of the database</li>
  </ul>
  <p dir="auto">You can instead use more finely-grained permissions from the default Datasette permissions collection:</p>
  <ul dir="auto">
  <li><code>create-table</code> allows users to create a new table. The <code>resource</code> will be the name of the database.</li>
  <li><code>drop-table</code> allows users to drop a table. The <code>resource</code> will be a tuple of <code>(database_name, table_name)</code>.</li>
  <li><code>alter-table</code> allows users to alter a table. The <code>resource</code> will be a tuple of <code>(database_name, table_name)</code>.</li>
  </ul>
  <p dir="auto">To rename a table a user must have both <code>drop-table</code> permission for that table and <code>create-table</code> permission for that database.</p>
  <p dir="auto">For example, to configure Datasette to allow the user with ID <code>pelican</code> to create, alter and drop tables in the <code>marketing</code> database and to alter just the <code>notes</code> table in the <code>sales</code> database, you could use the following configuration:</p>
  <div class="highlight highlight-source-yaml notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="databases:
    marketing:
      permissions:
        create-table:
          id: pelican
        drop-table:
          id: pelican
        alter-table:
          id: pelican
    sales:
      tables:
        notes:
          permissions:
            alter-table:
              id: pelican"><pre><span class="pl-ent">databases</span>:
    <span class="pl-ent">marketing</span>:
      <span class="pl-ent">permissions</span>:
        <span class="pl-ent">create-table</span>:
          <span class="pl-ent">id</span>: <span class="pl-s">pelican</span>
        <span class="pl-ent">drop-table</span>:
          <span class="pl-ent">id</span>: <span class="pl-s">pelican</span>
        <span class="pl-ent">alter-table</span>:
          <span class="pl-ent">id</span>: <span class="pl-s">pelican</span>
    <span class="pl-ent">sales</span>:
      <span class="pl-ent">tables</span>:
        <span class="pl-ent">notes</span>:
          <span class="pl-ent">permissions</span>:
            <span class="pl-ent">alter-table</span>:
              <span class="pl-ent">id</span>: <span class="pl-s">pelican</span></pre></div>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Events</h2><a id="user-content-events" class="anchor" aria-label="Permalink: Events" href="#events"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">This plugin fires <code>create-table</code>, <code>alter-table</code> and <code>drop-table</code> events when tables are modified, using the <a href="https://docs.datasette.io/en/latest/events.html" rel="nofollow">Datasette Events</a> system introduced in <a href="https://docs.datasette.io/en/latest/changelog.html#a8-2024-02-07" rel="nofollow">Datasette 1.0a8</a>.</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Screenshot</h2><a id="user-content-screenshot" class="anchor" aria-label="Permalink: Screenshot" href="#screenshot"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto"><a target="_blank" rel="noopener noreferrer nofollow" href="https://raw.githubusercontent.com/simonw/datasette-edit-schema/main/datasette-edit-schema.png"><img src="https://raw.githubusercontent.com/simonw/datasette-edit-schema/main/datasette-edit-schema.png" alt="datasette-edit-schema interface" style="max-width: 100%;"></a></p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Development</h2><a id="user-content-development" class="anchor" aria-label="Permalink: Development" href="#development"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">To set up this plugin locally, first checkout the code. Then create a new virtual environment:</p>
  <div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="cd datasette-edit-schema
  python3 -mvenv venv
  source venv/bin/activate"><pre><span class="pl-c1">cd</span> datasette-edit-schema
  python3 -mvenv venv
  <span class="pl-c1">source</span> venv/bin/activate</pre></div>
  <p dir="auto">Or if you are using <code>pipenv</code>:</p>
  <div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="pipenv shell"><pre>pipenv shell</pre></div>
  <p dir="auto">Now install the dependencies and tests:</p>
  <div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="pip install -e '.[test]'"><pre>pip install -e <span class="pl-s"><span class="pl-pds">'</span>.[test]<span class="pl-pds">'</span></span></pre></div>
  <p dir="auto">To run the tests:</p>
  <div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="pytest"><pre>pytest</pre></div>
  </article></div>


node_id: MDEwOlJlcG9zaXRvcnkyNDgzODUyOTk=
name: datasette-publish-fly
full_name: simonw/datasette-publish-fly
html_url: https://github.com/simonw/datasette-publish-fly
description: Datasette plugin for publishing data using Fly
created_at: 2020-03-19T01:47:01Z
updated_at: 2025-02-28T21:21:04Z
pushed_at: 2023-08-21T18:59:10Z
language: Python
license: apache-2.0
topics: ["datasette", "datasette-io", "datasette-plugin", "fly"]
visibility: public
default_branch: main
permissions: {"admin": false, "maintain": false, "push": false, "triage": false, "pull": true}
readme:
  # datasette-publish-fly

  [![PyPI](https://img.shields.io/pypi/v/datasette-publish-fly.svg)](https://pypi.org/project/datasette-publish-fly/)
  [![Changelog](https://img.shields.io/github/v/release/simonw/datasette-publish-fly?include_prereleases&label=changelog)](https://github.com/simonw/datasette-publish-fly/releases)
  [![Tests](https://github.com/simonw/datasette-publish-fly/workflows/Test/badge.svg)](https://github.com/simonw/datasette-publish-fly/actions?query=workflow%3ATest)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/datasette-publish-fly/blob/main/LICENSE)

  [Datasette](https://datasette.io/) plugin for deploying Datasette instances to [Fly.io](https://fly.io/).

  Project background: [Using SQLite and Datasette with Fly Volumes](https://simonwillison.net/2022/Feb/15/fly-volumes/)

  ## Installation

  Install this plugin in the same environment as Datasette.

      $ datasette install datasette-publish-fly

  ## Deploying read-only data

  First, install the `flyctl` command-line tool by [following their instructions](https://fly.io/docs/getting-started/installing-flyctl/).

  Run `flyctl auth signup` to create an account there, or `flyctl auth login` if you already have one.

  You can now use `datasette publish fly` to publish one or more SQLite database files:

      datasette publish fly my-database.db --app="my-data-app"

  The argument you pass to `--app` will be used for the URL of your application: `my-data-app.fly.dev`.

  To update an application, run the publish command passing the same application name to the `--app` option.

  Fly have [a free tier](https://fly.io/docs/about/pricing/#free-allowances), beyond which they will charge you monthly for each application you have live.  Details of their pricing can be [found on their site](https://fly.io/docs/pricing/).

  Your application will be deployed at `https://your-app-name.fly.io/` - be aware that it may take several minutes to start working the first time you deploy it.

  ## Using Fly volumes for writable databases

  Fly [Volumes](https://fly.io/docs/reference/volumes/) provide persistant disk storage for Fly applications. Volumes can be 1GB or more in size and the Fly free tier includes 3GB of volume space.

  Datasette plugins such as [datasette-uploads-csvs](https://datasette.io/plugins/datasette-upload-csvs) and [datasette-tiddlywiki](https://datasette.io/plugins/datasette-tiddlywiki) can be deployed to Fly and store their mutable data in a volume.

  > :warning: **You should only run a single instance of your application** if your database accepts writes. Fly has excellent support for running multiple instances in different geographical regions, but `datasette-publish-fly` with volumes is not yet compatible with that model. You should probably [use Fly PostgreSQL instead](https://fly.io/blog/globally-distributed-postgres/).

  Here's how to deploy `datasette-tiddlywiki` with authentication provided by `datasette-auth-passwords`.

  First, you'll need to create a root password hash to use to sign into the instance.

  You can do that by installing the plugin and running the `datasette hash-password` command, or by using [this hosted tool](https://datasette-auth-passwords-demo.datasette.io/-/password-tool).

  The hash should look like `pbkdf2_sha256$...` - you'll need this for the next step.

  In this example we're also deploying a read-only database called `content.db`.

  Pick a name for your new application, then run the following:

      datasette publish fly \
      content.db \
      --app your-application-name \
      --create-volume 1 \
      --create-db tiddlywiki \
      --install datasette-auth-passwords \
      --install datasette-tiddlywiki \
      --plugin-secret datasette-auth-passwords root_password_hash 'pbkdf2_sha256$...'

  This will create the new application, deploy the `content.db` read-only database, create a 1GB volume for that application, create a new database in that volume called `tiddlywiki.db`, then install the two plugins and configure the password you specified.

  ### Updating applications that use a volume

  Once you have deployed an application using a volume, you can update that application without needing the `--create-volume` or `--create-db` options. To add the [datasette-graphq](https://datasette.io/plugins/datasette-graphql) plugin to your deployed application you would run the following:

      datasette publish fly \
      content.db \
      --app your-application-name \
      --install datasette-auth-passwords \
      --install datasette-tiddlywiki \
      --install datasette-graphql \
      --plugin-secret datasette-auth-passwords root_password_hash 'pbkdf2_sha256$...' \

  Since the application name is the same you don't need the `--create-volume` or `--create-db` options - these are persisted automatically between deploys.

  You do need to specify the full list of plugins that you want to have installed, and any plugin secrets.

  You also need to include any read-only database files that are part of the instance - `content.db` in this example - otherwise the new deployment will not include them.

  ### Advanced volume usage

  `datasette publish fly` will add a volume called `datasette` to your Fly application. You can customize the name using the `--volume name custom_name` option.

  Fly can be used to scale applications to run multiple instances in multiple regions around the world. This works well with read-only Datasette but is not currently recommended using Datasette with volumes, since each Fly replica would need its own volume and data stored in one instance would not be visible in others.

  If you want to use multiple instances with volumes you will need to switch to using the `flyctl` command directly. The `--generate-dir` option, described below, can help with this.

  ## Generating without deploying

  Use the `--generate-dir` option to generate a directory that can be deployed to Fly rather than deploying directly:

      datasette publish fly my-database.db \
        --app="my-generated-app" \
        --generate-dir /tmp/deploy-this

  You can then manually deploy your generated application using the following:

      cd /tmp/deploy-this
      flyctl apps create my-generated-app
      flyctl deploy

  ## datasette publish fly --help

  <!-- [[[cog
  import cog
  from datasette import cli
  from click.testing import CliRunner
  runner = CliRunner()
  result = runner.invoke(cli.cli, ["publish", "fly", "--help"])
  help = result.output.replace("Usage: cli", "Usage: datasette")
  cog.out(
      "```\n{}```".format(help)
  )
  ]]] -->
  ```
  Usage: datasette publish fly [OPTIONS] [FILES]...

    Deploy an application to Fly that runs Datasette against the provided database
    files.

    Usage example:

        datasette publish fly my-database.db --app="my-data-app"

    Full documentation: https://datasette.io/plugins/datasette-publish-fly

  Options:
    -m, --metadata FILENAME         Path to JSON/YAML file containing metadata to
                                    publish
    --extra-options TEXT            Extra options to pass to datasette serve
    --branch TEXT                   Install datasette from a GitHub branch e.g.
                                    main
    --template-dir DIRECTORY        Path to directory containing custom templates
    --plugins-dir DIRECTORY         Path to directory containing custom plugins
    --static MOUNT:DIRECTORY        Serve static files from this directory at
                                    /MOUNT/...
    --install TEXT                  Additional packages (e.g. plugins) to install
    --plugin-secret <TEXT TEXT TEXT>...
                                    Secrets to pass to plugins, e.g. --plugin-
                                    secret datasette-auth-github client_id xxx
    --version-note TEXT             Additional note to show on /-/versions
    --secret TEXT                   Secret used for signing secure values, such as
                                    signed cookies
    --title TEXT                    Title for metadata
    --license TEXT                  License label for metadata
    --license_url TEXT              License URL for metadata
    --source TEXT                   Source label for metadata
    --source_url TEXT               Source URL for metadata
    --about TEXT                    About label for metadata
    --about_url TEXT                About URL for metadata
    --spatialite                    Enable SpatialLite extension
    --region TEXT                   Fly region to deploy to, e.g sjc - see
                                    https://fly.io/docs/reference/regions/
    --create-volume INTEGER RANGE   Create and attach volume of this size in GB
                                    [x>=1]
    --create-db TEXT                Names of read-write database files to create
    --volume-name TEXT              Volume name to use
    -a, --app TEXT                  Name of Fly app to deploy  [required]
    -o, --org TEXT                  Name of Fly org to deploy to
    --generate-dir DIRECTORY        Output generated application files and stop
                                    without deploying
    --show-files                    Output the generated Dockerfile, metadata.json
                                    and fly.toml
    --setting SETTING...            Setting, see
                                    docs.datasette.io/en/stable/settings.html
    --crossdb                       Enable cross-database SQL queries
    --help                          Show this message and exit.
  ```
  <!-- [[[end]]] -->

  ## Development

  To contribute to this tool, first checkout the code. Then create a new virtual environment:

      cd datasette-publish-fly
      python -m venv venv
      source venv/bin/activate

  Or if you are using `pipenv`:

      pipenv shell

  Now install the dependencies and test dependencies:

      pip install -e '.[test]'

  To run the tests:

      pytest

  ### Integration tests

  The tests in `tests/test_integration.py` make actual calls to Fly to deploy a test application.

  These tests are skipped by default. If you have `flyctl` installed and configured, you can run the integration tests like this:

      pytest --integration -s

  The `-s` option here ensures that output from the deploys will be visible to you - otherwise it can look like the tests have hung.

  The tests will create applications on Fly that start with the prefix `publish-fly-temp-` and then delete them at the end of the run.

readme_html:
  <div id="readme" class="md" data-path="README.md"><article class="markdown-body entry-content container-lg" itemprop="text"><div class="markdown-heading" dir="auto"><h1 class="heading-element" dir="auto">datasette-publish-fly</h1><a id="user-content-datasette-publish-fly" class="anchor" aria-label="Permalink: datasette-publish-fly" href="#datasette-publish-fly"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto"><a href="https://pypi.org/project/datasette-publish-fly/" rel="nofollow"><img src="https://img.shields.io/pypi/v/datasette-publish-fly.svg" alt="PyPI" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/21e9e342ac86518e46c22e12ea6ed81e4edcc2f4c18b0cffb6a605c023a9b184/68747470733a2f2f696d672e736869656c64732e696f2f707970692f762f6461746173657474652d7075626c6973682d666c792e737667"></a>
  <a href="https://github.com/simonw/datasette-publish-fly/releases"><img src="https://img.shields.io/github/v/release/simonw/datasette-publish-fly?include_prereleases&label=changelog" alt="Changelog" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/a7f57bd306fc359357023ddf4cdd4229cef6ceebe9ba56872396ec7adc72a25a/68747470733a2f2f696d672e736869656c64732e696f2f6769746875622f762f72656c656173652f73696d6f6e772f6461746173657474652d7075626c6973682d666c793f696e636c7564655f70726572656c6561736573266c6162656c3d6368616e67656c6f67"></a>
  <a href="https://github.com/simonw/datasette-publish-fly/actions?query=workflow%3ATest"><img src="https://github.com/simonw/datasette-publish-fly/workflows/Test/badge.svg" alt="Tests" style="max-width: 100%;"></a>
  <a href="https://github.com/simonw/datasette-publish-fly/blob/main/LICENSE"><img src="https://img.shields.io/badge/license-Apache%202.0-blue.svg" alt="License" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/c355f200ea90fddaa407b6eaab303663a669248ea3ca7b1fcf77dbe04ff5f48c/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6c6963656e73652d417061636865253230322e302d626c75652e737667"></a></p>
  <p dir="auto"><a href="https://datasette.io/" rel="nofollow">Datasette</a> plugin for deploying Datasette instances to <a href="https://fly.io/" rel="nofollow">Fly.io</a>.</p>
  <p dir="auto">Project background: <a href="https://simonwillison.net/2022/Feb/15/fly-volumes/" rel="nofollow">Using SQLite and Datasette with Fly Volumes</a></p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Installation</h2><a id="user-content-installation" class="anchor" aria-label="Permalink: Installation" href="#installation"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">Install this plugin in the same environment as Datasette.</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ datasette install datasette-publish-fly"><pre class="notranslate"><code>$ datasette install datasette-publish-fly
  </code></pre></div>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Deploying read-only data</h2><a id="user-content-deploying-read-only-data" class="anchor" aria-label="Permalink: Deploying read-only data" href="#deploying-read-only-data"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">First, install the <code>flyctl</code> command-line tool by <a href="https://fly.io/docs/getting-started/installing-flyctl/" rel="nofollow">following their instructions</a>.</p>
  <p dir="auto">Run <code>flyctl auth signup</code> to create an account there, or <code>flyctl auth login</code> if you already have one.</p>
  <p dir="auto">You can now use <code>datasette publish fly</code> to publish one or more SQLite database files:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="datasette publish fly my-database.db --app="my-data-app""><pre class="notranslate"><code>datasette publish fly my-database.db --app="my-data-app"
  </code></pre></div>
  <p dir="auto">The argument you pass to <code>--app</code> will be used for the URL of your application: <code>my-data-app.fly.dev</code>.</p>
  <p dir="auto">To update an application, run the publish command passing the same application name to the <code>--app</code> option.</p>
  <p dir="auto">Fly have <a href="https://fly.io/docs/about/pricing/#free-allowances" rel="nofollow">a free tier</a>, beyond which they will charge you monthly for each application you have live.  Details of their pricing can be <a href="https://fly.io/docs/pricing/" rel="nofollow">found on their site</a>.</p>
  <p dir="auto">Your application will be deployed at <code>https://your-app-name.fly.io/</code> - be aware that it may take several minutes to start working the first time you deploy it.</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Using Fly volumes for writable databases</h2><a id="user-content-using-fly-volumes-for-writable-databases" class="anchor" aria-label="Permalink: Using Fly volumes for writable databases" href="#using-fly-volumes-for-writable-databases"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">Fly <a href="https://fly.io/docs/reference/volumes/" rel="nofollow">Volumes</a> provide persistant disk storage for Fly applications. Volumes can be 1GB or more in size and the Fly free tier includes 3GB of volume space.</p>
  <p dir="auto">Datasette plugins such as <a href="https://datasette.io/plugins/datasette-upload-csvs" rel="nofollow">datasette-uploads-csvs</a> and <a href="https://datasette.io/plugins/datasette-tiddlywiki" rel="nofollow">datasette-tiddlywiki</a> can be deployed to Fly and store their mutable data in a volume.</p>
  <blockquote>
  <p dir="auto"><g-emoji class="g-emoji" alias="warning">⚠️</g-emoji> <strong>You should only run a single instance of your application</strong> if your database accepts writes. Fly has excellent support for running multiple instances in different geographical regions, but <code>datasette-publish-fly</code> with volumes is not yet compatible with that model. You should probably <a href="https://fly.io/blog/globally-distributed-postgres/" rel="nofollow">use Fly PostgreSQL instead</a>.</p>
  </blockquote>
  <p dir="auto">Here's how to deploy <code>datasette-tiddlywiki</code> with authentication provided by <code>datasette-auth-passwords</code>.</p>
  <p dir="auto">First, you'll need to create a root password hash to use to sign into the instance.</p>
  <p dir="auto">You can do that by installing the plugin and running the <code>datasette hash-password</code> command, or by using <a href="https://datasette-auth-passwords-demo.datasette.io/-/password-tool" rel="nofollow">this hosted tool</a>.</p>
  <p dir="auto">The hash should look like <code>pbkdf2_sha256$...</code> - you'll need this for the next step.</p>
  <p dir="auto">In this example we're also deploying a read-only database called <code>content.db</code>.</p>
  <p dir="auto">Pick a name for your new application, then run the following:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="datasette publish fly \
  content.db \
  --app your-application-name \
  --create-volume 1 \
  --create-db tiddlywiki \
  --install datasette-auth-passwords \
  --install datasette-tiddlywiki \
  --plugin-secret datasette-auth-passwords root_password_hash 'pbkdf2_sha256$...'"><pre class="notranslate"><code>datasette publish fly \
  content.db \
  --app your-application-name \
  --create-volume 1 \
  --create-db tiddlywiki \
  --install datasette-auth-passwords \
  --install datasette-tiddlywiki \
  --plugin-secret datasette-auth-passwords root_password_hash 'pbkdf2_sha256$...'
  </code></pre></div>
  <p dir="auto">This will create the new application, deploy the <code>content.db</code> read-only database, create a 1GB volume for that application, create a new database in that volume called <code>tiddlywiki.db</code>, then install the two plugins and configure the password you specified.</p>
  <div class="markdown-heading" dir="auto"><h3 class="heading-element" dir="auto">Updating applications that use a volume</h3><a id="user-content-updating-applications-that-use-a-volume" class="anchor" aria-label="Permalink: Updating applications that use a volume" href="#updating-applications-that-use-a-volume"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">Once you have deployed an application using a volume, you can update that application without needing the <code>--create-volume</code> or <code>--create-db</code> options. To add the <a href="https://datasette.io/plugins/datasette-graphql" rel="nofollow">datasette-graphq</a> plugin to your deployed application you would run the following:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="datasette publish fly \
  content.db \
  --app your-application-name \
  --install datasette-auth-passwords \
  --install datasette-tiddlywiki \
  --install datasette-graphql \
  --plugin-secret datasette-auth-passwords root_password_hash 'pbkdf2_sha256$...' \"><pre class="notranslate"><code>datasette publish fly \
  content.db \
  --app your-application-name \
  --install datasette-auth-passwords \
  --install datasette-tiddlywiki \
  --install datasette-graphql \
  --plugin-secret datasette-auth-passwords root_password_hash 'pbkdf2_sha256$...' \
  </code></pre></div>
  <p dir="auto">Since the application name is the same you don't need the <code>--create-volume</code> or <code>--create-db</code> options - these are persisted automatically between deploys.</p>
  <p dir="auto">You do need to specify the full list of plugins that you want to have installed, and any plugin secrets.</p>
  <p dir="auto">You also need to include any read-only database files that are part of the instance - <code>content.db</code> in this example - otherwise the new deployment will not include them.</p>
  <div class="markdown-heading" dir="auto"><h3 class="heading-element" dir="auto">Advanced volume usage</h3><a id="user-content-advanced-volume-usage" class="anchor" aria-label="Permalink: Advanced volume usage" href="#advanced-volume-usage"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto"><code>datasette publish fly</code> will add a volume called <code>datasette</code> to your Fly application. You can customize the name using the <code>--volume name custom_name</code> option.</p>
  <p dir="auto">Fly can be used to scale applications to run multiple instances in multiple regions around the world. This works well with read-only Datasette but is not currently recommended using Datasette with volumes, since each Fly replica would need its own volume and data stored in one instance would not be visible in others.</p>
  <p dir="auto">If you want to use multiple instances with volumes you will need to switch to using the <code>flyctl</code> command directly. The <code>--generate-dir</code> option, described below, can help with this.</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Generating without deploying</h2><a id="user-content-generating-without-deploying" class="anchor" aria-label="Permalink: Generating without deploying" href="#generating-without-deploying"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">Use the <code>--generate-dir</code> option to generate a directory that can be deployed to Fly rather than deploying directly:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="datasette publish fly my-database.db \
    --app="my-generated-app" \
    --generate-dir /tmp/deploy-this"><pre class="notranslate"><code>datasette publish fly my-database.db \
    --app="my-generated-app" \
    --generate-dir /tmp/deploy-this
  </code></pre></div>
  <p dir="auto">You can then manually deploy your generated application using the following:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="cd /tmp/deploy-this
  flyctl apps create my-generated-app
  flyctl deploy"><pre class="notranslate"><code>cd /tmp/deploy-this
  flyctl apps create my-generated-app
  flyctl deploy
  </code></pre></div>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">datasette publish fly --help</h2><a id="user-content-datasette-publish-fly---help" class="anchor" aria-label="Permalink: datasette publish fly --help" href="#datasette-publish-fly---help"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>

  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="Usage: datasette publish fly [OPTIONS] [FILES]...

    Deploy an application to Fly that runs Datasette against the provided database
    files.

    Usage example:

        datasette publish fly my-database.db --app="my-data-app"

    Full documentation: https://datasette.io/plugins/datasette-publish-fly

  Options:
    -m, --metadata FILENAME         Path to JSON/YAML file containing metadata to
                                    publish
    --extra-options TEXT            Extra options to pass to datasette serve
    --branch TEXT                   Install datasette from a GitHub branch e.g.
                                    main
    --template-dir DIRECTORY        Path to directory containing custom templates
    --plugins-dir DIRECTORY         Path to directory containing custom plugins
    --static MOUNT:DIRECTORY        Serve static files from this directory at
                                    /MOUNT/...
    --install TEXT                  Additional packages (e.g. plugins) to install
    --plugin-secret <TEXT TEXT TEXT>...
                                    Secrets to pass to plugins, e.g. --plugin-
                                    secret datasette-auth-github client_id xxx
    --version-note TEXT             Additional note to show on /-/versions
    --secret TEXT                   Secret used for signing secure values, such as
                                    signed cookies
    --title TEXT                    Title for metadata
    --license TEXT                  License label for metadata
    --license_url TEXT              License URL for metadata
    --source TEXT                   Source label for metadata
    --source_url TEXT               Source URL for metadata
    --about TEXT                    About label for metadata
    --about_url TEXT                About URL for metadata
    --spatialite                    Enable SpatialLite extension
    --region TEXT                   Fly region to deploy to, e.g sjc - see
                                    https://fly.io/docs/reference/regions/
    --create-volume INTEGER RANGE   Create and attach volume of this size in GB
                                    [x>=1]
    --create-db TEXT                Names of read-write database files to create
    --volume-name TEXT              Volume name to use
    -a, --app TEXT                  Name of Fly app to deploy  [required]
    -o, --org TEXT                  Name of Fly org to deploy to
    --generate-dir DIRECTORY        Output generated application files and stop
                                    without deploying
    --show-files                    Output the generated Dockerfile, metadata.json
                                    and fly.toml
    --setting SETTING...            Setting, see
                                    docs.datasette.io/en/stable/settings.html
    --crossdb                       Enable cross-database SQL queries
    --help                          Show this message and exit."><pre class="notranslate"><code>Usage: datasette publish fly [OPTIONS] [FILES]...

    Deploy an application to Fly that runs Datasette against the provided database
    files.

    Usage example:

        datasette publish fly my-database.db --app="my-data-app"

    Full documentation: https://datasette.io/plugins/datasette-publish-fly

  Options:
    -m, --metadata FILENAME         Path to JSON/YAML file containing metadata to
                                    publish
    --extra-options TEXT            Extra options to pass to datasette serve
    --branch TEXT                   Install datasette from a GitHub branch e.g.
                                    main
    --template-dir DIRECTORY        Path to directory containing custom templates
    --plugins-dir DIRECTORY         Path to directory containing custom plugins
    --static MOUNT:DIRECTORY        Serve static files from this directory at
                                    /MOUNT/...
    --install TEXT                  Additional packages (e.g. plugins) to install
    --plugin-secret <TEXT TEXT TEXT>...
                                    Secrets to pass to plugins, e.g. --plugin-
                                    secret datasette-auth-github client_id xxx
    --version-note TEXT             Additional note to show on /-/versions
    --secret TEXT                   Secret used for signing secure values, such as
                                    signed cookies
    --title TEXT                    Title for metadata
    --license TEXT                  License label for metadata
    --license_url TEXT              License URL for metadata
    --source TEXT                   Source label for metadata
    --source_url TEXT               Source URL for metadata
    --about TEXT                    About label for metadata
    --about_url TEXT                About URL for metadata
    --spatialite                    Enable SpatialLite extension
    --region TEXT                   Fly region to deploy to, e.g sjc - see
                                    https://fly.io/docs/reference/regions/
    --create-volume INTEGER RANGE   Create and attach volume of this size in GB
                                    [x>=1]
    --create-db TEXT                Names of read-write database files to create
    --volume-name TEXT              Volume name to use
    -a, --app TEXT                  Name of Fly app to deploy  [required]
    -o, --org TEXT                  Name of Fly org to deploy to
    --generate-dir DIRECTORY        Output generated application files and stop
                                    without deploying
    --show-files                    Output the generated Dockerfile, metadata.json
                                    and fly.toml
    --setting SETTING...            Setting, see
                                    docs.datasette.io/en/stable/settings.html
    --crossdb                       Enable cross-database SQL queries
    --help                          Show this message and exit.
  </code></pre></div>

  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Development</h2><a id="user-content-development" class="anchor" aria-label="Permalink: Development" href="#development"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">To contribute to this tool, first checkout the code. Then create a new virtual environment:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="cd datasette-publish-fly
  python -m venv venv
  source venv/bin/activate"><pre class="notranslate"><code>cd datasette-publish-fly
  python -m venv venv
  source venv/bin/activate
  </code></pre></div>
  <p dir="auto">Or if you are using <code>pipenv</code>:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="pipenv shell"><pre class="notranslate"><code>pipenv shell
  </code></pre></div>
  <p dir="auto">Now install the dependencies and test dependencies:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="pip install -e '.[test]'"><pre class="notranslate"><code>pip install -e '.[test]'
  </code></pre></div>
  <p dir="auto">To run the tests:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="pytest"><pre class="notranslate"><code>pytest
  </code></pre></div>
  <div class="markdown-heading" dir="auto"><h3 class="heading-element" dir="auto">Integration tests</h3><a id="user-content-integration-tests" class="anchor" aria-label="Permalink: Integration tests" href="#integration-tests"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">The tests in <code>tests/test_integration.py</code> make actual calls to Fly to deploy a test application.</p>
  <p dir="auto">These tests are skipped by default. If you have <code>flyctl</code> installed and configured, you can run the integration tests like this:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="pytest --integration -s"><pre class="notranslate"><code>pytest --integration -s
  </code></pre></div>
  <p dir="auto">The <code>-s</code> option here ensures that output from the deploys will be visible to you - otherwise it can look like the tests have hung.</p>
  <p dir="auto">The tests will create applications on Fly that start with the prefix <code>publish-fly-temp-</code> and then delete them at the end of the run.</p>
  </article></div>


node_id: MDEwOlJlcG9zaXRvcnkyNDg5MDM1NDQ=
name: hacker-news-to-sqlite
full_name: dogsheep/hacker-news-to-sqlite
html_url: https://github.com/dogsheep/hacker-news-to-sqlite
description: Create a SQLite database containing data pulled from Hacker News
created_at: 2020-03-21T04:02:05Z
updated_at: 2025-05-05T18:57:41Z
pushed_at: 2023-03-26T22:22:39Z
language: Python
license: apache-2.0
topics: ["datasette", "datasette-io", "datasette-tool", "dogsheep", "hacker-news"]
visibility: public
default_branch: main
permissions: {"admin": false, "maintain": false, "push": false, "triage": false, "pull": true}
custom_properties: {}
readme:
  # hacker-news-to-sqlite

  [![PyPI](https://img.shields.io/pypi/v/hacker-news-to-sqlite.svg)](https://pypi.org/project/hacker-news-to-sqlite/)
  [![Changelog](https://img.shields.io/github/v/release/dogsheep/hacker-news-to-sqlite?include_prereleases&label=changelog)](https://github.com/dogsheep/hacker-news-to-sqlite/releases)
  [![Tests](https://github.com/dogsheep/hacker-news-to-sqlite/workflows/Test/badge.svg)](https://github.com/dogsheep/hacker-news-to-sqlite/actions?query=workflow%3ATest)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/hacker-news-to-sqlite/blob/main/LICENSE)

  Create a SQLite database containing data fetched from [Hacker News](https://news.ycombinator.com/).

  ## How to install

      $ pip install hacker-news-to-sqlite

  ## Usage

      $ hacker-news-to-sqlite user hacker-news.db your-username
      Importing items:  37%|███████████                        | 845/2297 [05:09<11:02,  2.19it/s]

  Imports all of your Hacker News submissions and comments into a SQLite database called `hacker-news.db`.

      $ hacker-news-to-sqlite trees hacker-news.db 22640038 22643218

  Fetches the entire comments tree in which any of those content IDs appears.

  ## Browsing your data with Datasette

  You can use [Datasette](https://datasette.readthedocs.org/) to browse your data. Install Datasette like this:

      $ pip install datasette

  Now run it against your `hacker-news.db` file like so:

      $ datasette hacker-news.db

  Visit `https://localhost:8001/` to search and explore your data.

  You can improve the display of your data usinng the [datasette-render-timestamps](https://github.com/simonw/datasette-render-timestamps) and [datasette-render-html](https://github.com/simonw/datasette-render-html) plugins. Install them like this:

      $ pip install datasette-render-timestamps datasette-render-html

  Now save the following configuration in a file called `metadata.json`:

  ```json
  {
      "databases": {
          "hacker-news": {
              "tables": {
                  "items": {
                      "plugins": {
                          "datasette-render-html": {
                              "columns": [
                                  "text"
                              ]
                          },
                          "datasette-render-timestamps": {
                              "columns": [
                                  "time"
                              ]
                          }
                      }
                  },
                  "users": {
                      "plugins": {
                          "datasette-render-timestamps": {
                              "columns": [
                                  "created"
                              ]
                          }
                      }
                  }
              }
          }
      }
  }
  ```
  Run Datasette like this:

      $ datasette -m metadata.json hacker-news.db

  The timestamp columns will now be rendered as human-readable dates, and any HTML in your posts will be displayed as rendered HTML.

readme_html:
  <div id="readme" class="md" data-path="README.md"><article class="markdown-body entry-content container-lg" itemprop="text"><div class="markdown-heading" dir="auto"><h1 class="heading-element" dir="auto">hacker-news-to-sqlite</h1><a id="user-content-hacker-news-to-sqlite" class="anchor" aria-label="Permalink: hacker-news-to-sqlite" href="#hacker-news-to-sqlite"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto"><a href="https://pypi.org/project/hacker-news-to-sqlite/" rel="nofollow"><img src="https://img.shields.io/pypi/v/hacker-news-to-sqlite.svg" alt="PyPI" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/f5b2f5aaa48e6dc51fd7b0aff0894c4180d4723b94c7ee350c1fe66f9763646a/68747470733a2f2f696d672e736869656c64732e696f2f707970692f762f6861636b65722d6e6577732d746f2d73716c6974652e737667"></a>
  <a href="https://github.com/dogsheep/hacker-news-to-sqlite/releases"><img src="https://img.shields.io/github/v/release/dogsheep/hacker-news-to-sqlite?include_prereleases&label=changelog" alt="Changelog" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/2feee820a6a39d411cacc2c599e7f8edc129a882fedfa66df40020ad2fc0c3c1/68747470733a2f2f696d672e736869656c64732e696f2f6769746875622f762f72656c656173652f646f6773686565702f6861636b65722d6e6577732d746f2d73716c6974653f696e636c7564655f70726572656c6561736573266c6162656c3d6368616e67656c6f67"></a>
  <a href="https://github.com/dogsheep/hacker-news-to-sqlite/actions?query=workflow%3ATest"><img src="https://github.com/dogsheep/hacker-news-to-sqlite/workflows/Test/badge.svg" alt="Tests" style="max-width: 100%;"></a>
  <a href="https://github.com/simonw/hacker-news-to-sqlite/blob/main/LICENSE"><img src="https://img.shields.io/badge/license-Apache%202.0-blue.svg" alt="License" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/c355f200ea90fddaa407b6eaab303663a669248ea3ca7b1fcf77dbe04ff5f48c/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6c6963656e73652d417061636865253230322e302d626c75652e737667"></a></p>
  <p dir="auto">Create a SQLite database containing data fetched from <a href="https://news.ycombinator.com/" rel="nofollow">Hacker News</a>.</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">How to install</h2><a id="user-content-how-to-install" class="anchor" aria-label="Permalink: How to install" href="#how-to-install"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ pip install hacker-news-to-sqlite"><pre class="notranslate"><code>$ pip install hacker-news-to-sqlite
  </code></pre></div>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Usage</h2><a id="user-content-usage" class="anchor" aria-label="Permalink: Usage" href="#usage"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ hacker-news-to-sqlite user hacker-news.db your-username
  Importing items:  37%|███████████                        | 845/2297 [05:09<11:02,  2.19it/s]"><pre class="notranslate"><code>$ hacker-news-to-sqlite user hacker-news.db your-username
  Importing items:  37%|███████████                        | 845/2297 [05:09<11:02,  2.19it/s]
  </code></pre></div>
  <p dir="auto">Imports all of your Hacker News submissions and comments into a SQLite database called <code>hacker-news.db</code>.</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ hacker-news-to-sqlite trees hacker-news.db 22640038 22643218"><pre class="notranslate"><code>$ hacker-news-to-sqlite trees hacker-news.db 22640038 22643218
  </code></pre></div>
  <p dir="auto">Fetches the entire comments tree in which any of those content IDs appears.</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Browsing your data with Datasette</h2><a id="user-content-browsing-your-data-with-datasette" class="anchor" aria-label="Permalink: Browsing your data with Datasette" href="#browsing-your-data-with-datasette"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">You can use <a href="https://datasette.readthedocs.org/" rel="nofollow">Datasette</a> to browse your data. Install Datasette like this:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ pip install datasette"><pre class="notranslate"><code>$ pip install datasette
  </code></pre></div>
  <p dir="auto">Now run it against your <code>hacker-news.db</code> file like so:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ datasette hacker-news.db"><pre class="notranslate"><code>$ datasette hacker-news.db
  </code></pre></div>
  <p dir="auto">Visit <code>https://localhost:8001/</code> to search and explore your data.</p>
  <p dir="auto">You can improve the display of your data usinng the <a href="https://github.com/simonw/datasette-render-timestamps">datasette-render-timestamps</a> and <a href="https://github.com/simonw/datasette-render-html">datasette-render-html</a> plugins. Install them like this:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ pip install datasette-render-timestamps datasette-render-html"><pre class="notranslate"><code>$ pip install datasette-render-timestamps datasette-render-html
  </code></pre></div>
  <p dir="auto">Now save the following configuration in a file called <code>metadata.json</code>:</p>
  <div class="highlight highlight-source-json notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="{
      "databases": {
          "hacker-news": {
              "tables": {
                  "items": {
                      "plugins": {
                          "datasette-render-html": {
                              "columns": [
                                  "text"
                              ]
                          },
                          "datasette-render-timestamps": {
                              "columns": [
                                  "time"
                              ]
                          }
                      }
                  },
                  "users": {
                      "plugins": {
                          "datasette-render-timestamps": {
                              "columns": [
                                  "created"
                              ]
                          }
                      }
                  }
              }
          }
      }
  }"><pre>{
      <span class="pl-ent">"databases"</span>: {
          <span class="pl-ent">"hacker-news"</span>: {
              <span class="pl-ent">"tables"</span>: {
                  <span class="pl-ent">"items"</span>: {
                      <span class="pl-ent">"plugins"</span>: {
                          <span class="pl-ent">"datasette-render-html"</span>: {
                              <span class="pl-ent">"columns"</span>: [
                                  <span class="pl-s"><span class="pl-pds">"</span>text<span class="pl-pds">"</span></span>
                              ]
                          },
                          <span class="pl-ent">"datasette-render-timestamps"</span>: {
                              <span class="pl-ent">"columns"</span>: [
                                  <span class="pl-s"><span class="pl-pds">"</span>time<span class="pl-pds">"</span></span>
                              ]
                          }
                      }
                  },
                  <span class="pl-ent">"users"</span>: {
                      <span class="pl-ent">"plugins"</span>: {
                          <span class="pl-ent">"datasette-render-timestamps"</span>: {
                              <span class="pl-ent">"columns"</span>: [
                                  <span class="pl-s"><span class="pl-pds">"</span>created<span class="pl-pds">"</span></span>
                              ]
                          }
                      }
                  }
              }
          }
      }
  }</pre></div>
  <p dir="auto">Run Datasette like this:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ datasette -m metadata.json hacker-news.db"><pre class="notranslate"><code>$ datasette -m metadata.json hacker-news.db
  </code></pre></div>
  <p dir="auto">The timestamp columns will now be rendered as human-readable dates, and any HTML in your posts will be displayed as rendered HTML.</p>
  </article></div>


node_id: MDEwOlJlcG9zaXRvcnkyNDg5OTk5OTQ=
name: datasette-show-errors
full_name: simonw/datasette-show-errors
html_url: https://github.com/simonw/datasette-show-errors
description: Datasette plugin for displaying error tracebacks
created_at: 2020-03-21T15:06:04Z
updated_at: 2020-09-24T00:17:29Z
pushed_at: 2020-09-01T00:32:23Z
language: Python
license: apache-2.0
topics: ["asgi", "datasette", "datasette-io", "datasette-plugin", "starlette"]
visibility: public
default_branch: master
permissions: {"admin": false, "maintain": false, "push": false, "triage": false, "pull": true}
readme:
  # datasette-show-errors

  [![PyPI](https://img.shields.io/pypi/v/datasette-show-errors.svg)](https://pypi.org/project/datasette-show-errors/)
  [![CircleCI](https://circleci.com/gh/simonw/datasette-show-errors.svg?style=svg)](https://circleci.com/gh/simonw/datasette-show-errors)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/datasette-show-errors/blob/master/LICENSE)

  Datasette plugin for displaying error tracebacks.

  **This plugin does not work with current versions of Datasette.** See [issue #2](https://github.com/simonw/datasette-show-errors/issues/2).

  ## Installation

      pip install datasette-show-errors

  ## Usage

  Installing the plugin will cause any internal error to be displayed with a full traceback, rather than just a generic 500 page.

  Be careful not to use this in a context that might expose sensitive information.

readme_html:
  <div id="readme" class="md" data-path="README.md"><article class="markdown-body entry-content container-lg" itemprop="text"><div class="markdown-heading" dir="auto"><h1 class="heading-element" dir="auto">datasette-show-errors</h1><a id="user-content-datasette-show-errors" class="anchor" aria-label="Permalink: datasette-show-errors" href="#datasette-show-errors"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto"><a href="https://pypi.org/project/datasette-show-errors/" rel="nofollow"><img src="https://img.shields.io/pypi/v/datasette-show-errors.svg" alt="PyPI" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/6bff26ffe9a8340f853ffb142f2c1a65a2c66e2f4517f9d4dd180c5ddf50ffc7/68747470733a2f2f696d672e736869656c64732e696f2f707970692f762f6461746173657474652d73686f772d6572726f72732e737667"></a>
  <a href="https://circleci.com/gh/simonw/datasette-show-errors" rel="nofollow"><img src="https://circleci.com/gh/simonw/datasette-show-errors.svg?style=svg" alt="CircleCI" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/89cdd36c604299ef996fe0d48383a0222f2c4cb270660d0c7685b39ac91bd016/68747470733a2f2f636972636c6563692e636f6d2f67682f73696d6f6e772f6461746173657474652d73686f772d6572726f72732e7376673f7374796c653d737667"></a>
  <a href="https://github.com/simonw/datasette-show-errors/blob/master/LICENSE"><img src="https://img.shields.io/badge/license-Apache%202.0-blue.svg" alt="License" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/c355f200ea90fddaa407b6eaab303663a669248ea3ca7b1fcf77dbe04ff5f48c/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6c6963656e73652d417061636865253230322e302d626c75652e737667"></a></p>
  <p dir="auto">Datasette plugin for displaying error tracebacks.</p>
  <p dir="auto"><strong>This plugin does not work with current versions of Datasette.</strong> See <a href="https://github.com/simonw/datasette-show-errors/issues/2" data-hovercard-type="issue" data-hovercard-url="/simonw/datasette-show-errors/issues/2/hovercard">issue #2</a>.</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Installation</h2><a id="user-content-installation" class="anchor" aria-label="Permalink: Installation" href="#installation"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="pip install datasette-show-errors"><pre class="notranslate"><code>pip install datasette-show-errors
  </code></pre></div>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Usage</h2><a id="user-content-usage" class="anchor" aria-label="Permalink: Usage" href="#usage"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">Installing the plugin will cause any internal error to be displayed with a full traceback, rather than just a generic 500 page.</p>
  <p dir="auto">Be careful not to use this in a context that might expose sensitive information.</p>
  </article></div>


node_id: MDEwOlJlcG9zaXRvcnkyNTM2MzI5NDg=
name: datasette-publish-vercel
full_name: simonw/datasette-publish-vercel
html_url: https://github.com/simonw/datasette-publish-vercel
description: Datasette plugin for publishing data using Vercel
created_at: 2020-04-06T22:47:13Z
updated_at: 2024-12-09T05:48:49Z
pushed_at: 2022-08-24T17:43:41Z
language: Python
license: apache-2.0
topics: ["datasette", "datasette-io", "datasette-plugin", "vercel", "zeit-now"]
visibility: public
default_branch: main
permissions: {"admin": false, "maintain": false, "push": false, "triage": false, "pull": true}
readme:
  # datasette-publish-vercel

  [![PyPI](https://img.shields.io/pypi/v/datasette-publish-vercel.svg)](https://pypi.org/project/datasette-publish-vercel/)
  [![Changelog](https://img.shields.io/github/v/release/simonw/datasette-publish-vercel?include_prereleases&label=changelog)](https://github.com/simonw/datasette-publish-vercel/releases)
  [![Tests](https://github.com/simonw/datasette-publish-vercel/workflows/Test/badge.svg)](https://github.com/simonw/datasette-publish-vercel/actions?query=workflow%3ATest)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/datasette-publish-vercel/blob/main/LICENSE)

  Datasette plugin for publishing data using [Vercel](https://vercel.com/).

  ## Installation

  Install this plugin in the same environment as Datasette.

      $ datasette install datasette-publish-vercel

  ## Usage

  First, install the Vercel CLI tool by [following their instructions](https://vercel.com/download).

  Run `vercel login` to login to (or create) an account.

  Now you can use `datasette publish vercel` to publish your data:

      datasette publish vercel my-database.db --project=my-database

  The `--project` argument is required - it specifies the project name that should be used for your deployment. This will be used as part of the deployment's URL.

  ### Other options

  * `--no-prod` deploys to the project without updating the "production" URL alias to point to that new deployment. Without that option all deploys go directly to production.
  * `--debug` enables the Vercel CLI debug output.
  * `--token` allows you to pass a Now authentication token, rather than needing to first run `now login` to configure the tool. Tokens can be created in the Vercel web dashboard under Account Settings -> Tokens.
  * `--public` runs `vercel --public` to publish the application source code at `/_src` e.g. https://datasette-public.now.sh/_src and make recent logs visible at `/_logs` e.g. https://datasette-public.now.sh/_logs
  * `--generate-dir` - by default this tool generates a new Vercel app in a temporary directory, deploys it and then deletes the directory. Use `--generate-dir=my-app` to output the generated application files to a new directory of your choice instead. You can then deploy it by running `vercel` in that directory.
  * `--setting default_page_size 10` - use this to set Datasette settings, as described in [the documentation](https://docs.datasette.io/en/stable/settings.html). This is a replacement for the unsupported `--extra-options` option.

  ### Full help

  **Warning:** Some of these options are not yet implemented by this plugin. In particular, the following do not yet work:

  * `--extra-options` - use `--setting` described above instead.
  * `--plugin-secret`
  * `--version-note`

  ```
  $ datasette publish vercel --help

  Usage: datasette publish vercel [OPTIONS] [FILES]...

    Publish to https://vercel.com/

  Options:
    -m, --metadata FILENAME         Path to JSON/YAML file containing metadata to publish
    --extra-options TEXT            Extra options to pass to datasette serve
    --branch TEXT                   Install datasette from a GitHub branch e.g. main
    --template-dir DIRECTORY        Path to directory containing custom templates
    --plugins-dir DIRECTORY         Path to directory containing custom plugins
    --static MOUNT:DIRECTORY        Serve static files from this directory at /MOUNT/...
    --install TEXT                  Additional packages (e.g. plugins) to install
    --plugin-secret <TEXT TEXT TEXT>...
                                    Secrets to pass to plugins, e.g. --plugin-secret
                                    datasette-auth-github client_id xxx
    --version-note TEXT             Additional note to show on /-/versions
    --secret TEXT                   Secret used for signing secure values, such as signed
                                    cookies
    --title TEXT                    Title for metadata
    --license TEXT                  License label for metadata
    --license_url TEXT              License URL for metadata
    --source TEXT                   Source label for metadata
    --source_url TEXT               Source URL for metadata
    --about TEXT                    About label for metadata
    --about_url TEXT                About URL for metadata
    --token TEXT                    Auth token to use for deploy
    --project PROJECT               Vercel project name to use  [required]
    --scope TEXT                    Optional Vercel scope (e.g. a team name)
    --no-prod                       Don't deploy directly to production
    --debug                         Enable Vercel CLI debug output
    --public                        Publish source with Vercel CLI --public
    --generate-dir DIRECTORY        Output generated application files and stop without
                                    deploying
    --generate-vercel-json          Output generated vercel.json file and stop without
                                    deploying
    --vercel-json FILENAME          Custom vercel.json file to use instead of generating
                                    one
    --setting SETTING...            Setting, see docs.datasette.io/en/stable/settings.html
    --crossdb                       Enable cross-database SQL queries
    --help                          Show this message and exit.
  ```
  ## Using a custom `vercel.json` file

  If you want to add additional redirects or similar to your Vercel configuration you may want to provide a custom `vercel.json` file.

  To do this, first generate a configuration file (without running a deploy) using the `--generate-vercel-json` option:

      datasette publish vercel my-database.db \
        --project=my-database \
        --generate-vercel-json > vercel.json

  You can now edit the `vercel.json` file that this creates to add your custom options.

  Then run the deploy using:

      datasette publish vercel my-database.db \
        --project=my-database \
        --vercel-json=vercel.json

  ## Setting a `DATASETTE_SECRET`

  Datasette uses [a secret string](https://docs.datasette.io/en/stable/settings.html#configuring-the-secret) for purposes such as signing authentication cookies. This secret is reset when the server restarts, which will sign out any users who are authenticated using a signed cookie.

  You can avoid this by generating a `DATASETTE_SECRET` secret string and setting that as a [Vercel environment variable](https://vercel.com/docs/concepts/projects/environment-variables). If you do this the secret will stay consistent and your users will not be signed out.

  ## Using this with GitHub Actions

  This plugin can be used together with [GitHub Actions](https://github.com/features/actions) to deploy Datasette instances automatically on new pushes to a repo, or on a schedule.

  The GitHub Actions runners already have the Vercel deployment tool installed. You'll need to create an API token for your account at [vercel.com/account/tokens](https://vercel.com/account/tokens), and store that as a secret in your GitHub repository called `VERCEL_TOKEN`.

  Make sure your workflow has installed `datasette` and `datasette-publish-vercel` using `pip`, then add the following step to your GitHub Actions workflow:
  ```
      - name: Deploy Datasette using Vercel
        env:
          VERCEL_TOKEN: ${{ secrets.VERCEL_TOKEN }}
        run: |-
          datasette publish vercel mydb.db \
            --token $VERCEL_TOKEN \
            --project my-vercel-project
  ```
  You can see a full example of a workflow that uses Vercel in this way [in the simonw/til repository](https://github.com/simonw/til/blob/12b3f0d3679320cbeafa5df164bbc08ba703625d/.github/workflows/build.yml).

readme_html:
  <div id="readme" class="md" data-path="README.md"><article class="markdown-body entry-content container-lg" itemprop="text"><div class="markdown-heading" dir="auto"><h1 class="heading-element" dir="auto">datasette-publish-vercel</h1><a id="user-content-datasette-publish-vercel" class="anchor" aria-label="Permalink: datasette-publish-vercel" href="#datasette-publish-vercel"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto"><a href="https://pypi.org/project/datasette-publish-vercel/" rel="nofollow"><img src="https://img.shields.io/pypi/v/datasette-publish-vercel.svg" alt="PyPI" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/67082701607716165abff57a76c83a2ea7dff649d4bc3426b7510aeac8ebc1da/68747470733a2f2f696d672e736869656c64732e696f2f707970692f762f6461746173657474652d7075626c6973682d76657263656c2e737667"></a>
  <a href="https://github.com/simonw/datasette-publish-vercel/releases"><img src="https://img.shields.io/github/v/release/simonw/datasette-publish-vercel?include_prereleases&label=changelog" alt="Changelog" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/24ce49e0c3a0582b2afef6749202ca8d9d19c480188b5cca84812f30099f6c42/68747470733a2f2f696d672e736869656c64732e696f2f6769746875622f762f72656c656173652f73696d6f6e772f6461746173657474652d7075626c6973682d76657263656c3f696e636c7564655f70726572656c6561736573266c6162656c3d6368616e67656c6f67"></a>
  <a href="https://github.com/simonw/datasette-publish-vercel/actions?query=workflow%3ATest"><img src="https://github.com/simonw/datasette-publish-vercel/workflows/Test/badge.svg" alt="Tests" style="max-width: 100%;"></a>
  <a href="https://github.com/simonw/datasette-publish-vercel/blob/main/LICENSE"><img src="https://img.shields.io/badge/license-Apache%202.0-blue.svg" alt="License" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/c355f200ea90fddaa407b6eaab303663a669248ea3ca7b1fcf77dbe04ff5f48c/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6c6963656e73652d417061636865253230322e302d626c75652e737667"></a></p>
  <p dir="auto">Datasette plugin for publishing data using <a href="https://vercel.com/" rel="nofollow">Vercel</a>.</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Installation</h2><a id="user-content-installation" class="anchor" aria-label="Permalink: Installation" href="#installation"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">Install this plugin in the same environment as Datasette.</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ datasette install datasette-publish-vercel"><pre class="notranslate"><code>$ datasette install datasette-publish-vercel
  </code></pre></div>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Usage</h2><a id="user-content-usage" class="anchor" aria-label="Permalink: Usage" href="#usage"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">First, install the Vercel CLI tool by <a href="https://vercel.com/download" rel="nofollow">following their instructions</a>.</p>
  <p dir="auto">Run <code>vercel login</code> to login to (or create) an account.</p>
  <p dir="auto">Now you can use <code>datasette publish vercel</code> to publish your data:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="datasette publish vercel my-database.db --project=my-database"><pre class="notranslate"><code>datasette publish vercel my-database.db --project=my-database
  </code></pre></div>
  <p dir="auto">The <code>--project</code> argument is required - it specifies the project name that should be used for your deployment. This will be used as part of the deployment's URL.</p>
  <div class="markdown-heading" dir="auto"><h3 class="heading-element" dir="auto">Other options</h3><a id="user-content-other-options" class="anchor" aria-label="Permalink: Other options" href="#other-options"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <ul dir="auto">
  <li><code>--no-prod</code> deploys to the project without updating the "production" URL alias to point to that new deployment. Without that option all deploys go directly to production.</li>
  <li><code>--debug</code> enables the Vercel CLI debug output.</li>
  <li><code>--token</code> allows you to pass a Now authentication token, rather than needing to first run <code>now login</code> to configure the tool. Tokens can be created in the Vercel web dashboard under Account Settings -> Tokens.</li>
  <li><code>--public</code> runs <code>vercel --public</code> to publish the application source code at <code>/_src</code> e.g. <a href="https://datasette-public.now.sh/_src" rel="nofollow">https://datasette-public.now.sh/_src</a> and make recent logs visible at <code>/_logs</code> e.g. <a href="https://datasette-public.now.sh/_logs" rel="nofollow">https://datasette-public.now.sh/_logs</a></li>
  <li><code>--generate-dir</code> - by default this tool generates a new Vercel app in a temporary directory, deploys it and then deletes the directory. Use <code>--generate-dir=my-app</code> to output the generated application files to a new directory of your choice instead. You can then deploy it by running <code>vercel</code> in that directory.</li>
  <li><code>--setting default_page_size 10</code> - use this to set Datasette settings, as described in <a href="https://docs.datasette.io/en/stable/settings.html" rel="nofollow">the documentation</a>. This is a replacement for the unsupported <code>--extra-options</code> option.</li>
  </ul>
  <div class="markdown-heading" dir="auto"><h3 class="heading-element" dir="auto">Full help</h3><a id="user-content-full-help" class="anchor" aria-label="Permalink: Full help" href="#full-help"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto"><strong>Warning:</strong> Some of these options are not yet implemented by this plugin. In particular, the following do not yet work:</p>
  <ul dir="auto">
  <li><code>--extra-options</code> - use <code>--setting</code> described above instead.</li>
  <li><code>--plugin-secret</code></li>
  <li><code>--version-note</code></li>
  </ul>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ datasette publish vercel --help

  Usage: datasette publish vercel [OPTIONS] [FILES]...

    Publish to https://vercel.com/

  Options:
    -m, --metadata FILENAME         Path to JSON/YAML file containing metadata to publish
    --extra-options TEXT            Extra options to pass to datasette serve
    --branch TEXT                   Install datasette from a GitHub branch e.g. main
    --template-dir DIRECTORY        Path to directory containing custom templates
    --plugins-dir DIRECTORY         Path to directory containing custom plugins
    --static MOUNT:DIRECTORY        Serve static files from this directory at /MOUNT/...
    --install TEXT                  Additional packages (e.g. plugins) to install
    --plugin-secret <TEXT TEXT TEXT>...
                                    Secrets to pass to plugins, e.g. --plugin-secret
                                    datasette-auth-github client_id xxx
    --version-note TEXT             Additional note to show on /-/versions
    --secret TEXT                   Secret used for signing secure values, such as signed
                                    cookies
    --title TEXT                    Title for metadata
    --license TEXT                  License label for metadata
    --license_url TEXT              License URL for metadata
    --source TEXT                   Source label for metadata
    --source_url TEXT               Source URL for metadata
    --about TEXT                    About label for metadata
    --about_url TEXT                About URL for metadata
    --token TEXT                    Auth token to use for deploy
    --project PROJECT               Vercel project name to use  [required]
    --scope TEXT                    Optional Vercel scope (e.g. a team name)
    --no-prod                       Don't deploy directly to production
    --debug                         Enable Vercel CLI debug output
    --public                        Publish source with Vercel CLI --public
    --generate-dir DIRECTORY        Output generated application files and stop without
                                    deploying
    --generate-vercel-json          Output generated vercel.json file and stop without
                                    deploying
    --vercel-json FILENAME          Custom vercel.json file to use instead of generating
                                    one
    --setting SETTING...            Setting, see docs.datasette.io/en/stable/settings.html
    --crossdb                       Enable cross-database SQL queries
    --help                          Show this message and exit."><pre class="notranslate"><code>$ datasette publish vercel --help

  Usage: datasette publish vercel [OPTIONS] [FILES]...

    Publish to https://vercel.com/

  Options:
    -m, --metadata FILENAME         Path to JSON/YAML file containing metadata to publish
    --extra-options TEXT            Extra options to pass to datasette serve
    --branch TEXT                   Install datasette from a GitHub branch e.g. main
    --template-dir DIRECTORY        Path to directory containing custom templates
    --plugins-dir DIRECTORY         Path to directory containing custom plugins
    --static MOUNT:DIRECTORY        Serve static files from this directory at /MOUNT/...
    --install TEXT                  Additional packages (e.g. plugins) to install
    --plugin-secret <TEXT TEXT TEXT>...
                                    Secrets to pass to plugins, e.g. --plugin-secret
                                    datasette-auth-github client_id xxx
    --version-note TEXT             Additional note to show on /-/versions
    --secret TEXT                   Secret used for signing secure values, such as signed
                                    cookies
    --title TEXT                    Title for metadata
    --license TEXT                  License label for metadata
    --license_url TEXT              License URL for metadata
    --source TEXT                   Source label for metadata
    --source_url TEXT               Source URL for metadata
    --about TEXT                    About label for metadata
    --about_url TEXT                About URL for metadata
    --token TEXT                    Auth token to use for deploy
    --project PROJECT               Vercel project name to use  [required]
    --scope TEXT                    Optional Vercel scope (e.g. a team name)
    --no-prod                       Don't deploy directly to production
    --debug                         Enable Vercel CLI debug output
    --public                        Publish source with Vercel CLI --public
    --generate-dir DIRECTORY        Output generated application files and stop without
                                    deploying
    --generate-vercel-json          Output generated vercel.json file and stop without
                                    deploying
    --vercel-json FILENAME          Custom vercel.json file to use instead of generating
                                    one
    --setting SETTING...            Setting, see docs.datasette.io/en/stable/settings.html
    --crossdb                       Enable cross-database SQL queries
    --help                          Show this message and exit.
  </code></pre></div>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Using a custom <code>vercel.json</code> file</h2><a id="user-content-using-a-custom-verceljson-file" class="anchor" aria-label="Permalink: Using a custom vercel.json file" href="#using-a-custom-verceljson-file"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">If you want to add additional redirects or similar to your Vercel configuration you may want to provide a custom <code>vercel.json</code> file.</p>
  <p dir="auto">To do this, first generate a configuration file (without running a deploy) using the <code>--generate-vercel-json</code> option:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="datasette publish vercel my-database.db \
    --project=my-database \
    --generate-vercel-json > vercel.json"><pre class="notranslate"><code>datasette publish vercel my-database.db \
    --project=my-database \
    --generate-vercel-json > vercel.json
  </code></pre></div>
  <p dir="auto">You can now edit the <code>vercel.json</code> file that this creates to add your custom options.</p>
  <p dir="auto">Then run the deploy using:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="datasette publish vercel my-database.db \
    --project=my-database \
    --vercel-json=vercel.json"><pre class="notranslate"><code>datasette publish vercel my-database.db \
    --project=my-database \
    --vercel-json=vercel.json
  </code></pre></div>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Setting a <code>DATASETTE_SECRET</code></h2><a id="user-content-setting-a-datasette_secret" class="anchor" aria-label="Permalink: Setting a DATASETTE_SECRET" href="#setting-a-datasette_secret"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">Datasette uses <a href="https://docs.datasette.io/en/stable/settings.html#configuring-the-secret" rel="nofollow">a secret string</a> for purposes such as signing authentication cookies. This secret is reset when the server restarts, which will sign out any users who are authenticated using a signed cookie.</p>
  <p dir="auto">You can avoid this by generating a <code>DATASETTE_SECRET</code> secret string and setting that as a <a href="https://vercel.com/docs/concepts/projects/environment-variables" rel="nofollow">Vercel environment variable</a>. If you do this the secret will stay consistent and your users will not be signed out.</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Using this with GitHub Actions</h2><a id="user-content-using-this-with-github-actions" class="anchor" aria-label="Permalink: Using this with GitHub Actions" href="#using-this-with-github-actions"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">This plugin can be used together with <a href="https://github.com/features/actions">GitHub Actions</a> to deploy Datasette instances automatically on new pushes to a repo, or on a schedule.</p>
  <p dir="auto">The GitHub Actions runners already have the Vercel deployment tool installed. You'll need to create an API token for your account at <a href="https://vercel.com/account/tokens" rel="nofollow">vercel.com/account/tokens</a>, and store that as a secret in your GitHub repository called <code>VERCEL_TOKEN</code>.</p>
  <p dir="auto">Make sure your workflow has installed <code>datasette</code> and <code>datasette-publish-vercel</code> using <code>pip</code>, then add the following step to your GitHub Actions workflow:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="    - name: Deploy Datasette using Vercel
        env:
          VERCEL_TOKEN: ${{ secrets.VERCEL_TOKEN }}
        run: |-
          datasette publish vercel mydb.db \
            --token $VERCEL_TOKEN \
            --project my-vercel-project"><pre class="notranslate"><code>    - name: Deploy Datasette using Vercel
        env:
          VERCEL_TOKEN: ${{ secrets.VERCEL_TOKEN }}
        run: |-
          datasette publish vercel mydb.db \
            --token $VERCEL_TOKEN \
            --project my-vercel-project
  </code></pre></div>
  <p dir="auto">You can see a full example of a workflow that uses Vercel in this way <a href="https://github.com/simonw/til/blob/12b3f0d3679320cbeafa5df164bbc08ba703625d/.github/workflows/build.yml">in the simonw/til repository</a>.</p>
  </article></div>


node_id: MDEwOlJlcG9zaXRvcnkyNTU0NjAzNDc=
name: datasette-clone
full_name: simonw/datasette-clone
html_url: https://github.com/simonw/datasette-clone
description: Create a local copy of database files from a Datasette instance
created_at: 2020-04-13T23:05:41Z
updated_at: 2025-02-28T21:23:10Z
pushed_at: 2021-02-22T19:32:36Z
language: Python
license: apache-2.0
topics: ["datasette", "datasette-io", "datasette-tool"]
visibility: public
default_branch: main
permissions: {"admin": false, "maintain": false, "push": false, "triage": false, "pull": true}
readme:
  # datasette-clone

  [![PyPI](https://img.shields.io/pypi/v/datasette-clone.svg)](https://pypi.org/project/datasette-clone/)
  [![Changelog](https://img.shields.io/github/v/release/simonw/datasette-clone?include_prereleases&label=changelog)](https://github.com/simonw/datasette-clone/releases)
  [![Tests](https://github.com/simonw/datasette-clone/workflows/Test/badge.svg)](https://github.com/simonw/datasette-clone/actions?query=workflow%3ATest)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/datasette-clone/blob/main/LICENSE)

  Create a local copy of database files from a Datasette instance.

  See [datasette-clone](https://simonwillison.net/2020/Apr/14/datasette-clone/) on my blog for background on this project.

  ## How to install

      $ pip install datasette-clone

  ## Usage

  This only works against Datasette instances running immutable databases (with the `-i` option). Databases published using the `datasette publish` command should be compatible with this tool.

  To download copies of all `.db` files from an instance, run:

      datasette-clone https://latest.datasette.io

  You can provide an optional second argument to specify a directory:

      datasette-clone https://latest.datasette.io /tmp/here-please

  The command stores its own copy of a `databases.json` manifest and uses it to only download databases that have changed the next time you run the command.

  It also stores a copy of the instance's `metadata.json` to ensure you have a copy of any source and licensing information for the downloaded databases.

  If your instance is protected by an API token, you can use `--token` to provide it:

      datasette-clone https://latest.datasette.io --token=xyz

  For verbose output showing what the tool is doing, use `-v`.

readme_html:
  <div id="readme" class="md" data-path="README.md"><article class="markdown-body entry-content container-lg" itemprop="text"><div class="markdown-heading" dir="auto"><h1 class="heading-element" dir="auto">datasette-clone</h1><a id="user-content-datasette-clone" class="anchor" aria-label="Permalink: datasette-clone" href="#datasette-clone"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto"><a href="https://pypi.org/project/datasette-clone/" rel="nofollow"><img src="https://img.shields.io/pypi/v/datasette-clone.svg" alt="PyPI" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/bda32b055ce6c467b06a723b32959a4e5e7905a4d1f771e934c407c9b01d1e41/68747470733a2f2f696d672e736869656c64732e696f2f707970692f762f6461746173657474652d636c6f6e652e737667"></a>
  <a href="https://github.com/simonw/datasette-clone/releases"><img src="https://img.shields.io/github/v/release/simonw/datasette-clone?include_prereleases&label=changelog" alt="Changelog" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/17ce7d6db028b8bdc917052c1601f0245c6c733ab73f4d10ef9729e23a3a0239/68747470733a2f2f696d672e736869656c64732e696f2f6769746875622f762f72656c656173652f73696d6f6e772f6461746173657474652d636c6f6e653f696e636c7564655f70726572656c6561736573266c6162656c3d6368616e67656c6f67"></a>
  <a href="https://github.com/simonw/datasette-clone/actions?query=workflow%3ATest"><img src="https://github.com/simonw/datasette-clone/workflows/Test/badge.svg" alt="Tests" style="max-width: 100%;"></a>
  <a href="https://github.com/simonw/datasette-clone/blob/main/LICENSE"><img src="https://img.shields.io/badge/license-Apache%202.0-blue.svg" alt="License" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/c355f200ea90fddaa407b6eaab303663a669248ea3ca7b1fcf77dbe04ff5f48c/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6c6963656e73652d417061636865253230322e302d626c75652e737667"></a></p>
  <p dir="auto">Create a local copy of database files from a Datasette instance.</p>
  <p dir="auto">See <a href="https://simonwillison.net/2020/Apr/14/datasette-clone/" rel="nofollow">datasette-clone</a> on my blog for background on this project.</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">How to install</h2><a id="user-content-how-to-install" class="anchor" aria-label="Permalink: How to install" href="#how-to-install"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ pip install datasette-clone"><pre class="notranslate"><code>$ pip install datasette-clone
  </code></pre></div>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Usage</h2><a id="user-content-usage" class="anchor" aria-label="Permalink: Usage" href="#usage"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">This only works against Datasette instances running immutable databases (with the <code>-i</code> option). Databases published using the <code>datasette publish</code> command should be compatible with this tool.</p>
  <p dir="auto">To download copies of all <code>.db</code> files from an instance, run:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="datasette-clone https://latest.datasette.io"><pre class="notranslate"><code>datasette-clone https://latest.datasette.io
  </code></pre></div>
  <p dir="auto">You can provide an optional second argument to specify a directory:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="datasette-clone https://latest.datasette.io /tmp/here-please"><pre class="notranslate"><code>datasette-clone https://latest.datasette.io /tmp/here-please
  </code></pre></div>
  <p dir="auto">The command stores its own copy of a <code>databases.json</code> manifest and uses it to only download databases that have changed the next time you run the command.</p>
  <p dir="auto">It also stores a copy of the instance's <code>metadata.json</code> to ensure you have a copy of any source and licensing information for the downloaded databases.</p>
  <p dir="auto">If your instance is protected by an API token, you can use <code>--token</code> to provide it:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="datasette-clone https://latest.datasette.io --token=xyz"><pre class="notranslate"><code>datasette-clone https://latest.datasette.io --token=xyz
  </code></pre></div>
  <p dir="auto">For verbose output showing what the tool is doing, use <code>-v</code>.</p>
  </article></div>


node_id: MDEwOlJlcG9zaXRvcnkyNTY4MzQ5MDc=
name: dogsheep-photos
full_name: dogsheep/dogsheep-photos
html_url: https://github.com/dogsheep/dogsheep-photos
description: Upload your photos to S3 and import metadata about them into a SQLite database
created_at: 2020-04-18T19:22:13Z
updated_at: 2025-02-28T21:18:04Z
pushed_at: 2024-01-05T18:52:26Z
language: Python
license: apache-2.0
topics: ["datasette", "datasette-io", "datasette-tool", "dogsheep", "sqlite"]
visibility: public
default_branch: master
permissions: {"admin": false, "maintain": false, "push": false, "triage": false, "pull": true}
custom_properties: {}
readme:
  # dogsheep-photos

  [![PyPI](https://img.shields.io/pypi/v/dogsheep-photos.svg)](https://pypi.org/project/dogsheep-photos/)
  [![Changelog](https://img.shields.io/github/v/release/dogsheep/dogsheep-photos?include_prereleases&label=changelog)](https://github.com/dogsheep/dogsheep-photos/releases)
  [![CircleCI](https://circleci.com/gh/dogsheep/dogsheep-photos.svg?style=svg)](https://circleci.com/gh/dogsheep/dogsheep-photos)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/dogsheep/dogsheep-photos/blob/master/LICENSE)

  Save details of your photos to a SQLite database and upload them to S3.

  See [Using SQL to find my best photo of a pelican according to Apple Photos](https://simonwillison.net/2020/May/21/apple-photos-sqlite/) for background information on this project.

  ## What these tools do

  These tools are a work-in-progress mechanism for taking full ownership of your photos. The core idea is to help implement the following:

  * Every photo you have taken lives in a single, private Amazon S3 bucket
  * You have a single SQLite database file which stores metadata about those photos - potentially pulled from multiple different places. This may include EXIF data, Apple Photos, the results of running machine learning APIs against photos and much more besides.
  * You can then use [Datasette](https://github.com/simonw/datasette) to explore your own photos.

  I'm a heavy user of Apple Photos so the initial releases of this tool will have a bias towards that, but ideally I would like a subset of these tools to be useful to people no matter which core photo solution they are using.

  ## Installation

      $ pip install dogsheep-photos

  ## Authentication (if using S3)

  If you want to use S3 to store your photos, you will need to first create S3 credentials for a new, dedicated bucket.

  You may find the [s3-credentials tool](https://github.com/simonw/s3-credentials) useful for this.

  Run this command and paste in your credentials. You will need three values: the name of your S3 bucket, your Access key ID and your Secret access key.

      $ dogsheep-photos s3-auth

  This will create a file called `auth.json` in your current directory containing the required values. To save the file at a different path or filename, use the `--auth=myauth.json` option.

  ## Uploading photos

  Run this command to upload every photo in a specific directory to your S3 bucket:

      $ dogsheep-photos upload photos.db \
          ~/Pictures/Photos\ Library.photoslibrary/original

  The command will only upload photos that have not yet been uploaded, based on their sha256 hash.

  `photos.db` will be created with an `uploads` table containing details of which files were uploaded.

  To see what the command would do without uploading any files, use the `--dry-run` option.

  The sha256 hash of the photo contents will be used as the name of the file in the bucket, with an extension matching the type of file. This is an implementation of the [Content addressable storage](https://en.wikipedia.org/wiki/Content-addressable_storage) pattern.

  ## Importing Apple Photos metadata

  The `apple-photos` command imports metadata from your Apple Photos library.

      $ photo-to-sqlite apple-photos photos.db

  Imported metadata includes places, people, albums, quality scores and machine learning labels for the photo contents.

  ## Creating a subset database

  You can create a new, subset database of photos using the `create-subset` command.

  This is useful for creating a shareable SQLite database that only contains metadata for a selected set of photos.

  Since photo metadata contains latitude and longitude you may not want to share a database that includes photos taken at your home address.

  `create-subset` takes three arguments: an existing database file created using the `apple-photos` command, the name of the new, shareable database file you would like to create and a SQL query that returns the `sha256` hash values of the photos you would like to include in that database.

  For example, here's how to create a shareable database of just the photos that have been added to albums containing the word "Public":

      $ dogsheep-photos create-subset \
          photos.db \
          public.db \
          "select sha256 from apple_photos where albums like '%Public%'"

  ## Serving photos locally with datasette-media

  If you don't want to upload your photos to S3 but you still want to browse them using Datasette you can do so using the [datasette-media](https://github.com/simonw/datasette-media) plugin. This plugin adds the ability to serve images and other static files directly from disk, configured using a SQL query.

  To use it, first install Datasette and the plugin:

      $ pip install datasette datasette-media

  If any of your photos are `.HEIC` images taken by an iPhone you should also install the optional `pyheif` dependency:

      $ pip install pyheif

  Now create a `metadata.yaml` file configuring the plugin:

  ```yaml
  plugins:
    datasette-media:
      thumbnail:
        sql: |-
          select path as filepath, 200 as resize_height from apple_photos where uuid = :key
      large:
        sql: |-
          select path as filepath, 1024 as resize_height from apple_photos where uuid = :key
  ```
  This will configure two URL endpoints - one for 200 pixel high thumbnails and one for 1024 pixel high larger images.

  Create your `photos.db` database using the `apple-photos` command, then run Datasette like this:

      $ datasette -m metadata.yaml

  Your photos will be served on URLs that look like this:

      http://127.0.0.1:8001/-/media/thumbnail/F4469918-13F3-43D8-9EC1-734C0E6B60AD
      http://127.0.0.1:8001/-/media/large/F4469918-13F3-43D8-9EC1-734C0E6B60AD

  You can find the UUIDs for use in these URLs by running `select uuid from photos_with_apple_metadata`.

  ### Displaying images using datasette-json-html

  If you are using `datasette-media` to serve photos you can include images directly in Datasette query results using the [datasette-json-html](https://github.com/simonw/datasette-json-html) plugin.

  Run `pip install datasette-json-html` to install the plugin, then use queries like this to view your images:

  ```sql
  select
      json_object(
          'img_src',
          '/-/media/thumbnail/' || uuid
      ) as photo,
      uuid,
      date
  from
      apple_photos
  order by
      date desc
  limit 10;
  ```
  The `photo` column returned by this query should render as image tags that display the correct images.

  ### Displaying images using custom template pages

  Datasette's [custom pages](https://datasette.readthedocs.io/en/stable/custom_templates.html#custom-pages) feature lets you create custom pages for a Datasette instance by dropping HTML templates into a `templates/pages` directory and then running Datasette using `datasette --template-dir=templates/`.

  You can combine that ability with the [datasette-template-sql](https://github.com/simonw/datasette-template-sql) plugin to create custom template pages that directly display photos served by `datasette-media`.

  Install the plugin using `pip install datasette-template-sql`.

  Create a `templates/pages` folder and add the following files:

  `recent-photos.html`
  ```html+jinja
  <h1>Recent photos</h1>

  <div>
  {% for photo in sql("select * from apple_photos order by date desc limit 20") %}
      <img src="/-/media/photo/{{ photo['uuid'] }}">
  {% endfor %}
  </div>
  ```
  `random-photos.html`
  ```html+jinja
  <h1>Random photos</h1>

  <div>
  {% for photo in sql("with foo as (select * from apple_photos order by date desc limit 5000) select * from foo order by random() limit 20") %}
      <img src="/-/media/photo/{{ photo['uuid'] }}">
  {% endfor %}
  </div>
  ```
  Now run Datasette like this:

      $ datasette photos.db -m metadata.yaml --template-dir=templates/

  Visiting `http://localhost:8001/recent-photos` will display 20 recent photos. Visiting `http://localhost:8001/random-photos` will display 20 photos randomly selected from your 5,000 most recent.

readme_html:
  <div id="readme" class="md" data-path="README.md"><article class="markdown-body entry-content container-lg" itemprop="text"><div class="markdown-heading" dir="auto"><h1 class="heading-element" dir="auto">dogsheep-photos</h1><a id="user-content-dogsheep-photos" class="anchor" aria-label="Permalink: dogsheep-photos" href="#dogsheep-photos"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto"><a href="https://pypi.org/project/dogsheep-photos/" rel="nofollow"><img src="https://img.shields.io/pypi/v/dogsheep-photos.svg" alt="PyPI" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/b431261b4b7b1762ea89cefaa326e20a27840e80338f455ae95b0a39ee445efb/68747470733a2f2f696d672e736869656c64732e696f2f707970692f762f646f6773686565702d70686f746f732e737667"></a>
  <a href="https://github.com/dogsheep/dogsheep-photos/releases"><img src="https://img.shields.io/github/v/release/dogsheep/dogsheep-photos?include_prereleases&label=changelog" alt="Changelog" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/83c0f939dc71c6aa99dd6fb530f84f4b8300841722948b83818eec9a25aee61d/68747470733a2f2f696d672e736869656c64732e696f2f6769746875622f762f72656c656173652f646f6773686565702f646f6773686565702d70686f746f733f696e636c7564655f70726572656c6561736573266c6162656c3d6368616e67656c6f67"></a>
  <a href="https://circleci.com/gh/dogsheep/dogsheep-photos" rel="nofollow"><img src="https://circleci.com/gh/dogsheep/dogsheep-photos.svg?style=svg" alt="CircleCI" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/0e29645be05a7354e3e16bbc97ab30c7edbbacde1b7de4add00828d4d7e2dc1c/68747470733a2f2f636972636c6563692e636f6d2f67682f646f6773686565702f646f6773686565702d70686f746f732e7376673f7374796c653d737667"></a>
  <a href="https://github.com/dogsheep/dogsheep-photos/blob/master/LICENSE"><img src="https://img.shields.io/badge/license-Apache%202.0-blue.svg" alt="License" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/c355f200ea90fddaa407b6eaab303663a669248ea3ca7b1fcf77dbe04ff5f48c/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6c6963656e73652d417061636865253230322e302d626c75652e737667"></a></p>
  <p dir="auto">Save details of your photos to a SQLite database and upload them to S3.</p>
  <p dir="auto">See <a href="https://simonwillison.net/2020/May/21/apple-photos-sqlite/" rel="nofollow">Using SQL to find my best photo of a pelican according to Apple Photos</a> for background information on this project.</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">What these tools do</h2><a id="user-content-what-these-tools-do" class="anchor" aria-label="Permalink: What these tools do" href="#what-these-tools-do"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">These tools are a work-in-progress mechanism for taking full ownership of your photos. The core idea is to help implement the following:</p>
  <ul dir="auto">
  <li>Every photo you have taken lives in a single, private Amazon S3 bucket</li>
  <li>You have a single SQLite database file which stores metadata about those photos - potentially pulled from multiple different places. This may include EXIF data, Apple Photos, the results of running machine learning APIs against photos and much more besides.</li>
  <li>You can then use <a href="https://github.com/simonw/datasette">Datasette</a> to explore your own photos.</li>
  </ul>
  <p dir="auto">I'm a heavy user of Apple Photos so the initial releases of this tool will have a bias towards that, but ideally I would like a subset of these tools to be useful to people no matter which core photo solution they are using.</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Installation</h2><a id="user-content-installation" class="anchor" aria-label="Permalink: Installation" href="#installation"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ pip install dogsheep-photos"><pre class="notranslate"><code>$ pip install dogsheep-photos
  </code></pre></div>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Authentication (if using S3)</h2><a id="user-content-authentication-if-using-s3" class="anchor" aria-label="Permalink: Authentication (if using S3)" href="#authentication-if-using-s3"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">If you want to use S3 to store your photos, you will need to first create S3 credentials for a new, dedicated bucket.</p>
  <p dir="auto">You may find the <a href="https://github.com/simonw/s3-credentials">s3-credentials tool</a> useful for this.</p>
  <p dir="auto">Run this command and paste in your credentials. You will need three values: the name of your S3 bucket, your Access key ID and your Secret access key.</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ dogsheep-photos s3-auth"><pre class="notranslate"><code>$ dogsheep-photos s3-auth
  </code></pre></div>
  <p dir="auto">This will create a file called <code>auth.json</code> in your current directory containing the required values. To save the file at a different path or filename, use the <code>--auth=myauth.json</code> option.</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Uploading photos</h2><a id="user-content-uploading-photos" class="anchor" aria-label="Permalink: Uploading photos" href="#uploading-photos"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">Run this command to upload every photo in a specific directory to your S3 bucket:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ dogsheep-photos upload photos.db \
      ~/Pictures/Photos\ Library.photoslibrary/original"><pre class="notranslate"><code>$ dogsheep-photos upload photos.db \
      ~/Pictures/Photos\ Library.photoslibrary/original
  </code></pre></div>
  <p dir="auto">The command will only upload photos that have not yet been uploaded, based on their sha256 hash.</p>
  <p dir="auto"><code>photos.db</code> will be created with an <code>uploads</code> table containing details of which files were uploaded.</p>
  <p dir="auto">To see what the command would do without uploading any files, use the <code>--dry-run</code> option.</p>
  <p dir="auto">The sha256 hash of the photo contents will be used as the name of the file in the bucket, with an extension matching the type of file. This is an implementation of the <a href="https://en.wikipedia.org/wiki/Content-addressable_storage" rel="nofollow">Content addressable storage</a> pattern.</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Importing Apple Photos metadata</h2><a id="user-content-importing-apple-photos-metadata" class="anchor" aria-label="Permalink: Importing Apple Photos metadata" href="#importing-apple-photos-metadata"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">The <code>apple-photos</code> command imports metadata from your Apple Photos library.</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ photo-to-sqlite apple-photos photos.db"><pre class="notranslate"><code>$ photo-to-sqlite apple-photos photos.db
  </code></pre></div>
  <p dir="auto">Imported metadata includes places, people, albums, quality scores and machine learning labels for the photo contents.</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Creating a subset database</h2><a id="user-content-creating-a-subset-database" class="anchor" aria-label="Permalink: Creating a subset database" href="#creating-a-subset-database"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">You can create a new, subset database of photos using the <code>create-subset</code> command.</p>
  <p dir="auto">This is useful for creating a shareable SQLite database that only contains metadata for a selected set of photos.</p>
  <p dir="auto">Since photo metadata contains latitude and longitude you may not want to share a database that includes photos taken at your home address.</p>
  <p dir="auto"><code>create-subset</code> takes three arguments: an existing database file created using the <code>apple-photos</code> command, the name of the new, shareable database file you would like to create and a SQL query that returns the <code>sha256</code> hash values of the photos you would like to include in that database.</p>
  <p dir="auto">For example, here's how to create a shareable database of just the photos that have been added to albums containing the word "Public":</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ dogsheep-photos create-subset \
      photos.db \
      public.db \
      "select sha256 from apple_photos where albums like '%Public%'""><pre class="notranslate"><code>$ dogsheep-photos create-subset \
      photos.db \
      public.db \
      "select sha256 from apple_photos where albums like '%Public%'"
  </code></pre></div>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Serving photos locally with datasette-media</h2><a id="user-content-serving-photos-locally-with-datasette-media" class="anchor" aria-label="Permalink: Serving photos locally with datasette-media" href="#serving-photos-locally-with-datasette-media"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">If you don't want to upload your photos to S3 but you still want to browse them using Datasette you can do so using the <a href="https://github.com/simonw/datasette-media">datasette-media</a> plugin. This plugin adds the ability to serve images and other static files directly from disk, configured using a SQL query.</p>
  <p dir="auto">To use it, first install Datasette and the plugin:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ pip install datasette datasette-media"><pre class="notranslate"><code>$ pip install datasette datasette-media
  </code></pre></div>
  <p dir="auto">If any of your photos are <code>.HEIC</code> images taken by an iPhone you should also install the optional <code>pyheif</code> dependency:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ pip install pyheif"><pre class="notranslate"><code>$ pip install pyheif
  </code></pre></div>
  <p dir="auto">Now create a <code>metadata.yaml</code> file configuring the plugin:</p>
  <div class="highlight highlight-source-yaml notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="plugins:
    datasette-media:
      thumbnail:
        sql: |-
          select path as filepath, 200 as resize_height from apple_photos where uuid = :key
      large:
        sql: |-
          select path as filepath, 1024 as resize_height from apple_photos where uuid = :key"><pre><span class="pl-ent">plugins</span>:
    <span class="pl-ent">datasette-media</span>:
      <span class="pl-ent">thumbnail</span>:
        <span class="pl-ent">sql</span>: <span class="pl-s">|-</span>
  <span class="pl-s">        select path as filepath, 200 as resize_height from apple_photos where uuid = :key</span>
  <span class="pl-s"></span>    <span class="pl-ent">large</span>:
        <span class="pl-ent">sql</span>: <span class="pl-s">|-</span>
  <span class="pl-s">        select path as filepath, 1024 as resize_height from apple_photos where uuid = :key</span></pre></div>
  <p dir="auto">This will configure two URL endpoints - one for 200 pixel high thumbnails and one for 1024 pixel high larger images.</p>
  <p dir="auto">Create your <code>photos.db</code> database using the <code>apple-photos</code> command, then run Datasette like this:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ datasette -m metadata.yaml"><pre class="notranslate"><code>$ datasette -m metadata.yaml
  </code></pre></div>
  <p dir="auto">Your photos will be served on URLs that look like this:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="http://127.0.0.1:8001/-/media/thumbnail/F4469918-13F3-43D8-9EC1-734C0E6B60AD
  http://127.0.0.1:8001/-/media/large/F4469918-13F3-43D8-9EC1-734C0E6B60AD"><pre class="notranslate"><code>http://127.0.0.1:8001/-/media/thumbnail/F4469918-13F3-43D8-9EC1-734C0E6B60AD
  http://127.0.0.1:8001/-/media/large/F4469918-13F3-43D8-9EC1-734C0E6B60AD
  </code></pre></div>
  <p dir="auto">You can find the UUIDs for use in these URLs by running <code>select uuid from photos_with_apple_metadata</code>.</p>
  <div class="markdown-heading" dir="auto"><h3 class="heading-element" dir="auto">Displaying images using datasette-json-html</h3><a id="user-content-displaying-images-using-datasette-json-html" class="anchor" aria-label="Permalink: Displaying images using datasette-json-html" href="#displaying-images-using-datasette-json-html"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">If you are using <code>datasette-media</code> to serve photos you can include images directly in Datasette query results using the <a href="https://github.com/simonw/datasette-json-html">datasette-json-html</a> plugin.</p>
  <p dir="auto">Run <code>pip install datasette-json-html</code> to install the plugin, then use queries like this to view your images:</p>
  <div class="highlight highlight-source-sql notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="select
      json_object(
          'img_src',
          '/-/media/thumbnail/' || uuid
      ) as photo,
      uuid,
      date
  from
      apple_photos
  order by
      date desc
  limit 10;"><pre><span class="pl-k">select</span>
      json_object(
          <span class="pl-s"><span class="pl-pds">'</span>img_src<span class="pl-pds">'</span></span>,
          <span class="pl-s"><span class="pl-pds">'</span>/-/media/thumbnail/<span class="pl-pds">'</span></span> <span class="pl-k">||</span> uuid
      ) <span class="pl-k">as</span> photo,
      uuid,
      <span class="pl-k">date</span>
  <span class="pl-k">from</span>
      apple_photos
  <span class="pl-k">order by</span>
      <span class="pl-k">date</span> <span class="pl-k">desc</span>
  <span class="pl-k">limit</span> <span class="pl-c1">10</span>;</pre></div>
  <p dir="auto">The <code>photo</code> column returned by this query should render as image tags that display the correct images.</p>
  <div class="markdown-heading" dir="auto"><h3 class="heading-element" dir="auto">Displaying images using custom template pages</h3><a id="user-content-displaying-images-using-custom-template-pages" class="anchor" aria-label="Permalink: Displaying images using custom template pages" href="#displaying-images-using-custom-template-pages"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">Datasette's <a href="https://datasette.readthedocs.io/en/stable/custom_templates.html#custom-pages" rel="nofollow">custom pages</a> feature lets you create custom pages for a Datasette instance by dropping HTML templates into a <code>templates/pages</code> directory and then running Datasette using <code>datasette --template-dir=templates/</code>.</p>
  <p dir="auto">You can combine that ability with the <a href="https://github.com/simonw/datasette-template-sql">datasette-template-sql</a> plugin to create custom template pages that directly display photos served by <code>datasette-media</code>.</p>
  <p dir="auto">Install the plugin using <code>pip install datasette-template-sql</code>.</p>
  <p dir="auto">Create a <code>templates/pages</code> folder and add the following files:</p>
  <p dir="auto"><code>recent-photos.html</code></p>
  <div class="highlight highlight-text-html-django notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="<h1>Recent photos</h1>

  <div>
  {% for photo in sql("select * from apple_photos order by date desc limit 20") %}
      <img src="/-/media/photo/{{ photo['uuid'] }}">
  {% endfor %}
  </div>"><pre><<span class="pl-ent">h1</span>>Recent photos</<span class="pl-ent">h1</span>>

  <<span class="pl-ent">div</span>>
  <span class="pl-e">{%</span> <span class="pl-k">for</span> <span class="pl-s">photo</span> <span class="pl-k">in</span> <span class="pl-s">sql</span>(<span class="pl-s">"select * from apple_photos order by date desc limit 20"</span>) <span class="pl-e">%}</span>
      <<span class="pl-ent">img</span> <span class="pl-e">src</span>=<span class="pl-s"><span class="pl-pds">"</span>/-/media/photo/{{ photo['uuid'] }}<span class="pl-pds">"</span></span>>
  <span class="pl-e">{%</span> <span class="pl-k">endfor</span> <span class="pl-e">%}</span>
  </<span class="pl-ent">div</span>></pre></div>
  <p dir="auto"><code>random-photos.html</code></p>
  <div class="highlight highlight-text-html-django notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="<h1>Random photos</h1>

  <div>
  {% for photo in sql("with foo as (select * from apple_photos order by date desc limit 5000) select * from foo order by random() limit 20") %}
      <img src="/-/media/photo/{{ photo['uuid'] }}">
  {% endfor %}
  </div>"><pre><<span class="pl-ent">h1</span>>Random photos</<span class="pl-ent">h1</span>>

  <<span class="pl-ent">div</span>>
  <span class="pl-e">{%</span> <span class="pl-k">for</span> <span class="pl-s">photo</span> <span class="pl-k">in</span> <span class="pl-s">sql</span>(<span class="pl-s">"with foo as (select * from apple_photos order by date desc limit 5000) select * from foo order by random() limit 20"</span>) <span class="pl-e">%}</span>
      <<span class="pl-ent">img</span> <span class="pl-e">src</span>=<span class="pl-s"><span class="pl-pds">"</span>/-/media/photo/{{ photo['uuid'] }}<span class="pl-pds">"</span></span>>
  <span class="pl-e">{%</span> <span class="pl-k">endfor</span> <span class="pl-e">%}</span>
  </<span class="pl-ent">div</span>></pre></div>
  <p dir="auto">Now run Datasette like this:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ datasette photos.db -m metadata.yaml --template-dir=templates/"><pre class="notranslate"><code>$ datasette photos.db -m metadata.yaml --template-dir=templates/
  </code></pre></div>
  <p dir="auto">Visiting <code>http://localhost:8001/recent-photos</code> will display 20 recent photos. Visiting <code>http://localhost:8001/random-photos</code> will display 20 photos randomly selected from your 5,000 most recent.</p>
  </article></div>


node_id: MDEwOlJlcG9zaXRvcnkyNjE2MzQ4MDc=
name: datasette-media
full_name: simonw/datasette-media
html_url: https://github.com/simonw/datasette-media
description: Datasette plugin for serving media based on a SQL query
created_at: 2020-05-06T02:42:57Z
updated_at: 2025-04-06T09:24:40Z
pushed_at: 2022-12-13T20:03:51Z
language: Python
license: apache-2.0
topics: ["datasette", "datasette-io", "datasette-plugin"]
visibility: public
default_branch: main
permissions: {"admin": false, "maintain": false, "push": false, "triage": false, "pull": true}
readme:
  # datasette-media

  [![PyPI](https://img.shields.io/pypi/v/datasette-media.svg)](https://pypi.org/project/datasette-media/)
  [![Changelog](https://img.shields.io/github/v/release/simonw/datasette-media?include_prereleases&label=changelog)](https://github.com/simonw/datasette-media/releases)
  [![Tests](https://github.com/simonw/datasette-media/workflows/Test/badge.svg)](https://github.com/simonw/datasette-media/actions?query=workflow%3ATest)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/datasette-media/blob/main/LICENSE)

  Datasette plugin for serving media based on a SQL query.

  Use this when you have a database table containing references to files on disk - or binary content stored in BLOB columns - that you would like to be able to serve to your users.

  ## Installation

  Install this plugin in the same environment as Datasette.

      $ pip install datasette-media

  ### HEIC image support

  Modern iPhones save their photos using the [HEIC image format](https://en.wikipedia.org/wiki/High_Efficiency_Image_File_Format). Processing these images requires an additional dependency, [pyheif](https://pypi.org/project/pyheif/). You can include this dependency by running:

      $ pip install datasette-media[heif]

  ## Usage

  You can use this plugin to configure Datasette to serve static media based on SQL queries to an underlying database table.

  Media will be served from URLs that start with `/-/media/`. The full URL to each media asset will look like this:

      /-/media/type-of-media/media-key

  `type-of-media` will correspond to a configured SQL query, and might be something like `photo`. `media-key` will be an identifier that is used as part of the underlying SQL query to find which file should be served.

  ### Serving static files from disk

  The following ``metadata.json`` configuration will cause this plugin to serve files from disk, based on queries to a database table called `apple_photos`.

  ```json
  {
      "plugins": {
          "datasette-media": {
              "photo": {
                  "sql": "select filepath from apple_photos where uuid=:key"
              }
          }
      }
  }
  ```

  A request to `/-/media/photo/CF972D33-5324-44F2-8DAE-22CB3182CD31` will execute the following SQL query:

  ```sql
  select filepath from apple_photos where uuid=:key
  ```

  The value from the URL -  in this case `CF972D33-5324-44F2-8DAE-22CB3182CD31` - will be passed as the `:key` parameter to the query.

  The query returns a `filepath` value that has been read from the table. The plugin will then read that file from disk and serve it in response to the request.

  SQL queries default to running against the first connected database. You can specify a different database to execute the query against using `"database": "name_of_db"`. To execute against `photos.db`, use this:

  ```json
  {
      "plugins": {
          "datasette-media": {
              "photo": {
                  "sql": "select filepath from apple_photos where uuid=:key",
                  "database": "photos"
              }
          }
      }
  }
  ```

  See [dogsheep-photos](https://github.com/dogsheep/dogsheep-photos) for an example of an application that can benefit from this plugin.

  ### Serving binary content from BLOB columns

  If your SQL query returns a `content` column, this will be served directly to the user:

  ```json
  {
      "plugins": {
          "datasette-media": {
              "photo": {
                  "sql": "select thumbnail as content from photos where uuid=:key",
                  "database": "thumbs"
              }
          }
      }
  }
  ```

  You can also return a `content_type` column which will be used as the `Content-Type` header served to the user:

  ```json
  {
      "plugins": {
          "datasette-media": {
              "photo": {
                  "sql": "select body as content, 'text/html;charset=utf-8' as content_type from documents where id=:key",
                  "database": "documents"
              }
          }
      }
  }
  ```

  If you do not specify a `content_type` the default of `application/octet-stream` will be used.

  ### Serving content proxied from a URL

  To serve content that is itself fetched from elsewhere, return a `content_url` column. This can be particularly useful when combined with the ability to resize images (described in the next section).

  ```json
  {
      "plugins": {
          "datasette-media": {
              "photos": {
                  "sql": "select photo_url as content_url from photos where id=:key",
                  "database": "photos",
                  "enable_transform": true
              }
          }
      }
  }
  ```

  Now you can access resized versions of images from that URL like so:

      /-/media/photos/13?w=200

  ### Setting a download file name

  The `content_filename` column can be returned to force browsers to download the content using a specific file name.

  ```json
  {
      "plugins": {
          "datasette-media": {
              "hello": {
                  "sql": "select 'Hello ' || :key as content, 'hello.txt' as content_filename"
              }
          }
      }
  }
  ```

  Visiting `/-/media/hello/Groot` will cause your browser to download a file called `hello.txt` containing the text `Hello Groot`.

  ### Resizing or transforming images

  Your SQL query can specify that an image should be resized and/or converted to another format by returning additional columns. All three are optional.

  * `resize_width` - the width to resize the image to
  * `resize_width` - the height to resize the image to
  * `output_format` - the output format to use (e.g. `jpeg` or `png`) - any output format [supported by Pillow](https://pillow.readthedocs.io/en/stable/handbook/image-file-formats.html) is allowed here.

  If you specify one but not the other of `resize_width` or `resize_height` the unspecified one will be calculated automatically to maintain the aspect ratio of the image.

  Here's an example configuration that will resize all images to be JPEGs that are 200 pixels in height:

  ```json
  {
      "plugins": {
          "datasette-media": {
              "photo": {
                  "sql": "select filepath, 200 as resize_height, 'jpeg' as output_format from apple_photos where uuid=:key",
                  "database": "photos"
              }
          }
      }
  }
  ```

  If you enable the `enable_transform` configuration option you can instead specify transform parameters at runtime using querystring parameters. For example:

  - `/-/media/photo/CF972D33?w=200` to resize to a fixed width
  - `/-/media/photo/CF972D33?h=200` to resize to a fixed height
  - `/-/media/photo/CF972D33?format=jpeg` to convert to JPEG

  That option is added like so:

  ```json
  {
      "plugins": {
          "datasette-media": {
              "photo": {
                  "sql": "select filepath from apple_photos where uuid=:key",
                  "database": "photos",
                  "enable_transform": true
              }
          }
      }
  }
  ```

  The maximum allowed height or width is 4000 pixels. You can change this limit using the `"max_width_height"` option:

  ```json
  {
      "plugins": {
          "datasette-media": {
              "photo": {
                  "sql": "select filepath from apple_photos where uuid=:key",
                  "database": "photos",
                  "enable_transform": true,
                  "max_width_height": 1000
              }
          }
      }
  }
  ```

  ## Configuration

  In addition to the different named content types, the following special plugin configuration setting is available:

  - `transform_threads` - number of threads to use for running transformations (e.g. resizing). Defaults to 4.

  This can be used like this:

  ```json
  {
      "plugins": {
          "datasette-media": {
              "photo": {
                  "sql": "select filepath from apple_photos where uuid=:key",
                  "database": "photos"
              },
              "transform_threads": 8
          }
      }
  }
  ```

readme_html:
  <div id="readme" class="md" data-path="README.md"><article class="markdown-body entry-content container-lg" itemprop="text"><div class="markdown-heading" dir="auto"><h1 class="heading-element" dir="auto">datasette-media</h1><a id="user-content-datasette-media" class="anchor" aria-label="Permalink: datasette-media" href="#datasette-media"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto"><a href="https://pypi.org/project/datasette-media/" rel="nofollow"><img src="https://img.shields.io/pypi/v/datasette-media.svg" alt="PyPI" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/f4799178424abd16231e47c62e422c06665acfe27a900b59767dd53e27b759c7/68747470733a2f2f696d672e736869656c64732e696f2f707970692f762f6461746173657474652d6d656469612e737667"></a>
  <a href="https://github.com/simonw/datasette-media/releases"><img src="https://img.shields.io/github/v/release/simonw/datasette-media?include_prereleases&label=changelog" alt="Changelog" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/f0e1ff55363acb140c4762185071b41f0b2d173a0c14ca480bf210f0b8ab3981/68747470733a2f2f696d672e736869656c64732e696f2f6769746875622f762f72656c656173652f73696d6f6e772f6461746173657474652d6d656469613f696e636c7564655f70726572656c6561736573266c6162656c3d6368616e67656c6f67"></a>
  <a href="https://github.com/simonw/datasette-media/actions?query=workflow%3ATest"><img src="https://github.com/simonw/datasette-media/workflows/Test/badge.svg" alt="Tests" style="max-width: 100%;"></a>
  <a href="https://github.com/simonw/datasette-media/blob/main/LICENSE"><img src="https://img.shields.io/badge/license-Apache%202.0-blue.svg" alt="License" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/c355f200ea90fddaa407b6eaab303663a669248ea3ca7b1fcf77dbe04ff5f48c/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6c6963656e73652d417061636865253230322e302d626c75652e737667"></a></p>
  <p dir="auto">Datasette plugin for serving media based on a SQL query.</p>
  <p dir="auto">Use this when you have a database table containing references to files on disk - or binary content stored in BLOB columns - that you would like to be able to serve to your users.</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Installation</h2><a id="user-content-installation" class="anchor" aria-label="Permalink: Installation" href="#installation"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">Install this plugin in the same environment as Datasette.</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ pip install datasette-media"><pre class="notranslate"><code>$ pip install datasette-media
  </code></pre></div>
  <div class="markdown-heading" dir="auto"><h3 class="heading-element" dir="auto">HEIC image support</h3><a id="user-content-heic-image-support" class="anchor" aria-label="Permalink: HEIC image support" href="#heic-image-support"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">Modern iPhones save their photos using the <a href="https://en.wikipedia.org/wiki/High_Efficiency_Image_File_Format" rel="nofollow">HEIC image format</a>. Processing these images requires an additional dependency, <a href="https://pypi.org/project/pyheif/" rel="nofollow">pyheif</a>. You can include this dependency by running:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ pip install datasette-media[heif]"><pre class="notranslate"><code>$ pip install datasette-media[heif]
  </code></pre></div>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Usage</h2><a id="user-content-usage" class="anchor" aria-label="Permalink: Usage" href="#usage"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">You can use this plugin to configure Datasette to serve static media based on SQL queries to an underlying database table.</p>
  <p dir="auto">Media will be served from URLs that start with <code>/-/media/</code>. The full URL to each media asset will look like this:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="/-/media/type-of-media/media-key"><pre class="notranslate"><code>/-/media/type-of-media/media-key
  </code></pre></div>
  <p dir="auto"><code>type-of-media</code> will correspond to a configured SQL query, and might be something like <code>photo</code>. <code>media-key</code> will be an identifier that is used as part of the underlying SQL query to find which file should be served.</p>
  <div class="markdown-heading" dir="auto"><h3 class="heading-element" dir="auto">Serving static files from disk</h3><a id="user-content-serving-static-files-from-disk" class="anchor" aria-label="Permalink: Serving static files from disk" href="#serving-static-files-from-disk"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">The following <code>metadata.json</code> configuration will cause this plugin to serve files from disk, based on queries to a database table called <code>apple_photos</code>.</p>
  <div class="highlight highlight-source-json notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="{
      "plugins": {
          "datasette-media": {
              "photo": {
                  "sql": "select filepath from apple_photos where uuid=:key"
              }
          }
      }
  }"><pre>{
      <span class="pl-ent">"plugins"</span>: {
          <span class="pl-ent">"datasette-media"</span>: {
              <span class="pl-ent">"photo"</span>: {
                  <span class="pl-ent">"sql"</span>: <span class="pl-s"><span class="pl-pds">"</span>select filepath from apple_photos where uuid=:key<span class="pl-pds">"</span></span>
              }
          }
      }
  }</pre></div>
  <p dir="auto">A request to <code>/-/media/photo/CF972D33-5324-44F2-8DAE-22CB3182CD31</code> will execute the following SQL query:</p>
  <div class="highlight highlight-source-sql notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="select filepath from apple_photos where uuid=:key"><pre><span class="pl-k">select</span> filepath <span class="pl-k">from</span> apple_photos <span class="pl-k">where</span> uuid<span class="pl-k">=</span>:key</pre></div>
  <p dir="auto">The value from the URL -  in this case <code>CF972D33-5324-44F2-8DAE-22CB3182CD31</code> - will be passed as the <code>:key</code> parameter to the query.</p>
  <p dir="auto">The query returns a <code>filepath</code> value that has been read from the table. The plugin will then read that file from disk and serve it in response to the request.</p>
  <p dir="auto">SQL queries default to running against the first connected database. You can specify a different database to execute the query against using <code>"database": "name_of_db"</code>. To execute against <code>photos.db</code>, use this:</p>
  <div class="highlight highlight-source-json notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="{
      "plugins": {
          "datasette-media": {
              "photo": {
                  "sql": "select filepath from apple_photos where uuid=:key",
                  "database": "photos"
              }
          }
      }
  }"><pre>{
      <span class="pl-ent">"plugins"</span>: {
          <span class="pl-ent">"datasette-media"</span>: {
              <span class="pl-ent">"photo"</span>: {
                  <span class="pl-ent">"sql"</span>: <span class="pl-s"><span class="pl-pds">"</span>select filepath from apple_photos where uuid=:key<span class="pl-pds">"</span></span>,
                  <span class="pl-ent">"database"</span>: <span class="pl-s"><span class="pl-pds">"</span>photos<span class="pl-pds">"</span></span>
              }
          }
      }
  }</pre></div>
  <p dir="auto">See <a href="https://github.com/dogsheep/dogsheep-photos">dogsheep-photos</a> for an example of an application that can benefit from this plugin.</p>
  <div class="markdown-heading" dir="auto"><h3 class="heading-element" dir="auto">Serving binary content from BLOB columns</h3><a id="user-content-serving-binary-content-from-blob-columns" class="anchor" aria-label="Permalink: Serving binary content from BLOB columns" href="#serving-binary-content-from-blob-columns"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">If your SQL query returns a <code>content</code> column, this will be served directly to the user:</p>
  <div class="highlight highlight-source-json notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="{
      "plugins": {
          "datasette-media": {
              "photo": {
                  "sql": "select thumbnail as content from photos where uuid=:key",
                  "database": "thumbs"
              }
          }
      }
  }"><pre>{
      <span class="pl-ent">"plugins"</span>: {
          <span class="pl-ent">"datasette-media"</span>: {
              <span class="pl-ent">"photo"</span>: {
                  <span class="pl-ent">"sql"</span>: <span class="pl-s"><span class="pl-pds">"</span>select thumbnail as content from photos where uuid=:key<span class="pl-pds">"</span></span>,
                  <span class="pl-ent">"database"</span>: <span class="pl-s"><span class="pl-pds">"</span>thumbs<span class="pl-pds">"</span></span>
              }
          }
      }
  }</pre></div>
  <p dir="auto">You can also return a <code>content_type</code> column which will be used as the <code>Content-Type</code> header served to the user:</p>
  <div class="highlight highlight-source-json notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="{
      "plugins": {
          "datasette-media": {
              "photo": {
                  "sql": "select body as content, 'text/html;charset=utf-8' as content_type from documents where id=:key",
                  "database": "documents"
              }
          }
      }
  }"><pre>{
      <span class="pl-ent">"plugins"</span>: {
          <span class="pl-ent">"datasette-media"</span>: {
              <span class="pl-ent">"photo"</span>: {
                  <span class="pl-ent">"sql"</span>: <span class="pl-s"><span class="pl-pds">"</span>select body as content, 'text/html;charset=utf-8' as content_type from documents where id=:key<span class="pl-pds">"</span></span>,
                  <span class="pl-ent">"database"</span>: <span class="pl-s"><span class="pl-pds">"</span>documents<span class="pl-pds">"</span></span>
              }
          }
      }
  }</pre></div>
  <p dir="auto">If you do not specify a <code>content_type</code> the default of <code>application/octet-stream</code> will be used.</p>
  <div class="markdown-heading" dir="auto"><h3 class="heading-element" dir="auto">Serving content proxied from a URL</h3><a id="user-content-serving-content-proxied-from-a-url" class="anchor" aria-label="Permalink: Serving content proxied from a URL" href="#serving-content-proxied-from-a-url"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">To serve content that is itself fetched from elsewhere, return a <code>content_url</code> column. This can be particularly useful when combined with the ability to resize images (described in the next section).</p>
  <div class="highlight highlight-source-json notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="{
      "plugins": {
          "datasette-media": {
              "photos": {
                  "sql": "select photo_url as content_url from photos where id=:key",
                  "database": "photos",
                  "enable_transform": true
              }
          }
      }
  }"><pre>{
      <span class="pl-ent">"plugins"</span>: {
          <span class="pl-ent">"datasette-media"</span>: {
              <span class="pl-ent">"photos"</span>: {
                  <span class="pl-ent">"sql"</span>: <span class="pl-s"><span class="pl-pds">"</span>select photo_url as content_url from photos where id=:key<span class="pl-pds">"</span></span>,
                  <span class="pl-ent">"database"</span>: <span class="pl-s"><span class="pl-pds">"</span>photos<span class="pl-pds">"</span></span>,
                  <span class="pl-ent">"enable_transform"</span>: <span class="pl-c1">true</span>
              }
          }
      }
  }</pre></div>
  <p dir="auto">Now you can access resized versions of images from that URL like so:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="/-/media/photos/13?w=200"><pre class="notranslate"><code>/-/media/photos/13?w=200
  </code></pre></div>
  <div class="markdown-heading" dir="auto"><h3 class="heading-element" dir="auto">Setting a download file name</h3><a id="user-content-setting-a-download-file-name" class="anchor" aria-label="Permalink: Setting a download file name" href="#setting-a-download-file-name"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">The <code>content_filename</code> column can be returned to force browsers to download the content using a specific file name.</p>
  <div class="highlight highlight-source-json notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="{
      "plugins": {
          "datasette-media": {
              "hello": {
                  "sql": "select 'Hello ' || :key as content, 'hello.txt' as content_filename"
              }
          }
      }
  }"><pre>{
      <span class="pl-ent">"plugins"</span>: {
          <span class="pl-ent">"datasette-media"</span>: {
              <span class="pl-ent">"hello"</span>: {
                  <span class="pl-ent">"sql"</span>: <span class="pl-s"><span class="pl-pds">"</span>select 'Hello ' || :key as content, 'hello.txt' as content_filename<span class="pl-pds">"</span></span>
              }
          }
      }
  }</pre></div>
  <p dir="auto">Visiting <code>/-/media/hello/Groot</code> will cause your browser to download a file called <code>hello.txt</code> containing the text <code>Hello Groot</code>.</p>
  <div class="markdown-heading" dir="auto"><h3 class="heading-element" dir="auto">Resizing or transforming images</h3><a id="user-content-resizing-or-transforming-images" class="anchor" aria-label="Permalink: Resizing or transforming images" href="#resizing-or-transforming-images"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">Your SQL query can specify that an image should be resized and/or converted to another format by returning additional columns. All three are optional.</p>
  <ul dir="auto">
  <li><code>resize_width</code> - the width to resize the image to</li>
  <li><code>resize_width</code> - the height to resize the image to</li>
  <li><code>output_format</code> - the output format to use (e.g. <code>jpeg</code> or <code>png</code>) - any output format <a href="https://pillow.readthedocs.io/en/stable/handbook/image-file-formats.html" rel="nofollow">supported by Pillow</a> is allowed here.</li>
  </ul>
  <p dir="auto">If you specify one but not the other of <code>resize_width</code> or <code>resize_height</code> the unspecified one will be calculated automatically to maintain the aspect ratio of the image.</p>
  <p dir="auto">Here's an example configuration that will resize all images to be JPEGs that are 200 pixels in height:</p>
  <div class="highlight highlight-source-json notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="{
      "plugins": {
          "datasette-media": {
              "photo": {
                  "sql": "select filepath, 200 as resize_height, 'jpeg' as output_format from apple_photos where uuid=:key",
                  "database": "photos"
              }
          }
      }
  }"><pre>{
      <span class="pl-ent">"plugins"</span>: {
          <span class="pl-ent">"datasette-media"</span>: {
              <span class="pl-ent">"photo"</span>: {
                  <span class="pl-ent">"sql"</span>: <span class="pl-s"><span class="pl-pds">"</span>select filepath, 200 as resize_height, 'jpeg' as output_format from apple_photos where uuid=:key<span class="pl-pds">"</span></span>,
                  <span class="pl-ent">"database"</span>: <span class="pl-s"><span class="pl-pds">"</span>photos<span class="pl-pds">"</span></span>
              }
          }
      }
  }</pre></div>
  <p dir="auto">If you enable the <code>enable_transform</code> configuration option you can instead specify transform parameters at runtime using querystring parameters. For example:</p>
  <ul dir="auto">
  <li><code>/-/media/photo/CF972D33?w=200</code> to resize to a fixed width</li>
  <li><code>/-/media/photo/CF972D33?h=200</code> to resize to a fixed height</li>
  <li><code>/-/media/photo/CF972D33?format=jpeg</code> to convert to JPEG</li>
  </ul>
  <p dir="auto">That option is added like so:</p>
  <div class="highlight highlight-source-json notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="{
      "plugins": {
          "datasette-media": {
              "photo": {
                  "sql": "select filepath from apple_photos where uuid=:key",
                  "database": "photos",
                  "enable_transform": true
              }
          }
      }
  }"><pre>{
      <span class="pl-ent">"plugins"</span>: {
          <span class="pl-ent">"datasette-media"</span>: {
              <span class="pl-ent">"photo"</span>: {
                  <span class="pl-ent">"sql"</span>: <span class="pl-s"><span class="pl-pds">"</span>select filepath from apple_photos where uuid=:key<span class="pl-pds">"</span></span>,
                  <span class="pl-ent">"database"</span>: <span class="pl-s"><span class="pl-pds">"</span>photos<span class="pl-pds">"</span></span>,
                  <span class="pl-ent">"enable_transform"</span>: <span class="pl-c1">true</span>
              }
          }
      }
  }</pre></div>
  <p dir="auto">The maximum allowed height or width is 4000 pixels. You can change this limit using the <code>"max_width_height"</code> option:</p>
  <div class="highlight highlight-source-json notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="{
      "plugins": {
          "datasette-media": {
              "photo": {
                  "sql": "select filepath from apple_photos where uuid=:key",
                  "database": "photos",
                  "enable_transform": true,
                  "max_width_height": 1000
              }
          }
      }
  }"><pre>{
      <span class="pl-ent">"plugins"</span>: {
          <span class="pl-ent">"datasette-media"</span>: {
              <span class="pl-ent">"photo"</span>: {
                  <span class="pl-ent">"sql"</span>: <span class="pl-s"><span class="pl-pds">"</span>select filepath from apple_photos where uuid=:key<span class="pl-pds">"</span></span>,
                  <span class="pl-ent">"database"</span>: <span class="pl-s"><span class="pl-pds">"</span>photos<span class="pl-pds">"</span></span>,
                  <span class="pl-ent">"enable_transform"</span>: <span class="pl-c1">true</span>,
                  <span class="pl-ent">"max_width_height"</span>: <span class="pl-c1">1000</span>
              }
          }
      }
  }</pre></div>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Configuration</h2><a id="user-content-configuration" class="anchor" aria-label="Permalink: Configuration" href="#configuration"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">In addition to the different named content types, the following special plugin configuration setting is available:</p>
  <ul dir="auto">
  <li><code>transform_threads</code> - number of threads to use for running transformations (e.g. resizing). Defaults to 4.</li>
  </ul>
  <p dir="auto">This can be used like this:</p>
  <div class="highlight highlight-source-json notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="{
      "plugins": {
          "datasette-media": {
              "photo": {
                  "sql": "select filepath from apple_photos where uuid=:key",
                  "database": "photos"
              },
              "transform_threads": 8
          }
      }
  }"><pre>{
      <span class="pl-ent">"plugins"</span>: {
          <span class="pl-ent">"datasette-media"</span>: {
              <span class="pl-ent">"photo"</span>: {
                  <span class="pl-ent">"sql"</span>: <span class="pl-s"><span class="pl-pds">"</span>select filepath from apple_photos where uuid=:key<span class="pl-pds">"</span></span>,
                  <span class="pl-ent">"database"</span>: <span class="pl-s"><span class="pl-pds">"</span>photos<span class="pl-pds">"</span></span>
              },
              <span class="pl-ent">"transform_threads"</span>: <span class="pl-c1">8</span>
          }
      }
  }</pre></div>
  </article></div>


node_id: MDEwOlJlcG9zaXRvcnkyNzE0MDg4OTU=
name: datasette-permissions-sql
full_name: simonw/datasette-permissions-sql
html_url: https://github.com/simonw/datasette-permissions-sql
description: Datasette plugin for configuring permission checks using SQL queries
created_at: 2020-06-10T23:48:13Z
updated_at: 2025-02-28T21:23:18Z
pushed_at: 2020-06-12T07:06:15Z
language: Python
license: apache-2.0
topics: ["datasette", "datasette-io", "datasette-plugin"]
visibility: public
default_branch: master
permissions: {"admin": false, "maintain": false, "push": false, "triage": false, "pull": true}
readme:
  # datasette-permissions-sql

  [![PyPI](https://img.shields.io/pypi/v/datasette-permissions-sql.svg)](https://pypi.org/project/datasette-permissions-sql/)
  [![CircleCI](https://circleci.com/gh/simonw/datasette-permissions-sql.svg?style=svg)](https://circleci.com/gh/simonw/datasette-permissions-sql)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/datasette-permissions-sql/blob/master/LICENSE)

  Datasette plugin for configuring permission checks using SQL queries

  ## Installation

  Install this plugin in the same environment as Datasette.

      $ pip install datasette-permissions-sql

  ## Usage

  First, read up on how Datasette's [authentication and permissions system](https://datasette.readthedocs.io/en/latest/authentication.html) works.

  This plugin lets you define rules containing SQL queries that are executed to see if the currently authenticated actor has permission to perform certain actions.

  Consider a canned query which authenticated users should only be able to execute if a row in the `users` table says that they are a member of staff.

  That `users` table in the `mydatabase.db` database could look like this:

  | id | username | is_staff |
  |--|--------|--------|
  | 1 | cleopaws | 0 |
  | 2 | simon | 1 |

  Authenticated users have an `actor` that looks like this:

  ```json
  {
      "id": 2,
      "username": "simon"
  }
  ```

  To configure the canned query to only be executable by staff users, add the following to your `metadata.json`:

  ```json
  {
      "plugins": {
          "datasette-permissions-sql": [
              {
                  "action": "view-query",
                  "resource": ["mydatabase", "promote_to_staff"],
                  "sql": "SELECT * FROM users WHERE is_staff = 1 AND id = :actor_id"
              }
          ]
      },
      "databases": {
          "mydatabase": {
              "queries": {
                  "promote_to_staff": {
                      "sql": "UPDATE users SET is is_staff=1 WHERE id=:id",
                      "write": true
                  }
              }
          }
      }
  }
  ```

  The `"datasette-permissions-sql"` key is a list of rules. Each of those rules has the following shape:

  ```json
  {
      "action": "name-of-action",
      "resource": ["resource identifier to run this on"],
      "sql": "SQL query to execute",
      "database": "mydatabase"
  }
  ```

  Both `"action"` and `"resource"` are optional. If present, the SQL query will only be executed on permission checks that match the action and, if present, the resource indicators.

  `"database"` is also optional: it specifies the named database that the query should be executed against. If it is not present the first connected database will be used.

  The Datasette documentation includes a [list of built-in permissions](https://datasette.readthedocs.io/en/stable/authentication.html#built-in-permissions) that you might want to use here.

  ### The SQL query

  If the SQL query returns any rows the action will be allowed. If it returns no rows, the plugin hook will return `False` and deny access to that action.

  The SQL query is called with a number of named parameters. You can use any of these as part of the query.

  The list of parameters is as follows:

  * `action` - the action, e.g. `"view-database"`
  * `resource_1` - the first component of the resource, if one was passed
  * `resource_2` - the second component of the resource, if available
  * `actor_*` - a parameter for every key on the actor. Usually `actor_id` is present.

  If any rows are returned, the permission check passes. If no rows are returned the check fails.

  Another example table, this time granting explicit access to individual tables. Consider a table called `table_access` that looks like this:

  | user_id | database | table |
  | - | - | - |
  | 1 | mydb | dogs |
  | 2 | mydb | dogs |
  | 1 | mydb | cats |

  The following SQL query would grant access to the `dogs` ttable in the `mydb.db` database to users 1 and 2 - but would forbid access for user 2 to the `cats` table:

  ```sql
  SELECT
      *
  FROM
      table_access
  WHERE
      user_id = :actor_id
      AND "database" = :resource_1
      AND "table" = :resource_2
  ```
  In a `metadata.yaml` configuration file that would look like this:

  ```yaml
  databases:
    mydb:
      allow_sql: {}
  plugins:
    datasette-permissions-sql:
    - action: view-table
      sql: |-
        SELECT
          *
        FROM
          table_access
        WHERE
          user_id = :actor_id
          AND "database" = :resource_1
          AND "table" = :resource_2
  ```
  We're using `allow_sql: {}` here to disable arbitrary SQL queries. This prevents users from running `select * from cats` directly to work around the permissions limits.

  ### Fallback mode

  The default behaviour of this plugin is to take full control of specified permissions. The SQL query will directly control if the user is allowed or denied access to the permission.

  This means that the default policy for each permission (which in Datasette core is "allow" for `view-database` and friends) will be ignored. It also means that any other `permission_allowed` plugins will not get their turn once this plugin has executed.

  You can change this on a per-rule basis using ``"fallback": true``:

  ```json
  {
      "action": "view-table",
      "resource": ["mydatabase", "mytable"],
      "sql": "select * from admins where user_id = :actor_id",
      "fallback": true
  }
  ```

  When running in fallback mode, a query result returning no rows will cause the plugin hook to return ``None`` - which means "I have no opinion on this permission, fall back to other plugins or the default".

  In this mode you can still return `False` (for "deny access") by returning a single row with a single value of `-1`. For example:

  ```json
  {
      "action": "view-table",
      "resource": ["mydatabase", "mytable"],
      "sql": "select -1 from banned where user_id = :actor_id",
      "fallback": true
  }
  ```

readme_html:
  <div id="readme" class="md" data-path="README.md"><article class="markdown-body entry-content container-lg" itemprop="text"><div class="markdown-heading" dir="auto"><h1 class="heading-element" dir="auto">datasette-permissions-sql</h1><a id="user-content-datasette-permissions-sql" class="anchor" aria-label="Permalink: datasette-permissions-sql" href="#datasette-permissions-sql"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto"><a href="https://pypi.org/project/datasette-permissions-sql/" rel="nofollow"><img src="https://img.shields.io/pypi/v/datasette-permissions-sql.svg" alt="PyPI" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/9d0981123a595dd6c3b361fc282f6958f2ba6c6a7f426f5894485eca2c38dcff/68747470733a2f2f696d672e736869656c64732e696f2f707970692f762f6461746173657474652d7065726d697373696f6e732d73716c2e737667"></a>
  <a href="https://circleci.com/gh/simonw/datasette-permissions-sql" rel="nofollow"><img src="https://circleci.com/gh/simonw/datasette-permissions-sql.svg?style=svg" alt="CircleCI" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/80a68132c1ce2f25d8feac63c39f0f5793f04d07492bf7e8dd494ce09150e179/68747470733a2f2f636972636c6563692e636f6d2f67682f73696d6f6e772f6461746173657474652d7065726d697373696f6e732d73716c2e7376673f7374796c653d737667"></a>
  <a href="https://github.com/simonw/datasette-permissions-sql/blob/master/LICENSE"><img src="https://img.shields.io/badge/license-Apache%202.0-blue.svg" alt="License" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/c355f200ea90fddaa407b6eaab303663a669248ea3ca7b1fcf77dbe04ff5f48c/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6c6963656e73652d417061636865253230322e302d626c75652e737667"></a></p>
  <p dir="auto">Datasette plugin for configuring permission checks using SQL queries</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Installation</h2><a id="user-content-installation" class="anchor" aria-label="Permalink: Installation" href="#installation"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">Install this plugin in the same environment as Datasette.</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ pip install datasette-permissions-sql"><pre class="notranslate"><code>$ pip install datasette-permissions-sql
  </code></pre></div>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Usage</h2><a id="user-content-usage" class="anchor" aria-label="Permalink: Usage" href="#usage"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">First, read up on how Datasette's <a href="https://datasette.readthedocs.io/en/latest/authentication.html" rel="nofollow">authentication and permissions system</a> works.</p>
  <p dir="auto">This plugin lets you define rules containing SQL queries that are executed to see if the currently authenticated actor has permission to perform certain actions.</p>
  <p dir="auto">Consider a canned query which authenticated users should only be able to execute if a row in the <code>users</code> table says that they are a member of staff.</p>
  <p dir="auto">That <code>users</code> table in the <code>mydatabase.db</code> database could look like this:</p>
  <markdown-accessiblity-table><table>
  <thead>
  <tr>
  <th>id</th>
  <th>username</th>
  <th>is_staff</th>
  </tr>
  </thead>
  <tbody>
  <tr>
  <td>1</td>
  <td>cleopaws</td>
  <td>0</td>
  </tr>
  <tr>
  <td>2</td>
  <td>simon</td>
  <td>1</td>
  </tr>
  </tbody>
  </table></markdown-accessiblity-table>
  <p dir="auto">Authenticated users have an <code>actor</code> that looks like this:</p>
  <div class="highlight highlight-source-json notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="{
      "id": 2,
      "username": "simon"
  }"><pre>{
      <span class="pl-ent">"id"</span>: <span class="pl-c1">2</span>,
      <span class="pl-ent">"username"</span>: <span class="pl-s"><span class="pl-pds">"</span>simon<span class="pl-pds">"</span></span>
  }</pre></div>
  <p dir="auto">To configure the canned query to only be executable by staff users, add the following to your <code>metadata.json</code>:</p>
  <div class="highlight highlight-source-json notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="{
      "plugins": {
          "datasette-permissions-sql": [
              {
                  "action": "view-query",
                  "resource": ["mydatabase", "promote_to_staff"],
                  "sql": "SELECT * FROM users WHERE is_staff = 1 AND id = :actor_id"
              }
          ]
      },
      "databases": {
          "mydatabase": {
              "queries": {
                  "promote_to_staff": {
                      "sql": "UPDATE users SET is is_staff=1 WHERE id=:id",
                      "write": true
                  }
              }
          }
      }
  }"><pre>{
      <span class="pl-ent">"plugins"</span>: {
          <span class="pl-ent">"datasette-permissions-sql"</span>: [
              {
                  <span class="pl-ent">"action"</span>: <span class="pl-s"><span class="pl-pds">"</span>view-query<span class="pl-pds">"</span></span>,
                  <span class="pl-ent">"resource"</span>: [<span class="pl-s"><span class="pl-pds">"</span>mydatabase<span class="pl-pds">"</span></span>, <span class="pl-s"><span class="pl-pds">"</span>promote_to_staff<span class="pl-pds">"</span></span>],
                  <span class="pl-ent">"sql"</span>: <span class="pl-s"><span class="pl-pds">"</span>SELECT * FROM users WHERE is_staff = 1 AND id = :actor_id<span class="pl-pds">"</span></span>
              }
          ]
      },
      <span class="pl-ent">"databases"</span>: {
          <span class="pl-ent">"mydatabase"</span>: {
              <span class="pl-ent">"queries"</span>: {
                  <span class="pl-ent">"promote_to_staff"</span>: {
                      <span class="pl-ent">"sql"</span>: <span class="pl-s"><span class="pl-pds">"</span>UPDATE users SET is is_staff=1 WHERE id=:id<span class="pl-pds">"</span></span>,
                      <span class="pl-ent">"write"</span>: <span class="pl-c1">true</span>
                  }
              }
          }
      }
  }</pre></div>
  <p dir="auto">The <code>"datasette-permissions-sql"</code> key is a list of rules. Each of those rules has the following shape:</p>
  <div class="highlight highlight-source-json notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="{
      "action": "name-of-action",
      "resource": ["resource identifier to run this on"],
      "sql": "SQL query to execute",
      "database": "mydatabase"
  }"><pre>{
      <span class="pl-ent">"action"</span>: <span class="pl-s"><span class="pl-pds">"</span>name-of-action<span class="pl-pds">"</span></span>,
      <span class="pl-ent">"resource"</span>: [<span class="pl-s"><span class="pl-pds">"</span>resource identifier to run this on<span class="pl-pds">"</span></span>],
      <span class="pl-ent">"sql"</span>: <span class="pl-s"><span class="pl-pds">"</span>SQL query to execute<span class="pl-pds">"</span></span>,
      <span class="pl-ent">"database"</span>: <span class="pl-s"><span class="pl-pds">"</span>mydatabase<span class="pl-pds">"</span></span>
  }</pre></div>
  <p dir="auto">Both <code>"action"</code> and <code>"resource"</code> are optional. If present, the SQL query will only be executed on permission checks that match the action and, if present, the resource indicators.</p>
  <p dir="auto"><code>"database"</code> is also optional: it specifies the named database that the query should be executed against. If it is not present the first connected database will be used.</p>
  <p dir="auto">The Datasette documentation includes a <a href="https://datasette.readthedocs.io/en/stable/authentication.html#built-in-permissions" rel="nofollow">list of built-in permissions</a> that you might want to use here.</p>
  <div class="markdown-heading" dir="auto"><h3 class="heading-element" dir="auto">The SQL query</h3><a id="user-content-the-sql-query" class="anchor" aria-label="Permalink: The SQL query" href="#the-sql-query"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">If the SQL query returns any rows the action will be allowed. If it returns no rows, the plugin hook will return <code>False</code> and deny access to that action.</p>
  <p dir="auto">The SQL query is called with a number of named parameters. You can use any of these as part of the query.</p>
  <p dir="auto">The list of parameters is as follows:</p>
  <ul dir="auto">
  <li><code>action</code> - the action, e.g. <code>"view-database"</code></li>
  <li><code>resource_1</code> - the first component of the resource, if one was passed</li>
  <li><code>resource_2</code> - the second component of the resource, if available</li>
  <li><code>actor_*</code> - a parameter for every key on the actor. Usually <code>actor_id</code> is present.</li>
  </ul>
  <p dir="auto">If any rows are returned, the permission check passes. If no rows are returned the check fails.</p>
  <p dir="auto">Another example table, this time granting explicit access to individual tables. Consider a table called <code>table_access</code> that looks like this:</p>
  <markdown-accessiblity-table><table>
  <thead>
  <tr>
  <th>user_id</th>
  <th>database</th>
  <th>table</th>
  </tr>
  </thead>
  <tbody>
  <tr>
  <td>1</td>
  <td>mydb</td>
  <td>dogs</td>
  </tr>
  <tr>
  <td>2</td>
  <td>mydb</td>
  <td>dogs</td>
  </tr>
  <tr>
  <td>1</td>
  <td>mydb</td>
  <td>cats</td>
  </tr>
  </tbody>
  </table></markdown-accessiblity-table>
  <p dir="auto">The following SQL query would grant access to the <code>dogs</code> ttable in the <code>mydb.db</code> database to users 1 and 2 - but would forbid access for user 2 to the <code>cats</code> table:</p>
  <div class="highlight highlight-source-sql notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="SELECT
      *
  FROM
      table_access
  WHERE
      user_id = :actor_id
      AND "database" = :resource_1
      AND "table" = :resource_2"><pre><span class="pl-k">SELECT</span>
      <span class="pl-k">*</span>
  <span class="pl-k">FROM</span>
      table_access
  <span class="pl-k">WHERE</span>
      user_id <span class="pl-k">=</span> :actor_id
      <span class="pl-k">AND</span> <span class="pl-s"><span class="pl-pds">"</span>database<span class="pl-pds">"</span></span> <span class="pl-k">=</span> :resource_1
      <span class="pl-k">AND</span> <span class="pl-s"><span class="pl-pds">"</span>table<span class="pl-pds">"</span></span> <span class="pl-k">=</span> :resource_2</pre></div>
  <p dir="auto">In a <code>metadata.yaml</code> configuration file that would look like this:</p>
  <div class="highlight highlight-source-yaml notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="databases:
    mydb:
      allow_sql: {}
  plugins:
    datasette-permissions-sql:
    - action: view-table
      sql: |-
        SELECT
          *
        FROM
          table_access
        WHERE
          user_id = :actor_id
          AND "database" = :resource_1
          AND "table" = :resource_2"><pre><span class="pl-ent">databases</span>:
    <span class="pl-ent">mydb</span>:
      <span class="pl-ent">allow_sql</span>: <span class="pl-s">{}</span>
  <span class="pl-ent">plugins</span>:
    <span class="pl-ent">datasette-permissions-sql</span>:
    - <span class="pl-ent">action</span>: <span class="pl-s">view-table</span>
      <span class="pl-ent">sql</span>: <span class="pl-s">|-</span>
  <span class="pl-s">      SELECT</span>
  <span class="pl-s">        *</span>
  <span class="pl-s">      FROM</span>
  <span class="pl-s">        table_access</span>
  <span class="pl-s">      WHERE</span>
  <span class="pl-s">        user_id = :actor_id</span>
  <span class="pl-s">        AND "database" = :resource_1</span>
  <span class="pl-s">        AND "table" = :resource_2</span></pre></div>
  <p dir="auto">We're using <code>allow_sql: {}</code> here to disable arbitrary SQL queries. This prevents users from running <code>select * from cats</code> directly to work around the permissions limits.</p>
  <div class="markdown-heading" dir="auto"><h3 class="heading-element" dir="auto">Fallback mode</h3><a id="user-content-fallback-mode" class="anchor" aria-label="Permalink: Fallback mode" href="#fallback-mode"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">The default behaviour of this plugin is to take full control of specified permissions. The SQL query will directly control if the user is allowed or denied access to the permission.</p>
  <p dir="auto">This means that the default policy for each permission (which in Datasette core is "allow" for <code>view-database</code> and friends) will be ignored. It also means that any other <code>permission_allowed</code> plugins will not get their turn once this plugin has executed.</p>
  <p dir="auto">You can change this on a per-rule basis using <code>"fallback": true</code>:</p>
  <div class="highlight highlight-source-json notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="{
      "action": "view-table",
      "resource": ["mydatabase", "mytable"],
      "sql": "select * from admins where user_id = :actor_id",
      "fallback": true
  }"><pre>{
      <span class="pl-ent">"action"</span>: <span class="pl-s"><span class="pl-pds">"</span>view-table<span class="pl-pds">"</span></span>,
      <span class="pl-ent">"resource"</span>: [<span class="pl-s"><span class="pl-pds">"</span>mydatabase<span class="pl-pds">"</span></span>, <span class="pl-s"><span class="pl-pds">"</span>mytable<span class="pl-pds">"</span></span>],
      <span class="pl-ent">"sql"</span>: <span class="pl-s"><span class="pl-pds">"</span>select * from admins where user_id = :actor_id<span class="pl-pds">"</span></span>,
      <span class="pl-ent">"fallback"</span>: <span class="pl-c1">true</span>
  }</pre></div>
  <p dir="auto">When running in fallback mode, a query result returning no rows will cause the plugin hook to return <code>None</code> - which means "I have no opinion on this permission, fall back to other plugins or the default".</p>
  <p dir="auto">In this mode you can still return <code>False</code> (for "deny access") by returning a single row with a single value of <code>-1</code>. For example:</p>
  <div class="highlight highlight-source-json notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="{
      "action": "view-table",
      "resource": ["mydatabase", "mytable"],
      "sql": "select -1 from banned where user_id = :actor_id",
      "fallback": true
  }"><pre>{
      <span class="pl-ent">"action"</span>: <span class="pl-s"><span class="pl-pds">"</span>view-table<span class="pl-pds">"</span></span>,
      <span class="pl-ent">"resource"</span>: [<span class="pl-s"><span class="pl-pds">"</span>mydatabase<span class="pl-pds">"</span></span>, <span class="pl-s"><span class="pl-pds">"</span>mytable<span class="pl-pds">"</span></span>],
      <span class="pl-ent">"sql"</span>: <span class="pl-s"><span class="pl-pds">"</span>select -1 from banned where user_id = :actor_id<span class="pl-pds">"</span></span>,
      <span class="pl-ent">"fallback"</span>: <span class="pl-c1">true</span>
  }</pre></div>
  </article></div>


node_id: MDEwOlJlcG9zaXRvcnkyNzE2NjUzMzY=
name: datasette-auth-tokens
full_name: simonw/datasette-auth-tokens
html_url: https://github.com/simonw/datasette-auth-tokens
description: Datasette plugin for authenticating access using API tokens
created_at: 2020-06-11T23:23:30Z
updated_at: 2026-04-20T02:08:37Z
pushed_at: 2026-04-20T02:08:33Z
language: Python
license: apache-2.0
topics: ["datasette", "datasette-io", "datasette-plugin"]
visibility: public
default_branch: main
permissions: {"admin": false, "maintain": false, "push": false, "triage": false, "pull": false}
pull_request_creation_policy: all
readme:
  # datasette-auth-tokens

  [![PyPI](https://img.shields.io/pypi/v/datasette-auth-tokens.svg)](https://pypi.org/project/datasette-auth-tokens/)
  [![Changelog](https://img.shields.io/github/v/release/simonw/datasette-auth-tokens?include_prereleases&label=changelog)](https://github.com/simonw/datasette-auth-tokens/releases)
  [![Tests](https://github.com/simonw/datasette-auth-tokens/workflows/Test/badge.svg)](https://github.com/simonw/datasette-auth-tokens/actions?query=workflow%3ATest)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/datasette-auth-tokens/blob/main/LICENSE)

  Datasette plugin for authenticating access using API tokens

  ## Installation

  Install this plugin in the same environment as Datasette.
  ```bash
  datasette install datasette-auth-tokens
  ```
  ## Hard-coded tokens

  Read about Datasette's [authentication and permissions system](https://datasette.readthedocs.io/en/latest/authentication.html).

  This plugin lets you configure secret API tokens which can be used to make authenticated requests to Datasette.

  First, create a random API token. A useful recipe for doing that is the following:
  ```bash
  python -c 'import secrets; print(secrets.token_hex(32))'
  ```
  ```
  5f9a486dd807de632200b17508c75002bb66ca6fde1993db1de6cbd446362589
  ```
  Decide on the actor that this token should represent, for example:

  ```json
  {
      "bot_id": "my-bot"
  }
  ```

  You can then use `"allow"` blocks to provide that token with permission to access specific actions. To enable access to a configured writable SQL query you could use this in your `config.json` (for Datasette 1.0) or `metadata.json`: 

  ```json
  {
      "plugins": {
          "datasette-auth-tokens": {
              "tokens": [
                  {
                      "token": {
                          "$env": "BOT_TOKEN"
                      },
                      "actor": {
                          "bot_id": "my-bot"
                      }
                  }
              ]
          }
      },
      "databases": {
          ":memory:": {
              "queries": {
                  "show_version": {
                      "sql": "select sqlite_version()",
                      "allow": {
                          "bot_id": "my-bot"
                      }
                  }
              }
          }
      }
  }
  ```
  This uses Datasette's [secret configuration values mechanism](https://datasette.readthedocs.io/en/stable/plugins.html#secret-configuration-values) to allow the secret token to be passed as an environment variable.

  Run Datasette like this:
  ```bash
  BOT_TOKEN="this-is-the-secret-token" \
      datasette -c config.json
  ```
  You can now run authenticated API queries like this:
  ```bash
  curl -H 'Authorization: Bearer this-is-the-secret-token' \
    'http://127.0.0.1:8001/:memory:/show_version.json?_shape=array'
  ```
  ```json
  [{"sqlite_version()": "3.31.1"}]
  ```
  Additionally you can allow passing the token as a query string parameter, although that's disabled by default given the security implications of URLs with secret tokens included. This may be useful to easily allow embedding data between different services.

  Enable it using the `param` config value:

  ```json
  {
      "plugins": {
          "datasette-auth-tokens": {
              "tokens": [
                  {
                      "token": {
                          "$env": "BOT_TOKEN"
                      },
                      "actor": {
                          "bot_id": "my-bot"
                      },
                  }
              ],
              "param": "_auth_token"
          }
      },
      "databases": {
          ":memory:": {
              "queries": {
                  "show_version": {
                      "sql": "select sqlite_version()",
                      "allow": {
                          "bot_id": "my-bot"
                      }
                  }
              }
          }
      }
  }
  ```

  You can now run authenticated API queries like this:
  ```bash
  curl http://127.0.0.1:8001/:memory:/show_version.json?_shape=array&_auth_token=this-is-the-secret-token
  ```
  ```json
  [{"sqlite_version()": "3.31.1"}]
  ```
  ## Managed tokens mode

  `datasette-auth-tokens` provides a managed tokens mode, where tokens are stored in a SQLite database table and the plugin provides an interface for creating and revoking tokens.

  To turn this mode on, add `"manage_tokens": true` to your plugin configuration:

  ```json
  {
      "plugins": {
          "datasette-auth-tokens": {
              "manage_tokens": true
          }
      }
  }
  ```
  This will add a "Create API token" option to the Datasette menu.

  Tokens that are created will be kept in a new `_datasette_auth_tokens` table.

  Users need the `auth-tokens-create` permission to create tokens. One way to grant that is to add this `"permissions"` block to your configuration:

  ```json
  {
      "permissions": {
          "auth-tokens-create": {
              "id": "*"
          }
      }
  }
  ```

  Use the "Create API token" option in the Datasette menu or navigate to `/-/api/tokens` to create tokens and manage tokens.

  When you create a new token a signed token string will be presented to you. You need to store this, as it is not stored directly in the database table and can only be retrieved once.

  If you have multiple databases attached to Datasette you will need to specify which database should be used for the `_datasette_auth_tokens` table. You can do this with the `manage_tokens_database` setting:

  ```json
  {
      "plugins": {
          "datasette-auth-tokens": {
              "manage_tokens": true,
              "manage_tokens_database": "tokens"
          }
      }
  }
  ```
  Now start Datasette like this:
  ```bash
  datasette -c config.json mydb.db tokens.db --create
  ```
  The `--create` option can be used to tell Datasette to create the `tokens.db` database file if it does not already exist.

  In Datasette 1.0 you can instead use the `-s` option like this:
  ```bash
  datasette \
    -s plugins.datasette-auth-tokens.manage_tokens true \
    -s plugins.datasette-auth-tokens.manage_tokens_database tokens \
    -s permissions.auth-tokens-create.id '*' # to enable token creation
  ```

  ### Viewing tokens

  By default, users can only view tokens that they themselves have created on the `/-/api/tokens` page.

  Grant the `auth-tokens-view-all` permission to allow a user to view all tokens, even those created by other users.

  ### Revoking tokens

  A token can be revoked by the user that created it by clicking the "Revoke this token" button at the bottom of the token page that is linked to from `/-/api/tokens`.

  A user with the `auth-tokens-revoke-all` permission can revoke any token.

  ## Custom tokens from your database

  If you decide not to use managed tokens mode, you can instead configure `datasette-auth-tokens` to use tokens that are stored in your own custom database tables.

  You can do this by configuring a custom SQL query that will execute to test if an incoming token is valid.

  Your query needs to take a `:token_id` parameter and return at least two columns: one called `token_secret` and one called `actor_*` - usually `actor_id`. Further `actor_` prefixed columns can be returned to provide more details for the authenticated actor.

  Here's a simple example of a configuration query:

  ```sql
  select actor_id, actor_name, token_secret from tokens where token_id = :token_id
  ```

  This can run against a table like this one:

  | token_id | token_secret | actor_id | actor_name |
  | -------- | ------------ | -------- | ---------- |
  | 1        | bd3c94f51fcd | 78       | Cleopaws   |
  | 2        | 86681b4d6f66 | 32       | Pancakes   |

  The tokens are formed as the token ID, then a hyphen, then the token secret. For example:

  - `1-bd3c94f51fcd`
  - `2-86681b4d6f66`

  The SQL query will be executed with the portion before the hyphen as the `:token_id` parameter.

  The `token_secret` value returned by the query will be compared to the portion of the token after the hyphen to check if the token is valid.

  Columns with a prefix of `actor_` will be used to populate the actor dictionary. In the above example, a token of `2-86681b4d6f66` will become an actor dictionary of `{"id": 32, "name": "Pancakes"}`.

  To configure this, use a `"query"` block in your plugin configuration like this:

  ```json
  {
      "plugins": {
          "datasette-auth-tokens": {
              "query": {
                  "sql": "select actor_id, actor_name, token_secret from tokens where token_id = :token_id",
                  "database": "tokens"
              }
          }
      },
      "databases": {
          "tokens": {
              "allow": false
          }
      }
  }
  ```
  The `"sql"` key here contains the SQL query. The `"database"` key has the name of the attached database file that the query should be executed against - in this case it would execute against `tokens.db`.

  ### Securing your custom tokens

  If you implement the custom pattern above which reads `token_secret` from your own `tokens` table, you need to be aware that anyone with read access to your Datasette instance could read those tokens from your table. This probably isn't what you want!

  To avoid this, you should lock down access to that table. The configuration example above shows how to do this using an `"allow": false` block to deny all access to that `tokens` database.

  Consult Datasette's [Permissions documentation](https://datasette.readthedocs.io/en/stable/authentication.html#permissions) for more information about how to lock down this kind of access.

readme_html:
  <div id="readme" class="md" data-path="README.md"><article class="markdown-body entry-content container-lg" itemprop="text"><div class="markdown-heading" dir="auto"><h1 class="heading-element" dir="auto">datasette-auth-tokens</h1><a id="user-content-datasette-auth-tokens" class="anchor" aria-label="Permalink: datasette-auth-tokens" href="#datasette-auth-tokens"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto"><a href="https://pypi.org/project/datasette-auth-tokens/" rel="nofollow"><img src="https://img.shields.io/pypi/v/datasette-auth-tokens.svg" alt="PyPI" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/1c2eb8b55b38961e0358dd141a16e1aa22d7d22d5752e01ee9903c57a993ecde/68747470733a2f2f696d672e736869656c64732e696f2f707970692f762f6461746173657474652d617574682d746f6b656e732e737667"></a>
  <a href="https://github.com/simonw/datasette-auth-tokens/releases"><img src="https://img.shields.io/github/v/release/simonw/datasette-auth-tokens?include_prereleases&label=changelog" alt="Changelog" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/bbd9652e447d981474d97e2ba8c08e899fec4753d8dfdfb642c5dfe05238d11f/68747470733a2f2f696d672e736869656c64732e696f2f6769746875622f762f72656c656173652f73696d6f6e772f6461746173657474652d617574682d746f6b656e733f696e636c7564655f70726572656c6561736573266c6162656c3d6368616e67656c6f67"></a>
  <a href="https://github.com/simonw/datasette-auth-tokens/actions?query=workflow%3ATest"><img src="https://github.com/simonw/datasette-auth-tokens/workflows/Test/badge.svg" alt="Tests" style="max-width: 100%;"></a>
  <a href="https://github.com/simonw/datasette-auth-tokens/blob/main/LICENSE"><img src="https://img.shields.io/badge/license-Apache%202.0-blue.svg" alt="License" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/c355f200ea90fddaa407b6eaab303663a669248ea3ca7b1fcf77dbe04ff5f48c/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6c6963656e73652d417061636865253230322e302d626c75652e737667"></a></p>
  <p dir="auto">Datasette plugin for authenticating access using API tokens</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Installation</h2><a id="user-content-installation" class="anchor" aria-label="Permalink: Installation" href="#installation"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">Install this plugin in the same environment as Datasette.</p>
  <div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="datasette install datasette-auth-tokens"><pre>datasette install datasette-auth-tokens</pre></div>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Hard-coded tokens</h2><a id="user-content-hard-coded-tokens" class="anchor" aria-label="Permalink: Hard-coded tokens" href="#hard-coded-tokens"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">Read about Datasette's <a href="https://datasette.readthedocs.io/en/latest/authentication.html" rel="nofollow">authentication and permissions system</a>.</p>
  <p dir="auto">This plugin lets you configure secret API tokens which can be used to make authenticated requests to Datasette.</p>
  <p dir="auto">First, create a random API token. A useful recipe for doing that is the following:</p>
  <div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="python -c 'import secrets; print(secrets.token_hex(32))'"><pre>python -c <span class="pl-s"><span class="pl-pds">'</span>import secrets; print(secrets.token_hex(32))<span class="pl-pds">'</span></span></pre></div>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="5f9a486dd807de632200b17508c75002bb66ca6fde1993db1de6cbd446362589"><pre class="notranslate"><code>5f9a486dd807de632200b17508c75002bb66ca6fde1993db1de6cbd446362589
  </code></pre></div>
  <p dir="auto">Decide on the actor that this token should represent, for example:</p>
  <div class="highlight highlight-source-json notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="{
      "bot_id": "my-bot"
  }"><pre>{
      <span class="pl-ent">"bot_id"</span>: <span class="pl-s"><span class="pl-pds">"</span>my-bot<span class="pl-pds">"</span></span>
  }</pre></div>
  <p dir="auto">You can then use <code>"allow"</code> blocks to provide that token with permission to access specific actions. To enable access to a configured writable SQL query you could use this in your <code>config.json</code> (for Datasette 1.0) or <code>metadata.json</code>:</p>
  <div class="highlight highlight-source-json notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="{
      "plugins": {
          "datasette-auth-tokens": {
              "tokens": [
                  {
                      "token": {
                          "$env": "BOT_TOKEN"
                      },
                      "actor": {
                          "bot_id": "my-bot"
                      }
                  }
              ]
          }
      },
      "databases": {
          ":memory:": {
              "queries": {
                  "show_version": {
                      "sql": "select sqlite_version()",
                      "allow": {
                          "bot_id": "my-bot"
                      }
                  }
              }
          }
      }
  }"><pre>{
      <span class="pl-ent">"plugins"</span>: {
          <span class="pl-ent">"datasette-auth-tokens"</span>: {
              <span class="pl-ent">"tokens"</span>: [
                  {
                      <span class="pl-ent">"token"</span>: {
                          <span class="pl-ent">"$env"</span>: <span class="pl-s"><span class="pl-pds">"</span>BOT_TOKEN<span class="pl-pds">"</span></span>
                      },
                      <span class="pl-ent">"actor"</span>: {
                          <span class="pl-ent">"bot_id"</span>: <span class="pl-s"><span class="pl-pds">"</span>my-bot<span class="pl-pds">"</span></span>
                      }
                  }
              ]
          }
      },
      <span class="pl-ent">"databases"</span>: {
          <span class="pl-ent">":memory:"</span>: {
              <span class="pl-ent">"queries"</span>: {
                  <span class="pl-ent">"show_version"</span>: {
                      <span class="pl-ent">"sql"</span>: <span class="pl-s"><span class="pl-pds">"</span>select sqlite_version()<span class="pl-pds">"</span></span>,
                      <span class="pl-ent">"allow"</span>: {
                          <span class="pl-ent">"bot_id"</span>: <span class="pl-s"><span class="pl-pds">"</span>my-bot<span class="pl-pds">"</span></span>
                      }
                  }
              }
          }
      }
  }</pre></div>
  <p dir="auto">This uses Datasette's <a href="https://datasette.readthedocs.io/en/stable/plugins.html#secret-configuration-values" rel="nofollow">secret configuration values mechanism</a> to allow the secret token to be passed as an environment variable.</p>
  <p dir="auto">Run Datasette like this:</p>
  <div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="BOT_TOKEN="this-is-the-secret-token" \
      datasette -c config.json"><pre>BOT_TOKEN=<span class="pl-s"><span class="pl-pds">"</span>this-is-the-secret-token<span class="pl-pds">"</span></span> \
      datasette -c config.json</pre></div>
  <p dir="auto">You can now run authenticated API queries like this:</p>
  <div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="curl -H 'Authorization: Bearer this-is-the-secret-token' \
    'http://127.0.0.1:8001/:memory:/show_version.json?_shape=array'"><pre>curl -H <span class="pl-s"><span class="pl-pds">'</span>Authorization: Bearer this-is-the-secret-token<span class="pl-pds">'</span></span> \
    <span class="pl-s"><span class="pl-pds">'</span>http://127.0.0.1:8001/:memory:/show_version.json?_shape=array<span class="pl-pds">'</span></span></pre></div>
  <div class="highlight highlight-source-json notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="[{"sqlite_version()": "3.31.1"}]"><pre>[{<span class="pl-ent">"sqlite_version()"</span>: <span class="pl-s"><span class="pl-pds">"</span>3.31.1<span class="pl-pds">"</span></span>}]</pre></div>
  <p dir="auto">Additionally you can allow passing the token as a query string parameter, although that's disabled by default given the security implications of URLs with secret tokens included. This may be useful to easily allow embedding data between different services.</p>
  <p dir="auto">Enable it using the <code>param</code> config value:</p>
  <div class="highlight highlight-source-json notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="{
      "plugins": {
          "datasette-auth-tokens": {
              "tokens": [
                  {
                      "token": {
                          "$env": "BOT_TOKEN"
                      },
                      "actor": {
                          "bot_id": "my-bot"
                      },
                  }
              ],
              "param": "_auth_token"
          }
      },
      "databases": {
          ":memory:": {
              "queries": {
                  "show_version": {
                      "sql": "select sqlite_version()",
                      "allow": {
                          "bot_id": "my-bot"
                      }
                  }
              }
          }
      }
  }"><pre>{
      <span class="pl-ent">"plugins"</span>: {
          <span class="pl-ent">"datasette-auth-tokens"</span>: {
              <span class="pl-ent">"tokens"</span>: [
                  {
                      <span class="pl-ent">"token"</span>: {
                          <span class="pl-ent">"$env"</span>: <span class="pl-s"><span class="pl-pds">"</span>BOT_TOKEN<span class="pl-pds">"</span></span>
                      },
                      <span class="pl-ent">"actor"</span>: {
                          <span class="pl-ent">"bot_id"</span>: <span class="pl-s"><span class="pl-pds">"</span>my-bot<span class="pl-pds">"</span></span>
                      },
                  }
              ],
              <span class="pl-ent">"param"</span>: <span class="pl-s"><span class="pl-pds">"</span>_auth_token<span class="pl-pds">"</span></span>
          }
      },
      <span class="pl-ent">"databases"</span>: {
          <span class="pl-ent">":memory:"</span>: {
              <span class="pl-ent">"queries"</span>: {
                  <span class="pl-ent">"show_version"</span>: {
                      <span class="pl-ent">"sql"</span>: <span class="pl-s"><span class="pl-pds">"</span>select sqlite_version()<span class="pl-pds">"</span></span>,
                      <span class="pl-ent">"allow"</span>: {
                          <span class="pl-ent">"bot_id"</span>: <span class="pl-s"><span class="pl-pds">"</span>my-bot<span class="pl-pds">"</span></span>
                      }
                  }
              }
          }
      }
  }</pre></div>
  <p dir="auto">You can now run authenticated API queries like this:</p>
  <div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="curl http://127.0.0.1:8001/:memory:/show_version.json?_shape=array&_auth_token=this-is-the-secret-token"><pre>curl http://127.0.0.1:8001/:memory:/show_version.json<span class="pl-k">?</span>_shape=array<span class="pl-k">&</span>_auth_token=this-is-the-secret-token</pre></div>
  <div class="highlight highlight-source-json notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="[{"sqlite_version()": "3.31.1"}]"><pre>[{<span class="pl-ent">"sqlite_version()"</span>: <span class="pl-s"><span class="pl-pds">"</span>3.31.1<span class="pl-pds">"</span></span>}]</pre></div>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Managed tokens mode</h2><a id="user-content-managed-tokens-mode" class="anchor" aria-label="Permalink: Managed tokens mode" href="#managed-tokens-mode"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto"><code>datasette-auth-tokens</code> provides a managed tokens mode, where tokens are stored in a SQLite database table and the plugin provides an interface for creating and revoking tokens.</p>
  <p dir="auto">To turn this mode on, add <code>"manage_tokens": true</code> to your plugin configuration:</p>
  <div class="highlight highlight-source-json notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="{
      "plugins": {
          "datasette-auth-tokens": {
              "manage_tokens": true
          }
      }
  }"><pre>{
      <span class="pl-ent">"plugins"</span>: {
          <span class="pl-ent">"datasette-auth-tokens"</span>: {
              <span class="pl-ent">"manage_tokens"</span>: <span class="pl-c1">true</span>
          }
      }
  }</pre></div>
  <p dir="auto">This will add a "Create API token" option to the Datasette menu.</p>
  <p dir="auto">Tokens that are created will be kept in a new <code>_datasette_auth_tokens</code> table.</p>
  <p dir="auto">Users need the <code>auth-tokens-create</code> permission to create tokens. One way to grant that is to add this <code>"permissions"</code> block to your configuration:</p>
  <div class="highlight highlight-source-json notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="{
      "permissions": {
          "auth-tokens-create": {
              "id": "*"
          }
      }
  }"><pre>{
      <span class="pl-ent">"permissions"</span>: {
          <span class="pl-ent">"auth-tokens-create"</span>: {
              <span class="pl-ent">"id"</span>: <span class="pl-s"><span class="pl-pds">"</span>*<span class="pl-pds">"</span></span>
          }
      }
  }</pre></div>
  <p dir="auto">Use the "Create API token" option in the Datasette menu or navigate to <code>/-/api/tokens</code> to create tokens and manage tokens.</p>
  <p dir="auto">When you create a new token a signed token string will be presented to you. You need to store this, as it is not stored directly in the database table and can only be retrieved once.</p>
  <p dir="auto">If you have multiple databases attached to Datasette you will need to specify which database should be used for the <code>_datasette_auth_tokens</code> table. You can do this with the <code>manage_tokens_database</code> setting:</p>
  <div class="highlight highlight-source-json notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="{
      "plugins": {
          "datasette-auth-tokens": {
              "manage_tokens": true,
              "manage_tokens_database": "tokens"
          }
      }
  }"><pre>{
      <span class="pl-ent">"plugins"</span>: {
          <span class="pl-ent">"datasette-auth-tokens"</span>: {
              <span class="pl-ent">"manage_tokens"</span>: <span class="pl-c1">true</span>,
              <span class="pl-ent">"manage_tokens_database"</span>: <span class="pl-s"><span class="pl-pds">"</span>tokens<span class="pl-pds">"</span></span>
          }
      }
  }</pre></div>
  <p dir="auto">Now start Datasette like this:</p>
  <div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="datasette -c config.json mydb.db tokens.db --create"><pre>datasette -c config.json mydb.db tokens.db --create</pre></div>
  <p dir="auto">The <code>--create</code> option can be used to tell Datasette to create the <code>tokens.db</code> database file if it does not already exist.</p>
  <p dir="auto">In Datasette 1.0 you can instead use the <code>-s</code> option like this:</p>
  <div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="datasette \
    -s plugins.datasette-auth-tokens.manage_tokens true \
    -s plugins.datasette-auth-tokens.manage_tokens_database tokens \
    -s permissions.auth-tokens-create.id '*' # to enable token creation"><pre>datasette \
    -s plugins.datasette-auth-tokens.manage_tokens <span class="pl-c1">true</span> \
    -s plugins.datasette-auth-tokens.manage_tokens_database tokens \
    -s permissions.auth-tokens-create.id <span class="pl-s"><span class="pl-pds">'</span>*<span class="pl-pds">'</span></span> <span class="pl-c"><span class="pl-c">#</span> to enable token creation</span></pre></div>
  <div class="markdown-heading" dir="auto"><h3 class="heading-element" dir="auto">Viewing tokens</h3><a id="user-content-viewing-tokens" class="anchor" aria-label="Permalink: Viewing tokens" href="#viewing-tokens"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">By default, users can only view tokens that they themselves have created on the <code>/-/api/tokens</code> page.</p>
  <p dir="auto">Grant the <code>auth-tokens-view-all</code> permission to allow a user to view all tokens, even those created by other users.</p>
  <div class="markdown-heading" dir="auto"><h3 class="heading-element" dir="auto">Revoking tokens</h3><a id="user-content-revoking-tokens" class="anchor" aria-label="Permalink: Revoking tokens" href="#revoking-tokens"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">A token can be revoked by the user that created it by clicking the "Revoke this token" button at the bottom of the token page that is linked to from <code>/-/api/tokens</code>.</p>
  <p dir="auto">A user with the <code>auth-tokens-revoke-all</code> permission can revoke any token.</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Custom tokens from your database</h2><a id="user-content-custom-tokens-from-your-database" class="anchor" aria-label="Permalink: Custom tokens from your database" href="#custom-tokens-from-your-database"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">If you decide not to use managed tokens mode, you can instead configure <code>datasette-auth-tokens</code> to use tokens that are stored in your own custom database tables.</p>
  <p dir="auto">You can do this by configuring a custom SQL query that will execute to test if an incoming token is valid.</p>
  <p dir="auto">Your query needs to take a <code>:token_id</code> parameter and return at least two columns: one called <code>token_secret</code> and one called <code>actor_*</code> - usually <code>actor_id</code>. Further <code>actor_</code> prefixed columns can be returned to provide more details for the authenticated actor.</p>
  <p dir="auto">Here's a simple example of a configuration query:</p>
  <div class="highlight highlight-source-sql notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="select actor_id, actor_name, token_secret from tokens where token_id = :token_id"><pre><span class="pl-k">select</span> actor_id, actor_name, token_secret <span class="pl-k">from</span> tokens <span class="pl-k">where</span> token_id <span class="pl-k">=</span> :token_id</pre></div>
  <p dir="auto">This can run against a table like this one:</p>
  <markdown-accessiblity-table><table>
  <thead>
  <tr>
  <th>token_id</th>
  <th>token_secret</th>
  <th>actor_id</th>
  <th>actor_name</th>
  </tr>
  </thead>
  <tbody>
  <tr>
  <td>1</td>
  <td>bd3c94f51fcd</td>
  <td>78</td>
  <td>Cleopaws</td>
  </tr>
  <tr>
  <td>2</td>
  <td>86681b4d6f66</td>
  <td>32</td>
  <td>Pancakes</td>
  </tr>
  </tbody>
  </table></markdown-accessiblity-table>
  <p dir="auto">The tokens are formed as the token ID, then a hyphen, then the token secret. For example:</p>
  <ul dir="auto">
  <li><code>1-bd3c94f51fcd</code></li>
  <li><code>2-86681b4d6f66</code></li>
  </ul>
  <p dir="auto">The SQL query will be executed with the portion before the hyphen as the <code>:token_id</code> parameter.</p>
  <p dir="auto">The <code>token_secret</code> value returned by the query will be compared to the portion of the token after the hyphen to check if the token is valid.</p>
  <p dir="auto">Columns with a prefix of <code>actor_</code> will be used to populate the actor dictionary. In the above example, a token of <code>2-86681b4d6f66</code> will become an actor dictionary of <code>{"id": 32, "name": "Pancakes"}</code>.</p>
  <p dir="auto">To configure this, use a <code>"query"</code> block in your plugin configuration like this:</p>
  <div class="highlight highlight-source-json notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="{
      "plugins": {
          "datasette-auth-tokens": {
              "query": {
                  "sql": "select actor_id, actor_name, token_secret from tokens where token_id = :token_id",
                  "database": "tokens"
              }
          }
      },
      "databases": {
          "tokens": {
              "allow": false
          }
      }
  }"><pre>{
      <span class="pl-ent">"plugins"</span>: {
          <span class="pl-ent">"datasette-auth-tokens"</span>: {
              <span class="pl-ent">"query"</span>: {
                  <span class="pl-ent">"sql"</span>: <span class="pl-s"><span class="pl-pds">"</span>select actor_id, actor_name, token_secret from tokens where token_id = :token_id<span class="pl-pds">"</span></span>,
                  <span class="pl-ent">"database"</span>: <span class="pl-s"><span class="pl-pds">"</span>tokens<span class="pl-pds">"</span></span>
              }
          }
      },
      <span class="pl-ent">"databases"</span>: {
          <span class="pl-ent">"tokens"</span>: {
              <span class="pl-ent">"allow"</span>: <span class="pl-c1">false</span>
          }
      }
  }</pre></div>
  <p dir="auto">The <code>"sql"</code> key here contains the SQL query. The <code>"database"</code> key has the name of the attached database file that the query should be executed against - in this case it would execute against <code>tokens.db</code>.</p>
  <div class="markdown-heading" dir="auto"><h3 class="heading-element" dir="auto">Securing your custom tokens</h3><a id="user-content-securing-your-custom-tokens" class="anchor" aria-label="Permalink: Securing your custom tokens" href="#securing-your-custom-tokens"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">If you implement the custom pattern above which reads <code>token_secret</code> from your own <code>tokens</code> table, you need to be aware that anyone with read access to your Datasette instance could read those tokens from your table. This probably isn't what you want!</p>
  <p dir="auto">To avoid this, you should lock down access to that table. The configuration example above shows how to do this using an <code>"allow": false</code> block to deny all access to that <code>tokens</code> database.</p>
  <p dir="auto">Consult Datasette's <a href="https://datasette.readthedocs.io/en/stable/authentication.html#permissions" rel="nofollow">Permissions documentation</a> for more information about how to lock down this kind of access.</p>
  </article></div>


node_id: MDEwOlJlcG9zaXRvcnkyNzIwOTg0ODY=
name: datasette-psutil
full_name: simonw/datasette-psutil
html_url: https://github.com/simonw/datasette-psutil
description: Datasette plugin adding a /-/psutil debugging endpoint
created_at: 2020-06-13T22:57:07Z
updated_at: 2025-03-04T22:09:49Z
pushed_at: 2022-03-07T15:35:57Z
homepage: https://datasette.io/plugins/datasette-psutil
language: Python
license: apache-2.0
topics: ["datasette", "datasette-io", "datasette-plugin", "psutil"]
visibility: public
default_branch: main
permissions: {"admin": false, "maintain": false, "push": false, "triage": false, "pull": true}
readme:
  # datasette-psutil

  [![PyPI](https://img.shields.io/pypi/v/datasette-psutil.svg)](https://pypi.org/project/datasette-psutil/)
  [![Changelog](https://img.shields.io/github/v/release/simonw/datasette-psutil?include_prereleases&label=changelog)](https://github.com/simonw/datasette-psutil/releases)
  [![Tests](https://github.com/simonw/datasette-psutil/workflows/Test/badge.svg)](https://github.com/simonw/datasette-psutil/actions?query=workflow%3ATest)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/datasette-psutil/blob/main/LICENSE)

  Datasette plugin adding a `/-/psutil` debugging endpoint

  ## Installation

  Install this plugin in the same environment as Datasette.

      $ pip install datasette-psutil

  ## Usage

  Visit `/-/psutil` on your Datasette instance to see various information provided by [psutil](https://psutil.readthedocs.io/).

  ## Demo

  https://latest-with-plugins.datasette.io/-/psutil is a live demo of this plugin, hosted on Google Cloud Run.

readme_html:
  <div id="readme" class="md" data-path="README.md"><article class="markdown-body entry-content container-lg" itemprop="text"><div class="markdown-heading" dir="auto"><h1 class="heading-element" dir="auto">datasette-psutil</h1><a id="user-content-datasette-psutil" class="anchor" aria-label="Permalink: datasette-psutil" href="#datasette-psutil"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto"><a href="https://pypi.org/project/datasette-psutil/" rel="nofollow"><img src="https://img.shields.io/pypi/v/datasette-psutil.svg" alt="PyPI" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/998c909fa1e1b88e0e67f3f2f9201aa9be32ee0231d48465b19b0050e5a58c1b/68747470733a2f2f696d672e736869656c64732e696f2f707970692f762f6461746173657474652d70737574696c2e737667"></a>
  <a href="https://github.com/simonw/datasette-psutil/releases"><img src="https://img.shields.io/github/v/release/simonw/datasette-psutil?include_prereleases&label=changelog" alt="Changelog" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/78b5dafc988811056fa4c6fd1edce5c49bc8a9d34bbc7c6e9a1e8e0b7cbe3ad3/68747470733a2f2f696d672e736869656c64732e696f2f6769746875622f762f72656c656173652f73696d6f6e772f6461746173657474652d70737574696c3f696e636c7564655f70726572656c6561736573266c6162656c3d6368616e67656c6f67"></a>
  <a href="https://github.com/simonw/datasette-psutil/actions?query=workflow%3ATest"><img src="https://github.com/simonw/datasette-psutil/workflows/Test/badge.svg" alt="Tests" style="max-width: 100%;"></a>
  <a href="https://github.com/simonw/datasette-psutil/blob/main/LICENSE"><img src="https://img.shields.io/badge/license-Apache%202.0-blue.svg" alt="License" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/c355f200ea90fddaa407b6eaab303663a669248ea3ca7b1fcf77dbe04ff5f48c/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6c6963656e73652d417061636865253230322e302d626c75652e737667"></a></p>
  <p dir="auto">Datasette plugin adding a <code>/-/psutil</code> debugging endpoint</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Installation</h2><a id="user-content-installation" class="anchor" aria-label="Permalink: Installation" href="#installation"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">Install this plugin in the same environment as Datasette.</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ pip install datasette-psutil"><pre class="notranslate"><code>$ pip install datasette-psutil
  </code></pre></div>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Usage</h2><a id="user-content-usage" class="anchor" aria-label="Permalink: Usage" href="#usage"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">Visit <code>/-/psutil</code> on your Datasette instance to see various information provided by <a href="https://psutil.readthedocs.io/" rel="nofollow">psutil</a>.</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Demo</h2><a id="user-content-demo" class="anchor" aria-label="Permalink: Demo" href="#demo"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto"><a href="https://latest-with-plugins.datasette.io/-/psutil" rel="nofollow">https://latest-with-plugins.datasette.io/-/psutil</a> is a live demo of this plugin, hosted on Google Cloud Run.</p>
  </article></div>


node_id: MDEwOlJlcG9zaXRvcnkyNzM2MDk4Nzk=
name: datasette-saved-queries
full_name: simonw/datasette-saved-queries
html_url: https://github.com/simonw/datasette-saved-queries
description: Datasette plugin that lets users save and execute queries
created_at: 2020-06-20T00:20:42Z
updated_at: 2024-06-19T21:57:41Z
pushed_at: 2024-03-07T14:33:33Z
language: Python
topics: ["datasette", "datasette-io", "datasette-plugin"]
visibility: public
default_branch: main
permissions: {"admin": false, "maintain": false, "push": false, "triage": false, "pull": true}
readme:
  # datasette-saved-queries

  [![PyPI](https://img.shields.io/pypi/v/datasette-saved-queries.svg)](https://pypi.org/project/datasette-saved-queries/)
  [![Changelog](https://img.shields.io/github/v/release/simonw/datasette-saved-queries?label=changelog)](https://github.com/simonw/datasette-saved-queries/releases)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/datasette-saved-queries/blob/master/LICENSE)

  Datasette plugin that lets users save and execute queries

  ## Installation

  Install this plugin in the same environment as Datasette.

      $ pip install datasette-saved-queries

  ## Usage

  When the plugin is installed Datasette will automatically create a `saved_queries` table in the first connected database when it starts up.

  It also creates a `save_query` writable canned query which you can use to save new queries.

  Queries that you save will be added to the query list on the database page.

  ## Development

  To set up this plugin locally, first checkout the code. Then create a new virtual environment:

      cd datasette-saved-queries
      python -mvenv venv
      source venv/bin/activate

  Or if you are using `pipenv`:

      pipenv shell

  Now install the dependencies and tests:

      pip install -e '.[test]'

  To run the tests:

      pytest

readme_html:
  <div id="readme" class="md" data-path="README.md"><article class="markdown-body entry-content container-lg" itemprop="text"><div class="markdown-heading" dir="auto"><h1 class="heading-element" dir="auto">datasette-saved-queries</h1><a id="user-content-datasette-saved-queries" class="anchor" aria-label="Permalink: datasette-saved-queries" href="#datasette-saved-queries"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto"><a href="https://pypi.org/project/datasette-saved-queries/" rel="nofollow"><img src="https://img.shields.io/pypi/v/datasette-saved-queries.svg" alt="PyPI" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/2a88439260cded2322b53cd8c74fd0ff3b41d887f3875d7d6ef34dd19c441f8d/68747470733a2f2f696d672e736869656c64732e696f2f707970692f762f6461746173657474652d73617665642d717565726965732e737667"></a>
  <a href="https://github.com/simonw/datasette-saved-queries/releases"><img src="https://img.shields.io/github/v/release/simonw/datasette-saved-queries?label=changelog" alt="Changelog" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/9c1418ad4d96f68346777aff82ece983fbc6554be359368c466b62d4b3c51762/68747470733a2f2f696d672e736869656c64732e696f2f6769746875622f762f72656c656173652f73696d6f6e772f6461746173657474652d73617665642d717565726965733f6c6162656c3d6368616e67656c6f67"></a>
  <a href="https://github.com/simonw/datasette-saved-queries/blob/master/LICENSE"><img src="https://img.shields.io/badge/license-Apache%202.0-blue.svg" alt="License" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/c355f200ea90fddaa407b6eaab303663a669248ea3ca7b1fcf77dbe04ff5f48c/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6c6963656e73652d417061636865253230322e302d626c75652e737667"></a></p>
  <p dir="auto">Datasette plugin that lets users save and execute queries</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Installation</h2><a id="user-content-installation" class="anchor" aria-label="Permalink: Installation" href="#installation"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">Install this plugin in the same environment as Datasette.</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ pip install datasette-saved-queries"><pre class="notranslate"><code>$ pip install datasette-saved-queries
  </code></pre></div>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Usage</h2><a id="user-content-usage" class="anchor" aria-label="Permalink: Usage" href="#usage"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">When the plugin is installed Datasette will automatically create a <code>saved_queries</code> table in the first connected database when it starts up.</p>
  <p dir="auto">It also creates a <code>save_query</code> writable canned query which you can use to save new queries.</p>
  <p dir="auto">Queries that you save will be added to the query list on the database page.</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Development</h2><a id="user-content-development" class="anchor" aria-label="Permalink: Development" href="#development"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">To set up this plugin locally, first checkout the code. Then create a new virtual environment:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="cd datasette-saved-queries
  python -mvenv venv
  source venv/bin/activate"><pre class="notranslate"><code>cd datasette-saved-queries
  python -mvenv venv
  source venv/bin/activate
  </code></pre></div>
  <p dir="auto">Or if you are using <code>pipenv</code>:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="pipenv shell"><pre class="notranslate"><code>pipenv shell
  </code></pre></div>
  <p dir="auto">Now install the dependencies and tests:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="pip install -e '.[test]'"><pre class="notranslate"><code>pip install -e '.[test]'
  </code></pre></div>
  <p dir="auto">To run the tests:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="pytest"><pre class="notranslate"><code>pytest
  </code></pre></div>
  </article></div>


