# repos

node_id: MDEwOlJlcG9zaXRvcnkxOTUxNDU2Nzg=
name: sqlite-diffable
full_name: simonw/sqlite-diffable
html_url: https://github.com/simonw/sqlite-diffable
description: Tools for dumping/loading a SQLite database to diffable directory structure
created_at: 2019-07-04T00:58:46Z
updated_at: 2025-05-06T04:59:26Z
pushed_at: 2025-05-05T05:44:56Z
language: Python
license: apache-2.0
topics: ["datasette-io", "datasette-tool", "sqlite"]
visibility: public
default_branch: main
permissions: {"admin": false, "maintain": false, "push": false, "triage": false, "pull": true}
readme:
  # sqlite-diffable

  [![PyPI](https://img.shields.io/pypi/v/sqlite-diffable.svg)](https://pypi.org/project/sqlite-diffable/)
  [![Changelog](https://img.shields.io/github/v/release/simonw/sqlite-diffable?include_prereleases&label=changelog)](https://github.com/simonw/sqlite-diffable/releases)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/sqlite-diffable/blob/main/LICENSE)

  Tools for dumping/loading a SQLite database to diffable directory structure

  ## Installation

      pip install sqlite-diffable

  ## Demo

  The repository at [simonw/simonwillisonblog-backup](https://github.com/simonw/simonwillisonblog-backup) contains a backup of the database on my blog, https://simonwillison.net/ - created using this tool.

  ## Dumping a database

  Given a SQLite database called `fixtures.db` containing a table `facetable`, the following will dump out that table to the `dump/` directory:
  ```bash
  sqlite-diffable dump fixtures.db dump/ facetable
  ```
  To dump out every table in that database, use `--all`:
  ```bash
  sqlite-diffable dump fixtures.db dump/ --all
  ```
  To dump all table except some specific ones, use `--exclude` one or more times:
  ```bash
  sqlite-diffable dump fixtures.db dump/ --all \
    --exclude unwanted_first_table \
    --exclude unwanted_second_table
  ```
  ## Loading a database

  To load a previously dumped database, run the following:
  ```bash
  sqlite-diffable load restored.db dump/
  ```
  This will show an error if any of the tables that are being restored already exist in the database file.

  You can replace those tables (dropping them before restoring them) using the `--replace` option:
  ```bash
  sqlite-diffable load restored.db dump/ --replace
  ```
  ## Converting to JSON objects

  Table rows are stored in the `.ndjson` files as newline-delimited JSON arrays, like this:

  ```
  ["a", "a", "a-a", 63, null, 0.7364712141640124, "$null"]
  ["a", "b", "a-b", 51, null, 0.6020187290499803, "$null"]
  ```

  Sometimes it can be more convenient to work with a list of JSON objects.

  The `sqlite-diffable objects` command can read a `.ndjson` file and its accompanying `.metadata.json` file and output JSON objects to standard output:
  ```bash
  sqlite-diffable objects fixtures.db dump/sortable.ndjson
  ```
  The output of that command looks something like this:
  ```
  {"pk1": "a", "pk2": "a", "content": "a-a", "sortable": 63, "sortable_with_nulls": null, "sortable_with_nulls_2": 0.7364712141640124, "text": "$null"}
  {"pk1": "a", "pk2": "b", "content": "a-b", "sortable": 51, "sortable_with_nulls": null, "sortable_with_nulls_2": 0.6020187290499803, "text": "$null"}
  ```

  Add `-o` to write that output to a file:
  ```bash
  sqlite-diffable objects fixtures.db dump/sortable.ndjson -o output.txt
  ```
  Add `--array` to output a JSON array of objects, as opposed to a newline-delimited file:
  ```bash
  sqlite-diffable objects fixtures.db dump/sortable.ndjson --array
  ```
  Output:
  ```
  [
  {"pk1": "a", "pk2": "a", "content": "a-a", "sortable": 63, "sortable_with_nulls": null, "sortable_with_nulls_2": 0.7364712141640124, "text": "$null"},
  {"pk1": "a", "pk2": "b", "content": "a-b", "sortable": 51, "sortable_with_nulls": null, "sortable_with_nulls_2": 0.6020187290499803, "text": "$null"}
  ]
  ```

  ## Storage format

  Each table is represented as two files. The first, `table_name.metadata.json`, contains metadata describing the structure of the table. For a table called `redirects_redirect` that file might look like this:

  ```json
  {
      "name": "redirects_redirect",
      "columns": [
          "id",
          "domain",
          "path",
          "target",
          "created"
      ],
      "schema": "CREATE TABLE [redirects_redirect] (\n   [id] INTEGER PRIMARY KEY,\n   [domain] TEXT,\n   [path] TEXT,\n   [target] TEXT,\n   [created] TEXT\n)"
  }
  ```

  It is an object with three keys: `name` is the name of the table, `columns` is an array of column strings and `schema` is the SQL schema text used for tha table.

  The second file, `table_name.ndjson`, contains newline-delimited JSON (aka [JSON Lines](https://jsonlines.org/)) for every row in the table. Each row is represented as a JSON array with items corresponding to each of the columns defined in the metadata.

  That file for the `redirects_redirect.ndjson` table might look like this:

  ```
  [1, "feeds.simonwillison.net", "swn-everything", "https://simonwillison.net/atom/everything/", "2017-10-01T21:11:36.440537+00:00"]
  [2, "feeds.simonwillison.net", "swn-entries", "https://simonwillison.net/atom/entries/", "2017-10-01T21:12:32.478849+00:00"]
  [3, "feeds.simonwillison.net", "swn-links", "https://simonwillison.net/atom/links/", "2017-10-01T21:12:54.820729+00:00"]
  ```

readme_html:
  <div id="readme" class="md" data-path="README.md"><article class="markdown-body entry-content container-lg" itemprop="text"><div class="markdown-heading" dir="auto"><h1 class="heading-element" dir="auto">sqlite-diffable</h1><a id="user-content-sqlite-diffable" class="anchor" aria-label="Permalink: sqlite-diffable" href="#sqlite-diffable"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto"><a href="https://pypi.org/project/sqlite-diffable/" rel="nofollow"><img src="https://img.shields.io/pypi/v/sqlite-diffable.svg" alt="PyPI" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/746ad851f76dcf1857b734e710e07b1a36bc1c4daa8952018245e215a99e5425/68747470733a2f2f696d672e736869656c64732e696f2f707970692f762f73716c6974652d6469666661626c652e737667"></a>
  <a href="https://github.com/simonw/sqlite-diffable/releases"><img src="https://img.shields.io/github/v/release/simonw/sqlite-diffable?include_prereleases&label=changelog" alt="Changelog" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/3482943f3ab606753bdbcd8a90fe4d5c4c96bdb9f0fba42da2efb98c4da68576/68747470733a2f2f696d672e736869656c64732e696f2f6769746875622f762f72656c656173652f73696d6f6e772f73716c6974652d6469666661626c653f696e636c7564655f70726572656c6561736573266c6162656c3d6368616e67656c6f67"></a>
  <a href="https://github.com/simonw/sqlite-diffable/blob/main/LICENSE"><img src="https://img.shields.io/badge/license-Apache%202.0-blue.svg" alt="License" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/c355f200ea90fddaa407b6eaab303663a669248ea3ca7b1fcf77dbe04ff5f48c/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6c6963656e73652d417061636865253230322e302d626c75652e737667"></a></p>
  <p dir="auto">Tools for dumping/loading a SQLite database to diffable directory structure</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Installation</h2><a id="user-content-installation" class="anchor" aria-label="Permalink: Installation" href="#installation"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="pip install sqlite-diffable"><pre class="notranslate"><code>pip install sqlite-diffable
  </code></pre></div>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Demo</h2><a id="user-content-demo" class="anchor" aria-label="Permalink: Demo" href="#demo"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">The repository at <a href="https://github.com/simonw/simonwillisonblog-backup">simonw/simonwillisonblog-backup</a> contains a backup of the database on my blog, <a href="https://simonwillison.net/" rel="nofollow">https://simonwillison.net/</a> - created using this tool.</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Dumping a database</h2><a id="user-content-dumping-a-database" class="anchor" aria-label="Permalink: Dumping a database" href="#dumping-a-database"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">Given a SQLite database called <code>fixtures.db</code> containing a table <code>facetable</code>, the following will dump out that table to the <code>dump/</code> directory:</p>
  <div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="sqlite-diffable dump fixtures.db dump/ facetable"><pre>sqlite-diffable dump fixtures.db dump/ facetable</pre></div>
  <p dir="auto">To dump out every table in that database, use <code>--all</code>:</p>
  <div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="sqlite-diffable dump fixtures.db dump/ --all"><pre>sqlite-diffable dump fixtures.db dump/ --all</pre></div>
  <p dir="auto">To dump all table except some specific ones, use <code>--exclude</code> one or more times:</p>
  <div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="sqlite-diffable dump fixtures.db dump/ --all \
    --exclude unwanted_first_table \
    --exclude unwanted_second_table"><pre>sqlite-diffable dump fixtures.db dump/ --all \
    --exclude unwanted_first_table \
    --exclude unwanted_second_table</pre></div>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Loading a database</h2><a id="user-content-loading-a-database" class="anchor" aria-label="Permalink: Loading a database" href="#loading-a-database"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">To load a previously dumped database, run the following:</p>
  <div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="sqlite-diffable load restored.db dump/"><pre>sqlite-diffable load restored.db dump/</pre></div>
  <p dir="auto">This will show an error if any of the tables that are being restored already exist in the database file.</p>
  <p dir="auto">You can replace those tables (dropping them before restoring them) using the <code>--replace</code> option:</p>
  <div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="sqlite-diffable load restored.db dump/ --replace"><pre>sqlite-diffable load restored.db dump/ --replace</pre></div>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Converting to JSON objects</h2><a id="user-content-converting-to-json-objects" class="anchor" aria-label="Permalink: Converting to JSON objects" href="#converting-to-json-objects"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">Table rows are stored in the <code>.ndjson</code> files as newline-delimited JSON arrays, like this:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="["a", "a", "a-a", 63, null, 0.7364712141640124, "$null"]
  ["a", "b", "a-b", 51, null, 0.6020187290499803, "$null"]"><pre class="notranslate"><code>["a", "a", "a-a", 63, null, 0.7364712141640124, "$null"]
  ["a", "b", "a-b", 51, null, 0.6020187290499803, "$null"]
  </code></pre></div>
  <p dir="auto">Sometimes it can be more convenient to work with a list of JSON objects.</p>
  <p dir="auto">The <code>sqlite-diffable objects</code> command can read a <code>.ndjson</code> file and its accompanying <code>.metadata.json</code> file and output JSON objects to standard output:</p>
  <div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="sqlite-diffable objects fixtures.db dump/sortable.ndjson"><pre>sqlite-diffable objects fixtures.db dump/sortable.ndjson</pre></div>
  <p dir="auto">The output of that command looks something like this:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="{"pk1": "a", "pk2": "a", "content": "a-a", "sortable": 63, "sortable_with_nulls": null, "sortable_with_nulls_2": 0.7364712141640124, "text": "$null"}
  {"pk1": "a", "pk2": "b", "content": "a-b", "sortable": 51, "sortable_with_nulls": null, "sortable_with_nulls_2": 0.6020187290499803, "text": "$null"}"><pre class="notranslate"><code>{"pk1": "a", "pk2": "a", "content": "a-a", "sortable": 63, "sortable_with_nulls": null, "sortable_with_nulls_2": 0.7364712141640124, "text": "$null"}
  {"pk1": "a", "pk2": "b", "content": "a-b", "sortable": 51, "sortable_with_nulls": null, "sortable_with_nulls_2": 0.6020187290499803, "text": "$null"}
  </code></pre></div>
  <p dir="auto">Add <code>-o</code> to write that output to a file:</p>
  <div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="sqlite-diffable objects fixtures.db dump/sortable.ndjson -o output.txt"><pre>sqlite-diffable objects fixtures.db dump/sortable.ndjson -o output.txt</pre></div>
  <p dir="auto">Add <code>--array</code> to output a JSON array of objects, as opposed to a newline-delimited file:</p>
  <div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="sqlite-diffable objects fixtures.db dump/sortable.ndjson --array"><pre>sqlite-diffable objects fixtures.db dump/sortable.ndjson --array</pre></div>
  <p dir="auto">Output:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="[
  {"pk1": "a", "pk2": "a", "content": "a-a", "sortable": 63, "sortable_with_nulls": null, "sortable_with_nulls_2": 0.7364712141640124, "text": "$null"},
  {"pk1": "a", "pk2": "b", "content": "a-b", "sortable": 51, "sortable_with_nulls": null, "sortable_with_nulls_2": 0.6020187290499803, "text": "$null"}
  ]"><pre class="notranslate"><code>[
  {"pk1": "a", "pk2": "a", "content": "a-a", "sortable": 63, "sortable_with_nulls": null, "sortable_with_nulls_2": 0.7364712141640124, "text": "$null"},
  {"pk1": "a", "pk2": "b", "content": "a-b", "sortable": 51, "sortable_with_nulls": null, "sortable_with_nulls_2": 0.6020187290499803, "text": "$null"}
  ]
  </code></pre></div>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Storage format</h2><a id="user-content-storage-format" class="anchor" aria-label="Permalink: Storage format" href="#storage-format"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">Each table is represented as two files. The first, <code>table_name.metadata.json</code>, contains metadata describing the structure of the table. For a table called <code>redirects_redirect</code> that file might look like this:</p>
  <div class="highlight highlight-source-json notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="{
      "name": "redirects_redirect",
      "columns": [
          "id",
          "domain",
          "path",
          "target",
          "created"
      ],
      "schema": "CREATE TABLE [redirects_redirect] (\n   [id] INTEGER PRIMARY KEY,\n   [domain] TEXT,\n   [path] TEXT,\n   [target] TEXT,\n   [created] TEXT\n)"
  }"><pre>{
      <span class="pl-ent">"name"</span>: <span class="pl-s"><span class="pl-pds">"</span>redirects_redirect<span class="pl-pds">"</span></span>,
      <span class="pl-ent">"columns"</span>: [
          <span class="pl-s"><span class="pl-pds">"</span>id<span class="pl-pds">"</span></span>,
          <span class="pl-s"><span class="pl-pds">"</span>domain<span class="pl-pds">"</span></span>,
          <span class="pl-s"><span class="pl-pds">"</span>path<span class="pl-pds">"</span></span>,
          <span class="pl-s"><span class="pl-pds">"</span>target<span class="pl-pds">"</span></span>,
          <span class="pl-s"><span class="pl-pds">"</span>created<span class="pl-pds">"</span></span>
      ],
      <span class="pl-ent">"schema"</span>: <span class="pl-s"><span class="pl-pds">"</span>CREATE TABLE [redirects_redirect] (<span class="pl-cce">\n</span>   [id] INTEGER PRIMARY KEY,<span class="pl-cce">\n</span>   [domain] TEXT,<span class="pl-cce">\n</span>   [path] TEXT,<span class="pl-cce">\n</span>   [target] TEXT,<span class="pl-cce">\n</span>   [created] TEXT<span class="pl-cce">\n</span>)<span class="pl-pds">"</span></span>
  }</pre></div>
  <p dir="auto">It is an object with three keys: <code>name</code> is the name of the table, <code>columns</code> is an array of column strings and <code>schema</code> is the SQL schema text used for tha table.</p>
  <p dir="auto">The second file, <code>table_name.ndjson</code>, contains newline-delimited JSON (aka <a href="https://jsonlines.org/" rel="nofollow">JSON Lines</a>) for every row in the table. Each row is represented as a JSON array with items corresponding to each of the columns defined in the metadata.</p>
  <p dir="auto">That file for the <code>redirects_redirect.ndjson</code> table might look like this:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="[1, "feeds.simonwillison.net", "swn-everything", "https://simonwillison.net/atom/everything/", "2017-10-01T21:11:36.440537+00:00"]
  [2, "feeds.simonwillison.net", "swn-entries", "https://simonwillison.net/atom/entries/", "2017-10-01T21:12:32.478849+00:00"]
  [3, "feeds.simonwillison.net", "swn-links", "https://simonwillison.net/atom/links/", "2017-10-01T21:12:54.820729+00:00"]"><pre class="notranslate"><code>[1, "feeds.simonwillison.net", "swn-everything", "https://simonwillison.net/atom/everything/", "2017-10-01T21:11:36.440537+00:00"]
  [2, "feeds.simonwillison.net", "swn-entries", "https://simonwillison.net/atom/entries/", "2017-10-01T21:12:32.478849+00:00"]
  [3, "feeds.simonwillison.net", "swn-links", "https://simonwillison.net/atom/links/", "2017-10-01T21:12:54.820729+00:00"]
  </code></pre></div>
  </article></div>


node_id: MDEwOlJlcG9zaXRvcnkxOTU2OTY4MDQ=
name: datasette-cors
full_name: simonw/datasette-cors
html_url: https://github.com/simonw/datasette-cors
description: Datasette plugin for configuring CORS headers
created_at: 2019-07-07T21:03:11Z
updated_at: 2025-11-02T20:21:04Z
pushed_at: 2025-11-02T20:21:00Z
language: Python
license: apache-2.0
topics: ["datasette", "datasette-io", "datasette-plugin"]
visibility: public
default_branch: main
permissions: {"admin": false, "maintain": false, "push": false, "triage": false, "pull": false}
readme:
  # datasette-cors

  [![PyPI](https://img.shields.io/pypi/v/datasette-cors.svg)](https://pypi.org/project/datasette-cors/)
  [![Tests](https://github.com/simonw/datasette-cors/actions/workflows/test.yml/badge.svg)](https://github.com/simonw/datasette-cors/actions/workflows/test.yml)
  [![Changelog](https://img.shields.io/github/v/release/simonw/datasette-cors?include_prereleases&label=changelog)](https://github.com/simonw/datasette-cors/releases)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/datasette-cors/blob/main/LICENSE)

  Datasette plugin for configuring CORS headers, based on [asgi-cors](https://github.com/simonw/asgi-cors).

  You can use this plugin to allow JavaScript running on an allowlisted set of domains to make `fetch()` calls to the JSON API provided by your Datasette instance.

  ## Installation
  ```bash
  datasette install datasette-cors
  ```
  ## Configuration

  You need to add some plugin configuration for this plugin to take effect.

  To allowlist specific domains, use this:

  ```json
  {
      "plugins": {
          "datasette-cors": {
              "hosts": ["https://www.example.com"]
          }
      }
  }
  ```
  This affects the `access-control-allow-origin` header.

  You can also allowlist host patterns like this:

  ```json
  {
      "plugins": {
          "datasette-cors": {
              "host_wildcards": ["https://*.example.com"]
          }
      }
  }
  ```

  To allow all origins, use:

  ```json
  {
      "plugins": {
          "datasette-cors": {
              "allow_all": true
          }
      }
  }
  ```
  This sets the `access-control-allow-origin` header to `*`.

  You can specify allowed headers - with the `access-control-allow-headers` header - using the `headers` option:

  ```json
  {
      "plugins": {
          "datasette-cors": {
              "allow_all": true,
              "headers": ["Authorization", "Content-Type"]
          }
      }
  }
  ```

  To allow specific HTTP methods with the `access-control-allow-methods` header, use the `methods` option:

  ```json
  {
      "plugins": {
          "datasette-cors": {
              "allow_all": true,
              "methods": ["GET", "POST", "OPTIONS"]
          }
      }
  }
  ```

  You can set the `access-control-max-age` header using the `max_age` option:

  ```json
  {
      "plugins": {
          "datasette-cors": {
              "allow_all": true,
              "max_age": 3600
          }
      }
  }
  ```

  ## Testing it

  To test this plugin out, run it locally by saving one of the above examples as `metadata.json` and running this:
  ```bash
  datasette -m metadata.json
  ```
  With Datasette 1.0 use `-c config.json` instead, or try this:
  ```bash
  datasette -s plugins.datasette-cors.allow_all true
  ```

  Now visit https://www.example.com/ in your browser, open the browser developer console and paste in the following:

  ```javascript
  fetch("http://127.0.0.1:8001/_memory.json?sql=select+sqlite_version%28%29").then(r => r.json()).then(console.log)
  ```

  If the plugin is working correctly, you will see the JSON response output to the console.

readme_html:
  <div id="readme" class="md" data-path="README.md"><article class="markdown-body entry-content container-lg" itemprop="text"><div class="markdown-heading" dir="auto"><h1 class="heading-element" dir="auto">datasette-cors</h1><a id="user-content-datasette-cors" class="anchor" aria-label="Permalink: datasette-cors" href="#datasette-cors"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto"><a href="https://pypi.org/project/datasette-cors/" rel="nofollow"><img src="https://img.shields.io/pypi/v/datasette-cors.svg" alt="PyPI" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/ab52fc037b9d3a8a665b50788b237523cc5287182c54793c6934700333ced859/68747470733a2f2f696d672e736869656c64732e696f2f707970692f762f6461746173657474652d636f72732e737667"></a>
  <a href="https://github.com/simonw/datasette-cors/actions/workflows/test.yml"><img src="https://github.com/simonw/datasette-cors/actions/workflows/test.yml/badge.svg" alt="Tests" style="max-width: 100%;"></a>
  <a href="https://github.com/simonw/datasette-cors/releases"><img src="https://img.shields.io/github/v/release/simonw/datasette-cors?include_prereleases&label=changelog" alt="Changelog" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/93250246c3086e6b2401a7c23c86aff984e841685a3b3f7547ecd2e483033d7d/68747470733a2f2f696d672e736869656c64732e696f2f6769746875622f762f72656c656173652f73696d6f6e772f6461746173657474652d636f72733f696e636c7564655f70726572656c6561736573266c6162656c3d6368616e67656c6f67"></a>
  <a href="https://github.com/simonw/datasette-cors/blob/main/LICENSE"><img src="https://img.shields.io/badge/license-Apache%202.0-blue.svg" alt="License" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/c355f200ea90fddaa407b6eaab303663a669248ea3ca7b1fcf77dbe04ff5f48c/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6c6963656e73652d417061636865253230322e302d626c75652e737667"></a></p>
  <p dir="auto">Datasette plugin for configuring CORS headers, based on <a href="https://github.com/simonw/asgi-cors">asgi-cors</a>.</p>
  <p dir="auto">You can use this plugin to allow JavaScript running on an allowlisted set of domains to make <code>fetch()</code> calls to the JSON API provided by your Datasette instance.</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Installation</h2><a id="user-content-installation" class="anchor" aria-label="Permalink: Installation" href="#installation"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="datasette install datasette-cors"><pre>datasette install datasette-cors</pre></div>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Configuration</h2><a id="user-content-configuration" class="anchor" aria-label="Permalink: Configuration" href="#configuration"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">You need to add some plugin configuration for this plugin to take effect.</p>
  <p dir="auto">To allowlist specific domains, use this:</p>
  <div class="highlight highlight-source-json notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="{
      "plugins": {
          "datasette-cors": {
              "hosts": ["https://www.example.com"]
          }
      }
  }"><pre>{
      <span class="pl-ent">"plugins"</span>: {
          <span class="pl-ent">"datasette-cors"</span>: {
              <span class="pl-ent">"hosts"</span>: [<span class="pl-s"><span class="pl-pds">"</span>https://www.example.com<span class="pl-pds">"</span></span>]
          }
      }
  }</pre></div>
  <p dir="auto">This affects the <code>access-control-allow-origin</code> header.</p>
  <p dir="auto">You can also allowlist host patterns like this:</p>
  <div class="highlight highlight-source-json notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="{
      "plugins": {
          "datasette-cors": {
              "host_wildcards": ["https://*.example.com"]
          }
      }
  }"><pre>{
      <span class="pl-ent">"plugins"</span>: {
          <span class="pl-ent">"datasette-cors"</span>: {
              <span class="pl-ent">"host_wildcards"</span>: [<span class="pl-s"><span class="pl-pds">"</span>https://*.example.com<span class="pl-pds">"</span></span>]
          }
      }
  }</pre></div>
  <p dir="auto">To allow all origins, use:</p>
  <div class="highlight highlight-source-json notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="{
      "plugins": {
          "datasette-cors": {
              "allow_all": true
          }
      }
  }"><pre>{
      <span class="pl-ent">"plugins"</span>: {
          <span class="pl-ent">"datasette-cors"</span>: {
              <span class="pl-ent">"allow_all"</span>: <span class="pl-c1">true</span>
          }
      }
  }</pre></div>
  <p dir="auto">This sets the <code>access-control-allow-origin</code> header to <code>*</code>.</p>
  <p dir="auto">You can specify allowed headers - with the <code>access-control-allow-headers</code> header - using the <code>headers</code> option:</p>
  <div class="highlight highlight-source-json notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="{
      "plugins": {
          "datasette-cors": {
              "allow_all": true,
              "headers": ["Authorization", "Content-Type"]
          }
      }
  }"><pre>{
      <span class="pl-ent">"plugins"</span>: {
          <span class="pl-ent">"datasette-cors"</span>: {
              <span class="pl-ent">"allow_all"</span>: <span class="pl-c1">true</span>,
              <span class="pl-ent">"headers"</span>: [<span class="pl-s"><span class="pl-pds">"</span>Authorization<span class="pl-pds">"</span></span>, <span class="pl-s"><span class="pl-pds">"</span>Content-Type<span class="pl-pds">"</span></span>]
          }
      }
  }</pre></div>
  <p dir="auto">To allow specific HTTP methods with the <code>access-control-allow-methods</code> header, use the <code>methods</code> option:</p>
  <div class="highlight highlight-source-json notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="{
      "plugins": {
          "datasette-cors": {
              "allow_all": true,
              "methods": ["GET", "POST", "OPTIONS"]
          }
      }
  }"><pre>{
      <span class="pl-ent">"plugins"</span>: {
          <span class="pl-ent">"datasette-cors"</span>: {
              <span class="pl-ent">"allow_all"</span>: <span class="pl-c1">true</span>,
              <span class="pl-ent">"methods"</span>: [<span class="pl-s"><span class="pl-pds">"</span>GET<span class="pl-pds">"</span></span>, <span class="pl-s"><span class="pl-pds">"</span>POST<span class="pl-pds">"</span></span>, <span class="pl-s"><span class="pl-pds">"</span>OPTIONS<span class="pl-pds">"</span></span>]
          }
      }
  }</pre></div>
  <p dir="auto">You can set the <code>access-control-max-age</code> header using the <code>max_age</code> option:</p>
  <div class="highlight highlight-source-json notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="{
      "plugins": {
          "datasette-cors": {
              "allow_all": true,
              "max_age": 3600
          }
      }
  }"><pre>{
      <span class="pl-ent">"plugins"</span>: {
          <span class="pl-ent">"datasette-cors"</span>: {
              <span class="pl-ent">"allow_all"</span>: <span class="pl-c1">true</span>,
              <span class="pl-ent">"max_age"</span>: <span class="pl-c1">3600</span>
          }
      }
  }</pre></div>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Testing it</h2><a id="user-content-testing-it" class="anchor" aria-label="Permalink: Testing it" href="#testing-it"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">To test this plugin out, run it locally by saving one of the above examples as <code>metadata.json</code> and running this:</p>
  <div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="datasette -m metadata.json"><pre>datasette -m metadata.json</pre></div>
  <p dir="auto">With Datasette 1.0 use <code>-c config.json</code> instead, or try this:</p>
  <div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="datasette -s plugins.datasette-cors.allow_all true"><pre>datasette -s plugins.datasette-cors.allow_all <span class="pl-c1">true</span></pre></div>
  <p dir="auto">Now visit <a href="https://www.example.com/" rel="nofollow">https://www.example.com/</a> in your browser, open the browser developer console and paste in the following:</p>
  <div class="highlight highlight-source-js notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="fetch("http://127.0.0.1:8001/_memory.json?sql=select+sqlite_version%28%29").then(r => r.json()).then(console.log)"><pre><span class="pl-en">fetch</span><span class="pl-kos">(</span><span class="pl-s">"http://127.0.0.1:8001/_memory.json?sql=select+sqlite_version%28%29"</span><span class="pl-kos">)</span><span class="pl-kos">.</span><span class="pl-en">then</span><span class="pl-kos">(</span><span class="pl-s1">r</span> <span class="pl-c1">=></span> <span class="pl-s1">r</span><span class="pl-kos">.</span><span class="pl-en">json</span><span class="pl-kos">(</span><span class="pl-kos">)</span><span class="pl-kos">)</span><span class="pl-kos">.</span><span class="pl-en">then</span><span class="pl-kos">(</span><span class="pl-smi">console</span><span class="pl-kos">.</span><span class="pl-c1">log</span><span class="pl-kos">)</span></pre></div>
  <p dir="auto">If the plugin is working correctly, you will see the JSON response output to the console.</p>
  </article></div>


node_id: MDEwOlJlcG9zaXRvcnkxOTc0MzExMDk=
name: dogsheep-beta
full_name: dogsheep/dogsheep-beta
html_url: https://github.com/dogsheep/dogsheep-beta
created_at: 2019-07-17T17:07:26Z
updated_at: 2026-04-08T17:51:11Z
pushed_at: 2026-04-08T17:51:51Z
homepage: https://dogsheep.github.io/
language: Python
license: apache-2.0
topics: ["datasette", "datasette-io", "datasette-plugin", "datasette-tool", "dogsheep", "search"]
visibility: public
default_branch: main
permissions: {"admin": false, "maintain": false, "push": false, "triage": false, "pull": false}
custom_properties: {}
pull_request_creation_policy: all
description:
  Build a search index across content from multiple SQLite database tables and run faceted searches against it using Datasette

readme:
  # dogsheep-beta

  [![PyPI](https://img.shields.io/pypi/v/dogsheep-beta.svg)](https://pypi.org/project/dogsheep-beta/)
  [![Changelog](https://img.shields.io/github/v/release/dogsheep/beta?include_prereleases&label=changelog)](https://github.com/dogsheep/beta/releases)
  [![Tests](https://github.com/dogsheep/beta/workflows/Test/badge.svg)](https://github.com/dogsheep/beta/actions?query=workflow%3ATest)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/dogsheep/beta/blob/main/LICENSE)

  Build a search index across content from multiple SQLite database tables and run faceted searches against it using Datasette

  ## Example

  A live example of this plugin is running at https://datasette.io/-/beta - configured using [this YAML file](https://github.com/simonw/datasette.io/blob/main/templates/dogsheep-beta.yml).

  Read more about how this example works in [Building a search engine for datasette.io](https://simonwillison.net/2020/Dec/19/dogsheep-beta/).

  ## Installation

  Install this tool like so:

      $ pip install dogsheep-beta

  ## Usage

  Run the indexer using the `dogsheep-beta` command-line tool:

      $ dogsheep-beta index dogsheep.db config.yml

  The `config.yml` file contains details of the databases and document types that should be indexed:

  ```yaml
  twitter.db:
      tweets:
          sql: |-
              select
                  tweets.id as key,
                  'Tweet by @' || users.screen_name as title,
                  tweets.created_at as timestamp,
                  tweets.full_text as search_1
              from tweets join users on tweets.user = users.id
      users:
          sql: |-
              select
                  id as key,
                  name || ' @' || screen_name as title,
                  created_at as timestamp,
                  description as search_1
              from users
  ```

  This will create a `search_index` table in the `dogsheep.db` database populated by data from those SQL queries.

  By default the search index that this tool creates will be configured for Porter stemming. This means that searches for words like `run` will match documents containing `runs` or `running`.

  If you don't want to use Porter stemming, use the `--tokenize none` option:

      $ dogsheep-beta index dogsheep.db config.yml --tokenize none

  You can pass other SQLite tokenize argumenst here, see [the SQLite FTS tokenizers documentation](https://www.sqlite.org/fts5.html#tokenizers).

  ## Columns

  The columns that can be returned by our query are:

  - `key` - a unique (within that type) primary key
  - `title` - the title for the item
  - `timestamp` - an ISO8601 timestamp, e.g. `2020-09-02T21:00:21`
  - `search_1` - a larger chunk of text to be included in the search index
  - `category` - an integer category ID, see below
  - `is_public` - an integer (0 or 1, defaults to 0 if not set) specifying if this is public or not

  Public records are things like your public tweets, blog posts and GitHub commits.

  ## Categories

  Indexed items can be assigned a category. Categories are integers that correspond to records in the `categories` table, which defaults to containing the following:

  |   id | name       |
  |------|------------|
  |    1 | created    |
  |    2 | saved      |
  |    3 | received   |

  `created` is for items that have been created by the Dogsheep instance owner.

  `saved` is for items that they have saved, liked or favourited.

  `received` is for items that have been specifically sent to them by other people - incoming emails or direct messages for example.

  ## Datasette plugin

  Run `datasette install dogsheep-beta` (or use `pip install dogsheep-beta` in the same environment as Datasette) to install the Dogsheep Beta Datasette plugin.

  Once installed, a custom search interface will be made available at `/-/beta`. You can use this interface to execute searches.

  The Datasette plugin has some configuration options. You can set these by adding the following to your `metadata.json` configuration file:

  ```json
  {
      "plugins": {
          "dogsheep-beta": {
              "database": "beta",
              "config_file": "dogsheep-beta.yml",
              "template_debug": true
          }
      }
  }
  ```
  The configuration settings for the plugin are:
  - `database` - the database file that contains your search index. If the file is `beta.db` you should set `database` to `beta`.
  - `config_file` - the YAML file containing your Dogsheep Beta configuration.
  - `template_debug` - set this to `true` to enable debugging output if errors occur in your custom templates, see below.

  ## Custom results display

  Each indexed item type can define custom display HTML as part of the `config.yml` file. It can do this using a `display` key containing a fragment of Jinja template, and optionally a `display_sql` key with extra SQL to execute to fetch the data to display.

  Here's how to define a custom display template for a tweet:

  ```yaml
  twitter.db:
      tweets:
          sql: |-
              select
                  tweets.id as key,
                  'Tweet by @' || users.screen_name as title,
                  tweets.created_at as timestamp,
                  tweets.full_text as search_1
              from tweets join users on tweets.user = users.id
          display: |-
              <p>{{ title }} - tweeted at {{ timestamp }}</p>
              <blockquote>{{ search_1 }}</blockquote>
  ```
  This example reuses the value that were stored in the `search_index` table when the indexing query was run.

  To load in extra values to display in the template, use a `display_sql` query like this:

  ```yaml
  twitter.db:
      tweets:
          sql: |-
              select
                  tweets.id as key,
                  'Tweet by @' || users.screen_name as title,
                  tweets.created_at as timestamp,
                  tweets.full_text as search_1
              from tweets join users on tweets.user = users.id
          display_sql: |-
              select
                  users.screen_name,
                  tweets.full_text,
                  tweets.created_at
              from
                  tweets join users on tweets.user = users.id
              where
                  tweets.id = :key
          display: |-
              <p>{{ display.screen_name }} - tweeted at {{ display.created_at }}</p>
              <blockquote>{{ display.full_text }}</blockquote>
  ```
  The `display_sql` query will be executed for every search result, passing the key value from the `search_index` table as the `:key` parameter and the user's search term as the `:q` parameter.

  This performs well because [many small queries are efficient in SQLite](https://www.sqlite.org/np1queryprob.html).

  If an error occurs while rendering one of your templates the search results page will return a 500 error. You can use the `template_debug` configuration setting described above to instead output debugging information for the search results item that experienced the error.

  ## Displaying maps

  This plugin will eventually include a number of useful shortcuts for rendering interesting content.

  The first available shortcut is for displaying maps. Make your custom content output something like this:

  ```html
  <div
      data-map-latitude="{{ display.latitude }}"
      data-map-longitude="{{ display.longitude }}"
      style="display: none; float: right; width: 250px; height: 200px; background-color: #ccc;"
  ></div>
  ```
  JavaScript on the page will look for any elements with `data-map-latitude` and `data-map-longitude` and, if it finds any, will load Leaflet and convert those elements into maps centered on that location. The default zoom level will be 12, or you can set a `data-map-zoom` attribute to customize this.

  ## Development

  To set up this plugin locally, first checkout the code. Then create a new virtual environment:

      cd dogsheep-beta
      python3 -mvenv venv
      source venv/bin/activate

  Or if you are using `pipenv`:

      pipenv shell

  Now install the dependencies and tests:

      pip install -e '.[test]'

  To run the tests:

      pytest

readme_html:
  <div id="readme" class="md" data-path="README.md"><article class="markdown-body entry-content container-lg" itemprop="text"><div class="markdown-heading" dir="auto"><h1 class="heading-element" dir="auto">dogsheep-beta</h1><a id="user-content-dogsheep-beta" class="anchor" aria-label="Permalink: dogsheep-beta" href="#dogsheep-beta"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto"><a href="https://pypi.org/project/dogsheep-beta/" rel="nofollow"><img src="https://img.shields.io/pypi/v/dogsheep-beta.svg" alt="PyPI" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/d4be45b6c6861c8425f901798fedf1085f7b813ddc1f8a5a7f68ee542260bb2c/68747470733a2f2f696d672e736869656c64732e696f2f707970692f762f646f6773686565702d626574612e737667"></a>
  <a href="https://github.com/dogsheep/beta/releases"><img src="https://img.shields.io/github/v/release/dogsheep/beta?include_prereleases&label=changelog" alt="Changelog" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/46f835f2652d23ab823d214c9dd8d7c4abc4c44e7e55a30ded1dee93b4094ea2/68747470733a2f2f696d672e736869656c64732e696f2f6769746875622f762f72656c656173652f646f6773686565702f626574613f696e636c7564655f70726572656c6561736573266c6162656c3d6368616e67656c6f67"></a>
  <a href="https://github.com/dogsheep/beta/actions?query=workflow%3ATest"><img src="https://github.com/dogsheep/beta/workflows/Test/badge.svg" alt="Tests" style="max-width: 100%;"></a>
  <a href="https://github.com/dogsheep/beta/blob/main/LICENSE"><img src="https://img.shields.io/badge/license-Apache%202.0-blue.svg" alt="License" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/c355f200ea90fddaa407b6eaab303663a669248ea3ca7b1fcf77dbe04ff5f48c/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6c6963656e73652d417061636865253230322e302d626c75652e737667"></a></p>
  <p dir="auto">Build a search index across content from multiple SQLite database tables and run faceted searches against it using Datasette</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Example</h2><a id="user-content-example" class="anchor" aria-label="Permalink: Example" href="#example"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">A live example of this plugin is running at <a href="https://datasette.io/-/beta" rel="nofollow">https://datasette.io/-/beta</a> - configured using <a href="https://github.com/simonw/datasette.io/blob/main/templates/dogsheep-beta.yml">this YAML file</a>.</p>
  <p dir="auto">Read more about how this example works in <a href="https://simonwillison.net/2020/Dec/19/dogsheep-beta/" rel="nofollow">Building a search engine for datasette.io</a>.</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Installation</h2><a id="user-content-installation" class="anchor" aria-label="Permalink: Installation" href="#installation"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">Install this tool like so:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ pip install dogsheep-beta"><pre class="notranslate"><code>$ pip install dogsheep-beta
  </code></pre></div>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Usage</h2><a id="user-content-usage" class="anchor" aria-label="Permalink: Usage" href="#usage"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">Run the indexer using the <code>dogsheep-beta</code> command-line tool:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ dogsheep-beta index dogsheep.db config.yml"><pre class="notranslate"><code>$ dogsheep-beta index dogsheep.db config.yml
  </code></pre></div>
  <p dir="auto">The <code>config.yml</code> file contains details of the databases and document types that should be indexed:</p>
  <div class="highlight highlight-source-yaml notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="twitter.db:
      tweets:
          sql: |-
              select
                  tweets.id as key,
                  'Tweet by @' || users.screen_name as title,
                  tweets.created_at as timestamp,
                  tweets.full_text as search_1
              from tweets join users on tweets.user = users.id
      users:
          sql: |-
              select
                  id as key,
                  name || ' @' || screen_name as title,
                  created_at as timestamp,
                  description as search_1
              from users"><pre><span class="pl-ent">twitter.db</span>:
      <span class="pl-ent">tweets</span>:
          <span class="pl-ent">sql</span>: <span class="pl-s">|-</span>
  <span class="pl-s">            select</span>
  <span class="pl-s">                tweets.id as key,</span>
  <span class="pl-s">                'Tweet by @' || users.screen_name as title,</span>
  <span class="pl-s">                tweets.created_at as timestamp,</span>
  <span class="pl-s">                tweets.full_text as search_1</span>
  <span class="pl-s">            from tweets join users on tweets.user = users.id</span>
  <span class="pl-s"></span>    <span class="pl-ent">users</span>:
          <span class="pl-ent">sql</span>: <span class="pl-s">|-</span>
  <span class="pl-s">            select</span>
  <span class="pl-s">                id as key,</span>
  <span class="pl-s">                name || ' @' || screen_name as title,</span>
  <span class="pl-s">                created_at as timestamp,</span>
  <span class="pl-s">                description as search_1</span>
  <span class="pl-s">            from users</span></pre></div>
  <p dir="auto">This will create a <code>search_index</code> table in the <code>dogsheep.db</code> database populated by data from those SQL queries.</p>
  <p dir="auto">By default the search index that this tool creates will be configured for Porter stemming. This means that searches for words like <code>run</code> will match documents containing <code>runs</code> or <code>running</code>.</p>
  <p dir="auto">If you don't want to use Porter stemming, use the <code>--tokenize none</code> option:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ dogsheep-beta index dogsheep.db config.yml --tokenize none"><pre class="notranslate"><code>$ dogsheep-beta index dogsheep.db config.yml --tokenize none
  </code></pre></div>
  <p dir="auto">You can pass other SQLite tokenize argumenst here, see <a href="https://www.sqlite.org/fts5.html#tokenizers" rel="nofollow">the SQLite FTS tokenizers documentation</a>.</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Columns</h2><a id="user-content-columns" class="anchor" aria-label="Permalink: Columns" href="#columns"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">The columns that can be returned by our query are:</p>
  <ul dir="auto">
  <li><code>key</code> - a unique (within that type) primary key</li>
  <li><code>title</code> - the title for the item</li>
  <li><code>timestamp</code> - an ISO8601 timestamp, e.g. <code>2020-09-02T21:00:21</code></li>
  <li><code>search_1</code> - a larger chunk of text to be included in the search index</li>
  <li><code>category</code> - an integer category ID, see below</li>
  <li><code>is_public</code> - an integer (0 or 1, defaults to 0 if not set) specifying if this is public or not</li>
  </ul>
  <p dir="auto">Public records are things like your public tweets, blog posts and GitHub commits.</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Categories</h2><a id="user-content-categories" class="anchor" aria-label="Permalink: Categories" href="#categories"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">Indexed items can be assigned a category. Categories are integers that correspond to records in the <code>categories</code> table, which defaults to containing the following:</p>
  <markdown-accessiblity-table><table>
  <thead>
  <tr>
  <th>id</th>
  <th>name</th>
  </tr>
  </thead>
  <tbody>
  <tr>
  <td>1</td>
  <td>created</td>
  </tr>
  <tr>
  <td>2</td>
  <td>saved</td>
  </tr>
  <tr>
  <td>3</td>
  <td>received</td>
  </tr>
  </tbody>
  </table></markdown-accessiblity-table>
  <p dir="auto"><code>created</code> is for items that have been created by the Dogsheep instance owner.</p>
  <p dir="auto"><code>saved</code> is for items that they have saved, liked or favourited.</p>
  <p dir="auto"><code>received</code> is for items that have been specifically sent to them by other people - incoming emails or direct messages for example.</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Datasette plugin</h2><a id="user-content-datasette-plugin" class="anchor" aria-label="Permalink: Datasette plugin" href="#datasette-plugin"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">Run <code>datasette install dogsheep-beta</code> (or use <code>pip install dogsheep-beta</code> in the same environment as Datasette) to install the Dogsheep Beta Datasette plugin.</p>
  <p dir="auto">Once installed, a custom search interface will be made available at <code>/-/beta</code>. You can use this interface to execute searches.</p>
  <p dir="auto">The Datasette plugin has some configuration options. You can set these by adding the following to your <code>metadata.json</code> configuration file:</p>
  <div class="highlight highlight-source-json notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="{
      "plugins": {
          "dogsheep-beta": {
              "database": "beta",
              "config_file": "dogsheep-beta.yml",
              "template_debug": true
          }
      }
  }"><pre>{
      <span class="pl-ent">"plugins"</span>: {
          <span class="pl-ent">"dogsheep-beta"</span>: {
              <span class="pl-ent">"database"</span>: <span class="pl-s"><span class="pl-pds">"</span>beta<span class="pl-pds">"</span></span>,
              <span class="pl-ent">"config_file"</span>: <span class="pl-s"><span class="pl-pds">"</span>dogsheep-beta.yml<span class="pl-pds">"</span></span>,
              <span class="pl-ent">"template_debug"</span>: <span class="pl-c1">true</span>
          }
      }
  }</pre></div>
  <p dir="auto">The configuration settings for the plugin are:</p>
  <ul dir="auto">
  <li><code>database</code> - the database file that contains your search index. If the file is <code>beta.db</code> you should set <code>database</code> to <code>beta</code>.</li>
  <li><code>config_file</code> - the YAML file containing your Dogsheep Beta configuration.</li>
  <li><code>template_debug</code> - set this to <code>true</code> to enable debugging output if errors occur in your custom templates, see below.</li>
  </ul>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Custom results display</h2><a id="user-content-custom-results-display" class="anchor" aria-label="Permalink: Custom results display" href="#custom-results-display"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">Each indexed item type can define custom display HTML as part of the <code>config.yml</code> file. It can do this using a <code>display</code> key containing a fragment of Jinja template, and optionally a <code>display_sql</code> key with extra SQL to execute to fetch the data to display.</p>
  <p dir="auto">Here's how to define a custom display template for a tweet:</p>
  <div class="highlight highlight-source-yaml notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="twitter.db:
      tweets:
          sql: |-
              select
                  tweets.id as key,
                  'Tweet by @' || users.screen_name as title,
                  tweets.created_at as timestamp,
                  tweets.full_text as search_1
              from tweets join users on tweets.user = users.id
          display: |-
              <p>{{ title }} - tweeted at {{ timestamp }}</p>
              <blockquote>{{ search_1 }}</blockquote>"><pre><span class="pl-ent">twitter.db</span>:
      <span class="pl-ent">tweets</span>:
          <span class="pl-ent">sql</span>: <span class="pl-s">|-</span>
  <span class="pl-s">            select</span>
  <span class="pl-s">                tweets.id as key,</span>
  <span class="pl-s">                'Tweet by @' || users.screen_name as title,</span>
  <span class="pl-s">                tweets.created_at as timestamp,</span>
  <span class="pl-s">                tweets.full_text as search_1</span>
  <span class="pl-s">            from tweets join users on tweets.user = users.id</span>
  <span class="pl-s"></span>        <span class="pl-ent">display</span>: <span class="pl-s">|-</span>
  <span class="pl-s">            <p>{{ title }} - tweeted at {{ timestamp }}</p></span>
  <span class="pl-s">            <blockquote>{{ search_1 }}</blockquote></span></pre></div>
  <p dir="auto">This example reuses the value that were stored in the <code>search_index</code> table when the indexing query was run.</p>
  <p dir="auto">To load in extra values to display in the template, use a <code>display_sql</code> query like this:</p>
  <div class="highlight highlight-source-yaml notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="twitter.db:
      tweets:
          sql: |-
              select
                  tweets.id as key,
                  'Tweet by @' || users.screen_name as title,
                  tweets.created_at as timestamp,
                  tweets.full_text as search_1
              from tweets join users on tweets.user = users.id
          display_sql: |-
              select
                  users.screen_name,
                  tweets.full_text,
                  tweets.created_at
              from
                  tweets join users on tweets.user = users.id
              where
                  tweets.id = :key
          display: |-
              <p>{{ display.screen_name }} - tweeted at {{ display.created_at }}</p>
              <blockquote>{{ display.full_text }}</blockquote>"><pre><span class="pl-ent">twitter.db</span>:
      <span class="pl-ent">tweets</span>:
          <span class="pl-ent">sql</span>: <span class="pl-s">|-</span>
  <span class="pl-s">            select</span>
  <span class="pl-s">                tweets.id as key,</span>
  <span class="pl-s">                'Tweet by @' || users.screen_name as title,</span>
  <span class="pl-s">                tweets.created_at as timestamp,</span>
  <span class="pl-s">                tweets.full_text as search_1</span>
  <span class="pl-s">            from tweets join users on tweets.user = users.id</span>
  <span class="pl-s"></span>        <span class="pl-ent">display_sql</span>: <span class="pl-s">|-</span>
  <span class="pl-s">            select</span>
  <span class="pl-s">                users.screen_name,</span>
  <span class="pl-s">                tweets.full_text,</span>
  <span class="pl-s">                tweets.created_at</span>
  <span class="pl-s">            from</span>
  <span class="pl-s">                tweets join users on tweets.user = users.id</span>
  <span class="pl-s">            where</span>
  <span class="pl-s">                tweets.id = :key</span>
  <span class="pl-s"></span>        <span class="pl-ent">display</span>: <span class="pl-s">|-</span>
  <span class="pl-s">            <p>{{ display.screen_name }} - tweeted at {{ display.created_at }}</p></span>
  <span class="pl-s">            <blockquote>{{ display.full_text }}</blockquote></span></pre></div>
  <p dir="auto">The <code>display_sql</code> query will be executed for every search result, passing the key value from the <code>search_index</code> table as the <code>:key</code> parameter and the user's search term as the <code>:q</code> parameter.</p>
  <p dir="auto">This performs well because <a href="https://www.sqlite.org/np1queryprob.html" rel="nofollow">many small queries are efficient in SQLite</a>.</p>
  <p dir="auto">If an error occurs while rendering one of your templates the search results page will return a 500 error. You can use the <code>template_debug</code> configuration setting described above to instead output debugging information for the search results item that experienced the error.</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Displaying maps</h2><a id="user-content-displaying-maps" class="anchor" aria-label="Permalink: Displaying maps" href="#displaying-maps"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">This plugin will eventually include a number of useful shortcuts for rendering interesting content.</p>
  <p dir="auto">The first available shortcut is for displaying maps. Make your custom content output something like this:</p>
  <div class="highlight highlight-text-html-basic notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="<div
      data-map-latitude="{{ display.latitude }}"
      data-map-longitude="{{ display.longitude }}"
      style="display: none; float: right; width: 250px; height: 200px; background-color: #ccc;"
  ></div>"><pre><span class="pl-kos"><</span><span class="pl-ent">div</span>
      <span class="pl-c1">data-map-latitude</span>="<span class="pl-s">{{ display.latitude }}</span>"
      <span class="pl-c1">data-map-longitude</span>="<span class="pl-s">{{ display.longitude }}</span>"
      <span class="pl-c1">style</span>="<span class="pl-s">display: none; float: right; width: 250px; height: 200px; background-color: #ccc;</span>"
  <span class="pl-kos">></span><span class="pl-kos"></</span><span class="pl-ent">div</span><span class="pl-kos">></span></pre></div>
  <p dir="auto">JavaScript on the page will look for any elements with <code>data-map-latitude</code> and <code>data-map-longitude</code> and, if it finds any, will load Leaflet and convert those elements into maps centered on that location. The default zoom level will be 12, or you can set a <code>data-map-zoom</code> attribute to customize this.</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Development</h2><a id="user-content-development" class="anchor" aria-label="Permalink: Development" href="#development"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">To set up this plugin locally, first checkout the code. Then create a new virtual environment:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="cd dogsheep-beta
  python3 -mvenv venv
  source venv/bin/activate"><pre class="notranslate"><code>cd dogsheep-beta
  python3 -mvenv venv
  source venv/bin/activate
  </code></pre></div>
  <p dir="auto">Or if you are using <code>pipenv</code>:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="pipenv shell"><pre class="notranslate"><code>pipenv shell
  </code></pre></div>
  <p dir="auto">Now install the dependencies and tests:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="pip install -e '.[test]'"><pre class="notranslate"><code>pip install -e '.[test]'
  </code></pre></div>
  <p dir="auto">To run the tests:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="pytest"><pre class="notranslate"><code>pytest
  </code></pre></div>
  </article></div>


node_id: MDEwOlJlcG9zaXRvcnkxOTc4ODIzODI=
name: healthkit-to-sqlite
full_name: dogsheep/healthkit-to-sqlite
html_url: https://github.com/dogsheep/healthkit-to-sqlite
description: Convert an Apple Healthkit export zip to a SQLite database
created_at: 2019-07-20T05:03:12Z
updated_at: 2025-04-25T20:29:57Z
pushed_at: 2023-01-01T17:29:57Z
homepage: https://datasette.io/tools/healthkit-to-sqlite
language: Python
license: apache-2.0
topics: ["datasette", "datasette-io", "datasette-tool", "dogsheep", "healthkit", "sqlite"]
visibility: public
default_branch: main
permissions: {"admin": false, "maintain": false, "push": false, "triage": false, "pull": true}
custom_properties: {}
readme:
  # healthkit-to-sqlite

  [![PyPI](https://img.shields.io/pypi/v/healthkit-to-sqlite.svg)](https://pypi.org/project/healthkit-to-sqlite/)
  [![Changelog](https://img.shields.io/github/v/release/dogsheep/healthkit-to-sqlite?include_prereleases&label=changelog)](https://github.com/dogsheep/healthkit-to-sqlite/releases)
  [![Tests](https://github.com/dogsheep/healthkit-to-sqlite/workflows/Test/badge.svg)](https://github.com/dogsheep/healthkit-to-sqlite/actions?query=workflow%3ATest)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/dogsheep/healthkit-to-sqlite/blob/main/LICENSE)

  Convert an Apple Healthkit export zip to a SQLite database

  ## How to install

      $ pip install healthkit-to-sqlite

  ## How to use

  First you need to export your Apple HealthKit data.

  1. On your iPhone, open the "Health" app
  2. Click the profile icon in the top right
  3. Click "Export Health Data" at the bottom of that page
  4. Save the resulting file somewhere you can access it, or AirDrop it directly to your laptop.

  Now you can convert the resulting `export.zip` file to SQLite like so:

      $ healthkit-to-sqlite export.zip healthkit.db

  A progress bar will be displayed. You can disable this using `--silent`.

  ```
  Importing from HealthKit  [#-------------]    5%  00:01:33
  ```

  You can explore the resulting data using [Datasette](https://datasette.readthedocs.io/) like this:

      $ datasette healthkit.db

readme_html:
  <div id="readme" class="md" data-path="README.md"><article class="markdown-body entry-content container-lg" itemprop="text"><div class="markdown-heading" dir="auto"><h1 class="heading-element" dir="auto">healthkit-to-sqlite</h1><a id="user-content-healthkit-to-sqlite" class="anchor" aria-label="Permalink: healthkit-to-sqlite" href="#healthkit-to-sqlite"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto"><a href="https://pypi.org/project/healthkit-to-sqlite/" rel="nofollow"><img src="https://img.shields.io/pypi/v/healthkit-to-sqlite.svg" alt="PyPI" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/0028ee32440725629b6acfa907e490c197516f721eb078db7ae1aa068aad3007/68747470733a2f2f696d672e736869656c64732e696f2f707970692f762f6865616c74686b69742d746f2d73716c6974652e737667"></a>
  <a href="https://github.com/dogsheep/healthkit-to-sqlite/releases"><img src="https://img.shields.io/github/v/release/dogsheep/healthkit-to-sqlite?include_prereleases&label=changelog" alt="Changelog" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/95fd80fc59a47d96b84c03a29926996ce78dcdbcf69c2c8cd72c1d2660b563ee/68747470733a2f2f696d672e736869656c64732e696f2f6769746875622f762f72656c656173652f646f6773686565702f6865616c74686b69742d746f2d73716c6974653f696e636c7564655f70726572656c6561736573266c6162656c3d6368616e67656c6f67"></a>
  <a href="https://github.com/dogsheep/healthkit-to-sqlite/actions?query=workflow%3ATest"><img src="https://github.com/dogsheep/healthkit-to-sqlite/workflows/Test/badge.svg" alt="Tests" style="max-width: 100%;"></a>
  <a href="https://github.com/dogsheep/healthkit-to-sqlite/blob/main/LICENSE"><img src="https://img.shields.io/badge/license-Apache%202.0-blue.svg" alt="License" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/c355f200ea90fddaa407b6eaab303663a669248ea3ca7b1fcf77dbe04ff5f48c/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6c6963656e73652d417061636865253230322e302d626c75652e737667"></a></p>
  <p dir="auto">Convert an Apple Healthkit export zip to a SQLite database</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">How to install</h2><a id="user-content-how-to-install" class="anchor" aria-label="Permalink: How to install" href="#how-to-install"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ pip install healthkit-to-sqlite"><pre class="notranslate"><code>$ pip install healthkit-to-sqlite
  </code></pre></div>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">How to use</h2><a id="user-content-how-to-use" class="anchor" aria-label="Permalink: How to use" href="#how-to-use"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">First you need to export your Apple HealthKit data.</p>
  <ol dir="auto">
  <li>On your iPhone, open the "Health" app</li>
  <li>Click the profile icon in the top right</li>
  <li>Click "Export Health Data" at the bottom of that page</li>
  <li>Save the resulting file somewhere you can access it, or AirDrop it directly to your laptop.</li>
  </ol>
  <p dir="auto">Now you can convert the resulting <code>export.zip</code> file to SQLite like so:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ healthkit-to-sqlite export.zip healthkit.db"><pre class="notranslate"><code>$ healthkit-to-sqlite export.zip healthkit.db
  </code></pre></div>
  <p dir="auto">A progress bar will be displayed. You can disable this using <code>--silent</code>.</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="Importing from HealthKit  [#-------------]    5%  00:01:33"><pre class="notranslate"><code>Importing from HealthKit  [#-------------]    5%  00:01:33
  </code></pre></div>
  <p dir="auto">You can explore the resulting data using <a href="https://datasette.readthedocs.io/" rel="nofollow">Datasette</a> like this:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ datasette healthkit.db"><pre class="notranslate"><code>$ datasette healthkit.db
  </code></pre></div>
  </article></div>


node_id: MDEwOlJlcG9zaXRvcnkyMDU0MjkzNzU=
name: swarm-to-sqlite
full_name: dogsheep/swarm-to-sqlite
html_url: https://github.com/dogsheep/swarm-to-sqlite
description: Create a SQLite database containing your checkin history from Foursquare Swarm
created_at: 2019-08-30T17:37:29Z
updated_at: 2024-11-25T19:42:49Z
pushed_at: 2023-04-11T03:18:16Z
language: Python
license: apache-2.0
visibility: public
default_branch: main
permissions: {"admin": false, "maintain": false, "push": false, "triage": false, "pull": true}
custom_properties: {}
topics:
  ["datasette", "datasette-io", "datasette-tool", "dogsheep", "foursquare", "foursquare-api", "sqlite", "swarm"]

readme:
  # swarm-to-sqlite

  [![PyPI](https://img.shields.io/pypi/v/swarm-to-sqlite.svg)](https://pypi.org/project/swarm-to-sqlite/)
  [![Changelog](https://img.shields.io/github/v/release/dogsheep/swarm-to-sqlite?include_prereleases&label=changelog)](https://github.com/dogsheep/swarm-to-sqlite/releases)
  [![Tests](https://github.com/dogsheep/swarm-to-sqlite/workflows/Test/badge.svg)](https://github.com/dogsheep/swarm-to-sqlite/actions?query=workflow%3ATest)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/dogsheep/swarm-to-sqlite/blob/main/LICENSE)

  Create a SQLite database containing your checkin history from Foursquare Swarm.

  ## How to install

      $ pip install swarm-to-sqlite

  ## Usage

  You will need to first obtain a valid OAuth token for your Foursquare account. You can do so using this tool: https://your-foursquare-oauth-token.glitch.me/

  Simplest usage is to simply provide the name of the database file you wish to write to. The tool will prompt you to paste in your token, and will then download your checkins and store them in the specified database file.

      $ swarm-to-sqlite checkins.db
      Please provide your Foursquare OAuth token:
      Importing 3699 checkins  [#########-----------------------] 27% 00:02:31

  You can also pass the token as a command-line option:

      $ swarm-to-sqlite checkins.db --token=XXX

  Or as an environment variable:

      $ export FOURSQUARE_TOKEN=XXX
      $ swarm-to-sqlite checkins.db

  To retrieve just checkins within the past X hours, days or weeks, use the `--since=` option. For example, to pull only checkins that happened within the last 10 days use:

      $ swarm-to-sqlite checkins.db --token=XXX --since=10d

  Use `2w` for two weeks, `10h` for ten hours, `3d` for three days.

  In addition to saving the checkins to a database, you can also write them to a JSON file using the `--save` option:

      $ swarm-to-sqlite checkins.db --save=checkins.json

  Having done this, you can re-import checkins directly from that file (rather than making API calls to fetch data from Foursquare) like this:

      $ swarm-to-sqlite checkins.db --load=checkins.json

  ## Using with Datasette

  The SQLite database produced by this tool is designed to be browsed using [Datasette](https://datasette.io/).

  You can install the [datasette-cluster-map](https://datasette.io/plugins/datasette-cluster-map) plugin to view your checkins on a map.

readme_html:
  <div id="readme" class="md" data-path="README.md"><article class="markdown-body entry-content container-lg" itemprop="text"><div class="markdown-heading" dir="auto"><h1 class="heading-element" dir="auto">swarm-to-sqlite</h1><a id="user-content-swarm-to-sqlite" class="anchor" aria-label="Permalink: swarm-to-sqlite" href="#swarm-to-sqlite"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto"><a href="https://pypi.org/project/swarm-to-sqlite/" rel="nofollow"><img src="https://img.shields.io/pypi/v/swarm-to-sqlite.svg" alt="PyPI" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/d187669074d60b1bb8eb1418d9824521ea0270830bcafe019138b0a7085d5028/68747470733a2f2f696d672e736869656c64732e696f2f707970692f762f737761726d2d746f2d73716c6974652e737667"></a>
  <a href="https://github.com/dogsheep/swarm-to-sqlite/releases"><img src="https://img.shields.io/github/v/release/dogsheep/swarm-to-sqlite?include_prereleases&label=changelog" alt="Changelog" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/05858f2f2b7233da04d4964fab6d71aa5cb1b55ce2a91a06dc6a45b512cc41a7/68747470733a2f2f696d672e736869656c64732e696f2f6769746875622f762f72656c656173652f646f6773686565702f737761726d2d746f2d73716c6974653f696e636c7564655f70726572656c6561736573266c6162656c3d6368616e67656c6f67"></a>
  <a href="https://github.com/dogsheep/swarm-to-sqlite/actions?query=workflow%3ATest"><img src="https://github.com/dogsheep/swarm-to-sqlite/workflows/Test/badge.svg" alt="Tests" style="max-width: 100%;"></a>
  <a href="https://github.com/dogsheep/swarm-to-sqlite/blob/main/LICENSE"><img src="https://img.shields.io/badge/license-Apache%202.0-blue.svg" alt="License" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/c355f200ea90fddaa407b6eaab303663a669248ea3ca7b1fcf77dbe04ff5f48c/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6c6963656e73652d417061636865253230322e302d626c75652e737667"></a></p>
  <p dir="auto">Create a SQLite database containing your checkin history from Foursquare Swarm.</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">How to install</h2><a id="user-content-how-to-install" class="anchor" aria-label="Permalink: How to install" href="#how-to-install"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ pip install swarm-to-sqlite"><pre class="notranslate"><code>$ pip install swarm-to-sqlite
  </code></pre></div>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Usage</h2><a id="user-content-usage" class="anchor" aria-label="Permalink: Usage" href="#usage"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">You will need to first obtain a valid OAuth token for your Foursquare account. You can do so using this tool: <a href="https://your-foursquare-oauth-token.glitch.me/" rel="nofollow">https://your-foursquare-oauth-token.glitch.me/</a></p>
  <p dir="auto">Simplest usage is to simply provide the name of the database file you wish to write to. The tool will prompt you to paste in your token, and will then download your checkins and store them in the specified database file.</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ swarm-to-sqlite checkins.db
  Please provide your Foursquare OAuth token:
  Importing 3699 checkins  [#########-----------------------] 27% 00:02:31"><pre class="notranslate"><code>$ swarm-to-sqlite checkins.db
  Please provide your Foursquare OAuth token:
  Importing 3699 checkins  [#########-----------------------] 27% 00:02:31
  </code></pre></div>
  <p dir="auto">You can also pass the token as a command-line option:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ swarm-to-sqlite checkins.db --token=XXX"><pre class="notranslate"><code>$ swarm-to-sqlite checkins.db --token=XXX
  </code></pre></div>
  <p dir="auto">Or as an environment variable:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ export FOURSQUARE_TOKEN=XXX
  $ swarm-to-sqlite checkins.db"><pre class="notranslate"><code>$ export FOURSQUARE_TOKEN=XXX
  $ swarm-to-sqlite checkins.db
  </code></pre></div>
  <p dir="auto">To retrieve just checkins within the past X hours, days or weeks, use the <code>--since=</code> option. For example, to pull only checkins that happened within the last 10 days use:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ swarm-to-sqlite checkins.db --token=XXX --since=10d"><pre class="notranslate"><code>$ swarm-to-sqlite checkins.db --token=XXX --since=10d
  </code></pre></div>
  <p dir="auto">Use <code>2w</code> for two weeks, <code>10h</code> for ten hours, <code>3d</code> for three days.</p>
  <p dir="auto">In addition to saving the checkins to a database, you can also write them to a JSON file using the <code>--save</code> option:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ swarm-to-sqlite checkins.db --save=checkins.json"><pre class="notranslate"><code>$ swarm-to-sqlite checkins.db --save=checkins.json
  </code></pre></div>
  <p dir="auto">Having done this, you can re-import checkins directly from that file (rather than making API calls to fetch data from Foursquare) like this:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ swarm-to-sqlite checkins.db --load=checkins.json"><pre class="notranslate"><code>$ swarm-to-sqlite checkins.db --load=checkins.json
  </code></pre></div>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Using with Datasette</h2><a id="user-content-using-with-datasette" class="anchor" aria-label="Permalink: Using with Datasette" href="#using-with-datasette"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">The SQLite database produced by this tool is designed to be browsed using <a href="https://datasette.io/" rel="nofollow">Datasette</a>.</p>
  <p dir="auto">You can install the <a href="https://datasette.io/plugins/datasette-cluster-map" rel="nofollow">datasette-cluster-map</a> plugin to view your checkins on a map.</p>
  </article></div>


node_id: MDEwOlJlcG9zaXRvcnkyMDYxNTY4NjY=
name: twitter-to-sqlite
full_name: dogsheep/twitter-to-sqlite
html_url: https://github.com/dogsheep/twitter-to-sqlite
description: Save data from Twitter to a SQLite database
created_at: 2019-09-03T19:30:08Z
updated_at: 2025-04-25T20:30:49Z
pushed_at: 2022-12-28T23:33:31Z
language: Python
license: apache-2.0
topics: ["datasette", "datasette-io", "datasette-tool", "dogsheep", "sqlite", "twitter", "twitter-api"]
visibility: public
default_branch: main
permissions: {"admin": false, "maintain": false, "push": false, "triage": false, "pull": true}
custom_properties: {}
readme:
  # twitter-to-sqlite

  [![PyPI](https://img.shields.io/pypi/v/twitter-to-sqlite.svg)](https://pypi.org/project/twitter-to-sqlite/)
  [![Changelog](https://img.shields.io/github/v/release/dogsheep/twitter-to-sqlite?include_prereleases&label=changelog)](https://github.com/dogsheep/twitter-to-sqlite/releases)
  [![Tests](https://github.com/dogsheep/twitter-to-sqlite/workflows/Test/badge.svg)](https://github.com/dogsheep/twitter-to-sqlite/actions?query=workflow%3ATest)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/dogsheep/twitter-to-sqlite/blob/main/LICENSE)

  Save data from Twitter to a SQLite database.

  **This tool currently uses Twitter API v1**. You may be unable to use it if you do not have an API key for that version of the API.

  <!-- toc -->

  - [How to install](#how-to-install)
  - [Authentication](#authentication)
  - [Retrieving tweets by specific accounts](#retrieving-tweets-by-specific-accounts)
  - [Retrieve user profiles in bulk](#retrieve-user-profiles-in-bulk)
  - [Retrieve tweets in bulk](#retrieve-tweets-in-bulk)
  - [Retrieving Twitter followers](#retrieving-twitter-followers)
  - [Retrieving friends](#retrieving-friends)
  - [Retrieving favorited tweets](#retrieving-favorited-tweets)
  - [Retrieving Twitter lists](#retrieving-twitter-lists)
  - [Retrieving Twitter list memberships](#retrieving-twitter-list-memberships)
  - [Retrieving just follower and friend IDs](#retrieving-just-follower-and-friend-ids)
  - [Retrieving tweets from your home timeline](#retrieving-tweets-from-your-home-timeline)
  - [Retrieving your mentions](#retrieving-your-mentions)
  - [Providing input from a SQL query with --sql and --attach](#providing-input-from-a-sql-query-with---sql-and---attach)
  - [Running searches](#running-searches)
  - [Capturing tweets in real-time with track and follow](#capturing-tweets-in-real-time-with-track-and-follow)
    * [track](#track)
    * [follow](#follow)
  - [Importing data from your Twitter archive](#importing-data-from-your-twitter-archive)
  - [Design notes](#design-notes)

  <!-- tocstop -->

  ## How to install

      $ pip install twitter-to-sqlite

  ## Authentication

  First, you will need to create a Twitter application at https://developer.twitter.com/en/apps. You may need to apply for a Twitter developer account - if so, you may find this [example of an email application](https://raw.githubusercontent.com/dogsheep/twitter-to-sqlite/main/email.png) useful that has been approved in the past.

  Once you have created your application, navigate to the "Keys and tokens" page and make note of the following:

  * Your API key
  * Your API secret key
  * Your access token
  * Your access token secret

  You will need to save all four of these values to a JSON file in order to use this tool.

  You can create that JSON file by running the following command and pasting in the values at the prompts:

      $ twitter-to-sqlite auth
      Create an app here: https://developer.twitter.com/en/apps
      Then navigate to 'Keys and tokens' and paste in the following:

      API key: xxx
      API secret key: xxx
      Access token: xxx
      Access token secret: xxx

  This will create a file called `auth.json` in your current directory containing the required values. To save the file at a different path or filename, use the `--auth=myauth.json` option.

  ## Retrieving tweets by specific accounts

  The `user-timeline` command retrieves all of the tweets posted by the specified user accounts. It defaults to the account belonging to the authenticated user:

      $ twitter-to-sqlite user-timeline twitter.db
      Importing tweets  [#####-------------------------------]  2799/17780  00:01:39

  All of these commands assume that there is an `auth.json` file in the current directory. You can provide the path to your `auth.json` file using `-a`:

      $ twitter-to-sqlite user-timeline twitter.db -a /path/to/auth.json

  To load tweets for other users, pass their screen names as arguments:

      $ twitter-to-sqlite user-timeline twitter.db cleopaws nichemuseums

  Twitter's API only returns up to around 3,200 tweets for most user accounts, but you may find that it returns all available tweets for your own user account.

  You can pass numeric Twitter user IDs instead of screen names using the `--ids` parameter.

  You can use `--since` to retrieve every tweet since the last time you imported for that user, or `--since_id=xxx` to retrieve every tweet since a specific tweet ID.

  This command also accepts `--sql` and `--attach` options, documented below.

  ## Retrieve user profiles in bulk

  If you have a list of Twitter screen names (or user IDs) you can bulk fetch their fully inflated Twitter profiles using the `users-lookup` command:

      $ twitter-to-sqlite users-lookup users.db simonw cleopaws

  You can pass user IDs instead using the `--ids` option:

      $ twitter-to-sqlite users-lookup users.db 12497 3166449535 --ids

  This command also accepts `--sql` and `--attach` options, documented below.

  ## Retrieve tweets in bulk

  If you have a list of tweet IDS you can bulk fetch them using the `statuses-lookup` command:

      $ twitter-to-sqlite statuses-lookup tweets.db 1122154819815239680 1122154178493575169

  The `--sql` and `--attach` options are supported.

  Here's a recipe to retrieve any tweets that existing tweets are in-reply-to which have not yet been stored in your database:

      $ twitter-to-sqlite statuses-lookup tweets.db \
          --sql='
              select in_reply_to_status_id
              from tweets
              where in_reply_to_status_id is not null' \
          --skip-existing

  The `--skip-existing` option means that tweets that have already been stored in the database will not be fetched again.

  ## Retrieving Twitter followers

  The `followers` command retrieves details of every follower of the specified accounts. You can use it to retrieve your own followers, or you can pass one or more screen names to pull the followers for other accounts.

  The following command pulls your followers and saves them in a SQLite database file called `twitter.db`:

      $ twitter-to-sqlite followers twitter.db

  This command is **extremely slow**, because Twitter impose a rate limit of no more than one request per minute to this endpoint! If you are running it against an account with thousands of followers you should expect this to take several hours.

  To retrieve followers for another account, use:

      $ twitter-to-sqlite followers twitter.db cleopaws

  This command also accepts the `--ids`, `--sql` and `--attach` options.

  See [Analyzing my Twitter followers with Datasette](https://simonwillison.net/2018/Jan/28/analyzing-my-twitter-followers/) for the original inspiration for this command.

  ## Retrieving friends

  The `friends` command works like the `followers` command, but retrieves the specified (or currently authenticated) user's friends - defined as accounts that the user is following.

      $ twitter-to-sqlite friends twitter.db

  It takes the same options as the `followers` command.

  ## Retrieving favorited tweets

  The `favorites` command retrieves tweets that have been favorited by a specified user. Called without any extra arguments it retrieves tweets favorited by the currently authenticated user:

      $ twitter-to-sqlite favorites faves.db

  You can also use the `--screen_name` or `--user_id` arguments to retrieve favorite tweets for another user:

      $ twitter-to-sqlite favorites faves-obama.db --screen_name=BarackObama

  Use the `--stop_after=xxx` argument to retrieve only the most recent number of favorites, e.g. to get the authenticated user's 50 most recent favorites:

      $ twitter-to-sqlite favorites faves.db --stop_after=50

  ## Retrieving Twitter lists

  The `lists` command retrieves all of the lists belonging to one or more users.

      $ twitter-to-sqlite lists lists.db simonw dogsheep

  This command also accepts the `--sql` and `--attach` and `--ids` options.

  To additionally fetch the list of members for each list, use `--members`.

  ## Retrieving Twitter list memberships

  The `list-members` command can be used to retrieve details of one or more Twitter lists, including all of their members.

      $ twitter-to-sqlite list-members members.db simonw/the-good-place

  You can pass multiple `screen_name/list_slug` identifiers.

  If you know the numeric IDs of the lists instead, you can use `--ids`:

      $ twitter-to-sqlite list-members members.db 927913322841653248 --ids

  ## Retrieving just follower and friend IDs

  It's also possible to retrieve just the numeric Twitter IDs of the accounts that specific users are following ("friends" in Twitter's API terminology) or followed-by:

      $ twitter-to-sqlite followers-ids members.db simonw cleopaws

  This will populate the `following` table with `followed_id`/`follower_id` pairs for the two specified accounts, listing every account ID that is following either of those two accounts.

      $ twitter-to-sqlite friends-ids members.db simonw cleopaws

  This will do the same thing but pull the IDs that those accounts are following.

  Both of these commands also support `--sql` and `--attach` as an alternative to passing screen names as direct command-line arguments. You can use `--ids` to process the inputs as user IDs rather than screen names.

  The underlying Twitter APIs have a rate limit of 15 requests every 15 minutes - though they do return up to 5,000 IDs in each call. By default both of these subcommands will wait for 61 seconds between API calls in order to stay within the rate limit - you can adjust this behaviour down to just one second delay if you know you will not be making many calls using `--sleep=1`.

  ## Retrieving tweets from your home timeline

  The `home-timeline` command retrieves up to 800 tweets from the home timeline of the authenticated user - generally this means tweets from people you follow.

      $ twitter-to-sqlite home-timeline twitter.db
      Importing timeline  [#################--------]  591/800  00:01:14

  The tweets are stored in the `tweets` table, and a record is added to the `timeline_tweets` table noting that this tweet came in due to being spotted in the timeline of your user.

  You can use `--since` to retrieve just tweets that have been posted since the last time this command was run, or `--since_id=xxx` to explicitly pass in a tweet ID to use as the last position.

  You can then view your timeline in Datasette using the following URL:

  `/tweets/tweets?_where=id+in+(select+tweet+from+[timeline_tweets])&_sort_desc=id&_facet=user`

  This will filter your tweets table to just tweets that appear in your timeline, ordered by most recent first and use faceting to show you which users are responsible for the most tweets.

  ## Retrieving your mentions

  The `mentions-timeline` command works like `home-timeline` except it retrieves tweets that mention the authenticated user's account. It records the user account that was mentioned in a `mentions_tweets` table.

  It supports `--since` and `--since_id` in the same was as `home-timeline` does.

  ## Providing input from a SQL query with --sql and --attach

  This option is available for some subcommands - run `twitter-to-sqlite command-name --help` to check.

  You can provide Twitter screen names (or user IDs or tweet IDs) directly as command-line arguments, or you can provide those screen names or IDs by executing a SQL query.

  For example: consider a SQLite database with an `attendees` table listing names and Twitter accounts - something like this:

  | First   | Last       | Twitter      |
  |---------|------------|--------------|
  | Simon   | Willison   | simonw       |
  | Avril   | Lavigne    | AvrilLavigne |

  You can run the `users-lookup` command to pull the Twitter profile of every user listed in that database by loading the screen names using a `--sql` query:

      $ twitter-to-sqlite users-lookup my.db --sql="select Twitter from attendees"

  If your database table contains Twitter IDs, you can select those IDs and pass the `--ids` argument. For example, to fetch the profiles of users who have had their user IDs inserted into the `following` table using the `twitter-to-sqlite friends-ids` command:

      $ twitter-to-sqlite users-lookup my.db --sql="select follower_id from following" --ids

  Or to avoid re-fetching users that have already been fetched:

      $ twitter-to-sqlite users-lookup my.db \
          --sql="select followed_id from following where followed_id not in (
              select id from users)" --ids

  If your data lives in a separate database file you can attach it using `--attach`. For example, consider the attendees example above but the data lives in an `attendees.db` file, and you want to fetch the user profiles into a `tweets.db` file. You could do that like this:

      $ twitter-to-sqlite users-lookup tweets.db \
          --attach=attendees.db \
          --sql="select Twitter from attendees.attendees"

  The filename (without the extension) will be used as the database alias within SQLite. If you want a different alias for some reason you can specify that with a colon like this:

      $ twitter-to-sqlite users-lookup tweets.db \
          --attach=foo:attendees.db \
          --sql="select Twitter from foo.attendees"

  ## Running searches

  The `search` command runs a search against the Twitter [standard search API](https://developer.twitter.com/en/docs/tweets/search/api-reference/get-search-tweets).

      $ twitter-to-sqlite search tweets.db "dogsheep"

  This will import up to around 320 tweets that match that search term into the `tweets` table. It will also create a record in the `search_runs` table recording that the search took place, and many-to-many records in the `search_runs_tweets` table recording which tweets were seen for that search at that time.

  You can use the `--since` parameter to check for previous search runs with the same arguments and only retrieve tweets that were posted since the last retrieved matching tweet.

  The following additional options for `search` are supported:

  * `--geocode`: `latitude,longitude,radius` where radius is a number followed by mi or km
  * `--lang`: ISO 639-1 language code e.g. `en` or `es`
  * `--locale`: Locale: only `ja` is currently effective
  * `--result_type`: `mixed`, `recent` or `popular`. Defaults to `mixed`
  * `--count`: Number of results per page, defaults to the maximum of 100
  * `--stop_after`: Stop after this many results
  * `--since_id`: Pull tweets since this Tweet ID. You probably want to use `--since` instead of this.

  ## Capturing tweets in real-time with track and follow

  This functionality is **experimental**. Please [file bug reports](https://github.com/dogsheep/twitter-to-sqlite/issues) if you find any!

  Twitter provides a real-time API which can be used to subscribe to tweets as they happen. `twitter-to-sqlite` can use this API to continually update a SQLite database with tweets matching certain keywords, or referencing specific users.

  ### track

  To track keywords, use the `track` command:

      $ twitter-to-sqlite track tweets.db kakapo

  This command will continue to run until you hit Ctrl+C. It will capture any tweets mentioning the keyword [kakapo](https://en.wikipedia.org/wiki/Kakapo) and store them in the `tweets.db` database file.

  You can pass multiple keywords as a space separated list. This will capture tweets matching either of those keywords:

      $ twitter-to-sqlite track tweets.db kakapo raccoon

  You can enclose phrases in quotes to search for tweets matching both of those keywords:

      $ twitter-to-sqlite track tweets.db 'trash panda'

  See [the Twitter track documentation](https://developer.twitter.com/en/docs/tweets/filter-realtime/guides/basic-stream-parameters#track) for advanced tips on using this command.

  Add the `--verbose` option to see matching tweets (in their verbose JSON form) displayed to the terminal as they are captured:

      $ twitter-to-sqlite track tweets.db raccoon --verbose

  ### follow

  The `follow` command will capture all tweets that are relevant to one or more specific Twitter users.

      $ twitter-to-sqlite follow tweets.db nytimes

  This includes tweets by those users, tweets that reply to or quote those users and retweets by that user. See [the Twitter follow documentation](https://developer.twitter.com/en/docs/tweets/filter-realtime/guides/basic-stream-parameters#follow) for full details.

  The command accepts one or more screen names.

  You can feed it numeric Twitter user IDs instead of screen names by using the `--ids` flag.

  The command also supports the `--sql` and `--attach` options, and the `--verbose` option for displaying tweets as they are captured.

  Here's how to start following tweets from every user ID currently represented as being followed in the `following` table (populated using the `friends-ids` command):

      $ twitter-to-sqlite follow tweets.db \
          --sql="select distinct followed_id from following" \
          --ids

  ## Importing data from your Twitter archive

  You can request an archive of your Twitter data by [following these instructions](https://help.twitter.com/en/managing-your-account/how-to-download-your-twitter-archive).

  Twitter will send you a link to download a `.zip` file. You can import the contents of that file into a set of tables in a new database file called `archive.db` (each table beginning with the `archive_` prefix) using the `import` command:

      $ twitter-to-sqlite import archive.db ~/Downloads/twitter-2019-06-25-b31f2.zip

  This command does not populate any of the regular tables, since Twitter's export data does not exactly match the schema returned by the Twitter API.

  It will delete and recreate the corresponding `archive_*` tables every time you run it. If this is not what you want, run the command against a new SQLite database file name rather than running it against one that already exists.

  If you have already decompressed your archive, you can run this against the directory that you decompressed it to:

      $ twitter-to-sqlite import archive.db ~/Downloads/twitter-2019-06-25-b31f2/

  You can also run it against one or more specific files within that folder. For example, to import just the follower.js and following.js files:

      $ twitter-to-sqlite import archive.db \
          ~/Downloads/twitter-2019-06-25-b31f2/follower.js \
          ~/Downloads/twitter-2019-06-25-b31f2/following.js

  You may want to use other commands to populate tables based on data from the archive. For example, to retrieve full API versions of each of the tweets you have favourited in your archive, you could run the following:

      $ twitter-to-sqlite statuses-lookup archive.db \
          --sql='select tweetId from archive_like' \
          --skip-existing

  If you want these imported tweets to then be reflected in the `favorited_by` table, you can do so by applying the following SQL query:

      $ sqlite3 archive.db
      SQLite version 3.22.0 2018-01-22 18:45:57
      Enter ".help" for usage hints.
      sqlite> INSERT OR IGNORE INTO favorited_by (tweet, user)
         ...>     SELECT tweetId, 'YOUR_TWITTER_ID' FROM archive_like;
      <Ctrl+D>

  Replace YOUR_TWITTER_ID with your numeric Twitter ID. If you don't know that ID you can find it out by running the following:

      $ twitter-to-sqlite fetch \
          "https://api.twitter.com/1.1/account/verify_credentials.json" \
          | grep '"id"' | head -n 1

  ## Design notes

  * Tweet IDs are stored as integers, to afford sorting by ID in a sensible way
  * While we configure foreign key relationships between tables, we do not ask SQLite to enforce them. This is used by the `following` table to allow the `followers-ids` and `friends-ids` commands to populate it with user IDs even if the user accounts themselves are not yet present in the `users` table.

readme_html:
  <div id="readme" class="md" data-path="README.md"><article class="markdown-body entry-content container-lg" itemprop="text"><div class="markdown-heading" dir="auto"><h1 class="heading-element" dir="auto">twitter-to-sqlite</h1><a id="user-content-twitter-to-sqlite" class="anchor" aria-label="Permalink: twitter-to-sqlite" href="#twitter-to-sqlite"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto"><a href="https://pypi.org/project/twitter-to-sqlite/" rel="nofollow"><img src="https://img.shields.io/pypi/v/twitter-to-sqlite.svg" alt="PyPI" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/23ba852d891ae44346309a060302eb3c5d2eaea41868d074ec9cb1d67f6076a0/68747470733a2f2f696d672e736869656c64732e696f2f707970692f762f747769747465722d746f2d73716c6974652e737667"></a>
  <a href="https://github.com/dogsheep/twitter-to-sqlite/releases"><img src="https://img.shields.io/github/v/release/dogsheep/twitter-to-sqlite?include_prereleases&label=changelog" alt="Changelog" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/4b688faaddb3af7489ec91a38d556bccc185ed5f4a285bbe490c7514a8e07a42/68747470733a2f2f696d672e736869656c64732e696f2f6769746875622f762f72656c656173652f646f6773686565702f747769747465722d746f2d73716c6974653f696e636c7564655f70726572656c6561736573266c6162656c3d6368616e67656c6f67"></a>
  <a href="https://github.com/dogsheep/twitter-to-sqlite/actions?query=workflow%3ATest"><img src="https://github.com/dogsheep/twitter-to-sqlite/workflows/Test/badge.svg" alt="Tests" style="max-width: 100%;"></a>
  <a href="https://github.com/dogsheep/twitter-to-sqlite/blob/main/LICENSE"><img src="https://img.shields.io/badge/license-Apache%202.0-blue.svg" alt="License" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/c355f200ea90fddaa407b6eaab303663a669248ea3ca7b1fcf77dbe04ff5f48c/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6c6963656e73652d417061636865253230322e302d626c75652e737667"></a></p>
  <p dir="auto">Save data from Twitter to a SQLite database.</p>
  <p dir="auto"><strong>This tool currently uses Twitter API v1</strong>. You may be unable to use it if you do not have an API key for that version of the API.</p>

  <ul dir="auto">
  <li><a href="#how-to-install">How to install</a></li>
  <li><a href="#authentication">Authentication</a></li>
  <li><a href="#retrieving-tweets-by-specific-accounts">Retrieving tweets by specific accounts</a></li>
  <li><a href="#retrieve-user-profiles-in-bulk">Retrieve user profiles in bulk</a></li>
  <li><a href="#retrieve-tweets-in-bulk">Retrieve tweets in bulk</a></li>
  <li><a href="#retrieving-twitter-followers">Retrieving Twitter followers</a></li>
  <li><a href="#retrieving-friends">Retrieving friends</a></li>
  <li><a href="#retrieving-favorited-tweets">Retrieving favorited tweets</a></li>
  <li><a href="#retrieving-twitter-lists">Retrieving Twitter lists</a></li>
  <li><a href="#retrieving-twitter-list-memberships">Retrieving Twitter list memberships</a></li>
  <li><a href="#retrieving-just-follower-and-friend-ids">Retrieving just follower and friend IDs</a></li>
  <li><a href="#retrieving-tweets-from-your-home-timeline">Retrieving tweets from your home timeline</a></li>
  <li><a href="#retrieving-your-mentions">Retrieving your mentions</a></li>
  <li><a href="#providing-input-from-a-sql-query-with---sql-and---attach">Providing input from a SQL query with --sql and --attach</a></li>
  <li><a href="#running-searches">Running searches</a></li>
  <li><a href="#capturing-tweets-in-real-time-with-track-and-follow">Capturing tweets in real-time with track and follow</a>
  <ul dir="auto">
  <li><a href="#track">track</a></li>
  <li><a href="#follow">follow</a></li>
  </ul>
  </li>
  <li><a href="#importing-data-from-your-twitter-archive">Importing data from your Twitter archive</a></li>
  <li><a href="#design-notes">Design notes</a></li>
  </ul>

  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">How to install</h2><a id="user-content-how-to-install" class="anchor" aria-label="Permalink: How to install" href="#how-to-install"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ pip install twitter-to-sqlite"><pre class="notranslate"><code>$ pip install twitter-to-sqlite
  </code></pre></div>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Authentication</h2><a id="user-content-authentication" class="anchor" aria-label="Permalink: Authentication" href="#authentication"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">First, you will need to create a Twitter application at <a href="https://developer.twitter.com/en/apps" rel="nofollow">https://developer.twitter.com/en/apps</a>. You may need to apply for a Twitter developer account - if so, you may find this <a href="https://raw.githubusercontent.com/dogsheep/twitter-to-sqlite/main/email.png" rel="nofollow">example of an email application</a> useful that has been approved in the past.</p>
  <p dir="auto">Once you have created your application, navigate to the "Keys and tokens" page and make note of the following:</p>
  <ul dir="auto">
  <li>Your API key</li>
  <li>Your API secret key</li>
  <li>Your access token</li>
  <li>Your access token secret</li>
  </ul>
  <p dir="auto">You will need to save all four of these values to a JSON file in order to use this tool.</p>
  <p dir="auto">You can create that JSON file by running the following command and pasting in the values at the prompts:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ twitter-to-sqlite auth
  Create an app here: https://developer.twitter.com/en/apps
  Then navigate to 'Keys and tokens' and paste in the following:

  API key: xxx
  API secret key: xxx
  Access token: xxx
  Access token secret: xxx"><pre class="notranslate"><code>$ twitter-to-sqlite auth
  Create an app here: https://developer.twitter.com/en/apps
  Then navigate to 'Keys and tokens' and paste in the following:

  API key: xxx
  API secret key: xxx
  Access token: xxx
  Access token secret: xxx
  </code></pre></div>
  <p dir="auto">This will create a file called <code>auth.json</code> in your current directory containing the required values. To save the file at a different path or filename, use the <code>--auth=myauth.json</code> option.</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Retrieving tweets by specific accounts</h2><a id="user-content-retrieving-tweets-by-specific-accounts" class="anchor" aria-label="Permalink: Retrieving tweets by specific accounts" href="#retrieving-tweets-by-specific-accounts"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">The <code>user-timeline</code> command retrieves all of the tweets posted by the specified user accounts. It defaults to the account belonging to the authenticated user:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ twitter-to-sqlite user-timeline twitter.db
  Importing tweets  [#####-------------------------------]  2799/17780  00:01:39"><pre class="notranslate"><code>$ twitter-to-sqlite user-timeline twitter.db
  Importing tweets  [#####-------------------------------]  2799/17780  00:01:39
  </code></pre></div>
  <p dir="auto">All of these commands assume that there is an <code>auth.json</code> file in the current directory. You can provide the path to your <code>auth.json</code> file using <code>-a</code>:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ twitter-to-sqlite user-timeline twitter.db -a /path/to/auth.json"><pre class="notranslate"><code>$ twitter-to-sqlite user-timeline twitter.db -a /path/to/auth.json
  </code></pre></div>
  <p dir="auto">To load tweets for other users, pass their screen names as arguments:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ twitter-to-sqlite user-timeline twitter.db cleopaws nichemuseums"><pre class="notranslate"><code>$ twitter-to-sqlite user-timeline twitter.db cleopaws nichemuseums
  </code></pre></div>
  <p dir="auto">Twitter's API only returns up to around 3,200 tweets for most user accounts, but you may find that it returns all available tweets for your own user account.</p>
  <p dir="auto">You can pass numeric Twitter user IDs instead of screen names using the <code>--ids</code> parameter.</p>
  <p dir="auto">You can use <code>--since</code> to retrieve every tweet since the last time you imported for that user, or <code>--since_id=xxx</code> to retrieve every tweet since a specific tweet ID.</p>
  <p dir="auto">This command also accepts <code>--sql</code> and <code>--attach</code> options, documented below.</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Retrieve user profiles in bulk</h2><a id="user-content-retrieve-user-profiles-in-bulk" class="anchor" aria-label="Permalink: Retrieve user profiles in bulk" href="#retrieve-user-profiles-in-bulk"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">If you have a list of Twitter screen names (or user IDs) you can bulk fetch their fully inflated Twitter profiles using the <code>users-lookup</code> command:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ twitter-to-sqlite users-lookup users.db simonw cleopaws"><pre class="notranslate"><code>$ twitter-to-sqlite users-lookup users.db simonw cleopaws
  </code></pre></div>
  <p dir="auto">You can pass user IDs instead using the <code>--ids</code> option:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ twitter-to-sqlite users-lookup users.db 12497 3166449535 --ids"><pre class="notranslate"><code>$ twitter-to-sqlite users-lookup users.db 12497 3166449535 --ids
  </code></pre></div>
  <p dir="auto">This command also accepts <code>--sql</code> and <code>--attach</code> options, documented below.</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Retrieve tweets in bulk</h2><a id="user-content-retrieve-tweets-in-bulk" class="anchor" aria-label="Permalink: Retrieve tweets in bulk" href="#retrieve-tweets-in-bulk"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">If you have a list of tweet IDS you can bulk fetch them using the <code>statuses-lookup</code> command:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ twitter-to-sqlite statuses-lookup tweets.db 1122154819815239680 1122154178493575169"><pre class="notranslate"><code>$ twitter-to-sqlite statuses-lookup tweets.db 1122154819815239680 1122154178493575169
  </code></pre></div>
  <p dir="auto">The <code>--sql</code> and <code>--attach</code> options are supported.</p>
  <p dir="auto">Here's a recipe to retrieve any tweets that existing tweets are in-reply-to which have not yet been stored in your database:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ twitter-to-sqlite statuses-lookup tweets.db \
      --sql='
          select in_reply_to_status_id
          from tweets
          where in_reply_to_status_id is not null' \
      --skip-existing"><pre class="notranslate"><code>$ twitter-to-sqlite statuses-lookup tweets.db \
      --sql='
          select in_reply_to_status_id
          from tweets
          where in_reply_to_status_id is not null' \
      --skip-existing
  </code></pre></div>
  <p dir="auto">The <code>--skip-existing</code> option means that tweets that have already been stored in the database will not be fetched again.</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Retrieving Twitter followers</h2><a id="user-content-retrieving-twitter-followers" class="anchor" aria-label="Permalink: Retrieving Twitter followers" href="#retrieving-twitter-followers"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">The <code>followers</code> command retrieves details of every follower of the specified accounts. You can use it to retrieve your own followers, or you can pass one or more screen names to pull the followers for other accounts.</p>
  <p dir="auto">The following command pulls your followers and saves them in a SQLite database file called <code>twitter.db</code>:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ twitter-to-sqlite followers twitter.db"><pre class="notranslate"><code>$ twitter-to-sqlite followers twitter.db
  </code></pre></div>
  <p dir="auto">This command is <strong>extremely slow</strong>, because Twitter impose a rate limit of no more than one request per minute to this endpoint! If you are running it against an account with thousands of followers you should expect this to take several hours.</p>
  <p dir="auto">To retrieve followers for another account, use:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ twitter-to-sqlite followers twitter.db cleopaws"><pre class="notranslate"><code>$ twitter-to-sqlite followers twitter.db cleopaws
  </code></pre></div>
  <p dir="auto">This command also accepts the <code>--ids</code>, <code>--sql</code> and <code>--attach</code> options.</p>
  <p dir="auto">See <a href="https://simonwillison.net/2018/Jan/28/analyzing-my-twitter-followers/" rel="nofollow">Analyzing my Twitter followers with Datasette</a> for the original inspiration for this command.</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Retrieving friends</h2><a id="user-content-retrieving-friends" class="anchor" aria-label="Permalink: Retrieving friends" href="#retrieving-friends"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">The <code>friends</code> command works like the <code>followers</code> command, but retrieves the specified (or currently authenticated) user's friends - defined as accounts that the user is following.</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ twitter-to-sqlite friends twitter.db"><pre class="notranslate"><code>$ twitter-to-sqlite friends twitter.db
  </code></pre></div>
  <p dir="auto">It takes the same options as the <code>followers</code> command.</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Retrieving favorited tweets</h2><a id="user-content-retrieving-favorited-tweets" class="anchor" aria-label="Permalink: Retrieving favorited tweets" href="#retrieving-favorited-tweets"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">The <code>favorites</code> command retrieves tweets that have been favorited by a specified user. Called without any extra arguments it retrieves tweets favorited by the currently authenticated user:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ twitter-to-sqlite favorites faves.db"><pre class="notranslate"><code>$ twitter-to-sqlite favorites faves.db
  </code></pre></div>
  <p dir="auto">You can also use the <code>--screen_name</code> or <code>--user_id</code> arguments to retrieve favorite tweets for another user:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ twitter-to-sqlite favorites faves-obama.db --screen_name=BarackObama"><pre class="notranslate"><code>$ twitter-to-sqlite favorites faves-obama.db --screen_name=BarackObama
  </code></pre></div>
  <p dir="auto">Use the <code>--stop_after=xxx</code> argument to retrieve only the most recent number of favorites, e.g. to get the authenticated user's 50 most recent favorites:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ twitter-to-sqlite favorites faves.db --stop_after=50"><pre class="notranslate"><code>$ twitter-to-sqlite favorites faves.db --stop_after=50
  </code></pre></div>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Retrieving Twitter lists</h2><a id="user-content-retrieving-twitter-lists" class="anchor" aria-label="Permalink: Retrieving Twitter lists" href="#retrieving-twitter-lists"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">The <code>lists</code> command retrieves all of the lists belonging to one or more users.</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ twitter-to-sqlite lists lists.db simonw dogsheep"><pre class="notranslate"><code>$ twitter-to-sqlite lists lists.db simonw dogsheep
  </code></pre></div>
  <p dir="auto">This command also accepts the <code>--sql</code> and <code>--attach</code> and <code>--ids</code> options.</p>
  <p dir="auto">To additionally fetch the list of members for each list, use <code>--members</code>.</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Retrieving Twitter list memberships</h2><a id="user-content-retrieving-twitter-list-memberships" class="anchor" aria-label="Permalink: Retrieving Twitter list memberships" href="#retrieving-twitter-list-memberships"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">The <code>list-members</code> command can be used to retrieve details of one or more Twitter lists, including all of their members.</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ twitter-to-sqlite list-members members.db simonw/the-good-place"><pre class="notranslate"><code>$ twitter-to-sqlite list-members members.db simonw/the-good-place
  </code></pre></div>
  <p dir="auto">You can pass multiple <code>screen_name/list_slug</code> identifiers.</p>
  <p dir="auto">If you know the numeric IDs of the lists instead, you can use <code>--ids</code>:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ twitter-to-sqlite list-members members.db 927913322841653248 --ids"><pre class="notranslate"><code>$ twitter-to-sqlite list-members members.db 927913322841653248 --ids
  </code></pre></div>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Retrieving just follower and friend IDs</h2><a id="user-content-retrieving-just-follower-and-friend-ids" class="anchor" aria-label="Permalink: Retrieving just follower and friend IDs" href="#retrieving-just-follower-and-friend-ids"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">It's also possible to retrieve just the numeric Twitter IDs of the accounts that specific users are following ("friends" in Twitter's API terminology) or followed-by:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ twitter-to-sqlite followers-ids members.db simonw cleopaws"><pre class="notranslate"><code>$ twitter-to-sqlite followers-ids members.db simonw cleopaws
  </code></pre></div>
  <p dir="auto">This will populate the <code>following</code> table with <code>followed_id</code>/<code>follower_id</code> pairs for the two specified accounts, listing every account ID that is following either of those two accounts.</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ twitter-to-sqlite friends-ids members.db simonw cleopaws"><pre class="notranslate"><code>$ twitter-to-sqlite friends-ids members.db simonw cleopaws
  </code></pre></div>
  <p dir="auto">This will do the same thing but pull the IDs that those accounts are following.</p>
  <p dir="auto">Both of these commands also support <code>--sql</code> and <code>--attach</code> as an alternative to passing screen names as direct command-line arguments. You can use <code>--ids</code> to process the inputs as user IDs rather than screen names.</p>
  <p dir="auto">The underlying Twitter APIs have a rate limit of 15 requests every 15 minutes - though they do return up to 5,000 IDs in each call. By default both of these subcommands will wait for 61 seconds between API calls in order to stay within the rate limit - you can adjust this behaviour down to just one second delay if you know you will not be making many calls using <code>--sleep=1</code>.</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Retrieving tweets from your home timeline</h2><a id="user-content-retrieving-tweets-from-your-home-timeline" class="anchor" aria-label="Permalink: Retrieving tweets from your home timeline" href="#retrieving-tweets-from-your-home-timeline"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">The <code>home-timeline</code> command retrieves up to 800 tweets from the home timeline of the authenticated user - generally this means tweets from people you follow.</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ twitter-to-sqlite home-timeline twitter.db
  Importing timeline  [#################--------]  591/800  00:01:14"><pre class="notranslate"><code>$ twitter-to-sqlite home-timeline twitter.db
  Importing timeline  [#################--------]  591/800  00:01:14
  </code></pre></div>
  <p dir="auto">The tweets are stored in the <code>tweets</code> table, and a record is added to the <code>timeline_tweets</code> table noting that this tweet came in due to being spotted in the timeline of your user.</p>
  <p dir="auto">You can use <code>--since</code> to retrieve just tweets that have been posted since the last time this command was run, or <code>--since_id=xxx</code> to explicitly pass in a tweet ID to use as the last position.</p>
  <p dir="auto">You can then view your timeline in Datasette using the following URL:</p>
  <p dir="auto"><code>/tweets/tweets?_where=id+in+(select+tweet+from+[timeline_tweets])&_sort_desc=id&_facet=user</code></p>
  <p dir="auto">This will filter your tweets table to just tweets that appear in your timeline, ordered by most recent first and use faceting to show you which users are responsible for the most tweets.</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Retrieving your mentions</h2><a id="user-content-retrieving-your-mentions" class="anchor" aria-label="Permalink: Retrieving your mentions" href="#retrieving-your-mentions"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">The <code>mentions-timeline</code> command works like <code>home-timeline</code> except it retrieves tweets that mention the authenticated user's account. It records the user account that was mentioned in a <code>mentions_tweets</code> table.</p>
  <p dir="auto">It supports <code>--since</code> and <code>--since_id</code> in the same was as <code>home-timeline</code> does.</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Providing input from a SQL query with --sql and --attach</h2><a id="user-content-providing-input-from-a-sql-query-with---sql-and---attach" class="anchor" aria-label="Permalink: Providing input from a SQL query with --sql and --attach" href="#providing-input-from-a-sql-query-with---sql-and---attach"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">This option is available for some subcommands - run <code>twitter-to-sqlite command-name --help</code> to check.</p>
  <p dir="auto">You can provide Twitter screen names (or user IDs or tweet IDs) directly as command-line arguments, or you can provide those screen names or IDs by executing a SQL query.</p>
  <p dir="auto">For example: consider a SQLite database with an <code>attendees</code> table listing names and Twitter accounts - something like this:</p>
  <markdown-accessiblity-table><table>
  <thead>
  <tr>
  <th>First</th>
  <th>Last</th>
  <th>Twitter</th>
  </tr>
  </thead>
  <tbody>
  <tr>
  <td>Simon</td>
  <td>Willison</td>
  <td>simonw</td>
  </tr>
  <tr>
  <td>Avril</td>
  <td>Lavigne</td>
  <td>AvrilLavigne</td>
  </tr>
  </tbody>
  </table></markdown-accessiblity-table>
  <p dir="auto">You can run the <code>users-lookup</code> command to pull the Twitter profile of every user listed in that database by loading the screen names using a <code>--sql</code> query:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ twitter-to-sqlite users-lookup my.db --sql="select Twitter from attendees""><pre class="notranslate"><code>$ twitter-to-sqlite users-lookup my.db --sql="select Twitter from attendees"
  </code></pre></div>
  <p dir="auto">If your database table contains Twitter IDs, you can select those IDs and pass the <code>--ids</code> argument. For example, to fetch the profiles of users who have had their user IDs inserted into the <code>following</code> table using the <code>twitter-to-sqlite friends-ids</code> command:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ twitter-to-sqlite users-lookup my.db --sql="select follower_id from following" --ids"><pre class="notranslate"><code>$ twitter-to-sqlite users-lookup my.db --sql="select follower_id from following" --ids
  </code></pre></div>
  <p dir="auto">Or to avoid re-fetching users that have already been fetched:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ twitter-to-sqlite users-lookup my.db \
      --sql="select followed_id from following where followed_id not in (
          select id from users)" --ids"><pre class="notranslate"><code>$ twitter-to-sqlite users-lookup my.db \
      --sql="select followed_id from following where followed_id not in (
          select id from users)" --ids
  </code></pre></div>
  <p dir="auto">If your data lives in a separate database file you can attach it using <code>--attach</code>. For example, consider the attendees example above but the data lives in an <code>attendees.db</code> file, and you want to fetch the user profiles into a <code>tweets.db</code> file. You could do that like this:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ twitter-to-sqlite users-lookup tweets.db \
      --attach=attendees.db \
      --sql="select Twitter from attendees.attendees""><pre class="notranslate"><code>$ twitter-to-sqlite users-lookup tweets.db \
      --attach=attendees.db \
      --sql="select Twitter from attendees.attendees"
  </code></pre></div>
  <p dir="auto">The filename (without the extension) will be used as the database alias within SQLite. If you want a different alias for some reason you can specify that with a colon like this:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ twitter-to-sqlite users-lookup tweets.db \
      --attach=foo:attendees.db \
      --sql="select Twitter from foo.attendees""><pre class="notranslate"><code>$ twitter-to-sqlite users-lookup tweets.db \
      --attach=foo:attendees.db \
      --sql="select Twitter from foo.attendees"
  </code></pre></div>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Running searches</h2><a id="user-content-running-searches" class="anchor" aria-label="Permalink: Running searches" href="#running-searches"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">The <code>search</code> command runs a search against the Twitter <a href="https://developer.twitter.com/en/docs/tweets/search/api-reference/get-search-tweets" rel="nofollow">standard search API</a>.</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ twitter-to-sqlite search tweets.db "dogsheep""><pre class="notranslate"><code>$ twitter-to-sqlite search tweets.db "dogsheep"
  </code></pre></div>
  <p dir="auto">This will import up to around 320 tweets that match that search term into the <code>tweets</code> table. It will also create a record in the <code>search_runs</code> table recording that the search took place, and many-to-many records in the <code>search_runs_tweets</code> table recording which tweets were seen for that search at that time.</p>
  <p dir="auto">You can use the <code>--since</code> parameter to check for previous search runs with the same arguments and only retrieve tweets that were posted since the last retrieved matching tweet.</p>
  <p dir="auto">The following additional options for <code>search</code> are supported:</p>
  <ul dir="auto">
  <li><code>--geocode</code>: <code>latitude,longitude,radius</code> where radius is a number followed by mi or km</li>
  <li><code>--lang</code>: ISO 639-1 language code e.g. <code>en</code> or <code>es</code></li>
  <li><code>--locale</code>: Locale: only <code>ja</code> is currently effective</li>
  <li><code>--result_type</code>: <code>mixed</code>, <code>recent</code> or <code>popular</code>. Defaults to <code>mixed</code></li>
  <li><code>--count</code>: Number of results per page, defaults to the maximum of 100</li>
  <li><code>--stop_after</code>: Stop after this many results</li>
  <li><code>--since_id</code>: Pull tweets since this Tweet ID. You probably want to use <code>--since</code> instead of this.</li>
  </ul>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Capturing tweets in real-time with track and follow</h2><a id="user-content-capturing-tweets-in-real-time-with-track-and-follow" class="anchor" aria-label="Permalink: Capturing tweets in real-time with track and follow" href="#capturing-tweets-in-real-time-with-track-and-follow"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">This functionality is <strong>experimental</strong>. Please <a href="https://github.com/dogsheep/twitter-to-sqlite/issues">file bug reports</a> if you find any!</p>
  <p dir="auto">Twitter provides a real-time API which can be used to subscribe to tweets as they happen. <code>twitter-to-sqlite</code> can use this API to continually update a SQLite database with tweets matching certain keywords, or referencing specific users.</p>
  <div class="markdown-heading" dir="auto"><h3 class="heading-element" dir="auto">track</h3><a id="user-content-track" class="anchor" aria-label="Permalink: track" href="#track"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">To track keywords, use the <code>track</code> command:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ twitter-to-sqlite track tweets.db kakapo"><pre class="notranslate"><code>$ twitter-to-sqlite track tweets.db kakapo
  </code></pre></div>
  <p dir="auto">This command will continue to run until you hit Ctrl+C. It will capture any tweets mentioning the keyword <a href="https://en.wikipedia.org/wiki/Kakapo" rel="nofollow">kakapo</a> and store them in the <code>tweets.db</code> database file.</p>
  <p dir="auto">You can pass multiple keywords as a space separated list. This will capture tweets matching either of those keywords:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ twitter-to-sqlite track tweets.db kakapo raccoon"><pre class="notranslate"><code>$ twitter-to-sqlite track tweets.db kakapo raccoon
  </code></pre></div>
  <p dir="auto">You can enclose phrases in quotes to search for tweets matching both of those keywords:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ twitter-to-sqlite track tweets.db 'trash panda'"><pre class="notranslate"><code>$ twitter-to-sqlite track tweets.db 'trash panda'
  </code></pre></div>
  <p dir="auto">See <a href="https://developer.twitter.com/en/docs/tweets/filter-realtime/guides/basic-stream-parameters#track" rel="nofollow">the Twitter track documentation</a> for advanced tips on using this command.</p>
  <p dir="auto">Add the <code>--verbose</code> option to see matching tweets (in their verbose JSON form) displayed to the terminal as they are captured:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ twitter-to-sqlite track tweets.db raccoon --verbose"><pre class="notranslate"><code>$ twitter-to-sqlite track tweets.db raccoon --verbose
  </code></pre></div>
  <div class="markdown-heading" dir="auto"><h3 class="heading-element" dir="auto">follow</h3><a id="user-content-follow" class="anchor" aria-label="Permalink: follow" href="#follow"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">The <code>follow</code> command will capture all tweets that are relevant to one or more specific Twitter users.</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ twitter-to-sqlite follow tweets.db nytimes"><pre class="notranslate"><code>$ twitter-to-sqlite follow tweets.db nytimes
  </code></pre></div>
  <p dir="auto">This includes tweets by those users, tweets that reply to or quote those users and retweets by that user. See <a href="https://developer.twitter.com/en/docs/tweets/filter-realtime/guides/basic-stream-parameters#follow" rel="nofollow">the Twitter follow documentation</a> for full details.</p>
  <p dir="auto">The command accepts one or more screen names.</p>
  <p dir="auto">You can feed it numeric Twitter user IDs instead of screen names by using the <code>--ids</code> flag.</p>
  <p dir="auto">The command also supports the <code>--sql</code> and <code>--attach</code> options, and the <code>--verbose</code> option for displaying tweets as they are captured.</p>
  <p dir="auto">Here's how to start following tweets from every user ID currently represented as being followed in the <code>following</code> table (populated using the <code>friends-ids</code> command):</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ twitter-to-sqlite follow tweets.db \
      --sql="select distinct followed_id from following" \
      --ids"><pre class="notranslate"><code>$ twitter-to-sqlite follow tweets.db \
      --sql="select distinct followed_id from following" \
      --ids
  </code></pre></div>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Importing data from your Twitter archive</h2><a id="user-content-importing-data-from-your-twitter-archive" class="anchor" aria-label="Permalink: Importing data from your Twitter archive" href="#importing-data-from-your-twitter-archive"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">You can request an archive of your Twitter data by <a href="https://help.twitter.com/en/managing-your-account/how-to-download-your-twitter-archive" rel="nofollow">following these instructions</a>.</p>
  <p dir="auto">Twitter will send you a link to download a <code>.zip</code> file. You can import the contents of that file into a set of tables in a new database file called <code>archive.db</code> (each table beginning with the <code>archive_</code> prefix) using the <code>import</code> command:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ twitter-to-sqlite import archive.db ~/Downloads/twitter-2019-06-25-b31f2.zip"><pre class="notranslate"><code>$ twitter-to-sqlite import archive.db ~/Downloads/twitter-2019-06-25-b31f2.zip
  </code></pre></div>
  <p dir="auto">This command does not populate any of the regular tables, since Twitter's export data does not exactly match the schema returned by the Twitter API.</p>
  <p dir="auto">It will delete and recreate the corresponding <code>archive_*</code> tables every time you run it. If this is not what you want, run the command against a new SQLite database file name rather than running it against one that already exists.</p>
  <p dir="auto">If you have already decompressed your archive, you can run this against the directory that you decompressed it to:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ twitter-to-sqlite import archive.db ~/Downloads/twitter-2019-06-25-b31f2/"><pre class="notranslate"><code>$ twitter-to-sqlite import archive.db ~/Downloads/twitter-2019-06-25-b31f2/
  </code></pre></div>
  <p dir="auto">You can also run it against one or more specific files within that folder. For example, to import just the follower.js and following.js files:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ twitter-to-sqlite import archive.db \
      ~/Downloads/twitter-2019-06-25-b31f2/follower.js \
      ~/Downloads/twitter-2019-06-25-b31f2/following.js"><pre class="notranslate"><code>$ twitter-to-sqlite import archive.db \
      ~/Downloads/twitter-2019-06-25-b31f2/follower.js \
      ~/Downloads/twitter-2019-06-25-b31f2/following.js
  </code></pre></div>
  <p dir="auto">You may want to use other commands to populate tables based on data from the archive. For example, to retrieve full API versions of each of the tweets you have favourited in your archive, you could run the following:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ twitter-to-sqlite statuses-lookup archive.db \
      --sql='select tweetId from archive_like' \
      --skip-existing"><pre class="notranslate"><code>$ twitter-to-sqlite statuses-lookup archive.db \
      --sql='select tweetId from archive_like' \
      --skip-existing
  </code></pre></div>
  <p dir="auto">If you want these imported tweets to then be reflected in the <code>favorited_by</code> table, you can do so by applying the following SQL query:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ sqlite3 archive.db
  SQLite version 3.22.0 2018-01-22 18:45:57
  Enter ".help" for usage hints.
  sqlite> INSERT OR IGNORE INTO favorited_by (tweet, user)
     ...>     SELECT tweetId, 'YOUR_TWITTER_ID' FROM archive_like;
  <Ctrl+D>"><pre class="notranslate"><code>$ sqlite3 archive.db
  SQLite version 3.22.0 2018-01-22 18:45:57
  Enter ".help" for usage hints.
  sqlite> INSERT OR IGNORE INTO favorited_by (tweet, user)
     ...>     SELECT tweetId, 'YOUR_TWITTER_ID' FROM archive_like;
  <Ctrl+D>
  </code></pre></div>
  <p dir="auto">Replace YOUR_TWITTER_ID with your numeric Twitter ID. If you don't know that ID you can find it out by running the following:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ twitter-to-sqlite fetch \
      "https://api.twitter.com/1.1/account/verify_credentials.json" \
      | grep '"id"' | head -n 1"><pre class="notranslate"><code>$ twitter-to-sqlite fetch \
      "https://api.twitter.com/1.1/account/verify_credentials.json" \
      | grep '"id"' | head -n 1
  </code></pre></div>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Design notes</h2><a id="user-content-design-notes" class="anchor" aria-label="Permalink: Design notes" href="#design-notes"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <ul dir="auto">
  <li>Tweet IDs are stored as integers, to afford sorting by ID in a sensible way</li>
  <li>While we configure foreign key relationships between tables, we do not ask SQLite to enforce them. This is used by the <code>following</code> table to allow the <code>followers-ids</code> and <code>friends-ids</code> commands to populate it with user IDs even if the user accounts themselves are not yet present in the <code>users</code> table.</li>
  </ul>
  </article></div>


node_id: MDEwOlJlcG9zaXRvcnkyMDYyMDI4NjQ=
name: inaturalist-to-sqlite
full_name: dogsheep/inaturalist-to-sqlite
html_url: https://github.com/dogsheep/inaturalist-to-sqlite
description: Create a SQLite database containing your observation history from iNaturalist
created_at: 2019-09-04T01:21:21Z
updated_at: 2025-01-11T07:51:05Z
pushed_at: 2020-10-22T00:08:58Z
language: Python
license: apache-2.0
topics: ["datasette", "datasette-io", "datasette-tool", "dogsheep", "inaturalist", "sqlite"]
visibility: public
default_branch: master
permissions: {"admin": false, "maintain": false, "push": false, "triage": false, "pull": true}
custom_properties: {}
readme:
  # inaturalist-to-sqlite

  [![PyPI](https://img.shields.io/pypi/v/inaturalist-to-sqlite.svg)](https://pypi.org/project/inaturalist-to-sqlite/)
  [![CircleCI](https://circleci.com/gh/dogsheep/inaturalist-to-sqlite.svg?style=svg)](https://circleci.com/gh/dogsheep/inaturalist-to-sqlite)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/dogsheep/inaturalist-to-sqlite/blob/master/LICENSE)

  Create a SQLite database containing your observation history from [iNaturalist](https://www.inaturalist.org/).

  ## How to install

      $ pip install inaturalist-to-sqlite

  ## Usage

      $ inaturalist-to-sqlite inaturalist.db yourusername

  (Or try `simonw` if you don't yet have an iNaturalist account)

  This will import all of your iNaturalist observations into a SQLite database called `inaturalist.db`.

readme_html:
  <div id="readme" class="md" data-path="README.md"><article class="markdown-body entry-content container-lg" itemprop="text"><div class="markdown-heading" dir="auto"><h1 class="heading-element" dir="auto">inaturalist-to-sqlite</h1><a id="user-content-inaturalist-to-sqlite" class="anchor" aria-label="Permalink: inaturalist-to-sqlite" href="#inaturalist-to-sqlite"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto"><a href="https://pypi.org/project/inaturalist-to-sqlite/" rel="nofollow"><img src="https://img.shields.io/pypi/v/inaturalist-to-sqlite.svg" alt="PyPI" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/228fa36746cf3911c34f561a803b76e0aa9412aebdeb291fb301cfc4021060a1/68747470733a2f2f696d672e736869656c64732e696f2f707970692f762f696e61747572616c6973742d746f2d73716c6974652e737667"></a>
  <a href="https://circleci.com/gh/dogsheep/inaturalist-to-sqlite" rel="nofollow"><img src="https://circleci.com/gh/dogsheep/inaturalist-to-sqlite.svg?style=svg" alt="CircleCI" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/064577bb7eed309f0d8e0b7112eb7411d661db2ee6ad366a8d06d8216fe1c650/68747470733a2f2f636972636c6563692e636f6d2f67682f646f6773686565702f696e61747572616c6973742d746f2d73716c6974652e7376673f7374796c653d737667"></a>
  <a href="https://github.com/dogsheep/inaturalist-to-sqlite/blob/master/LICENSE"><img src="https://img.shields.io/badge/license-Apache%202.0-blue.svg" alt="License" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/c355f200ea90fddaa407b6eaab303663a669248ea3ca7b1fcf77dbe04ff5f48c/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6c6963656e73652d417061636865253230322e302d626c75652e737667"></a></p>
  <p dir="auto">Create a SQLite database containing your observation history from <a href="https://www.inaturalist.org/" rel="nofollow">iNaturalist</a>.</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">How to install</h2><a id="user-content-how-to-install" class="anchor" aria-label="Permalink: How to install" href="#how-to-install"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ pip install inaturalist-to-sqlite"><pre class="notranslate"><code>$ pip install inaturalist-to-sqlite
  </code></pre></div>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Usage</h2><a id="user-content-usage" class="anchor" aria-label="Permalink: Usage" href="#usage"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ inaturalist-to-sqlite inaturalist.db yourusername"><pre class="notranslate"><code>$ inaturalist-to-sqlite inaturalist.db yourusername
  </code></pre></div>
  <p dir="auto">(Or try <code>simonw</code> if you don't yet have an iNaturalist account)</p>
  <p dir="auto">This will import all of your iNaturalist observations into a SQLite database called <code>inaturalist.db</code>.</p>
  </article></div>


node_id: MDEwOlJlcG9zaXRvcnkyMDcwNTI4ODI=
name: github-to-sqlite
full_name: dogsheep/github-to-sqlite
html_url: https://github.com/dogsheep/github-to-sqlite
description: Save data from GitHub to a SQLite database
created_at: 2019-09-08T02:50:28Z
updated_at: 2025-05-12T03:49:17Z
pushed_at: 2024-01-15T05:56:13Z
homepage: https://github-to-sqlite.dogsheep.net/
language: Python
license: apache-2.0
topics: ["datasette", "datasette-io", "datasette-tool", "dogsheep", "github-api", "sqlite"]
visibility: public
default_branch: main
permissions: {"admin": false, "maintain": false, "push": false, "triage": false, "pull": true}
custom_properties: {}
readme:
  # github-to-sqlite

  [![PyPI](https://img.shields.io/pypi/v/github-to-sqlite.svg)](https://pypi.org/project/github-to-sqlite/)
  [![Changelog](https://img.shields.io/github/v/release/dogsheep/github-to-sqlite?include_prereleases&label=changelog)](https://github.com/dogsheep/github-to-sqlite/releases)
  [![Tests](https://github.com/dogsheep/github-to-sqlite/workflows/Test/badge.svg)](https://github.com/dogsheep/github-to-sqlite/actions?query=workflow%3ATest)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/dogsheep/github-to-sqlite/blob/main/LICENSE)

  Save data from GitHub to a SQLite database.

  <!-- toc -->

  - [Demo](#demo)
  - [How to install](#how-to-install)
  - [Authentication](#authentication)
  - [Fetching issues for a repository](#fetching-issues-for-a-repository)
  - [Fetching pull requests for a repository](#fetching-pull-requests-for-a-repository)
  - [Fetching issue comments for a repository](#fetching-issue-comments-for-a-repository)
  - [Fetching commits for a repository](#fetching-commits-for-a-repository)
  - [Fetching releases for a repository](#fetching-releases-for-a-repository)
  - [Fetching tags for a repository](#fetching-tags-for-a-repository)
  - [Fetching contributors to a repository](#fetching-contributors-to-a-repository)
  - [Fetching repos belonging to a user or organization](#fetching-repos-belonging-to-a-user-or-organization)
  - [Fetching specific repositories](#fetching-specific-repositories)
  - [Fetching repos that have been starred by a user](#fetching-repos-that-have-been-starred-by-a-user)
  - [Fetching users that have starred specific repos](#fetching-users-that-have-starred-specific-repos)
  - [Fetching GitHub Actions workflows](#fetching-github-actions-workflows)
  - [Scraping dependents for a repository](#scraping-dependents-for-a-repository)
  - [Fetching emojis](#fetching-emojis)
  - [Making authenticated API calls](#making-authenticated-api-calls)

  <!-- tocstop -->

  ## Demo

  https://github-to-sqlite.dogsheep.net/ hosts a [Datasette](https://datasette.io/) demo of a database created by [running this tool](https://github.com/dogsheep/github-to-sqlite/blob/main/.github/workflows/deploy-demo.yml#L40-L60) against all of the repositories in the [Dogsheep GitHub organization](https://github.com/dogsheep), plus the [datasette](https://github.com/simonw/datasette) and [sqlite-utils](https://github.com/simonw/sqlite-utils) repositories.

  ## How to install

      $ pip install github-to-sqlite

  ## Authentication

  Create a GitHub personal access token: https://github.com/settings/tokens

  Run this command and paste in your new token:

      $ github-to-sqlite auth

  This will create a file called `auth.json` in your current directory containing the required value. To save the file at a different path or filename, use the `--auth=myauth.json` option.

  As an alternative to using an `auth.json` file you can add your access token to an environment variable called `GITHUB_TOKEN`.

  ## Fetching issues for a repository

  The `issues` command retrieves all of the issues belonging to a specified repository.

      $ github-to-sqlite issues github.db simonw/datasette

  If an `auth.json` file is present it will use the token from that file. It works without authentication for public repositories but you should be aware that GitHub have strict IP-based rate limits for unauthenticated requests.

  You can point to a different location of `auth.json` using `-a`:

      $ github-to-sqlite issues github.db simonw/datasette -a /path/to/auth.json

  You can use the `--issue` option one or more times to load specific issues:

      $ github-to-sqlite issues github.db simonw/datasette --issue=1

  Example: [issues table](https://github-to-sqlite.dogsheep.net/github/issues)

  ## Fetching pull requests for a repository

  While pull requests are a type of issue, you will get more information on pull requests by pulling them separately. For example, whether a pull request has been merged and when.

  Following the API of issues, the `pull-requests` command retrieves all of the pull requests belonging to a specified repository.

      $ github-to-sqlite pull-requests github.db simonw/datasette

  You can use the `--pull-request` option one or more times to load specific pull request:

      $ github-to-sqlite pull-requests github.db simonw/datasette --pull-request=81

  Note that the `merged_by` column on the `pull_requests` table will only be populated for pull requests that are loaded using the `--pull-request` option - the GitHub API does not return this field for pull requests that are loaded in bulk.

  You can load only pull requests in a certain state with the `--state` option:

      $ github-to-sqlite pull-requests --state=open github.db simonw/datasette

  Pull requests across an entire organization (or more than one) can be loaded with `--org`:

      $ github-to-sqlite pull-requests --state=open --org=psf --org=python github.db

  You can use a search query to find pull requests.  Note that no more than 1000 will be loaded (this is a GitHub API limitation), and some data will be missing (base and head SHAs).  When using searches, other filters are ignored; put all criteria into the search itself:

      $ github-to-sqlite pull-requests --search='org:python defaultdict state:closed created:<2023-09-01' github.db

  Example: [pull_requests table](https://github-to-sqlite.dogsheep.net/github/pull_requests)

  ## Fetching issue comments for a repository

  The `issue-comments` command retrieves all of the comments on all of the issues in a repository.

  It is recommended you run `issues` first, so that each imported comment can have a foreign key pointing to its issue.

      $ github-to-sqlite issues github.db simonw/datasette
      $ github-to-sqlite issue-comments github.db simonw/datasette

  You can use the `--issue` option to only load comments for a specific issue within that repository, for example:

      $ github-to-sqlite issue-comments github.db simonw/datasette --issue=1

  Example: [issue_comments table](https://github-to-sqlite.dogsheep.net/github/issue_comments)

  ## Fetching commits for a repository

  The `commits` command retrieves details of all of the commits for one or more repositories. It currently fetches the SHA, commit message and author and committer details; it does not retrieve the full commit body.

      $ github-to-sqlite commits github.db simonw/datasette simonw/sqlite-utils

  The command accepts one or more repositories.

  By default it will stop as soon as it sees a commit that has previously been retrieved. You can force it to retrieve all commits (including those that have been previously inserted) using `--all`.

  Example: [commits table](https://github-to-sqlite.dogsheep.net/github/commits)

  ## Fetching releases for a repository

  The `releases` command retrieves the releases for one or more repositories.

      $ github-to-sqlite releases github.db simonw/datasette simonw/sqlite-utils

  The command accepts one or more repositories.

  Example: [releases table](https://github-to-sqlite.dogsheep.net/github/releases)

  ## Fetching tags for a repository

  The `tags` command retrieves all of the tags for one or more repositories.

      $ github-to-sqlite tags github.db simonw/datasette simonw/sqlite-utils

  Example: [tags table](https://github-to-sqlite.dogsheep.net/github/tags)

  ## Fetching contributors to a repository

  The `contributors` command retrieves details of all of the contributors for one or more repositories.

      $ github-to-sqlite contributors github.db simonw/datasette simonw/sqlite-utils

  The command accepts one or more repositories. It populates a `contributors` table, with foreign keys to `repos` and `users` and a `contributions` table listing the number of commits to that repository for each contributor.

  Example: [contributors table](https://github-to-sqlite.dogsheep.net/github/contributors)

  ## Fetching repos belonging to a user or organization

  The `repos` command fetches repos belonging to a user or organization.

  Without any other arguments, this command will fetch all repos that the currently authenticated user owns, collaborates on or can access via one of their organizations:

      $ github-to-sqlite repos github.db

  To fetch repos belonging to a specific user or organization, provide their username as an argument:

      $ github-to-sqlite repos github.db dogsheep # organization
      $ github-to-sqlite repos github.db simonw # user

  You can pass more than one username to fetch for multiple users or organizations at once:

      $ github-to-sqlite repos github.db simonw dogsheep

  Add the `--readme` option to save the README for the repo in a column called `readme`. Add `--readme-html` to save the HTML rendered version of the README into a column called `readme_html`.

  Example: [repos table](https://github-to-sqlite.dogsheep.net/github/repos)

  ## Fetching specific repositories

  You can use `-r` with the `repos` command one or more times to fetch just specific repositories.

      $ github-to-sqlite repos github.db -r simonw/datasette -r dogsheep/github-to-sqlite

  ## Fetching repos that have been starred by a user

  The `starred` command fetches the repos that have been starred by a user.

      $ github-to-sqlite starred github.db simonw

  If you are using an `auth.json` file you can omit the username to retrieve the starred repos for the authenticated user.

  Example: [stars table](https://github-to-sqlite.dogsheep.net/github/stars)

  ## Fetching users that have starred specific repos

  The `stargazers` command fetches the users that have starred the specified repos.

      $ github-to-sqlite stargazers github.db simonw/datasette dogsheep/github-to-sqlite

  You can specify one or more repository using `owner/repo` syntax.

  Users fetched using this command will be inserted into the `users` table. Many-to-many records showing which repository they starred will be added to the `stars` table.

  ## Fetching GitHub Actions workflows

  The `workflows` command fetches the YAML workflow configurations from each repository's `.github/workflows` directory and parses them to populate `workflows`, `jobs` and `steps` tables.

      $ github-to-sqlite workflows github.db simonw/datasette dogsheep/github-to-sqlite

  You can specify one or more repository using `owner/repo` syntax.

  Example: [workflows table](https://github-to-sqlite.dogsheep.net/github/workflows), [jobs table](https://github-to-sqlite.dogsheep.net/github/jobs), [steps table](https://github-to-sqlite.dogsheep.net/github/steps)

  ## Scraping dependents for a repository

  The GitHub dependency graph can show other GitHub projects that depend on a specific repo, for example [simonw/datasette/network/dependents](https://github.com/simonw/datasette/network/dependents).

  This data is not yet available through the GitHub API. The `scrape-dependents` command scrapes those pages and uses the GitHub API to load full versions of the dependent repositories.

      $ github-to-sqlite scrape-dependents github.db simonw/datasette

  The command accepts one or more repositories.

  Add `-v` for verbose output.

  Example: [dependents table](https://github-to-sqlite.dogsheep.net/github/dependents?_sort_desc=first_seen_utc)

  ## Fetching emojis

  You can fetch a list of every emoji supported by GitHub using the `emojis` command:

      $ github-to-sqlite emojis github.db

  This will create a table called `emojis` with a primary key `name` and a `url` column.

  If you add the `--fetch` option the command will also fetch the binary content of the images and place them in an `image` column:

      $ github-to-sqlite emojis emojis.db -f
      [########----------------------------]  397/1799   22%  00:03:43

  You can then use the [datasette-render-images](https://github.com/simonw/datasette-render-images) plugin to browse them visually.

  Example: [emojis table](https://github-to-sqlite.dogsheep.net/github/emojis)

  ## Making authenticated API calls

  The `github-to-sqlite get` command provides a convenient shortcut for making authenticated calls to the API. Once you have created your `auth.json` file (or set a `GITHUB_TOKEN` environment variable) you can use it like this:

      $ github-to-sqlite get https://api.github.com/gists

  This will make an authenticated call to the URL you provide and pretty-print the resulting JSON to the console.

  You can omit the `https://api.github.com/` prefix, for example:

      $ github-to-sqlite get /gists

  Many GitHub APIs are [paginated using the HTTP Link header](https://docs.github.com/en/rest/guides/traversing-with-pagination). You can follow this pagination and output a list of all of the resulting items using `--paginate`:

      $ github-to-sqlite get /users/simonw/repos --paginate

  You can outline newline-delimited JSON for each item using `--nl`. This can be useful for streaming items into another tool.

      $ github-to-sqlite get /users/simonw/repos --nl

readme_html:
  <div id="readme" class="md" data-path="README.md"><article class="markdown-body entry-content container-lg" itemprop="text"><div class="markdown-heading" dir="auto"><h1 class="heading-element" dir="auto">github-to-sqlite</h1><a id="user-content-github-to-sqlite" class="anchor" aria-label="Permalink: github-to-sqlite" href="#github-to-sqlite"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto"><a href="https://pypi.org/project/github-to-sqlite/" rel="nofollow"><img src="https://img.shields.io/pypi/v/github-to-sqlite.svg" alt="PyPI" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/fa09b71e17527835dc970e0238963f46118140185da019e37ba206cd7c81b33f/68747470733a2f2f696d672e736869656c64732e696f2f707970692f762f6769746875622d746f2d73716c6974652e737667"></a>
  <a href="https://github.com/dogsheep/github-to-sqlite/releases"><img src="https://img.shields.io/github/v/release/dogsheep/github-to-sqlite?include_prereleases&label=changelog" alt="Changelog" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/04fb849b5b372c8ef9014a6b620f7b489218781cbdbd122198f80c7aa70ec06c/68747470733a2f2f696d672e736869656c64732e696f2f6769746875622f762f72656c656173652f646f6773686565702f6769746875622d746f2d73716c6974653f696e636c7564655f70726572656c6561736573266c6162656c3d6368616e67656c6f67"></a>
  <a href="https://github.com/dogsheep/github-to-sqlite/actions?query=workflow%3ATest"><img src="https://github.com/dogsheep/github-to-sqlite/workflows/Test/badge.svg" alt="Tests" style="max-width: 100%;"></a>
  <a href="https://github.com/dogsheep/github-to-sqlite/blob/main/LICENSE"><img src="https://img.shields.io/badge/license-Apache%202.0-blue.svg" alt="License" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/c355f200ea90fddaa407b6eaab303663a669248ea3ca7b1fcf77dbe04ff5f48c/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6c6963656e73652d417061636865253230322e302d626c75652e737667"></a></p>
  <p dir="auto">Save data from GitHub to a SQLite database.</p>

  <ul dir="auto">
  <li><a href="#demo">Demo</a></li>
  <li><a href="#how-to-install">How to install</a></li>
  <li><a href="#authentication">Authentication</a></li>
  <li><a href="#fetching-issues-for-a-repository">Fetching issues for a repository</a></li>
  <li><a href="#fetching-pull-requests-for-a-repository">Fetching pull requests for a repository</a></li>
  <li><a href="#fetching-issue-comments-for-a-repository">Fetching issue comments for a repository</a></li>
  <li><a href="#fetching-commits-for-a-repository">Fetching commits for a repository</a></li>
  <li><a href="#fetching-releases-for-a-repository">Fetching releases for a repository</a></li>
  <li><a href="#fetching-tags-for-a-repository">Fetching tags for a repository</a></li>
  <li><a href="#fetching-contributors-to-a-repository">Fetching contributors to a repository</a></li>
  <li><a href="#fetching-repos-belonging-to-a-user-or-organization">Fetching repos belonging to a user or organization</a></li>
  <li><a href="#fetching-specific-repositories">Fetching specific repositories</a></li>
  <li><a href="#fetching-repos-that-have-been-starred-by-a-user">Fetching repos that have been starred by a user</a></li>
  <li><a href="#fetching-users-that-have-starred-specific-repos">Fetching users that have starred specific repos</a></li>
  <li><a href="#fetching-github-actions-workflows">Fetching GitHub Actions workflows</a></li>
  <li><a href="#scraping-dependents-for-a-repository">Scraping dependents for a repository</a></li>
  <li><a href="#fetching-emojis">Fetching emojis</a></li>
  <li><a href="#making-authenticated-api-calls">Making authenticated API calls</a></li>
  </ul>

  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Demo</h2><a id="user-content-demo" class="anchor" aria-label="Permalink: Demo" href="#demo"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto"><a href="https://github-to-sqlite.dogsheep.net/" rel="nofollow">https://github-to-sqlite.dogsheep.net/</a> hosts a <a href="https://datasette.io/" rel="nofollow">Datasette</a> demo of a database created by <a href="https://github.com/dogsheep/github-to-sqlite/blob/main/.github/workflows/deploy-demo.yml#L40-L60">running this tool</a> against all of the repositories in the <a href="https://github.com/dogsheep">Dogsheep GitHub organization</a>, plus the <a href="https://github.com/simonw/datasette">datasette</a> and <a href="https://github.com/simonw/sqlite-utils">sqlite-utils</a> repositories.</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">How to install</h2><a id="user-content-how-to-install" class="anchor" aria-label="Permalink: How to install" href="#how-to-install"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ pip install github-to-sqlite"><pre class="notranslate"><code>$ pip install github-to-sqlite
  </code></pre></div>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Authentication</h2><a id="user-content-authentication" class="anchor" aria-label="Permalink: Authentication" href="#authentication"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">Create a GitHub personal access token: <a href="https://github.com/settings/tokens">https://github.com/settings/tokens</a></p>
  <p dir="auto">Run this command and paste in your new token:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ github-to-sqlite auth"><pre class="notranslate"><code>$ github-to-sqlite auth
  </code></pre></div>
  <p dir="auto">This will create a file called <code>auth.json</code> in your current directory containing the required value. To save the file at a different path or filename, use the <code>--auth=myauth.json</code> option.</p>
  <p dir="auto">As an alternative to using an <code>auth.json</code> file you can add your access token to an environment variable called <code>GITHUB_TOKEN</code>.</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Fetching issues for a repository</h2><a id="user-content-fetching-issues-for-a-repository" class="anchor" aria-label="Permalink: Fetching issues for a repository" href="#fetching-issues-for-a-repository"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">The <code>issues</code> command retrieves all of the issues belonging to a specified repository.</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ github-to-sqlite issues github.db simonw/datasette"><pre class="notranslate"><code>$ github-to-sqlite issues github.db simonw/datasette
  </code></pre></div>
  <p dir="auto">If an <code>auth.json</code> file is present it will use the token from that file. It works without authentication for public repositories but you should be aware that GitHub have strict IP-based rate limits for unauthenticated requests.</p>
  <p dir="auto">You can point to a different location of <code>auth.json</code> using <code>-a</code>:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ github-to-sqlite issues github.db simonw/datasette -a /path/to/auth.json"><pre class="notranslate"><code>$ github-to-sqlite issues github.db simonw/datasette -a /path/to/auth.json
  </code></pre></div>
  <p dir="auto">You can use the <code>--issue</code> option one or more times to load specific issues:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ github-to-sqlite issues github.db simonw/datasette --issue=1"><pre class="notranslate"><code>$ github-to-sqlite issues github.db simonw/datasette --issue=1
  </code></pre></div>
  <p dir="auto">Example: <a href="https://github-to-sqlite.dogsheep.net/github/issues" rel="nofollow">issues table</a></p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Fetching pull requests for a repository</h2><a id="user-content-fetching-pull-requests-for-a-repository" class="anchor" aria-label="Permalink: Fetching pull requests for a repository" href="#fetching-pull-requests-for-a-repository"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">While pull requests are a type of issue, you will get more information on pull requests by pulling them separately. For example, whether a pull request has been merged and when.</p>
  <p dir="auto">Following the API of issues, the <code>pull-requests</code> command retrieves all of the pull requests belonging to a specified repository.</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ github-to-sqlite pull-requests github.db simonw/datasette"><pre class="notranslate"><code>$ github-to-sqlite pull-requests github.db simonw/datasette
  </code></pre></div>
  <p dir="auto">You can use the <code>--pull-request</code> option one or more times to load specific pull request:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ github-to-sqlite pull-requests github.db simonw/datasette --pull-request=81"><pre class="notranslate"><code>$ github-to-sqlite pull-requests github.db simonw/datasette --pull-request=81
  </code></pre></div>
  <p dir="auto">Note that the <code>merged_by</code> column on the <code>pull_requests</code> table will only be populated for pull requests that are loaded using the <code>--pull-request</code> option - the GitHub API does not return this field for pull requests that are loaded in bulk.</p>
  <p dir="auto">You can load only pull requests in a certain state with the <code>--state</code> option:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ github-to-sqlite pull-requests --state=open github.db simonw/datasette"><pre class="notranslate"><code>$ github-to-sqlite pull-requests --state=open github.db simonw/datasette
  </code></pre></div>
  <p dir="auto">Pull requests across an entire organization (or more than one) can be loaded with <code>--org</code>:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ github-to-sqlite pull-requests --state=open --org=psf --org=python github.db"><pre class="notranslate"><code>$ github-to-sqlite pull-requests --state=open --org=psf --org=python github.db
  </code></pre></div>
  <p dir="auto">You can use a search query to find pull requests.  Note that no more than 1000 will be loaded (this is a GitHub API limitation), and some data will be missing (base and head SHAs).  When using searches, other filters are ignored; put all criteria into the search itself:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ github-to-sqlite pull-requests --search='org:python defaultdict state:closed created:<2023-09-01' github.db"><pre class="notranslate"><code>$ github-to-sqlite pull-requests --search='org:python defaultdict state:closed created:<2023-09-01' github.db
  </code></pre></div>
  <p dir="auto">Example: <a href="https://github-to-sqlite.dogsheep.net/github/pull_requests" rel="nofollow">pull_requests table</a></p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Fetching issue comments for a repository</h2><a id="user-content-fetching-issue-comments-for-a-repository" class="anchor" aria-label="Permalink: Fetching issue comments for a repository" href="#fetching-issue-comments-for-a-repository"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">The <code>issue-comments</code> command retrieves all of the comments on all of the issues in a repository.</p>
  <p dir="auto">It is recommended you run <code>issues</code> first, so that each imported comment can have a foreign key pointing to its issue.</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ github-to-sqlite issues github.db simonw/datasette
  $ github-to-sqlite issue-comments github.db simonw/datasette"><pre class="notranslate"><code>$ github-to-sqlite issues github.db simonw/datasette
  $ github-to-sqlite issue-comments github.db simonw/datasette
  </code></pre></div>
  <p dir="auto">You can use the <code>--issue</code> option to only load comments for a specific issue within that repository, for example:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ github-to-sqlite issue-comments github.db simonw/datasette --issue=1"><pre class="notranslate"><code>$ github-to-sqlite issue-comments github.db simonw/datasette --issue=1
  </code></pre></div>
  <p dir="auto">Example: <a href="https://github-to-sqlite.dogsheep.net/github/issue_comments" rel="nofollow">issue_comments table</a></p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Fetching commits for a repository</h2><a id="user-content-fetching-commits-for-a-repository" class="anchor" aria-label="Permalink: Fetching commits for a repository" href="#fetching-commits-for-a-repository"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">The <code>commits</code> command retrieves details of all of the commits for one or more repositories. It currently fetches the SHA, commit message and author and committer details; it does not retrieve the full commit body.</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ github-to-sqlite commits github.db simonw/datasette simonw/sqlite-utils"><pre class="notranslate"><code>$ github-to-sqlite commits github.db simonw/datasette simonw/sqlite-utils
  </code></pre></div>
  <p dir="auto">The command accepts one or more repositories.</p>
  <p dir="auto">By default it will stop as soon as it sees a commit that has previously been retrieved. You can force it to retrieve all commits (including those that have been previously inserted) using <code>--all</code>.</p>
  <p dir="auto">Example: <a href="https://github-to-sqlite.dogsheep.net/github/commits" rel="nofollow">commits table</a></p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Fetching releases for a repository</h2><a id="user-content-fetching-releases-for-a-repository" class="anchor" aria-label="Permalink: Fetching releases for a repository" href="#fetching-releases-for-a-repository"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">The <code>releases</code> command retrieves the releases for one or more repositories.</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ github-to-sqlite releases github.db simonw/datasette simonw/sqlite-utils"><pre class="notranslate"><code>$ github-to-sqlite releases github.db simonw/datasette simonw/sqlite-utils
  </code></pre></div>
  <p dir="auto">The command accepts one or more repositories.</p>
  <p dir="auto">Example: <a href="https://github-to-sqlite.dogsheep.net/github/releases" rel="nofollow">releases table</a></p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Fetching tags for a repository</h2><a id="user-content-fetching-tags-for-a-repository" class="anchor" aria-label="Permalink: Fetching tags for a repository" href="#fetching-tags-for-a-repository"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">The <code>tags</code> command retrieves all of the tags for one or more repositories.</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ github-to-sqlite tags github.db simonw/datasette simonw/sqlite-utils"><pre class="notranslate"><code>$ github-to-sqlite tags github.db simonw/datasette simonw/sqlite-utils
  </code></pre></div>
  <p dir="auto">Example: <a href="https://github-to-sqlite.dogsheep.net/github/tags" rel="nofollow">tags table</a></p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Fetching contributors to a repository</h2><a id="user-content-fetching-contributors-to-a-repository" class="anchor" aria-label="Permalink: Fetching contributors to a repository" href="#fetching-contributors-to-a-repository"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">The <code>contributors</code> command retrieves details of all of the contributors for one or more repositories.</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ github-to-sqlite contributors github.db simonw/datasette simonw/sqlite-utils"><pre class="notranslate"><code>$ github-to-sqlite contributors github.db simonw/datasette simonw/sqlite-utils
  </code></pre></div>
  <p dir="auto">The command accepts one or more repositories. It populates a <code>contributors</code> table, with foreign keys to <code>repos</code> and <code>users</code> and a <code>contributions</code> table listing the number of commits to that repository for each contributor.</p>
  <p dir="auto">Example: <a href="https://github-to-sqlite.dogsheep.net/github/contributors" rel="nofollow">contributors table</a></p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Fetching repos belonging to a user or organization</h2><a id="user-content-fetching-repos-belonging-to-a-user-or-organization" class="anchor" aria-label="Permalink: Fetching repos belonging to a user or organization" href="#fetching-repos-belonging-to-a-user-or-organization"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">The <code>repos</code> command fetches repos belonging to a user or organization.</p>
  <p dir="auto">Without any other arguments, this command will fetch all repos that the currently authenticated user owns, collaborates on or can access via one of their organizations:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ github-to-sqlite repos github.db"><pre class="notranslate"><code>$ github-to-sqlite repos github.db
  </code></pre></div>
  <p dir="auto">To fetch repos belonging to a specific user or organization, provide their username as an argument:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ github-to-sqlite repos github.db dogsheep # organization
  $ github-to-sqlite repos github.db simonw # user"><pre class="notranslate"><code>$ github-to-sqlite repos github.db dogsheep # organization
  $ github-to-sqlite repos github.db simonw # user
  </code></pre></div>
  <p dir="auto">You can pass more than one username to fetch for multiple users or organizations at once:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ github-to-sqlite repos github.db simonw dogsheep"><pre class="notranslate"><code>$ github-to-sqlite repos github.db simonw dogsheep
  </code></pre></div>
  <p dir="auto">Add the <code>--readme</code> option to save the README for the repo in a column called <code>readme</code>. Add <code>--readme-html</code> to save the HTML rendered version of the README into a column called <code>readme_html</code>.</p>
  <p dir="auto">Example: <a href="https://github-to-sqlite.dogsheep.net/github/repos" rel="nofollow">repos table</a></p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Fetching specific repositories</h2><a id="user-content-fetching-specific-repositories" class="anchor" aria-label="Permalink: Fetching specific repositories" href="#fetching-specific-repositories"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">You can use <code>-r</code> with the <code>repos</code> command one or more times to fetch just specific repositories.</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ github-to-sqlite repos github.db -r simonw/datasette -r dogsheep/github-to-sqlite"><pre class="notranslate"><code>$ github-to-sqlite repos github.db -r simonw/datasette -r dogsheep/github-to-sqlite
  </code></pre></div>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Fetching repos that have been starred by a user</h2><a id="user-content-fetching-repos-that-have-been-starred-by-a-user" class="anchor" aria-label="Permalink: Fetching repos that have been starred by a user" href="#fetching-repos-that-have-been-starred-by-a-user"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">The <code>starred</code> command fetches the repos that have been starred by a user.</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ github-to-sqlite starred github.db simonw"><pre class="notranslate"><code>$ github-to-sqlite starred github.db simonw
  </code></pre></div>
  <p dir="auto">If you are using an <code>auth.json</code> file you can omit the username to retrieve the starred repos for the authenticated user.</p>
  <p dir="auto">Example: <a href="https://github-to-sqlite.dogsheep.net/github/stars" rel="nofollow">stars table</a></p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Fetching users that have starred specific repos</h2><a id="user-content-fetching-users-that-have-starred-specific-repos" class="anchor" aria-label="Permalink: Fetching users that have starred specific repos" href="#fetching-users-that-have-starred-specific-repos"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">The <code>stargazers</code> command fetches the users that have starred the specified repos.</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ github-to-sqlite stargazers github.db simonw/datasette dogsheep/github-to-sqlite"><pre class="notranslate"><code>$ github-to-sqlite stargazers github.db simonw/datasette dogsheep/github-to-sqlite
  </code></pre></div>
  <p dir="auto">You can specify one or more repository using <code>owner/repo</code> syntax.</p>
  <p dir="auto">Users fetched using this command will be inserted into the <code>users</code> table. Many-to-many records showing which repository they starred will be added to the <code>stars</code> table.</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Fetching GitHub Actions workflows</h2><a id="user-content-fetching-github-actions-workflows" class="anchor" aria-label="Permalink: Fetching GitHub Actions workflows" href="#fetching-github-actions-workflows"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">The <code>workflows</code> command fetches the YAML workflow configurations from each repository's <code>.github/workflows</code> directory and parses them to populate <code>workflows</code>, <code>jobs</code> and <code>steps</code> tables.</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ github-to-sqlite workflows github.db simonw/datasette dogsheep/github-to-sqlite"><pre class="notranslate"><code>$ github-to-sqlite workflows github.db simonw/datasette dogsheep/github-to-sqlite
  </code></pre></div>
  <p dir="auto">You can specify one or more repository using <code>owner/repo</code> syntax.</p>
  <p dir="auto">Example: <a href="https://github-to-sqlite.dogsheep.net/github/workflows" rel="nofollow">workflows table</a>, <a href="https://github-to-sqlite.dogsheep.net/github/jobs" rel="nofollow">jobs table</a>, <a href="https://github-to-sqlite.dogsheep.net/github/steps" rel="nofollow">steps table</a></p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Scraping dependents for a repository</h2><a id="user-content-scraping-dependents-for-a-repository" class="anchor" aria-label="Permalink: Scraping dependents for a repository" href="#scraping-dependents-for-a-repository"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">The GitHub dependency graph can show other GitHub projects that depend on a specific repo, for example <a href="https://github.com/simonw/datasette/network/dependents">simonw/datasette/network/dependents</a>.</p>
  <p dir="auto">This data is not yet available through the GitHub API. The <code>scrape-dependents</code> command scrapes those pages and uses the GitHub API to load full versions of the dependent repositories.</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ github-to-sqlite scrape-dependents github.db simonw/datasette"><pre class="notranslate"><code>$ github-to-sqlite scrape-dependents github.db simonw/datasette
  </code></pre></div>
  <p dir="auto">The command accepts one or more repositories.</p>
  <p dir="auto">Add <code>-v</code> for verbose output.</p>
  <p dir="auto">Example: <a href="https://github-to-sqlite.dogsheep.net/github/dependents?_sort_desc=first_seen_utc" rel="nofollow">dependents table</a></p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Fetching emojis</h2><a id="user-content-fetching-emojis" class="anchor" aria-label="Permalink: Fetching emojis" href="#fetching-emojis"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">You can fetch a list of every emoji supported by GitHub using the <code>emojis</code> command:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ github-to-sqlite emojis github.db"><pre class="notranslate"><code>$ github-to-sqlite emojis github.db
  </code></pre></div>
  <p dir="auto">This will create a table called <code>emojis</code> with a primary key <code>name</code> and a <code>url</code> column.</p>
  <p dir="auto">If you add the <code>--fetch</code> option the command will also fetch the binary content of the images and place them in an <code>image</code> column:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ github-to-sqlite emojis emojis.db -f
  [########----------------------------]  397/1799   22%  00:03:43"><pre class="notranslate"><code>$ github-to-sqlite emojis emojis.db -f
  [########----------------------------]  397/1799   22%  00:03:43
  </code></pre></div>
  <p dir="auto">You can then use the <a href="https://github.com/simonw/datasette-render-images">datasette-render-images</a> plugin to browse them visually.</p>
  <p dir="auto">Example: <a href="https://github-to-sqlite.dogsheep.net/github/emojis" rel="nofollow">emojis table</a></p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Making authenticated API calls</h2><a id="user-content-making-authenticated-api-calls" class="anchor" aria-label="Permalink: Making authenticated API calls" href="#making-authenticated-api-calls"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">The <code>github-to-sqlite get</code> command provides a convenient shortcut for making authenticated calls to the API. Once you have created your <code>auth.json</code> file (or set a <code>GITHUB_TOKEN</code> environment variable) you can use it like this:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ github-to-sqlite get https://api.github.com/gists"><pre class="notranslate"><code>$ github-to-sqlite get https://api.github.com/gists
  </code></pre></div>
  <p dir="auto">This will make an authenticated call to the URL you provide and pretty-print the resulting JSON to the console.</p>
  <p dir="auto">You can omit the <code>https://api.github.com/</code> prefix, for example:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ github-to-sqlite get /gists"><pre class="notranslate"><code>$ github-to-sqlite get /gists
  </code></pre></div>
  <p dir="auto">Many GitHub APIs are <a href="https://docs.github.com/en/rest/guides/traversing-with-pagination">paginated using the HTTP Link header</a>. You can follow this pagination and output a list of all of the resulting items using <code>--paginate</code>:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ github-to-sqlite get /users/simonw/repos --paginate"><pre class="notranslate"><code>$ github-to-sqlite get /users/simonw/repos --paginate
  </code></pre></div>
  <p dir="auto">You can outline newline-delimited JSON for each item using <code>--nl</code>. This can be useful for streaming items into another tool.</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ github-to-sqlite get /users/simonw/repos --nl"><pre class="notranslate"><code>$ github-to-sqlite get /users/simonw/repos --nl
  </code></pre></div>
  </article></div>


node_id: MDEwOlJlcG9zaXRvcnkyMDc2MzAxNzQ=
name: datasette-rure
full_name: simonw/datasette-rure
html_url: https://github.com/simonw/datasette-rure
created_at: 2019-09-10T18:09:33Z
updated_at: 2025-02-10T15:11:30Z
pushed_at: 2019-09-11T22:59:38Z
language: Python
license: apache-2.0
topics: ["datasette", "datasette-io", "datasette-plugin", "regular-expressions", "sqlite"]
visibility: public
default_branch: master
permissions: {"admin": false, "maintain": false, "push": false, "triage": false, "pull": true}
description:
  Datasette plugin that adds a custom SQL function for executing matches using the Rust regular expression engine

readme:
  # datasette-rure

  [![PyPI](https://img.shields.io/pypi/v/datasette-rure.svg)](https://pypi.org/project/datasette-rure/)
  [![CircleCI](https://circleci.com/gh/simonw/datasette-rure.svg?style=svg)](https://circleci.com/gh/simonw/datasette-rure)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/datasette-rure/blob/master/LICENSE)

  Datasette plugin that adds a custom SQL function for executing matches using the Rust regular expression engine

  Install this plugin in the same environment as Datasette to enable the `regexp()` SQL function.

      $ pip install datasette-rure

  The plugin is built on top of the [rure-python](https://github.com/davidblewett/rure-python) library by David Blewett.

  ## regexp() to test regular expressions

  You can test if a value matches a regular expression like this:

      select regexp('hi.*there', 'hi there')
      -- returns 1
      select regexp('not.*there', 'hi there')
      -- returns 0

  You can also use SQLite's custom syntax to run matches:

      select 'hi there' REGEXP 'hi.*there'
      -- returns 1

  This means you can select rows based on regular expression matches - for example, to select every article where the title begins with an E or an F:

      select * from articles where title REGEXP '^[EF]'

  Try this out: [REGEXP interactive demo](https://datasette-rure-demo.datasette.io/24ways?sql=select+*+from+articles+where+title+REGEXP+%27%5E%5BEF%5D%27)

  ## regexp_match() to extract groups

  You can extract captured subsets of a pattern using `regexp_match()`.

      select regexp_match('.*( and .*)', title) as n from articles where n is not null
      -- Returns the ' and X' component of any matching titles, e.g.
      --     and Recognition
      --     and Transitions Their Place
      -- etc

  This will return the first parenthesis match when called with two arguments. You can call it with three arguments to indicate which match you would like to extract:

      select regexp_match('.*(and)(.*)', title, 2) as n from articles where n is not null

  The function will return `null` for invalid inputs e.g. a pattern without capture groups.

  Try this out: [regexp_match() interactive demo](https://datasette-rure-demo.datasette.io/24ways?sql=select+%27WHY+%27+%7C%7C+regexp_match%28%27Why+%28.*%29%27%2C+title%29+as+t+from+articles+where+t+is+not+null)

  ## regexp_matches() to extract multiple matches at once

  The `regexp_matches()` function can be used to extract multiple patterns from a single string. The result is returned as a JSON array, which can then be further processed using SQLite's [JSON functions](https://www.sqlite.org/json1.html).

  The first argument is a regular expression with named capture groups. The second argument is the string to be matched.

      select regexp_matches(
          'hello (?P<name>\w+) the (?P<species>\w+)',
          'hello bob the dog, hello maggie the cat, hello tarquin the otter'
      )

  This will return a list of JSON objects, each one representing the named captures from the original regular expression:

      [
          {"name": "bob", "species": "dog"},
          {"name": "maggie", "species": "cat"},
          {"name": "tarquin", "species": "otter"}
      ]

  Try this out: [regexp_matches() interactive demo](https://datasette-rure-demo.datasette.io/24ways?sql=select+regexp_matches%28%0D%0A++++%27hello+%28%3FP%3Cname%3E%5Cw%2B%29+the+%28%3FP%3Cspecies%3E%5Cw%2B%29%27%2C%0D%0A++++%27hello+bob+the+dog%2C+hello+maggie+the+cat%2C+hello+tarquin+the+otter%27%0D%0A%29)

readme_html:
  <div id="readme" class="md" data-path="README.md"><article class="markdown-body entry-content container-lg" itemprop="text"><div class="markdown-heading" dir="auto"><h1 class="heading-element" dir="auto">datasette-rure</h1><a id="user-content-datasette-rure" class="anchor" aria-label="Permalink: datasette-rure" href="#datasette-rure"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto"><a href="https://pypi.org/project/datasette-rure/" rel="nofollow"><img src="https://img.shields.io/pypi/v/datasette-rure.svg" alt="PyPI" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/cdd751b79b920a00e823ad779af3c310f0285483c4638b72ab009cdf8d72be92/68747470733a2f2f696d672e736869656c64732e696f2f707970692f762f6461746173657474652d727572652e737667"></a>
  <a href="https://circleci.com/gh/simonw/datasette-rure" rel="nofollow"><img src="https://circleci.com/gh/simonw/datasette-rure.svg?style=svg" alt="CircleCI" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/e80ea86cad502ba9a84bf54585ae712311a37e7ed44fa46d755274d3403e60c3/68747470733a2f2f636972636c6563692e636f6d2f67682f73696d6f6e772f6461746173657474652d727572652e7376673f7374796c653d737667"></a>
  <a href="https://github.com/simonw/datasette-rure/blob/master/LICENSE"><img src="https://img.shields.io/badge/license-Apache%202.0-blue.svg" alt="License" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/c355f200ea90fddaa407b6eaab303663a669248ea3ca7b1fcf77dbe04ff5f48c/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6c6963656e73652d417061636865253230322e302d626c75652e737667"></a></p>
  <p dir="auto">Datasette plugin that adds a custom SQL function for executing matches using the Rust regular expression engine</p>
  <p dir="auto">Install this plugin in the same environment as Datasette to enable the <code>regexp()</code> SQL function.</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ pip install datasette-rure"><pre class="notranslate"><code>$ pip install datasette-rure
  </code></pre></div>
  <p dir="auto">The plugin is built on top of the <a href="https://github.com/davidblewett/rure-python">rure-python</a> library by David Blewett.</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">regexp() to test regular expressions</h2><a id="user-content-regexp-to-test-regular-expressions" class="anchor" aria-label="Permalink: regexp() to test regular expressions" href="#regexp-to-test-regular-expressions"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">You can test if a value matches a regular expression like this:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="select regexp('hi.*there', 'hi there')
  -- returns 1
  select regexp('not.*there', 'hi there')
  -- returns 0"><pre class="notranslate"><code>select regexp('hi.*there', 'hi there')
  -- returns 1
  select regexp('not.*there', 'hi there')
  -- returns 0
  </code></pre></div>
  <p dir="auto">You can also use SQLite's custom syntax to run matches:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="select 'hi there' REGEXP 'hi.*there'
  -- returns 1"><pre class="notranslate"><code>select 'hi there' REGEXP 'hi.*there'
  -- returns 1
  </code></pre></div>
  <p dir="auto">This means you can select rows based on regular expression matches - for example, to select every article where the title begins with an E or an F:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="select * from articles where title REGEXP '^[EF]'"><pre class="notranslate"><code>select * from articles where title REGEXP '^[EF]'
  </code></pre></div>
  <p dir="auto">Try this out: <a href="https://datasette-rure-demo.datasette.io/24ways?sql=select+*+from+articles+where+title+REGEXP+%27%5E%5BEF%5D%27" rel="nofollow">REGEXP interactive demo</a></p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">regexp_match() to extract groups</h2><a id="user-content-regexp_match-to-extract-groups" class="anchor" aria-label="Permalink: regexp_match() to extract groups" href="#regexp_match-to-extract-groups"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">You can extract captured subsets of a pattern using <code>regexp_match()</code>.</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="select regexp_match('.*( and .*)', title) as n from articles where n is not null
  -- Returns the ' and X' component of any matching titles, e.g.
  --     and Recognition
  --     and Transitions Their Place
  -- etc"><pre class="notranslate"><code>select regexp_match('.*( and .*)', title) as n from articles where n is not null
  -- Returns the ' and X' component of any matching titles, e.g.
  --     and Recognition
  --     and Transitions Their Place
  -- etc
  </code></pre></div>
  <p dir="auto">This will return the first parenthesis match when called with two arguments. You can call it with three arguments to indicate which match you would like to extract:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="select regexp_match('.*(and)(.*)', title, 2) as n from articles where n is not null"><pre class="notranslate"><code>select regexp_match('.*(and)(.*)', title, 2) as n from articles where n is not null
  </code></pre></div>
  <p dir="auto">The function will return <code>null</code> for invalid inputs e.g. a pattern without capture groups.</p>
  <p dir="auto">Try this out: <a href="https://datasette-rure-demo.datasette.io/24ways?sql=select+%27WHY+%27+%7C%7C+regexp_match%28%27Why+%28.*%29%27%2C+title%29+as+t+from+articles+where+t+is+not+null" rel="nofollow">regexp_match() interactive demo</a></p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">regexp_matches() to extract multiple matches at once</h2><a id="user-content-regexp_matches-to-extract-multiple-matches-at-once" class="anchor" aria-label="Permalink: regexp_matches() to extract multiple matches at once" href="#regexp_matches-to-extract-multiple-matches-at-once"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">The <code>regexp_matches()</code> function can be used to extract multiple patterns from a single string. The result is returned as a JSON array, which can then be further processed using SQLite's <a href="https://www.sqlite.org/json1.html" rel="nofollow">JSON functions</a>.</p>
  <p dir="auto">The first argument is a regular expression with named capture groups. The second argument is the string to be matched.</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="select regexp_matches(
      'hello (?P<name>\w+) the (?P<species>\w+)',
      'hello bob the dog, hello maggie the cat, hello tarquin the otter'
  )"><pre class="notranslate"><code>select regexp_matches(
      'hello (?P<name>\w+) the (?P<species>\w+)',
      'hello bob the dog, hello maggie the cat, hello tarquin the otter'
  )
  </code></pre></div>
  <p dir="auto">This will return a list of JSON objects, each one representing the named captures from the original regular expression:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="[
      {"name": "bob", "species": "dog"},
      {"name": "maggie", "species": "cat"},
      {"name": "tarquin", "species": "otter"}
  ]"><pre class="notranslate"><code>[
      {"name": "bob", "species": "dog"},
      {"name": "maggie", "species": "cat"},
      {"name": "tarquin", "species": "otter"}
  ]
  </code></pre></div>
  <p dir="auto">Try this out: <a href="https://datasette-rure-demo.datasette.io/24ways?sql=select+regexp_matches%28%0D%0A++++%27hello+%28%3FP%3Cname%3E%5Cw%2B%29+the+%28%3FP%3Cspecies%3E%5Cw%2B%29%27%2C%0D%0A++++%27hello+bob+the+dog%2C+hello+maggie+the+cat%2C+hello+tarquin+the+otter%27%0D%0A%29" rel="nofollow">regexp_matches() interactive demo</a></p>
  </article></div>


node_id: MDEwOlJlcG9zaXRvcnkyMDkwOTEyNTY=
name: datasette-atom
full_name: simonw/datasette-atom
html_url: https://github.com/simonw/datasette-atom
description: Datasette plugin that adds a .atom output format
created_at: 2019-09-17T15:31:01Z
updated_at: 2026-04-08T18:01:35Z
pushed_at: 2026-04-08T18:02:43Z
language: Python
license: apache-2.0
topics: ["datasette", "datasette-io", "datasette-plugin"]
visibility: public
default_branch: main
permissions: {"admin": false, "maintain": false, "push": false, "triage": false, "pull": false}
pull_request_creation_policy: all
readme:
  # datasette-atom

  [![PyPI](https://img.shields.io/pypi/v/datasette-atom.svg)](https://pypi.org/project/datasette-atom/)
  [![Changelog](https://img.shields.io/github/v/release/simonw/datasette-atom?include_prereleases&label=changelog)](https://github.com/simonw/datasette-atom/releases)
  [![Tests](https://github.com/simonw/datasette-atom/workflows/Test/badge.svg)](https://github.com/simonw/datasette-atom/actions?query=workflow%3ATest)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/datasette-atom/blob/main/LICENSE)

  Datasette plugin that adds support for generating [Atom feeds](https://validator.w3.org/feed/docs/atom.html) with the results of a SQL query.

  ## Installation

  Install this plugin in the same environment as Datasette to enable the `.atom` output extension.

      $ pip install datasette-atom

  ## Usage

  To create an Atom feed you need to define a custom SQL query that returns a required set of columns:

  * `atom_id` - a unique ID for each row. [This article](https://web.archive.org/web/20080211143232/http://diveintomark.org/archives/2004/05/28/howto-atom-id) has suggestions about ways to create these IDs.
  * `atom_title` - a title for that row.
  * `atom_updated` - an [RFC 3339](http://www.faqs.org/rfcs/rfc3339.html) timestamp representing the last time the entry was modified in a significant way. This can usually be the time that the row was created.

  The following columns are optional:

  * `atom_content` - content that should be shown in the feed. This will be treated as a regular string, so any embedded HTML tags will be escaped when they are displayed.
  * `atom_content_html` - content that should be shown in the feed. This will be treated as an HTML string, and will be sanitized using [Bleach](https://github.com/mozilla/bleach) to ensure it does not have any malicious code in it before being returned as part of a `<content type="html">` Atom element. If both are provided, this will be used in place of `atom_content`.
  * `atom_link` - a URL that should be used as the link that the feed entry points to.
  * `atom_author_name` - the name of the author of the entry. If you provide this you can also provide `atom_author_uri` and `atom_author_email` with a URL and e-mail address for that author.

  A query that returns these columns can then be returned as an Atom feed by adding the `.atom` extension.

  ## Example

  Here is an example SQL query which generates an Atom feed for new entries on [www.niche-museums.com](https://www.niche-museums.com/):

  ```sql
  select
    'tag:niche-museums.com,' || substr(created, 0, 11) || ':' || id as atom_id,
    name as atom_title,
    created as atom_updated,
    'https://www.niche-museums.com/browse/museums/' || id as atom_link,
    coalesce(
      '<img src="' || photo_url || '?w=800&amp;h=400&amp;fit=crop&amp;auto=compress">',
      ''
    ) || '<p>' || description || '</p>' as atom_content_html
  from
    museums
  order by
    created desc
  limit
    15
  ```

  You can try this query by [pasting it in here](https://www.niche-museums.com/browse) - then click the `.atom` link to see it as an Atom feed.

  ## Using a canned query

  Datasette's [canned query mechanism](https://docs.datasette.io/en/stable/sql_queries.html#canned-queries) is a useful way to configure feeds. If a canned query definition has a `title` that will be used as the title of the Atom feed.

  Here's an example, defined using a `metadata.yaml` file:

  ```yaml
  databases:
    browse:
      queries:
        feed:
          title: Niche Museums
          sql: |-
            select
              'tag:niche-museums.com,' || substr(created, 0, 11) || ':' || id as atom_id,
              name as atom_title,
              created as atom_updated,
              'https://www.niche-museums.com/browse/museums/' || id as atom_link,
              coalesce(
                '<img src="' || photo_url || '?w=800&amp;h=400&amp;fit=crop&amp;auto=compress">',
                ''
              ) || '<p>' || description || '</p>' as atom_content_html
            from
              museums
            order by
              created desc
            limit
              15
  ```
  ## Disabling HTML filtering

  The HTML allow-list used by Bleach for the `atom_content_html` column can be found in the `clean(html)` function at the bottom of [datasette_atom/__init__.py](https://github.com/simonw/datasette-atom/blob/main/datasette_atom/__init__.py).

  You can disable Bleach entirely for Atom feeds generated using a canned query. You should only do this if you are certain that no user-provided HTML could be included in that value.

  Here's how to do that in `metadata.json`:

  ```json
  {
    "plugins": {
      "datasette-atom": {
        "allow_unsafe_html_in_canned_queries": true
      }
    }
  }
  ```
  Setting this to `true` will disable Bleach filtering for all canned queries across all databases.

  You can disable Bleach filtering just for a specific list of canned queries like so:

  ```json
  {
    "plugins": {
      "datasette-atom": {
        "allow_unsafe_html_in_canned_queries": {
          "museums": ["latest", "moderation"]
        }
      }
    }
  }
  ```
  This will disable Bleach just for the canned queries called `latest` and `moderation` in the `museums.db` database.

readme_html:
  <div id="readme" class="md" data-path="README.md"><article class="markdown-body entry-content container-lg" itemprop="text"><div class="markdown-heading" dir="auto"><h1 class="heading-element" dir="auto">datasette-atom</h1><a id="user-content-datasette-atom" class="anchor" aria-label="Permalink: datasette-atom" href="#datasette-atom"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto"><a href="https://pypi.org/project/datasette-atom/" rel="nofollow"><img src="https://img.shields.io/pypi/v/datasette-atom.svg" alt="PyPI" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/09e3c940d9fc62ec61319fc9089b4401e043f6ff8cbd2e9bfb974c666d8a9738/68747470733a2f2f696d672e736869656c64732e696f2f707970692f762f6461746173657474652d61746f6d2e737667"></a>
  <a href="https://github.com/simonw/datasette-atom/releases"><img src="https://img.shields.io/github/v/release/simonw/datasette-atom?include_prereleases&label=changelog" alt="Changelog" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/bb4b1dd86c36941c2c9edee8293afb6be48972e0b8da1ca2d1bcd979dbd13153/68747470733a2f2f696d672e736869656c64732e696f2f6769746875622f762f72656c656173652f73696d6f6e772f6461746173657474652d61746f6d3f696e636c7564655f70726572656c6561736573266c6162656c3d6368616e67656c6f67"></a>
  <a href="https://github.com/simonw/datasette-atom/actions?query=workflow%3ATest"><img src="https://github.com/simonw/datasette-atom/workflows/Test/badge.svg" alt="Tests" style="max-width: 100%;"></a>
  <a href="https://github.com/simonw/datasette-atom/blob/main/LICENSE"><img src="https://img.shields.io/badge/license-Apache%202.0-blue.svg" alt="License" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/c355f200ea90fddaa407b6eaab303663a669248ea3ca7b1fcf77dbe04ff5f48c/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6c6963656e73652d417061636865253230322e302d626c75652e737667"></a></p>
  <p dir="auto">Datasette plugin that adds support for generating <a href="https://validator.w3.org/feed/docs/atom.html" rel="nofollow">Atom feeds</a> with the results of a SQL query.</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Installation</h2><a id="user-content-installation" class="anchor" aria-label="Permalink: Installation" href="#installation"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">Install this plugin in the same environment as Datasette to enable the <code>.atom</code> output extension.</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ pip install datasette-atom"><pre class="notranslate"><code>$ pip install datasette-atom
  </code></pre></div>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Usage</h2><a id="user-content-usage" class="anchor" aria-label="Permalink: Usage" href="#usage"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">To create an Atom feed you need to define a custom SQL query that returns a required set of columns:</p>
  <ul dir="auto">
  <li><code>atom_id</code> - a unique ID for each row. <a href="https://web.archive.org/web/20080211143232/http://diveintomark.org/archives/2004/05/28/howto-atom-id" rel="nofollow">This article</a> has suggestions about ways to create these IDs.</li>
  <li><code>atom_title</code> - a title for that row.</li>
  <li><code>atom_updated</code> - an <a href="http://www.faqs.org/rfcs/rfc3339.html" rel="nofollow">RFC 3339</a> timestamp representing the last time the entry was modified in a significant way. This can usually be the time that the row was created.</li>
  </ul>
  <p dir="auto">The following columns are optional:</p>
  <ul dir="auto">
  <li><code>atom_content</code> - content that should be shown in the feed. This will be treated as a regular string, so any embedded HTML tags will be escaped when they are displayed.</li>
  <li><code>atom_content_html</code> - content that should be shown in the feed. This will be treated as an HTML string, and will be sanitized using <a href="https://github.com/mozilla/bleach">Bleach</a> to ensure it does not have any malicious code in it before being returned as part of a <code><content type="html"></code> Atom element. If both are provided, this will be used in place of <code>atom_content</code>.</li>
  <li><code>atom_link</code> - a URL that should be used as the link that the feed entry points to.</li>
  <li><code>atom_author_name</code> - the name of the author of the entry. If you provide this you can also provide <code>atom_author_uri</code> and <code>atom_author_email</code> with a URL and e-mail address for that author.</li>
  </ul>
  <p dir="auto">A query that returns these columns can then be returned as an Atom feed by adding the <code>.atom</code> extension.</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Example</h2><a id="user-content-example" class="anchor" aria-label="Permalink: Example" href="#example"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">Here is an example SQL query which generates an Atom feed for new entries on <a href="https://www.niche-museums.com/" rel="nofollow">www.niche-museums.com</a>:</p>
  <div class="highlight highlight-source-sql notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="select
    'tag:niche-museums.com,' || substr(created, 0, 11) || ':' || id as atom_id,
    name as atom_title,
    created as atom_updated,
    'https://www.niche-museums.com/browse/museums/' || id as atom_link,
    coalesce(
      '<img src="' || photo_url || '?w=800&amp;h=400&amp;fit=crop&amp;auto=compress">',
      ''
    ) || '<p>' || description || '</p>' as atom_content_html
  from
    museums
  order by
    created desc
  limit
    15"><pre><span class="pl-k">select</span>
    <span class="pl-s"><span class="pl-pds">'</span>tag:niche-museums.com,<span class="pl-pds">'</span></span> <span class="pl-k">||</span> substr(created, <span class="pl-c1">0</span>, <span class="pl-c1">11</span>) <span class="pl-k">||</span> <span class="pl-s"><span class="pl-pds">'</span>:<span class="pl-pds">'</span></span> <span class="pl-k">||</span> id <span class="pl-k">as</span> atom_id,
    name <span class="pl-k">as</span> atom_title,
    created <span class="pl-k">as</span> atom_updated,
    <span class="pl-s"><span class="pl-pds">'</span>https://www.niche-museums.com/browse/museums/<span class="pl-pds">'</span></span> <span class="pl-k">||</span> id <span class="pl-k">as</span> atom_link,
    coalesce(
      <span class="pl-s"><span class="pl-pds">'</span><img src="<span class="pl-pds">'</span></span> <span class="pl-k">||</span> photo_url <span class="pl-k">||</span> <span class="pl-s"><span class="pl-pds">'</span>?w=800&amp;h=400&amp;fit=crop&amp;auto=compress"><span class="pl-pds">'</span></span>,
      <span class="pl-s"><span class="pl-pds">'</span><span class="pl-pds">'</span></span>
    ) <span class="pl-k">||</span> <span class="pl-s"><span class="pl-pds">'</span><p><span class="pl-pds">'</span></span> <span class="pl-k">||</span> description <span class="pl-k">||</span> <span class="pl-s"><span class="pl-pds">'</span></p><span class="pl-pds">'</span></span> <span class="pl-k">as</span> atom_content_html
  <span class="pl-k">from</span>
    museums
  <span class="pl-k">order by</span>
    created <span class="pl-k">desc</span>
  <span class="pl-k">limit</span>
    <span class="pl-c1">15</span></pre></div>
  <p dir="auto">You can try this query by <a href="https://www.niche-museums.com/browse" rel="nofollow">pasting it in here</a> - then click the <code>.atom</code> link to see it as an Atom feed.</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Using a canned query</h2><a id="user-content-using-a-canned-query" class="anchor" aria-label="Permalink: Using a canned query" href="#using-a-canned-query"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">Datasette's <a href="https://docs.datasette.io/en/stable/sql_queries.html#canned-queries" rel="nofollow">canned query mechanism</a> is a useful way to configure feeds. If a canned query definition has a <code>title</code> that will be used as the title of the Atom feed.</p>
  <p dir="auto">Here's an example, defined using a <code>metadata.yaml</code> file:</p>
  <div class="highlight highlight-source-yaml notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="databases:
    browse:
      queries:
        feed:
          title: Niche Museums
          sql: |-
            select
              'tag:niche-museums.com,' || substr(created, 0, 11) || ':' || id as atom_id,
              name as atom_title,
              created as atom_updated,
              'https://www.niche-museums.com/browse/museums/' || id as atom_link,
              coalesce(
                '<img src="' || photo_url || '?w=800&amp;h=400&amp;fit=crop&amp;auto=compress">',
                ''
              ) || '<p>' || description || '</p>' as atom_content_html
            from
              museums
            order by
              created desc
            limit
              15"><pre><span class="pl-ent">databases</span>:
    <span class="pl-ent">browse</span>:
      <span class="pl-ent">queries</span>:
        <span class="pl-ent">feed</span>:
          <span class="pl-ent">title</span>: <span class="pl-s">Niche Museums</span>
          <span class="pl-ent">sql</span>: <span class="pl-s">|-</span>
  <span class="pl-s">          select</span>
  <span class="pl-s">            'tag:niche-museums.com,' || substr(created, 0, 11) || ':' || id as atom_id,</span>
  <span class="pl-s">            name as atom_title,</span>
  <span class="pl-s">            created as atom_updated,</span>
  <span class="pl-s">            'https://www.niche-museums.com/browse/museums/' || id as atom_link,</span>
  <span class="pl-s">            coalesce(</span>
  <span class="pl-s">              '<img src="' || photo_url || '?w=800&amp;h=400&amp;fit=crop&amp;auto=compress">',</span>
  <span class="pl-s">              ''</span>
  <span class="pl-s">            ) || '<p>' || description || '</p>' as atom_content_html</span>
  <span class="pl-s">          from</span>
  <span class="pl-s">            museums</span>
  <span class="pl-s">          order by</span>
  <span class="pl-s">            created desc</span>
  <span class="pl-s">          limit</span>
  <span class="pl-s">            15</span></pre></div>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Disabling HTML filtering</h2><a id="user-content-disabling-html-filtering" class="anchor" aria-label="Permalink: Disabling HTML filtering" href="#disabling-html-filtering"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">The HTML allow-list used by Bleach for the <code>atom_content_html</code> column can be found in the <code>clean(html)</code> function at the bottom of <a href="https://github.com/simonw/datasette-atom/blob/main/datasette_atom/__init__.py">datasette_atom/<strong>init</strong>.py</a>.</p>
  <p dir="auto">You can disable Bleach entirely for Atom feeds generated using a canned query. You should only do this if you are certain that no user-provided HTML could be included in that value.</p>
  <p dir="auto">Here's how to do that in <code>metadata.json</code>:</p>
  <div class="highlight highlight-source-json notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="{
    "plugins": {
      "datasette-atom": {
        "allow_unsafe_html_in_canned_queries": true
      }
    }
  }"><pre>{
    <span class="pl-ent">"plugins"</span>: {
      <span class="pl-ent">"datasette-atom"</span>: {
        <span class="pl-ent">"allow_unsafe_html_in_canned_queries"</span>: <span class="pl-c1">true</span>
      }
    }
  }</pre></div>
  <p dir="auto">Setting this to <code>true</code> will disable Bleach filtering for all canned queries across all databases.</p>
  <p dir="auto">You can disable Bleach filtering just for a specific list of canned queries like so:</p>
  <div class="highlight highlight-source-json notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="{
    "plugins": {
      "datasette-atom": {
        "allow_unsafe_html_in_canned_queries": {
          "museums": ["latest", "moderation"]
        }
      }
    }
  }"><pre>{
    <span class="pl-ent">"plugins"</span>: {
      <span class="pl-ent">"datasette-atom"</span>: {
        <span class="pl-ent">"allow_unsafe_html_in_canned_queries"</span>: {
          <span class="pl-ent">"museums"</span>: [<span class="pl-s"><span class="pl-pds">"</span>latest<span class="pl-pds">"</span></span>, <span class="pl-s"><span class="pl-pds">"</span>moderation<span class="pl-pds">"</span></span>]
        }
      }
    }
  }</pre></div>
  <p dir="auto">This will disable Bleach just for the canned queries called <code>latest</code> and <code>moderation</code> in the <code>museums.db</code> database.</p>
  </article></div>


node_id: MDEwOlJlcG9zaXRvcnkyMDk1OTAzNDU=
name: genome-to-sqlite
full_name: dogsheep/genome-to-sqlite
html_url: https://github.com/dogsheep/genome-to-sqlite
description: Import your genome into a SQLite database
created_at: 2019-09-19T15:38:39Z
updated_at: 2024-08-23T16:15:52Z
pushed_at: 2019-09-19T15:41:17Z
language: Python
license: apache-2.0
visibility: public
default_branch: master
permissions: {"admin": false, "maintain": false, "push": false, "triage": false, "pull": true}
custom_properties: {}
topics:
  ["23andme", "datasette", "datasette-io", "datasette-tool", "dogsheep", "genetics", "personal-analytics", "sqlite"]

readme:
  # genome-to-sqlite

  [![PyPI](https://img.shields.io/pypi/v/genome-to-sqlite.svg)](https://pypi.org/project/genome-to-sqlite/)
  [![CircleCI](https://circleci.com/gh/dogsheep/genome-to-sqlite.svg?style=svg)](https://circleci.com/gh/dogsheep/genome-to-sqlite)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/dogsheep/genome-to-sqlite/blob/master/LICENSE)

  Import your genome into a SQLite database.

  ## How to install

      $ pip install genome-to-sqlite

  ## How to use

  First, export your genome. This tool has only been tested against 23andMe so far. You can request an export of your genome from https://you.23andme.com/tools/data/download/

  Now you can convert the resulting `export.zip` file to SQLite like so:

      $ genome-to-sqlite export.zip genome.db

  A progress bar will be displayed. You can disable this using `--silent`.

  ```
  Importing genome  [#----------------]    5%  00:01:33
  ```

  You can explore the resulting data using [Datasette](https://datasette.readthedocs.io/) like this:

      $ datasette genome.db --config facet_time_limit_ms:1000

  Bumping up the facet time limit is useful in order to enable faceting by chromosome:

  http://127.0.0.1:8001/genome/genome?_facet=chromosome&_sort=position

readme_html:
  <div id="readme" class="md" data-path="README.md"><article class="markdown-body entry-content container-lg" itemprop="text"><div class="markdown-heading" dir="auto"><h1 class="heading-element" dir="auto">genome-to-sqlite</h1><a id="user-content-genome-to-sqlite" class="anchor" aria-label="Permalink: genome-to-sqlite" href="#genome-to-sqlite"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto"><a href="https://pypi.org/project/genome-to-sqlite/" rel="nofollow"><img src="https://img.shields.io/pypi/v/genome-to-sqlite.svg" alt="PyPI" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/54b5d4caab359e45a66536b64f86895d2829312ee6eb95e63163e86fb6a0ece3/68747470733a2f2f696d672e736869656c64732e696f2f707970692f762f67656e6f6d652d746f2d73716c6974652e737667"></a>
  <a href="https://circleci.com/gh/dogsheep/genome-to-sqlite" rel="nofollow"><img src="https://circleci.com/gh/dogsheep/genome-to-sqlite.svg?style=svg" alt="CircleCI" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/51d5eb48796b70325d1bb50f8b955d00ca378c1111fa636f9818a26a094c3737/68747470733a2f2f636972636c6563692e636f6d2f67682f646f6773686565702f67656e6f6d652d746f2d73716c6974652e7376673f7374796c653d737667"></a>
  <a href="https://github.com/dogsheep/genome-to-sqlite/blob/master/LICENSE"><img src="https://img.shields.io/badge/license-Apache%202.0-blue.svg" alt="License" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/c355f200ea90fddaa407b6eaab303663a669248ea3ca7b1fcf77dbe04ff5f48c/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6c6963656e73652d417061636865253230322e302d626c75652e737667"></a></p>
  <p dir="auto">Import your genome into a SQLite database.</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">How to install</h2><a id="user-content-how-to-install" class="anchor" aria-label="Permalink: How to install" href="#how-to-install"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ pip install genome-to-sqlite"><pre class="notranslate"><code>$ pip install genome-to-sqlite
  </code></pre></div>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">How to use</h2><a id="user-content-how-to-use" class="anchor" aria-label="Permalink: How to use" href="#how-to-use"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">First, export your genome. This tool has only been tested against 23andMe so far. You can request an export of your genome from <a href="https://you.23andme.com/tools/data/download/" rel="nofollow">https://you.23andme.com/tools/data/download/</a></p>
  <p dir="auto">Now you can convert the resulting <code>export.zip</code> file to SQLite like so:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ genome-to-sqlite export.zip genome.db"><pre class="notranslate"><code>$ genome-to-sqlite export.zip genome.db
  </code></pre></div>
  <p dir="auto">A progress bar will be displayed. You can disable this using <code>--silent</code>.</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="Importing genome  [#----------------]    5%  00:01:33"><pre class="notranslate"><code>Importing genome  [#----------------]    5%  00:01:33
  </code></pre></div>
  <p dir="auto">You can explore the resulting data using <a href="https://datasette.readthedocs.io/" rel="nofollow">Datasette</a> like this:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ datasette genome.db --config facet_time_limit_ms:1000"><pre class="notranslate"><code>$ datasette genome.db --config facet_time_limit_ms:1000
  </code></pre></div>
  <p dir="auto">Bumping up the facet time limit is useful in order to enable faceting by chromosome:</p>
  <p dir="auto"><a href="http://127.0.0.1:8001/genome/genome?_facet=chromosome&_sort=position" rel="nofollow">http://127.0.0.1:8001/genome/genome?_facet=chromosome&_sort=position</a></p>
  </article></div>


node_id: MDEwOlJlcG9zaXRvcnkyMTMyODY3NTI=
name: pocket-to-sqlite
full_name: dogsheep/pocket-to-sqlite
html_url: https://github.com/dogsheep/pocket-to-sqlite
description: Create a SQLite database containing data from your Pocket account
created_at: 2019-10-07T03:24:14Z
updated_at: 2025-05-10T16:23:47Z
pushed_at: 2023-07-10T19:24:26Z
language: Python
license: apache-2.0
topics: ["datasette", "datasette-io", "datasette-tool", "dogsheep", "pocket", "pocket-api", "sqlite"]
visibility: public
default_branch: main
permissions: {"admin": false, "maintain": false, "push": false, "triage": false, "pull": true}
custom_properties: {}
readme:
  # pocket-to-sqlite

  [![PyPI](https://img.shields.io/pypi/v/pocket-to-sqlite.svg)](https://pypi.org/project/pocket-to-sqlite/)
  [![Changelog](https://img.shields.io/github/v/release/dogsheep/pocket-to-sqlite?include_prereleases&label=changelog)](https://github.com/dogsheep/pocket-to-sqlite/releases)
  [![Tests](https://github.com/dogsheep/pocket-to-sqlite/workflows/Test/badge.svg)](https://github.com/dogsheep/pocket-to-sqlite/actions?query=workflow%3ATest)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/dogsheep/pocket-to-sqlite/blob/main/LICENSE)

  Create a SQLite database containing data from your [Pocket](https://getpocket.com/) account.

  ## How to install
  ```bash
  pip install pocket-to-sqlite
  ```
  ## Usage

  You will need to first obtain a valid OAuth token for your Pocket account. You can do this by running the `auth` command and following the prompts:
  ```bash
  pocket-to-sqlite auth
  ```
  Which looks like this:
  ```
  Visit this page and sign in with your Pocket account:

  https://getpocket.com/auth/author...

  Once you have signed in there, hit <enter> to continue
  Authentication tokens written to auth.json
  ```

  Now you can fetch all of your items from Pocket like this:

  ```bash
  pocket-to-sqlite fetch pocket.db
  ```

  The first time you run this command it will fetch all of your items, and display a progress bar while it does it.

  On subsequent runs it will only fetch new items.

  You can force it to fetch everything from the beginning again using `--all`. Use `--silent` to disable the progress bar.

  ## Using with Datasette

  The SQLite database produced by this tool is designed to be browsed using [Datasette](https://datasette.readthedocs.io/). Use the [datasette-render-timestamps](https://github.com/simonw/datasette-render-timestamps) plugin to improve the display of the timestamp values.

readme_html:
  <div id="readme" class="md" data-path="README.md"><article class="markdown-body entry-content container-lg" itemprop="text"><div class="markdown-heading" dir="auto"><h1 class="heading-element" dir="auto">pocket-to-sqlite</h1><a id="user-content-pocket-to-sqlite" class="anchor" aria-label="Permalink: pocket-to-sqlite" href="#pocket-to-sqlite"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto"><a href="https://pypi.org/project/pocket-to-sqlite/" rel="nofollow"><img src="https://img.shields.io/pypi/v/pocket-to-sqlite.svg" alt="PyPI" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/3150b11251557bf8e872973d909df2d9dac3ebdeb17824d17ae5f03d2dcd18ea/68747470733a2f2f696d672e736869656c64732e696f2f707970692f762f706f636b65742d746f2d73716c6974652e737667"></a>
  <a href="https://github.com/dogsheep/pocket-to-sqlite/releases"><img src="https://img.shields.io/github/v/release/dogsheep/pocket-to-sqlite?include_prereleases&label=changelog" alt="Changelog" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/1a7f2cb999bb1f6d9b0d6199055fa597d8806034210e40caa91e130f89d214fa/68747470733a2f2f696d672e736869656c64732e696f2f6769746875622f762f72656c656173652f646f6773686565702f706f636b65742d746f2d73716c6974653f696e636c7564655f70726572656c6561736573266c6162656c3d6368616e67656c6f67"></a>
  <a href="https://github.com/dogsheep/pocket-to-sqlite/actions?query=workflow%3ATest"><img src="https://github.com/dogsheep/pocket-to-sqlite/workflows/Test/badge.svg" alt="Tests" style="max-width: 100%;"></a>
  <a href="https://github.com/dogsheep/pocket-to-sqlite/blob/main/LICENSE"><img src="https://img.shields.io/badge/license-Apache%202.0-blue.svg" alt="License" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/c355f200ea90fddaa407b6eaab303663a669248ea3ca7b1fcf77dbe04ff5f48c/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6c6963656e73652d417061636865253230322e302d626c75652e737667"></a></p>
  <p dir="auto">Create a SQLite database containing data from your <a href="https://getpocket.com/" rel="nofollow">Pocket</a> account.</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">How to install</h2><a id="user-content-how-to-install" class="anchor" aria-label="Permalink: How to install" href="#how-to-install"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="pip install pocket-to-sqlite"><pre>pip install pocket-to-sqlite</pre></div>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Usage</h2><a id="user-content-usage" class="anchor" aria-label="Permalink: Usage" href="#usage"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">You will need to first obtain a valid OAuth token for your Pocket account. You can do this by running the <code>auth</code> command and following the prompts:</p>
  <div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="pocket-to-sqlite auth"><pre>pocket-to-sqlite auth</pre></div>
  <p dir="auto">Which looks like this:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="Visit this page and sign in with your Pocket account:

  https://getpocket.com/auth/author...

  Once you have signed in there, hit <enter> to continue
  Authentication tokens written to auth.json"><pre class="notranslate"><code>Visit this page and sign in with your Pocket account:

  https://getpocket.com/auth/author...

  Once you have signed in there, hit <enter> to continue
  Authentication tokens written to auth.json
  </code></pre></div>
  <p dir="auto">Now you can fetch all of your items from Pocket like this:</p>
  <div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="pocket-to-sqlite fetch pocket.db"><pre>pocket-to-sqlite fetch pocket.db</pre></div>
  <p dir="auto">The first time you run this command it will fetch all of your items, and display a progress bar while it does it.</p>
  <p dir="auto">On subsequent runs it will only fetch new items.</p>
  <p dir="auto">You can force it to fetch everything from the beginning again using <code>--all</code>. Use <code>--silent</code> to disable the progress bar.</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Using with Datasette</h2><a id="user-content-using-with-datasette" class="anchor" aria-label="Permalink: Using with Datasette" href="#using-with-datasette"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">The SQLite database produced by this tool is designed to be browsed using <a href="https://datasette.readthedocs.io/" rel="nofollow">Datasette</a>. Use the <a href="https://github.com/simonw/datasette-render-timestamps">datasette-render-timestamps</a> plugin to improve the display of the timestamp values.</p>
  </article></div>


node_id: MDEwOlJlcG9zaXRvcnkyMTQyOTkyNjc=
name: datasette-render-timestamps
full_name: simonw/datasette-render-timestamps
html_url: https://github.com/simonw/datasette-render-timestamps
description: Datasette plugin for rendering timestamps
created_at: 2019-10-10T22:50:50Z
updated_at: 2025-02-28T21:22:27Z
pushed_at: 2023-04-01T04:58:29Z
language: Python
license: apache-2.0
topics: ["datasette", "datasette-io", "datasette-plugin"]
visibility: public
default_branch: master
permissions: {"admin": false, "maintain": false, "push": false, "triage": false, "pull": true}
readme:
  # datasette-render-timestamps

  [![PyPI](https://img.shields.io/pypi/v/datasette-render-timestamps.svg)](https://pypi.org/project/datasette-render-timestamps/)
  [![CircleCI](https://circleci.com/gh/simonw/datasette-render-timestamps.svg?style=svg)](https://circleci.com/gh/simonw/datasette-render-timestamps)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/datasette-render-timestamps/blob/master/LICENSE)

  Datasette plugin for rendering timestamps.

  ## Installation

  Install this plugin in the same environment as Datasette to enable this new functionality:

      pip install datasette-render-timestamps

  The plugin will then look out for integer numbers that are likely to be timestamps - anything that would be a number of seconds from 5 years ago to 5 years in the future.

  These will then be rendered in a more readable format.

  ## Configuration

  You can disable automatic column detection in favour of explicitly listing the columns that you would like to render using [plugin configuration](https://datasette.readthedocs.io/en/stable/plugins.html#plugin-configuration) in a `metadata.json` file.

  Add a `"datasette-render-timestamps"` configuration block and use a `"columns"` key to list the columns you would like to treat as timestamp values:

  ```json
  {
      "plugins": {
          "datasette-render-timestamps": {
              "columns": ["created", "updated"]
          }
      }
  }
  ```
  This will cause any `created` or `updated` columns in any table to be treated as timestamps and rendered.

  Save this to `metadata.json` and run datasette with the `--metadata` flag to load this configuration:

      datasette serve mydata.db --metadata metadata.json

  To disable automatic timestamp detection entirely, you can use `"columnns": []`.

  This configuration block can be used at the top level, or it can be applied just to specific databases or tables. Here's how to apply it to just the `entries` table in the `news.db` database:

  ```json
  {
      "databases": {
          "news": {
              "tables": {
                  "entries": {
                      "plugins": {
                          "datasette-render-timestamps": {
                              "columns": ["created", "updated"]
                          }
                      }
                  }
              }
          }
      }
  }
  ```

  And here's how to apply it to every `created` column in every table in the `news.db` database:

  ```json
  {
      "databases": {
          "news": {
              "plugins": {
                  "datasette-render-timestamps": {
                      "columns": ["created", "updated"]
                  }
              }
          }
      }
  }
  ```

  ### Customizing the date format

  The default format is `%B %d, %Y - %H:%M:%S UTC` which renders for example: `October 10, 2019 - 07:18:29 UTC`. If you want another format, the date format can be customized using plugin configuration. Any format string supported by [strftime](http://strftime.org/) may be used. For example:

  ```json
  {
      "plugins": {
          "datasette-render-timestamps": {
              "format": "%Y-%m-%d-%H:%M:%S"
          }
      }
  }
  ```

readme_html:
  <div id="readme" class="md" data-path="README.md"><article class="markdown-body entry-content container-lg" itemprop="text"><div class="markdown-heading" dir="auto"><h1 class="heading-element" dir="auto">datasette-render-timestamps</h1><a id="user-content-datasette-render-timestamps" class="anchor" aria-label="Permalink: datasette-render-timestamps" href="#datasette-render-timestamps"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto"><a href="https://pypi.org/project/datasette-render-timestamps/" rel="nofollow"><img src="https://img.shields.io/pypi/v/datasette-render-timestamps.svg" alt="PyPI" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/5e50a40d7ddd59e5a4c23e02ab7d5fab929478d8431be583d769ed7d2dd5c2b3/68747470733a2f2f696d672e736869656c64732e696f2f707970692f762f6461746173657474652d72656e6465722d74696d657374616d70732e737667"></a>
  <a href="https://circleci.com/gh/simonw/datasette-render-timestamps" rel="nofollow"><img src="https://circleci.com/gh/simonw/datasette-render-timestamps.svg?style=svg" alt="CircleCI" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/d9f99171b4beef0b7d7211f63f7fb0dd2e4253538e0601898fb354c2a97a32db/68747470733a2f2f636972636c6563692e636f6d2f67682f73696d6f6e772f6461746173657474652d72656e6465722d74696d657374616d70732e7376673f7374796c653d737667"></a>
  <a href="https://github.com/simonw/datasette-render-timestamps/blob/master/LICENSE"><img src="https://img.shields.io/badge/license-Apache%202.0-blue.svg" alt="License" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/c355f200ea90fddaa407b6eaab303663a669248ea3ca7b1fcf77dbe04ff5f48c/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6c6963656e73652d417061636865253230322e302d626c75652e737667"></a></p>
  <p dir="auto">Datasette plugin for rendering timestamps.</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Installation</h2><a id="user-content-installation" class="anchor" aria-label="Permalink: Installation" href="#installation"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">Install this plugin in the same environment as Datasette to enable this new functionality:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="pip install datasette-render-timestamps"><pre class="notranslate"><code>pip install datasette-render-timestamps
  </code></pre></div>
  <p dir="auto">The plugin will then look out for integer numbers that are likely to be timestamps - anything that would be a number of seconds from 5 years ago to 5 years in the future.</p>
  <p dir="auto">These will then be rendered in a more readable format.</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Configuration</h2><a id="user-content-configuration" class="anchor" aria-label="Permalink: Configuration" href="#configuration"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">You can disable automatic column detection in favour of explicitly listing the columns that you would like to render using <a href="https://datasette.readthedocs.io/en/stable/plugins.html#plugin-configuration" rel="nofollow">plugin configuration</a> in a <code>metadata.json</code> file.</p>
  <p dir="auto">Add a <code>"datasette-render-timestamps"</code> configuration block and use a <code>"columns"</code> key to list the columns you would like to treat as timestamp values:</p>
  <div class="highlight highlight-source-json notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="{
      "plugins": {
          "datasette-render-timestamps": {
              "columns": ["created", "updated"]
          }
      }
  }"><pre>{
      <span class="pl-ent">"plugins"</span>: {
          <span class="pl-ent">"datasette-render-timestamps"</span>: {
              <span class="pl-ent">"columns"</span>: [<span class="pl-s"><span class="pl-pds">"</span>created<span class="pl-pds">"</span></span>, <span class="pl-s"><span class="pl-pds">"</span>updated<span class="pl-pds">"</span></span>]
          }
      }
  }</pre></div>
  <p dir="auto">This will cause any <code>created</code> or <code>updated</code> columns in any table to be treated as timestamps and rendered.</p>
  <p dir="auto">Save this to <code>metadata.json</code> and run datasette with the <code>--metadata</code> flag to load this configuration:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="datasette serve mydata.db --metadata metadata.json"><pre class="notranslate"><code>datasette serve mydata.db --metadata metadata.json
  </code></pre></div>
  <p dir="auto">To disable automatic timestamp detection entirely, you can use <code>"columnns": []</code>.</p>
  <p dir="auto">This configuration block can be used at the top level, or it can be applied just to specific databases or tables. Here's how to apply it to just the <code>entries</code> table in the <code>news.db</code> database:</p>
  <div class="highlight highlight-source-json notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="{
      "databases": {
          "news": {
              "tables": {
                  "entries": {
                      "plugins": {
                          "datasette-render-timestamps": {
                              "columns": ["created", "updated"]
                          }
                      }
                  }
              }
          }
      }
  }"><pre>{
      <span class="pl-ent">"databases"</span>: {
          <span class="pl-ent">"news"</span>: {
              <span class="pl-ent">"tables"</span>: {
                  <span class="pl-ent">"entries"</span>: {
                      <span class="pl-ent">"plugins"</span>: {
                          <span class="pl-ent">"datasette-render-timestamps"</span>: {
                              <span class="pl-ent">"columns"</span>: [<span class="pl-s"><span class="pl-pds">"</span>created<span class="pl-pds">"</span></span>, <span class="pl-s"><span class="pl-pds">"</span>updated<span class="pl-pds">"</span></span>]
                          }
                      }
                  }
              }
          }
      }
  }</pre></div>
  <p dir="auto">And here's how to apply it to every <code>created</code> column in every table in the <code>news.db</code> database:</p>
  <div class="highlight highlight-source-json notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="{
      "databases": {
          "news": {
              "plugins": {
                  "datasette-render-timestamps": {
                      "columns": ["created", "updated"]
                  }
              }
          }
      }
  }"><pre>{
      <span class="pl-ent">"databases"</span>: {
          <span class="pl-ent">"news"</span>: {
              <span class="pl-ent">"plugins"</span>: {
                  <span class="pl-ent">"datasette-render-timestamps"</span>: {
                      <span class="pl-ent">"columns"</span>: [<span class="pl-s"><span class="pl-pds">"</span>created<span class="pl-pds">"</span></span>, <span class="pl-s"><span class="pl-pds">"</span>updated<span class="pl-pds">"</span></span>]
                  }
              }
          }
      }
  }</pre></div>
  <div class="markdown-heading" dir="auto"><h3 class="heading-element" dir="auto">Customizing the date format</h3><a id="user-content-customizing-the-date-format" class="anchor" aria-label="Permalink: Customizing the date format" href="#customizing-the-date-format"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">The default format is <code>%B %d, %Y - %H:%M:%S UTC</code> which renders for example: <code>October 10, 2019 - 07:18:29 UTC</code>. If you want another format, the date format can be customized using plugin configuration. Any format string supported by <a href="http://strftime.org/" rel="nofollow">strftime</a> may be used. For example:</p>
  <div class="highlight highlight-source-json notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="{
      "plugins": {
          "datasette-render-timestamps": {
              "format": "%Y-%m-%d-%H:%M:%S"
          }
      }
  }"><pre>{
      <span class="pl-ent">"plugins"</span>: {
          <span class="pl-ent">"datasette-render-timestamps"</span>: {
              <span class="pl-ent">"format"</span>: <span class="pl-s"><span class="pl-pds">"</span>%Y-%m-%d-%H:%M:%S<span class="pl-pds">"</span></span>
          }
      }
  }</pre></div>
  </article></div>


node_id: MDEwOlJlcG9zaXRvcnkyMTcyMTY3ODc=
name: datasette-haversine
full_name: simonw/datasette-haversine
html_url: https://github.com/simonw/datasette-haversine
description: Datasette plugin that adds a custom SQL function for haversine distances
created_at: 2019-10-24T05:16:14Z
updated_at: 2023-12-29T17:54:13Z
pushed_at: 2023-12-29T18:04:33Z
language: Python
license: apache-2.0
topics: ["datasette", "datasette-io", "datasette-plugin"]
visibility: public
default_branch: main
permissions: {"admin": false, "maintain": false, "push": false, "triage": false, "pull": true}
readme:
  # datasette-haversine

  [![PyPI](https://img.shields.io/pypi/v/datasette-haversine.svg)](https://pypi.org/project/datasette-haversine/)
  [![Changelog](https://img.shields.io/github/v/release/simonw/datasette-haversine?include_prereleases&label=changelog)](https://github.com/simonw/datasette-haversine/releases)
  [![Tests](https://github.com/simonw/datasette-haversine/workflows/Test/badge.svg)](https://github.com/simonw/datasette-haversine/actions?query=workflow%3ATest)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/datasette-haversine/blob/main/LICENSE)

  Datasette plugin that adds a custom SQL function for haversine distances

  Install this plugin in the same environment as Datasette to enable the `haversine()` SQL function.
  ```bash
  datasette install datasette-haversine
  ```
  The plugin is built on top of the [haversine](https://github.com/mapado/haversine) library.

  ## haversine() to calculate distances

  ```sql
  select haversine(lat1, lon1, lat2, lon2);
  ```

  This will return the distance in kilometers between the point defined by `(lat1, lon1)` and the point defined by `(lat2, lon2)`.

  ## Demo

  You can try this function [here using Datasette Lite](https://lite.datasette.io/?install=datasette-haversine#/content?sql=select+haversine%28%0A++40.7128%2C+-74.0060%2C+--+New+York%0A++37.7749%2C+-122.4194%2C+--+San+Francisco%0A++%27mi%27%0A%29+as+distance_in_miles%3B).

  ```sql
  select haversine(
    40.7128, -74.0060, -- New York
    37.7749, -122.4194, -- San Francisco
    'mi'
  ) as distance_in_miles;
  ```

  ## Custom units

  By default `haversine()` returns results in km. You can pass an optional third argument to get results in a different unit:

  - `ft` for feet
  - `m` for meters
  - `in` for inches
  - `mi` for miles
  - `nmi` for nautical miles
  - `km` for kilometers (the default)

  ```sql
  select haversine(lat1, lon1, lat2, lon2, 'mi');
  ```

readme_html:
  <div id="readme" class="md" data-path="README.md"><article class="markdown-body entry-content container-lg" itemprop="text"><div class="markdown-heading" dir="auto"><h1 class="heading-element" dir="auto">datasette-haversine</h1><a id="user-content-datasette-haversine" class="anchor" aria-label="Permalink: datasette-haversine" href="#datasette-haversine"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto"><a href="https://pypi.org/project/datasette-haversine/" rel="nofollow"><img src="https://img.shields.io/pypi/v/datasette-haversine.svg" alt="PyPI" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/51104573b3c8bed6cd8d818b4e502878968620234f7d5c968b5b7cbf5dac20d6/68747470733a2f2f696d672e736869656c64732e696f2f707970692f762f6461746173657474652d686176657273696e652e737667"></a>
  <a href="https://github.com/simonw/datasette-haversine/releases"><img src="https://img.shields.io/github/v/release/simonw/datasette-haversine?include_prereleases&label=changelog" alt="Changelog" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/1a7be6199487537eae770dd07f7919bab1202dec83bfc65b05d4df1e3dccac19/68747470733a2f2f696d672e736869656c64732e696f2f6769746875622f762f72656c656173652f73696d6f6e772f6461746173657474652d686176657273696e653f696e636c7564655f70726572656c6561736573266c6162656c3d6368616e67656c6f67"></a>
  <a href="https://github.com/simonw/datasette-haversine/actions?query=workflow%3ATest"><img src="https://github.com/simonw/datasette-haversine/workflows/Test/badge.svg" alt="Tests" style="max-width: 100%;"></a>
  <a href="https://github.com/simonw/datasette-haversine/blob/main/LICENSE"><img src="https://img.shields.io/badge/license-Apache%202.0-blue.svg" alt="License" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/c355f200ea90fddaa407b6eaab303663a669248ea3ca7b1fcf77dbe04ff5f48c/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6c6963656e73652d417061636865253230322e302d626c75652e737667"></a></p>
  <p dir="auto">Datasette plugin that adds a custom SQL function for haversine distances</p>
  <p dir="auto">Install this plugin in the same environment as Datasette to enable the <code>haversine()</code> SQL function.</p>
  <div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="datasette install datasette-haversine"><pre>datasette install datasette-haversine</pre></div>
  <p dir="auto">The plugin is built on top of the <a href="https://github.com/mapado/haversine">haversine</a> library.</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">haversine() to calculate distances</h2><a id="user-content-haversine-to-calculate-distances" class="anchor" aria-label="Permalink: haversine() to calculate distances" href="#haversine-to-calculate-distances"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <div class="highlight highlight-source-sql notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="select haversine(lat1, lon1, lat2, lon2);"><pre><span class="pl-k">select</span> haversine(lat1, lon1, lat2, lon2);</pre></div>
  <p dir="auto">This will return the distance in kilometers between the point defined by <code>(lat1, lon1)</code> and the point defined by <code>(lat2, lon2)</code>.</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Demo</h2><a id="user-content-demo" class="anchor" aria-label="Permalink: Demo" href="#demo"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">You can try this function <a href="https://lite.datasette.io/?install=datasette-haversine#/content?sql=select+haversine%28%0A++40.7128%2C+-74.0060%2C+--+New+York%0A++37.7749%2C+-122.4194%2C+--+San+Francisco%0A++%27mi%27%0A%29+as+distance_in_miles%3B" rel="nofollow">here using Datasette Lite</a>.</p>
  <div class="highlight highlight-source-sql notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="select haversine(
    40.7128, -74.0060, -- New York
    37.7749, -122.4194, -- San Francisco
    'mi'
  ) as distance_in_miles;"><pre><span class="pl-k">select</span> haversine(
    <span class="pl-c1">40</span>.<span class="pl-c1">7128</span>, <span class="pl-k">-</span><span class="pl-c1">74</span>.<span class="pl-c1">0060</span>, <span class="pl-c"><span class="pl-c">--</span> New York</span>
    <span class="pl-c1">37</span>.<span class="pl-c1">7749</span>, <span class="pl-k">-</span><span class="pl-c1">122</span>.<span class="pl-c1">4194</span>, <span class="pl-c"><span class="pl-c">--</span> San Francisco</span>
    <span class="pl-s"><span class="pl-pds">'</span>mi<span class="pl-pds">'</span></span>
  ) <span class="pl-k">as</span> distance_in_miles;</pre></div>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Custom units</h2><a id="user-content-custom-units" class="anchor" aria-label="Permalink: Custom units" href="#custom-units"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">By default <code>haversine()</code> returns results in km. You can pass an optional third argument to get results in a different unit:</p>
  <ul dir="auto">
  <li><code>ft</code> for feet</li>
  <li><code>m</code> for meters</li>
  <li><code>in</code> for inches</li>
  <li><code>mi</code> for miles</li>
  <li><code>nmi</code> for nautical miles</li>
  <li><code>km</code> for kilometers (the default)</li>
  </ul>
  <div class="highlight highlight-source-sql notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="select haversine(lat1, lon1, lat2, lon2, 'mi');"><pre><span class="pl-k">select</span> haversine(lat1, lon1, lat2, lon2, <span class="pl-s"><span class="pl-pds">'</span>mi<span class="pl-pds">'</span></span>);</pre></div>
  </article></div>


node_id: MDEwOlJlcG9zaXRvcnkyMTkzNzIxMzM=
name: sqlite-transform
full_name: simonw/sqlite-transform
html_url: https://github.com/simonw/sqlite-transform
description: Tool for running transformations on columns in a SQLite database
created_at: 2019-11-03T22:07:53Z
updated_at: 2025-02-28T21:20:13Z
pushed_at: 2021-08-02T22:07:57Z
language: Python
license: apache-2.0
topics: ["datasette-io", "datasette-tool", "sqlite"]
visibility: public
default_branch: main
permissions: {"admin": false, "maintain": false, "push": false, "triage": false, "pull": true}
readme:
  # sqlite-transform

  ![No longer maintained](https://img.shields.io/badge/no%20longer-maintained-red)
  [![PyPI](https://img.shields.io/pypi/v/sqlite-transform.svg)](https://pypi.org/project/sqlite-transform/)
  [![Changelog](https://img.shields.io/github/v/release/simonw/sqlite-transform?include_prereleases&label=changelog)](https://github.com/simonw/sqlite-transform/releases)
  [![Tests](https://github.com/simonw/sqlite-transform/workflows/Test/badge.svg)](https://github.com/simonw/sqlite-transform/actions?query=workflow%3ATest)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/dogsheep/sqlite-transform/blob/main/LICENSE)

  Tool for running transformations on columns in a SQLite database.

  > **:warning: This tool is no longer maintained**
  >
  > I added a new tool to [sqlite-utils](https://sqlite-utils.datasette.io/) called [sqlite-utils convert](https://sqlite-utils.datasette.io/en/stable/cli.html#converting-data-in-columns) which provides a super-set of the functionality originally provided here. `sqlite-transform` is no longer maintained, and I recommend switching to using `sqlite-utils convert` instead.

  ## How to install

      pip install sqlite-transform

  ## parsedate and parsedatetime

  These subcommands will run all values in the specified column through `dateutils.parser.parse()` and replace them with the result, formatted as an ISO timestamp or ISO date.

  For example, if a row in the database has an `opened` column which contains `10/10/2019 08:10:00 PM`, running the following command:

      sqlite-transform parsedatetime my.db mytable opened

  Will result in that value being replaced by `2019-10-10T20:10:00`.

  Using the `parsedate` subcommand here would result in `2019-10-10` instead.

  In the case of ambiguous dates such as `03/04/05` these commands both default to assuming American-style `mm/dd/yy` format. You can pass `--dayfirst` to specify that the day should be assumed to be first, or `--yearfirst` for the year.

  ## jsonsplit

  The `jsonsplit` subcommand takes columns that contain a comma-separated list, for example a `tags` column containing records like `"trees,park,dogs"` and converts it into a JSON array `["trees", "park", "dogs"]`.

  This is useful for taking advantage of Datasette's [Facet by JSON array](https://docs.datasette.io/en/stable/facets.html#facet-by-json-array) feature.

      sqlite-transform jsonsplit my.db mytable tags

  It defaults to splitting on commas, but you can specify a different delimiter character using the `--delimiter` option, for example:

      sqlite-transform jsonsplit \
          my.db mytable tags --delimiter ';'

  Values within the array will be treated as strings, so a column containing `123,552,775` will be converted into the JSON array `["123", "552", "775"]`.

  You can specify a different type for these values using `--type int` or `--type float`, for example:

      sqlite-transform jsonsplit \
          my.db mytable tags --type int

  This will result in that column being converted into `[123, 552, 775]`.

  ## lambda for executing your own code

  The `lambda` subcommand lets you specify Python code which will be executed against the column.

  Here's how to convert a column to uppercase:

      sqlite-transform lambda my.db mytable mycolumn --code='str(value).upper()'

  The code you provide will be compiled into a function that takes `value` as a single argument. You can break your function body into multiple lines, provided the last line is a `return` statement:

      sqlite-transform lambda my.db mytable mycolumn --code='value = str(value)
      return value.upper()'

  You can also specify Python modules that should be imported and made available to your code using one or more `--import` options:

      sqlite-transform lambda my.db mytable mycolumn \
          --code='"\n".join(textwrap.wrap(value, 10))' \
          --import=textwrap

  The `--dry-run` option will output a preview of the transformation against the first ten rows, without modifying the database.

  ## Saving the result to a separate column

  Each of these commands accepts optional `--output` and `--output-type` options. These can be used to save the result of the transformation to a separate column, which will be created if the column does not already exist.

  To save the result of `jsonsplit` to a new column called `json_tags`, use the following:

      sqlite-transform jsonsplit my.db mytable tags \
        --output json_tags

  The type of the created column defaults to `text`, but a different column type can be specified using `--output-type`. This example will create a new floating point column called `float_id` with a copy of each item's ID increased by 0.5:

      sqlite-transform lambda my.db mytable id \
        --code 'float(value) + 0.5' \
        --output float_id \
        --output-type float

  You can drop the original column at the end of the operation by adding `--drop`.

  ## Splitting a column into multiple columns

  Sometimes you may wish to convert a single column into multiple derived columns. For example, you may have a `location` column containing `latitude,longitude` values which you wish to split out into separate `latitude` and `longitude` columns.

  You can achieve this using the `--multi` option to `sqlite-transform lambda`. This option expects your `--code` function to return a Python dictionary: new columns well be created and populated for each of the keys in that dictionary.

  For the `latitude,longitude` example you would use the following:

      sqlite-transform lambda demo.db places location \
        --code 'return {
          "latitude": float(value.split(",")[0]),
          "longitude": float(value.split(",")[1]),
        }' --multi

  The type of the returned values will be taken into account when creating the new columns. In this example, the resulting database schema will look like this:

  ```sql
  CREATE TABLE [places] (
      [location] TEXT,
      [latitude] FLOAT,
      [longitude] FLOAT
  );
  ```
  The code function can also return `None`, in which case its output will be ignored.

  You can drop the original column at the end of the operation by adding `--drop`.

  ## Disabling the progress bar

  By default each command will show a progress bar. Pass `-s` or `--silent` to hide that progress bar.

readme_html:
  <div id="readme" class="md" data-path="README.md"><article class="markdown-body entry-content container-lg" itemprop="text"><div class="markdown-heading" dir="auto"><h1 class="heading-element" dir="auto">sqlite-transform</h1><a id="user-content-sqlite-transform" class="anchor" aria-label="Permalink: sqlite-transform" href="#sqlite-transform"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto"><a target="_blank" rel="noopener noreferrer nofollow" href="https://camo.githubusercontent.com/556c84fc2b13b7b5fe8a48b19c61cd679a17959572fba1f6d23851cc78a67144/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6e6f2532306c6f6e6765722d6d61696e7461696e65642d726564"><img src="https://img.shields.io/badge/no%20longer-maintained-red" alt="No longer maintained" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/556c84fc2b13b7b5fe8a48b19c61cd679a17959572fba1f6d23851cc78a67144/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6e6f2532306c6f6e6765722d6d61696e7461696e65642d726564"></a>
  <a href="https://pypi.org/project/sqlite-transform/" rel="nofollow"><img src="https://img.shields.io/pypi/v/sqlite-transform.svg" alt="PyPI" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/5afdddb02bc2d730d804918ccd0a102e24daa2326a4cf9f11f87930b3a847cd8/68747470733a2f2f696d672e736869656c64732e696f2f707970692f762f73716c6974652d7472616e73666f726d2e737667"></a>
  <a href="https://github.com/simonw/sqlite-transform/releases"><img src="https://img.shields.io/github/v/release/simonw/sqlite-transform?include_prereleases&label=changelog" alt="Changelog" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/28d7293fed82e1cbfcbaa2062b6214cfe1db1a329e467373d6fb3a630ab2f9b5/68747470733a2f2f696d672e736869656c64732e696f2f6769746875622f762f72656c656173652f73696d6f6e772f73716c6974652d7472616e73666f726d3f696e636c7564655f70726572656c6561736573266c6162656c3d6368616e67656c6f67"></a>
  <a href="https://github.com/simonw/sqlite-transform/actions?query=workflow%3ATest"><img src="https://github.com/simonw/sqlite-transform/workflows/Test/badge.svg" alt="Tests" style="max-width: 100%;"></a>
  <a href="https://github.com/dogsheep/sqlite-transform/blob/main/LICENSE"><img src="https://img.shields.io/badge/license-Apache%202.0-blue.svg" alt="License" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/c355f200ea90fddaa407b6eaab303663a669248ea3ca7b1fcf77dbe04ff5f48c/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6c6963656e73652d417061636865253230322e302d626c75652e737667"></a></p>
  <p dir="auto">Tool for running transformations on columns in a SQLite database.</p>
  <blockquote>
  <p dir="auto"><strong><g-emoji class="g-emoji" alias="warning">⚠️</g-emoji> This tool is no longer maintained</strong></p>
  <p dir="auto">I added a new tool to <a href="https://sqlite-utils.datasette.io/" rel="nofollow">sqlite-utils</a> called <a href="https://sqlite-utils.datasette.io/en/stable/cli.html#converting-data-in-columns" rel="nofollow">sqlite-utils convert</a> which provides a super-set of the functionality originally provided here. <code>sqlite-transform</code> is no longer maintained, and I recommend switching to using <code>sqlite-utils convert</code> instead.</p>
  </blockquote>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">How to install</h2><a id="user-content-how-to-install" class="anchor" aria-label="Permalink: How to install" href="#how-to-install"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="pip install sqlite-transform"><pre class="notranslate"><code>pip install sqlite-transform
  </code></pre></div>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">parsedate and parsedatetime</h2><a id="user-content-parsedate-and-parsedatetime" class="anchor" aria-label="Permalink: parsedate and parsedatetime" href="#parsedate-and-parsedatetime"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">These subcommands will run all values in the specified column through <code>dateutils.parser.parse()</code> and replace them with the result, formatted as an ISO timestamp or ISO date.</p>
  <p dir="auto">For example, if a row in the database has an <code>opened</code> column which contains <code>10/10/2019 08:10:00 PM</code>, running the following command:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="sqlite-transform parsedatetime my.db mytable opened"><pre class="notranslate"><code>sqlite-transform parsedatetime my.db mytable opened
  </code></pre></div>
  <p dir="auto">Will result in that value being replaced by <code>2019-10-10T20:10:00</code>.</p>
  <p dir="auto">Using the <code>parsedate</code> subcommand here would result in <code>2019-10-10</code> instead.</p>
  <p dir="auto">In the case of ambiguous dates such as <code>03/04/05</code> these commands both default to assuming American-style <code>mm/dd/yy</code> format. You can pass <code>--dayfirst</code> to specify that the day should be assumed to be first, or <code>--yearfirst</code> for the year.</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">jsonsplit</h2><a id="user-content-jsonsplit" class="anchor" aria-label="Permalink: jsonsplit" href="#jsonsplit"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">The <code>jsonsplit</code> subcommand takes columns that contain a comma-separated list, for example a <code>tags</code> column containing records like <code>"trees,park,dogs"</code> and converts it into a JSON array <code>["trees", "park", "dogs"]</code>.</p>
  <p dir="auto">This is useful for taking advantage of Datasette's <a href="https://docs.datasette.io/en/stable/facets.html#facet-by-json-array" rel="nofollow">Facet by JSON array</a> feature.</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="sqlite-transform jsonsplit my.db mytable tags"><pre class="notranslate"><code>sqlite-transform jsonsplit my.db mytable tags
  </code></pre></div>
  <p dir="auto">It defaults to splitting on commas, but you can specify a different delimiter character using the <code>--delimiter</code> option, for example:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="sqlite-transform jsonsplit \
      my.db mytable tags --delimiter ';'"><pre class="notranslate"><code>sqlite-transform jsonsplit \
      my.db mytable tags --delimiter ';'
  </code></pre></div>
  <p dir="auto">Values within the array will be treated as strings, so a column containing <code>123,552,775</code> will be converted into the JSON array <code>["123", "552", "775"]</code>.</p>
  <p dir="auto">You can specify a different type for these values using <code>--type int</code> or <code>--type float</code>, for example:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="sqlite-transform jsonsplit \
      my.db mytable tags --type int"><pre class="notranslate"><code>sqlite-transform jsonsplit \
      my.db mytable tags --type int
  </code></pre></div>
  <p dir="auto">This will result in that column being converted into <code>[123, 552, 775]</code>.</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">lambda for executing your own code</h2><a id="user-content-lambda-for-executing-your-own-code" class="anchor" aria-label="Permalink: lambda for executing your own code" href="#lambda-for-executing-your-own-code"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">The <code>lambda</code> subcommand lets you specify Python code which will be executed against the column.</p>
  <p dir="auto">Here's how to convert a column to uppercase:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="sqlite-transform lambda my.db mytable mycolumn --code='str(value).upper()'"><pre class="notranslate"><code>sqlite-transform lambda my.db mytable mycolumn --code='str(value).upper()'
  </code></pre></div>
  <p dir="auto">The code you provide will be compiled into a function that takes <code>value</code> as a single argument. You can break your function body into multiple lines, provided the last line is a <code>return</code> statement:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="sqlite-transform lambda my.db mytable mycolumn --code='value = str(value)
  return value.upper()'"><pre class="notranslate"><code>sqlite-transform lambda my.db mytable mycolumn --code='value = str(value)
  return value.upper()'
  </code></pre></div>
  <p dir="auto">You can also specify Python modules that should be imported and made available to your code using one or more <code>--import</code> options:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="sqlite-transform lambda my.db mytable mycolumn \
      --code='"\n".join(textwrap.wrap(value, 10))' \
      --import=textwrap"><pre class="notranslate"><code>sqlite-transform lambda my.db mytable mycolumn \
      --code='"\n".join(textwrap.wrap(value, 10))' \
      --import=textwrap
  </code></pre></div>
  <p dir="auto">The <code>--dry-run</code> option will output a preview of the transformation against the first ten rows, without modifying the database.</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Saving the result to a separate column</h2><a id="user-content-saving-the-result-to-a-separate-column" class="anchor" aria-label="Permalink: Saving the result to a separate column" href="#saving-the-result-to-a-separate-column"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">Each of these commands accepts optional <code>--output</code> and <code>--output-type</code> options. These can be used to save the result of the transformation to a separate column, which will be created if the column does not already exist.</p>
  <p dir="auto">To save the result of <code>jsonsplit</code> to a new column called <code>json_tags</code>, use the following:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="sqlite-transform jsonsplit my.db mytable tags \
    --output json_tags"><pre class="notranslate"><code>sqlite-transform jsonsplit my.db mytable tags \
    --output json_tags
  </code></pre></div>
  <p dir="auto">The type of the created column defaults to <code>text</code>, but a different column type can be specified using <code>--output-type</code>. This example will create a new floating point column called <code>float_id</code> with a copy of each item's ID increased by 0.5:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="sqlite-transform lambda my.db mytable id \
    --code 'float(value) + 0.5' \
    --output float_id \
    --output-type float"><pre class="notranslate"><code>sqlite-transform lambda my.db mytable id \
    --code 'float(value) + 0.5' \
    --output float_id \
    --output-type float
  </code></pre></div>
  <p dir="auto">You can drop the original column at the end of the operation by adding <code>--drop</code>.</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Splitting a column into multiple columns</h2><a id="user-content-splitting-a-column-into-multiple-columns" class="anchor" aria-label="Permalink: Splitting a column into multiple columns" href="#splitting-a-column-into-multiple-columns"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">Sometimes you may wish to convert a single column into multiple derived columns. For example, you may have a <code>location</code> column containing <code>latitude,longitude</code> values which you wish to split out into separate <code>latitude</code> and <code>longitude</code> columns.</p>
  <p dir="auto">You can achieve this using the <code>--multi</code> option to <code>sqlite-transform lambda</code>. This option expects your <code>--code</code> function to return a Python dictionary: new columns well be created and populated for each of the keys in that dictionary.</p>
  <p dir="auto">For the <code>latitude,longitude</code> example you would use the following:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="sqlite-transform lambda demo.db places location \
    --code 'return {
      "latitude": float(value.split(",")[0]),
      "longitude": float(value.split(",")[1]),
    }' --multi"><pre class="notranslate"><code>sqlite-transform lambda demo.db places location \
    --code 'return {
      "latitude": float(value.split(",")[0]),
      "longitude": float(value.split(",")[1]),
    }' --multi
  </code></pre></div>
  <p dir="auto">The type of the returned values will be taken into account when creating the new columns. In this example, the resulting database schema will look like this:</p>
  <div class="highlight highlight-source-sql notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="CREATE TABLE [places] (
      [location] TEXT,
      [latitude] FLOAT,
      [longitude] FLOAT
  );"><pre>CREATE TABLE [places] (
      [location] <span class="pl-k">TEXT</span>,
      [latitude] FLOAT,
      [longitude] FLOAT
  );</pre></div>
  <p dir="auto">The code function can also return <code>None</code>, in which case its output will be ignored.</p>
  <p dir="auto">You can drop the original column at the end of the operation by adding <code>--drop</code>.</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Disabling the progress bar</h2><a id="user-content-disabling-the-progress-bar" class="anchor" aria-label="Permalink: Disabling the progress bar" href="#disabling-the-progress-bar"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">By default each command will show a progress bar. Pass <code>-s</code> or <code>--silent</code> to hide that progress bar.</p>
  </article></div>


node_id: MDEwOlJlcG9zaXRvcnkyMjA3MTY4MjI=
name: datasette-render-markdown
full_name: simonw/datasette-render-markdown
html_url: https://github.com/simonw/datasette-render-markdown
description: Datasette plugin for rendering Markdown
created_at: 2019-11-09T23:28:31Z
updated_at: 2025-03-18T21:40:45Z
pushed_at: 2023-08-15T03:10:29Z
language: Python
license: apache-2.0
topics: ["datasette", "datasette-io", "datasette-plugin", "markdown"]
visibility: public
default_branch: main
permissions: {"admin": false, "maintain": false, "push": false, "triage": false, "pull": true}
readme:
  # datasette-render-markdown

  [![PyPI](https://img.shields.io/pypi/v/datasette-render-markdown.svg)](https://pypi.org/project/datasette-render-markdown/)
  [![Changelog](https://img.shields.io/github/v/release/simonw/datasette-render-markdown?include_prereleases&label=changelog)](https://github.com/simonw/datasette-render-markdown/releases)
  [![Tests](https://github.com/simonw/datasette-render-markdown/workflows/Test/badge.svg)](https://github.com/simonw/datasette-render-markdown/actions?query=workflow%3ATest)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/datasette-render-markdown/blob/main/LICENSE)

  [Datasette](https://datasette.io/) plugin for rendering Markdown.

  ## Installation

  Install this plugin in the same environment as Datasette to enable this new functionality:
  ```bash
  datasette install datasette-render-markdown
  ```
  ## Usage

  You can explicitly list the columns you would like to treat as Markdown using [plugin configuration](https://datasette.readthedocs.io/en/stable/plugins.html#plugin-configuration) in a `metadata.json` file.

  Add a `"datasette-render-markdown"` configuration block and use a `"columns"` key to list the columns you would like to treat as Markdown values:

  ```json
  {
      "plugins": {
          "datasette-render-markdown": {
              "columns": ["body"]
          }
      }
  }
  ```

  This will cause any `body` column in any table to be treated as markdown and safely rendered using [Python-Markdown](https://python-markdown.github.io/). The resulting HTML is then run through [Bleach](https://bleach.readthedocs.io/) to avoid the risk of XSS security problems.

  Save this to `metadata.json` and run Datasette with the `--metadata` flag to load this configuration:

      $ datasette serve mydata.db --metadata metadata.json

  The configuration block can be used at the top level, or it can be applied just to specific databases or tables. Here's how to apply it to just the `entries` table in the `news.db` database:

  ```json
  {
      "databases": {
          "news": {
              "tables": {
                  "entries": {
                      "plugins": {
                          "datasette-render-markdown": {
                              "columns": ["body"]
                          }
                      }
                  }
              }
          }
      }
  }
  ```

  And here's how to apply it to every `body` column in every table in the `news.db` database:

  ```json
  {
      "databases": {
          "news": {
              "plugins": {
                  "datasette-render-markdown": {
                      "columns": ["body"]
                  }
              }
          }
      }
  }
  ```

  ## Columns that match a naming convention

  This plugin can also render markdown in any columns that match a specific naming convention.

  By default, columns that have a name ending in `_markdown` will be rendered.

  You can try this out using the following query:

  ```sql
  select '# Hello there

  * This is a list
  * of items

  [And a link](https://github.com/simonw/datasette-render-markdown).'
  as demo_markdown
  ```

  You can configure a different list of wildcard patterns using the `"patterns"` configuration key. Here's how to render columns that end in either `_markdown` or `_md`:

  ```json
  {
      "plugins": {
          "datasette-render-markdown": {
              "patterns": ["*_markdown", "*_md"]
          }
      }
  }
  ```

  To disable wildcard column matching entirely, set `"patterns": []` in your plugin metadata configuration.

  ## Markdown extensions

  The [Python-Markdown library](https://python-markdown.github.io/) that powers this plugin supports extensions, both [bundled](https://python-markdown.github.io/extensions/) and [third-party](https://github.com/Python-Markdown/markdown/wiki/Third-Party-Extensions). These can be used to enable additional Markdown features such as [table support](https://python-markdown.github.io/extensions/tables/).

  You can configure support for extensions using the `"extensions"` key in your plugin metadata configuration.

  Since extensions may introduce new HTML tags, you will also need to add those tags to the list of tags that are allowed by the [Bleach](https://bleach.readthedocs.io/) sanitizer. You can do that using the `"extra_tags"` key, and you can allow-list additional HTML attributes using `"extra_attrs"`. See [the Bleach documentation](https://bleach.readthedocs.io/en/latest/clean.html#allowed-tags-tags) for more information on this.

  Here's how to enable support for [Markdown tables](https://python-markdown.github.io/extensions/tables/):

  ```json
  {
      "plugins": {
          "datasette-render-markdown": {
              "extensions": ["tables"],
              "extra_tags": ["table", "thead", "tr", "th", "td", "tbody"]
          }
      }
  }
  ```

  ### GitHub-Flavored Markdown

  Enabling [GitHub-Flavored Markdown](https://help.github.com/en/github/writing-on-github) (useful for if you are working with data imported from GitHub using [github-to-sqlite](https://github.com/dogsheep/github-to-sqlite)) is a little more complicated.

  First, you will need to install the [py-gfm](https://py-gfm.readthedocs.io) package:

      $ pip install py-gfm

  Note that `py-gfm` has [a bug](https://github.com/Zopieux/py-gfm/issues/13) that causes it to pin to `Markdown<3.0` - so if you are using it you should install it _before_ installing `datasette-render-markdown` to ensure you get a compatibly version of that dependency.

  Now you can configure it like this. Note that the extension name is `mdx_gfm:GithubFlavoredMarkdownExtension` and you need to allow-list several extra HTML tags and attributes:

  ```json
  {
      "plugins": {
          "datasette-render-markdown": {
              "extra_tags": [
                  "hr",
                  "br",
                  "details",
                  "summary",
                  "input"
              ],
              "extra_attrs": {
                  "input": [
                      "type",
                      "disabled",
                      "checked"
                  ],
              },
              "extensions": [
                  "mdx_gfm:GithubFlavoredMarkdownExtension"
              ]
          }
      }
  }
  ```

  The `<input type="" checked disabled>` attributes are needed to support rendering checkboxes in issue descriptions.

  ## Markdown in templates

  The plugin introduces a new template tag: `{% markdown %}...{% endmarkdown %}` - which can be used to render Markdown in your Jinja templates.

  ```html+jinja
  {% markdown %}
  # This will be rendered as markdown
  {% endmarkdown %}
  ```
  You can use attributes on the `{% markdown %}` tag to enable extensions and allow-list additional tags and attributes:
  ```html+jinja
  {% markdown
    extensions="tables"
    extra_tags="table thead tr th td tbody" 
    extra_attrs="p:id,class a:name,href" %}
  ## Markdown table

  First Header  | Second Header
  ------------- | -------------
  Content Cell  | Content Cell
  Content Cell  | Content Cell

  <a href="https://www.example.com/" name="namehere">Example</a>
  <p id="paragraph" class="klass">Paragraph</p>
  {% endmarkdown %}
  ```
  The `extensions=` and `extra_tags=` attributes accept a space-separated list of values.

  The `extra_attrs=` attribute accepts a space-separated list of `tag:attr1,attr2` values - each tag can specify one or more attributes that should be allowed.

  You can also use the `{{ render_markdown(...) }}` function, like this:

  ```html+jinja
  {{ render_markdown("""
  ## Markdown table

  First Header  | Second Header
  ------------- | -------------
  Content Cell  | Content Cell
  Content Cell  | Content Cell
  """, extensions=["tables"],
      extra_tags=["table", "thead", "tr", "th", "td", "tbody"])) }}
  ```

  The `{% markdown %}` tag is recommended, as it avoids the need to `\"` escape quotes in your Markdown content.

readme_html:
  <div id="readme" class="md" data-path="README.md"><article class="markdown-body entry-content container-lg" itemprop="text"><div class="markdown-heading" dir="auto"><h1 class="heading-element" dir="auto">datasette-render-markdown</h1><a id="user-content-datasette-render-markdown" class="anchor" aria-label="Permalink: datasette-render-markdown" href="#datasette-render-markdown"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto"><a href="https://pypi.org/project/datasette-render-markdown/" rel="nofollow"><img src="https://img.shields.io/pypi/v/datasette-render-markdown.svg" alt="PyPI" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/5f6cc294ca00068e1623317da168640302a0e9a69d637a930f6c873a346c5359/68747470733a2f2f696d672e736869656c64732e696f2f707970692f762f6461746173657474652d72656e6465722d6d61726b646f776e2e737667"></a>
  <a href="https://github.com/simonw/datasette-render-markdown/releases"><img src="https://img.shields.io/github/v/release/simonw/datasette-render-markdown?include_prereleases&label=changelog" alt="Changelog" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/2bec036af2e86342b47e73d26b4e1655de2fd092ba27e55e05003195893f93d6/68747470733a2f2f696d672e736869656c64732e696f2f6769746875622f762f72656c656173652f73696d6f6e772f6461746173657474652d72656e6465722d6d61726b646f776e3f696e636c7564655f70726572656c6561736573266c6162656c3d6368616e67656c6f67"></a>
  <a href="https://github.com/simonw/datasette-render-markdown/actions?query=workflow%3ATest"><img src="https://github.com/simonw/datasette-render-markdown/workflows/Test/badge.svg" alt="Tests" style="max-width: 100%;"></a>
  <a href="https://github.com/simonw/datasette-render-markdown/blob/main/LICENSE"><img src="https://img.shields.io/badge/license-Apache%202.0-blue.svg" alt="License" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/c355f200ea90fddaa407b6eaab303663a669248ea3ca7b1fcf77dbe04ff5f48c/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6c6963656e73652d417061636865253230322e302d626c75652e737667"></a></p>
  <p dir="auto"><a href="https://datasette.io/" rel="nofollow">Datasette</a> plugin for rendering Markdown.</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Installation</h2><a id="user-content-installation" class="anchor" aria-label="Permalink: Installation" href="#installation"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">Install this plugin in the same environment as Datasette to enable this new functionality:</p>
  <div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="datasette install datasette-render-markdown"><pre>datasette install datasette-render-markdown</pre></div>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Usage</h2><a id="user-content-usage" class="anchor" aria-label="Permalink: Usage" href="#usage"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">You can explicitly list the columns you would like to treat as Markdown using <a href="https://datasette.readthedocs.io/en/stable/plugins.html#plugin-configuration" rel="nofollow">plugin configuration</a> in a <code>metadata.json</code> file.</p>
  <p dir="auto">Add a <code>"datasette-render-markdown"</code> configuration block and use a <code>"columns"</code> key to list the columns you would like to treat as Markdown values:</p>
  <div class="highlight highlight-source-json notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="{
      "plugins": {
          "datasette-render-markdown": {
              "columns": ["body"]
          }
      }
  }"><pre>{
      <span class="pl-ent">"plugins"</span>: {
          <span class="pl-ent">"datasette-render-markdown"</span>: {
              <span class="pl-ent">"columns"</span>: [<span class="pl-s"><span class="pl-pds">"</span>body<span class="pl-pds">"</span></span>]
          }
      }
  }</pre></div>
  <p dir="auto">This will cause any <code>body</code> column in any table to be treated as markdown and safely rendered using <a href="https://python-markdown.github.io/" rel="nofollow">Python-Markdown</a>. The resulting HTML is then run through <a href="https://bleach.readthedocs.io/" rel="nofollow">Bleach</a> to avoid the risk of XSS security problems.</p>
  <p dir="auto">Save this to <code>metadata.json</code> and run Datasette with the <code>--metadata</code> flag to load this configuration:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ datasette serve mydata.db --metadata metadata.json"><pre class="notranslate"><code>$ datasette serve mydata.db --metadata metadata.json
  </code></pre></div>
  <p dir="auto">The configuration block can be used at the top level, or it can be applied just to specific databases or tables. Here's how to apply it to just the <code>entries</code> table in the <code>news.db</code> database:</p>
  <div class="highlight highlight-source-json notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="{
      "databases": {
          "news": {
              "tables": {
                  "entries": {
                      "plugins": {
                          "datasette-render-markdown": {
                              "columns": ["body"]
                          }
                      }
                  }
              }
          }
      }
  }"><pre>{
      <span class="pl-ent">"databases"</span>: {
          <span class="pl-ent">"news"</span>: {
              <span class="pl-ent">"tables"</span>: {
                  <span class="pl-ent">"entries"</span>: {
                      <span class="pl-ent">"plugins"</span>: {
                          <span class="pl-ent">"datasette-render-markdown"</span>: {
                              <span class="pl-ent">"columns"</span>: [<span class="pl-s"><span class="pl-pds">"</span>body<span class="pl-pds">"</span></span>]
                          }
                      }
                  }
              }
          }
      }
  }</pre></div>
  <p dir="auto">And here's how to apply it to every <code>body</code> column in every table in the <code>news.db</code> database:</p>
  <div class="highlight highlight-source-json notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="{
      "databases": {
          "news": {
              "plugins": {
                  "datasette-render-markdown": {
                      "columns": ["body"]
                  }
              }
          }
      }
  }"><pre>{
      <span class="pl-ent">"databases"</span>: {
          <span class="pl-ent">"news"</span>: {
              <span class="pl-ent">"plugins"</span>: {
                  <span class="pl-ent">"datasette-render-markdown"</span>: {
                      <span class="pl-ent">"columns"</span>: [<span class="pl-s"><span class="pl-pds">"</span>body<span class="pl-pds">"</span></span>]
                  }
              }
          }
      }
  }</pre></div>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Columns that match a naming convention</h2><a id="user-content-columns-that-match-a-naming-convention" class="anchor" aria-label="Permalink: Columns that match a naming convention" href="#columns-that-match-a-naming-convention"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">This plugin can also render markdown in any columns that match a specific naming convention.</p>
  <p dir="auto">By default, columns that have a name ending in <code>_markdown</code> will be rendered.</p>
  <p dir="auto">You can try this out using the following query:</p>
  <div class="highlight highlight-source-sql notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="select '# Hello there

  * This is a list
  * of items

  [And a link](https://github.com/simonw/datasette-render-markdown).'
  as demo_markdown"><pre><span class="pl-k">select</span> <span class="pl-s"><span class="pl-pds">'</span># Hello there</span>
  <span class="pl-s"></span>
  <span class="pl-s">* This is a list</span>
  <span class="pl-s">* of items</span>
  <span class="pl-s"></span>
  <span class="pl-s">[And a link](https://github.com/simonw/datasette-render-markdown).<span class="pl-pds">'</span></span>
  <span class="pl-k">as</span> demo_markdown</pre></div>
  <p dir="auto">You can configure a different list of wildcard patterns using the <code>"patterns"</code> configuration key. Here's how to render columns that end in either <code>_markdown</code> or <code>_md</code>:</p>
  <div class="highlight highlight-source-json notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="{
      "plugins": {
          "datasette-render-markdown": {
              "patterns": ["*_markdown", "*_md"]
          }
      }
  }"><pre>{
      <span class="pl-ent">"plugins"</span>: {
          <span class="pl-ent">"datasette-render-markdown"</span>: {
              <span class="pl-ent">"patterns"</span>: [<span class="pl-s"><span class="pl-pds">"</span>*_markdown<span class="pl-pds">"</span></span>, <span class="pl-s"><span class="pl-pds">"</span>*_md<span class="pl-pds">"</span></span>]
          }
      }
  }</pre></div>
  <p dir="auto">To disable wildcard column matching entirely, set <code>"patterns": []</code> in your plugin metadata configuration.</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Markdown extensions</h2><a id="user-content-markdown-extensions" class="anchor" aria-label="Permalink: Markdown extensions" href="#markdown-extensions"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">The <a href="https://python-markdown.github.io/" rel="nofollow">Python-Markdown library</a> that powers this plugin supports extensions, both <a href="https://python-markdown.github.io/extensions/" rel="nofollow">bundled</a> and <a href="https://github.com/Python-Markdown/markdown/wiki/Third-Party-Extensions">third-party</a>. These can be used to enable additional Markdown features such as <a href="https://python-markdown.github.io/extensions/tables/" rel="nofollow">table support</a>.</p>
  <p dir="auto">You can configure support for extensions using the <code>"extensions"</code> key in your plugin metadata configuration.</p>
  <p dir="auto">Since extensions may introduce new HTML tags, you will also need to add those tags to the list of tags that are allowed by the <a href="https://bleach.readthedocs.io/" rel="nofollow">Bleach</a> sanitizer. You can do that using the <code>"extra_tags"</code> key, and you can allow-list additional HTML attributes using <code>"extra_attrs"</code>. See <a href="https://bleach.readthedocs.io/en/latest/clean.html#allowed-tags-tags" rel="nofollow">the Bleach documentation</a> for more information on this.</p>
  <p dir="auto">Here's how to enable support for <a href="https://python-markdown.github.io/extensions/tables/" rel="nofollow">Markdown tables</a>:</p>
  <div class="highlight highlight-source-json notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="{
      "plugins": {
          "datasette-render-markdown": {
              "extensions": ["tables"],
              "extra_tags": ["table", "thead", "tr", "th", "td", "tbody"]
          }
      }
  }"><pre>{
      <span class="pl-ent">"plugins"</span>: {
          <span class="pl-ent">"datasette-render-markdown"</span>: {
              <span class="pl-ent">"extensions"</span>: [<span class="pl-s"><span class="pl-pds">"</span>tables<span class="pl-pds">"</span></span>],
              <span class="pl-ent">"extra_tags"</span>: [<span class="pl-s"><span class="pl-pds">"</span>table<span class="pl-pds">"</span></span>, <span class="pl-s"><span class="pl-pds">"</span>thead<span class="pl-pds">"</span></span>, <span class="pl-s"><span class="pl-pds">"</span>tr<span class="pl-pds">"</span></span>, <span class="pl-s"><span class="pl-pds">"</span>th<span class="pl-pds">"</span></span>, <span class="pl-s"><span class="pl-pds">"</span>td<span class="pl-pds">"</span></span>, <span class="pl-s"><span class="pl-pds">"</span>tbody<span class="pl-pds">"</span></span>]
          }
      }
  }</pre></div>
  <div class="markdown-heading" dir="auto"><h3 class="heading-element" dir="auto">GitHub-Flavored Markdown</h3><a id="user-content-github-flavored-markdown" class="anchor" aria-label="Permalink: GitHub-Flavored Markdown" href="#github-flavored-markdown"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">Enabling <a href="https://help.github.com/en/github/writing-on-github">GitHub-Flavored Markdown</a> (useful for if you are working with data imported from GitHub using <a href="https://github.com/dogsheep/github-to-sqlite">github-to-sqlite</a>) is a little more complicated.</p>
  <p dir="auto">First, you will need to install the <a href="https://py-gfm.readthedocs.io" rel="nofollow">py-gfm</a> package:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ pip install py-gfm"><pre class="notranslate"><code>$ pip install py-gfm
  </code></pre></div>
  <p dir="auto">Note that <code>py-gfm</code> has <a href="https://github.com/Zopieux/py-gfm/issues/13" data-hovercard-type="issue" data-hovercard-url="/zopieux/py-gfm/issues/13/hovercard">a bug</a> that causes it to pin to <code>Markdown<3.0</code> - so if you are using it you should install it <em>before</em> installing <code>datasette-render-markdown</code> to ensure you get a compatibly version of that dependency.</p>
  <p dir="auto">Now you can configure it like this. Note that the extension name is <code>mdx_gfm:GithubFlavoredMarkdownExtension</code> and you need to allow-list several extra HTML tags and attributes:</p>
  <div class="highlight highlight-source-json notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="{
      "plugins": {
          "datasette-render-markdown": {
              "extra_tags": [
                  "hr",
                  "br",
                  "details",
                  "summary",
                  "input"
              ],
              "extra_attrs": {
                  "input": [
                      "type",
                      "disabled",
                      "checked"
                  ],
              },
              "extensions": [
                  "mdx_gfm:GithubFlavoredMarkdownExtension"
              ]
          }
      }
  }"><pre>{
      <span class="pl-ent">"plugins"</span>: {
          <span class="pl-ent">"datasette-render-markdown"</span>: {
              <span class="pl-ent">"extra_tags"</span>: [
                  <span class="pl-s"><span class="pl-pds">"</span>hr<span class="pl-pds">"</span></span>,
                  <span class="pl-s"><span class="pl-pds">"</span>br<span class="pl-pds">"</span></span>,
                  <span class="pl-s"><span class="pl-pds">"</span>details<span class="pl-pds">"</span></span>,
                  <span class="pl-s"><span class="pl-pds">"</span>summary<span class="pl-pds">"</span></span>,
                  <span class="pl-s"><span class="pl-pds">"</span>input<span class="pl-pds">"</span></span>
              ],
              <span class="pl-ent">"extra_attrs"</span>: {
                  <span class="pl-ent">"input"</span>: [
                      <span class="pl-s"><span class="pl-pds">"</span>type<span class="pl-pds">"</span></span>,
                      <span class="pl-s"><span class="pl-pds">"</span>disabled<span class="pl-pds">"</span></span>,
                      <span class="pl-s"><span class="pl-pds">"</span>checked<span class="pl-pds">"</span></span>
                  ],
              },
              <span class="pl-ent">"extensions"</span>: [
                  <span class="pl-s"><span class="pl-pds">"</span>mdx_gfm:GithubFlavoredMarkdownExtension<span class="pl-pds">"</span></span>
              ]
          }
      }
  }</pre></div>
  <p dir="auto">The <code><input type="" checked disabled></code> attributes are needed to support rendering checkboxes in issue descriptions.</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Markdown in templates</h2><a id="user-content-markdown-in-templates" class="anchor" aria-label="Permalink: Markdown in templates" href="#markdown-in-templates"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">The plugin introduces a new template tag: <code>{% markdown %}...{% endmarkdown %}</code> - which can be used to render Markdown in your Jinja templates.</p>
  <div class="highlight highlight-text-html-django notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="{% markdown %}
  # This will be rendered as markdown
  {% endmarkdown %}"><pre><span class="pl-e">{%</span> <span class="pl-s">markdown</span> <span class="pl-e">%}</span>
  # This will be rendered as markdown
  <span class="pl-e">{%</span> <span class="pl-s">endmarkdown</span> <span class="pl-e">%}</span></pre></div>
  <p dir="auto">You can use attributes on the <code>{% markdown %}</code> tag to enable extensions and allow-list additional tags and attributes:</p>
  <div class="highlight highlight-text-html-django notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="{% markdown
    extensions="tables"
    extra_tags="table thead tr th td tbody" 
    extra_attrs="p:id,class a:name,href" %}
  ## Markdown table

  First Header  | Second Header
  ------------- | -------------
  Content Cell  | Content Cell
  Content Cell  | Content Cell

  <a href="https://www.example.com/" name="namehere">Example</a>
  <p id="paragraph" class="klass">Paragraph</p>
  {% endmarkdown %}"><pre><span class="pl-e">{%</span> <span class="pl-s">markdown</span>
    <span class="pl-s">extensions</span>=<span class="pl-s">"tables"</span>
    <span class="pl-s">extra_tags</span>=<span class="pl-s">"table thead tr th td tbody"</span> 
    <span class="pl-s">extra_attrs</span>=<span class="pl-s">"p:id,class a:name,href"</span> <span class="pl-e">%}</span>
  ## Markdown table

  First Header  | Second Header
  ------------- | -------------
  Content Cell  | Content Cell
  Content Cell  | Content Cell

  <<span class="pl-ent">a</span> <span class="pl-e">href</span>=<span class="pl-s"><span class="pl-pds">"</span>https://www.example.com/<span class="pl-pds">"</span></span> <span class="pl-e">name</span>=<span class="pl-s"><span class="pl-pds">"</span>namehere<span class="pl-pds">"</span></span>>Example</<span class="pl-ent">a</span>>
  <<span class="pl-ent">p</span> <span class="pl-e">id</span>=<span class="pl-s"><span class="pl-pds">"</span>paragraph<span class="pl-pds">"</span></span> <span class="pl-e">class</span>=<span class="pl-s"><span class="pl-pds">"</span>klass<span class="pl-pds">"</span></span>>Paragraph</<span class="pl-ent">p</span>>
  <span class="pl-e">{%</span> <span class="pl-s">endmarkdown</span> <span class="pl-e">%}</span></pre></div>
  <p dir="auto">The <code>extensions=</code> and <code>extra_tags=</code> attributes accept a space-separated list of values.</p>
  <p dir="auto">The <code>extra_attrs=</code> attribute accepts a space-separated list of <code>tag:attr1,attr2</code> values - each tag can specify one or more attributes that should be allowed.</p>
  <p dir="auto">You can also use the <code>{{ render_markdown(...) }}</code> function, like this:</p>
  <div class="highlight highlight-text-html-django notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="{{ render_markdown("""
  ## Markdown table

  First Header  | Second Header
  ------------- | -------------
  Content Cell  | Content Cell
  Content Cell  | Content Cell
  """, extensions=["tables"],
      extra_tags=["table", "thead", "tr", "th", "td", "tbody"])) }}"><pre>{{ render_markdown("""
  ## Markdown table

  First Header  | Second Header
  ------------- | -------------
  Content Cell  | Content Cell
  Content Cell  | Content Cell
  """, extensions=["tables<span class="pl-smi">"]</span>,
      extra_tags=["table", "thead", "tr", "th", "td", "tbody<span class="pl-smi">"]</span>)) }}</pre></div>
  <p dir="auto">The <code>{% markdown %}</code> tag is recommended, as it avoids the need to <code>\"</code> escape quotes in your Markdown content.</p>
  </article></div>


node_id: MDEwOlJlcG9zaXRvcnkyMjE4MDIyOTY=
name: datasette-template-sql
full_name: simonw/datasette-template-sql
html_url: https://github.com/simonw/datasette-template-sql
description: Datasette plugin for executing SQL queries from templates
created_at: 2019-11-14T23:05:34Z
updated_at: 2026-04-08T17:27:57Z
pushed_at: 2026-04-08T17:28:26Z
homepage: https://datasette.io/plugins/datasette-template-sql
language: Python
license: apache-2.0
topics: ["datasette", "datasette-io", "datasette-plugin"]
visibility: public
default_branch: main
permissions: {"admin": false, "maintain": false, "push": false, "triage": false, "pull": false}
pull_request_creation_policy: all
readme:
  # datasette-template-sql

  [![PyPI](https://img.shields.io/pypi/v/datasette-template-sql.svg)](https://pypi.org/project/datasette-template-sql/)
  [![Changelog](https://img.shields.io/github/v/release/simonw/datasette-template-sql?include_prereleases&label=changelog)](https://github.com/simonw/datasette-template-sql/releases)
  [![Tests](https://github.com/simonw/datasette-template-sql/workflows/Test/badge.svg)](https://github.com/simonw/datasette-template-sql/actions?query=workflow%3ATest)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/datasette-template-sql/blob/main/LICENSE)

  Datasette plugin for executing SQL queries from templates.

  ## Examples

  [datasette.io](https://datasette.io/) uses this plugin extensively with [custom page templates](https://docs.datasette.io/en/stable/custom_templates.html#custom-pages), check out [simonw/datasette.io](https://github.com/simonw/datasette.io) to see how it works.

  [www.niche-museums.com](https://www.niche-museums.com/) uses this plugin to run a custom themed website on top of Datasette. The full source code for the site [is here](https://github.com/simonw/museums) - see also [niche-museums.com, powered by Datasette](https://simonwillison.net/2019/Nov/25/niche-museums/).

  [simonw/til](https://github.com/simonw/til) is another simple example, described in [Using a self-rewriting README powered by GitHub Actions to track TILs](https://simonwillison.net/2020/Apr/20/self-rewriting-readme/).

  ## Installation

  Run this command to install the plugin in the same environment as Datasette:

      $ pip install datasette-template-sql

  ## Usage

  This plugin makes a new function, `sql(sql_query)`, available to your Datasette templates.

  You can use it like this:

  ```html+jinja
  {% for row in sql("select 1 + 1 as two, 2 * 4 as eight") %}
      {% for key in row.keys() %}
          {{ key }}: {{ row[key] }}<br>
      {% endfor %}
  {% endfor %}
  ```

  The plugin will execute SQL against the current database for the page in  `database.html`, `table.html` and `row.html` templates. If a template does not have a current database (`index.html` for example) the query will execute against the first attached database.

  ### Queries with arguments

  You can construct a SQL query using `?` or `:name` parameter syntax by passing a list or dictionary as a second argument:

  ```html+jinja
  {% for row in sql("select distinct topic from til order by topic") %}
      <h2>{{ row.topic }}</h2>
      <ul>
          {% for til in sql("select * from til where topic = ?", [row.topic]) %}
              <li><a href="{{ til.url }}">{{ til.title }}</a> - {{ til.created[:10] }}</li>
          {% endfor %}
      </ul>
  {% endfor %}
  ```

  Here's the same example using the `:topic` style of parameters:

  ```html+jinja
  {% for row in sql("select distinct topic from til order by topic") %}
      <h2>{{ row.topic }}</h2>
      <ul>
          {% for til in sql("select * from til where topic = :topic", {"topic": row.topic}) %}
              <li><a href="{{ til.url }}">{{ til.title }}</a> - {{ til.created[:10] }}</li>
          {% endfor %}
      </ul>
  {% endfor %}
  ```

  ### Querying a different database

  You can pass an optional `database=` argument to specify a named database to use for the query. For example, if you have attached a `news.db` database you could use this:

  ```html+jinja
  {% for article in sql(
      "select headline, date, summary from articles order by date desc limit 5",
      database="news"
  ) %}
      <h3>{{ article.headline }}</h2>
      <p class="date">{{ article.date }}</p>
      <p>{{ article.summary }}</p>
  {% endfor %}
  ```

readme_html:
  <div id="readme" class="md" data-path="README.md"><article class="markdown-body entry-content container-lg" itemprop="text"><div class="markdown-heading" dir="auto"><h1 class="heading-element" dir="auto">datasette-template-sql</h1><a id="user-content-datasette-template-sql" class="anchor" aria-label="Permalink: datasette-template-sql" href="#datasette-template-sql"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto"><a href="https://pypi.org/project/datasette-template-sql/" rel="nofollow"><img src="https://img.shields.io/pypi/v/datasette-template-sql.svg" alt="PyPI" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/510b7c19ea37fbc8744e86558a24b7485d19f84e37a1b8431c594e96006b5994/68747470733a2f2f696d672e736869656c64732e696f2f707970692f762f6461746173657474652d74656d706c6174652d73716c2e737667"></a>
  <a href="https://github.com/simonw/datasette-template-sql/releases"><img src="https://img.shields.io/github/v/release/simonw/datasette-template-sql?include_prereleases&label=changelog" alt="Changelog" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/1d80b2a904009731708b0d6ff7514be34bcf7cf0e08be67362968296d7d413a5/68747470733a2f2f696d672e736869656c64732e696f2f6769746875622f762f72656c656173652f73696d6f6e772f6461746173657474652d74656d706c6174652d73716c3f696e636c7564655f70726572656c6561736573266c6162656c3d6368616e67656c6f67"></a>
  <a href="https://github.com/simonw/datasette-template-sql/actions?query=workflow%3ATest"><img src="https://github.com/simonw/datasette-template-sql/workflows/Test/badge.svg" alt="Tests" style="max-width: 100%;"></a>
  <a href="https://github.com/simonw/datasette-template-sql/blob/main/LICENSE"><img src="https://img.shields.io/badge/license-Apache%202.0-blue.svg" alt="License" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/c355f200ea90fddaa407b6eaab303663a669248ea3ca7b1fcf77dbe04ff5f48c/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6c6963656e73652d417061636865253230322e302d626c75652e737667"></a></p>
  <p dir="auto">Datasette plugin for executing SQL queries from templates.</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Examples</h2><a id="user-content-examples" class="anchor" aria-label="Permalink: Examples" href="#examples"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto"><a href="https://datasette.io/" rel="nofollow">datasette.io</a> uses this plugin extensively with <a href="https://docs.datasette.io/en/stable/custom_templates.html#custom-pages" rel="nofollow">custom page templates</a>, check out <a href="https://github.com/simonw/datasette.io">simonw/datasette.io</a> to see how it works.</p>
  <p dir="auto"><a href="https://www.niche-museums.com/" rel="nofollow">www.niche-museums.com</a> uses this plugin to run a custom themed website on top of Datasette. The full source code for the site <a href="https://github.com/simonw/museums">is here</a> - see also <a href="https://simonwillison.net/2019/Nov/25/niche-museums/" rel="nofollow">niche-museums.com, powered by Datasette</a>.</p>
  <p dir="auto"><a href="https://github.com/simonw/til">simonw/til</a> is another simple example, described in <a href="https://simonwillison.net/2020/Apr/20/self-rewriting-readme/" rel="nofollow">Using a self-rewriting README powered by GitHub Actions to track TILs</a>.</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Installation</h2><a id="user-content-installation" class="anchor" aria-label="Permalink: Installation" href="#installation"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">Run this command to install the plugin in the same environment as Datasette:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ pip install datasette-template-sql"><pre class="notranslate"><code>$ pip install datasette-template-sql
  </code></pre></div>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Usage</h2><a id="user-content-usage" class="anchor" aria-label="Permalink: Usage" href="#usage"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">This plugin makes a new function, <code>sql(sql_query)</code>, available to your Datasette templates.</p>
  <p dir="auto">You can use it like this:</p>
  <div class="highlight highlight-text-html-django notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="{% for row in sql("select 1 + 1 as two, 2 * 4 as eight") %}
      {% for key in row.keys() %}
          {{ key }}: {{ row[key] }}<br>
      {% endfor %}
  {% endfor %}"><pre><span class="pl-e">{%</span> <span class="pl-k">for</span> <span class="pl-s">row</span> <span class="pl-k">in</span> <span class="pl-s">sql</span>(<span class="pl-s">"select 1 + 1 as two, 2 * 4 as eight"</span>) <span class="pl-e">%}</span>
      <span class="pl-e">{%</span> <span class="pl-k">for</span> <span class="pl-s">key</span> <span class="pl-k">in</span> <span class="pl-s">row</span>.<span class="pl-s">keys</span>() <span class="pl-e">%}</span>
          {{ key }}: {{ row[ke<span class="pl-smi">y]</span> }}<<span class="pl-ent">br</span>>
      <span class="pl-e">{%</span> <span class="pl-k">endfor</span> <span class="pl-e">%}</span>
  <span class="pl-e">{%</span> <span class="pl-k">endfor</span> <span class="pl-e">%}</span></pre></div>
  <p dir="auto">The plugin will execute SQL against the current database for the page in  <code>database.html</code>, <code>table.html</code> and <code>row.html</code> templates. If a template does not have a current database (<code>index.html</code> for example) the query will execute against the first attached database.</p>
  <div class="markdown-heading" dir="auto"><h3 class="heading-element" dir="auto">Queries with arguments</h3><a id="user-content-queries-with-arguments" class="anchor" aria-label="Permalink: Queries with arguments" href="#queries-with-arguments"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">You can construct a SQL query using <code>?</code> or <code>:name</code> parameter syntax by passing a list or dictionary as a second argument:</p>
  <div class="highlight highlight-text-html-django notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="{% for row in sql("select distinct topic from til order by topic") %}
      <h2>{{ row.topic }}</h2>
      <ul>
          {% for til in sql("select * from til where topic = ?", [row.topic]) %}
              <li><a href="{{ til.url }}">{{ til.title }}</a> - {{ til.created[:10] }}</li>
          {% endfor %}
      </ul>
  {% endfor %}"><pre><span class="pl-e">{%</span> <span class="pl-k">for</span> <span class="pl-s">row</span> <span class="pl-k">in</span> <span class="pl-s">sql</span>(<span class="pl-s">"select distinct topic from til order by topic"</span>) <span class="pl-e">%}</span>
      <<span class="pl-ent">h2</span>>{{ row.topic }}</<span class="pl-ent">h2</span>>
      <<span class="pl-ent">ul</span>>
          <span class="pl-e">{%</span> <span class="pl-k">for</span> <span class="pl-s">til</span> <span class="pl-k">in</span> <span class="pl-s">sql</span>(<span class="pl-s">"select * from til where topic = ?"</span>, [<span class="pl-s">row</span>.<span class="pl-s">topic</span>]) <span class="pl-e">%}</span>
              <<span class="pl-ent">li</span>><<span class="pl-ent">a</span> <span class="pl-e">href</span>=<span class="pl-s"><span class="pl-pds">"</span>{{ til.url }}<span class="pl-pds">"</span></span>>{{ til.title }}</<span class="pl-ent">a</span>> - {{ til.created[:1<span class="pl-smi">0]</span> }}</<span class="pl-ent">li</span>>
          <span class="pl-e">{%</span> <span class="pl-k">endfor</span> <span class="pl-e">%}</span>
      </<span class="pl-ent">ul</span>>
  <span class="pl-e">{%</span> <span class="pl-k">endfor</span> <span class="pl-e">%}</span></pre></div>
  <p dir="auto">Here's the same example using the <code>:topic</code> style of parameters:</p>
  <div class="highlight highlight-text-html-django notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="{% for row in sql("select distinct topic from til order by topic") %}
      <h2>{{ row.topic }}</h2>
      <ul>
          {% for til in sql("select * from til where topic = :topic", {"topic": row.topic}) %}
              <li><a href="{{ til.url }}">{{ til.title }}</a> - {{ til.created[:10] }}</li>
          {% endfor %}
      </ul>
  {% endfor %}"><pre><span class="pl-e">{%</span> <span class="pl-k">for</span> <span class="pl-s">row</span> <span class="pl-k">in</span> <span class="pl-s">sql</span>(<span class="pl-s">"select distinct topic from til order by topic"</span>) <span class="pl-e">%}</span>
      <<span class="pl-ent">h2</span>>{{ row.topic }}</<span class="pl-ent">h2</span>>
      <<span class="pl-ent">ul</span>>
          <span class="pl-e">{%</span> <span class="pl-k">for</span> <span class="pl-s">til</span> <span class="pl-k">in</span> <span class="pl-s">sql</span>(<span class="pl-s">"select * from til where topic = :topic"</span>, {<span class="pl-s">"topic"</span>: <span class="pl-s">row</span>.<span class="pl-s">topic</span>}) <span class="pl-e">%}</span>
              <<span class="pl-ent">li</span>><<span class="pl-ent">a</span> <span class="pl-e">href</span>=<span class="pl-s"><span class="pl-pds">"</span>{{ til.url }}<span class="pl-pds">"</span></span>>{{ til.title }}</<span class="pl-ent">a</span>> - {{ til.created[:1<span class="pl-smi">0]</span> }}</<span class="pl-ent">li</span>>
          <span class="pl-e">{%</span> <span class="pl-k">endfor</span> <span class="pl-e">%}</span>
      </<span class="pl-ent">ul</span>>
  <span class="pl-e">{%</span> <span class="pl-k">endfor</span> <span class="pl-e">%}</span></pre></div>
  <div class="markdown-heading" dir="auto"><h3 class="heading-element" dir="auto">Querying a different database</h3><a id="user-content-querying-a-different-database" class="anchor" aria-label="Permalink: Querying a different database" href="#querying-a-different-database"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">You can pass an optional <code>database=</code> argument to specify a named database to use for the query. For example, if you have attached a <code>news.db</code> database you could use this:</p>
  <div class="highlight highlight-text-html-django notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="{% for article in sql(
      "select headline, date, summary from articles order by date desc limit 5",
      database="news"
  ) %}
      <h3>{{ article.headline }}</h2>
      <p class="date">{{ article.date }}</p>
      <p>{{ article.summary }}</p>
  {% endfor %}"><pre><span class="pl-e">{%</span> <span class="pl-k">for</span> <span class="pl-s">article</span> <span class="pl-k">in</span> <span class="pl-s">sql</span>(
      <span class="pl-s">"select headline, date, summary from articles order by date desc limit 5"</span>,
      <span class="pl-s">database</span>=<span class="pl-s">"news"</span>
  ) <span class="pl-e">%}</span>
      <<span class="pl-ent">h3</span>>{{ article.headline }}</<span class="pl-ent">h2</span>>
      <<span class="pl-ent">p</span> <span class="pl-e">class</span>=<span class="pl-s"><span class="pl-pds">"</span>date<span class="pl-pds">"</span></span>>{{ article.date }}</<span class="pl-ent">p</span>>
      <<span class="pl-ent">p</span>>{{ article.summary }}</<span class="pl-ent">p</span>>
  <span class="pl-e">{%</span> <span class="pl-k">endfor</span> <span class="pl-e">%}</span></pre></div>
  </article></div>


node_id: MDEwOlJlcG9zaXRvcnkyMzQ4MjU3OTA=
name: datasette-upload-csvs
full_name: simonw/datasette-upload-csvs
html_url: https://github.com/simonw/datasette-upload-csvs
description: Datasette plugin for uploading CSV files and converting them to database tables
created_at: 2020-01-19T02:07:05Z
updated_at: 2026-03-27T23:17:20Z
pushed_at: 2025-11-10T19:08:50Z
homepage: https://datasette.io/plugins/datasette-upload-csvs
language: Python
license: apache-2.0
topics: ["csvs", "datasette", "datasette-io", "datasette-plugin"]
visibility: public
default_branch: main
permissions: {"admin": false, "maintain": false, "push": false, "triage": false, "pull": false}
pull_request_creation_policy: all
readme:
  # datasette-upload-csvs

  [![PyPI](https://img.shields.io/pypi/v/datasette-upload-csvs.svg)](https://pypi.org/project/datasette-upload-csvs/)
  [![Changelog](https://img.shields.io/github/v/release/simonw/datasette-upload-csvs?include_prereleases&label=changelog)](https://github.com/simonw/datasette-upload-csvs/releases)
  [![Tests](https://github.com/simonw/datasette-upload-csvs/workflows/Test/badge.svg)](https://github.com/simonw/datasette-upload-csvs/actions?query=workflow%3ATest)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/datasette-upload-csvs/blob/main/LICENSE)

  Datasette plugin for uploading CSV files and converting them to database tables

  ## Installation

      datasette install datasette-upload-csvs

  ## Usage

  The plugin adds an interface at `/-/upload-csvs` for uploading a CSV file and using it to create a new database table.

  By default only [the root actor](https://datasette.readthedocs.io/en/stable/authentication.html#using-the-root-actor) can access the page - so you'll need to run Datasette with the `--root` option and click on the link shown in the terminal to sign in and access the page.

  The `upload-csvs` permission governs access. You can use permission plugins such as [datasette-permissions-sql](https://github.com/simonw/datasette-permissions-sql) to grant additional access to the write interface.

readme_html:
  <div id="readme" class="md" data-path="README.md"><article class="markdown-body entry-content container-lg" itemprop="text"><div class="markdown-heading" dir="auto"><h1 class="heading-element" dir="auto">datasette-upload-csvs</h1><a id="user-content-datasette-upload-csvs" class="anchor" aria-label="Permalink: datasette-upload-csvs" href="#datasette-upload-csvs"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto"><a href="https://pypi.org/project/datasette-upload-csvs/" rel="nofollow"><img src="https://img.shields.io/pypi/v/datasette-upload-csvs.svg" alt="PyPI" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/9a33769398b25017fb8e61881dc0368937072fa48b06fc38a773666886fd1160/68747470733a2f2f696d672e736869656c64732e696f2f707970692f762f6461746173657474652d75706c6f61642d637376732e737667"></a>
  <a href="https://github.com/simonw/datasette-upload-csvs/releases"><img src="https://img.shields.io/github/v/release/simonw/datasette-upload-csvs?include_prereleases&label=changelog" alt="Changelog" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/aac82e4ca211ceea15e7452d6f73179beb3781615bea52d3e747d272de161067/68747470733a2f2f696d672e736869656c64732e696f2f6769746875622f762f72656c656173652f73696d6f6e772f6461746173657474652d75706c6f61642d637376733f696e636c7564655f70726572656c6561736573266c6162656c3d6368616e67656c6f67"></a>
  <a href="https://github.com/simonw/datasette-upload-csvs/actions?query=workflow%3ATest"><img src="https://github.com/simonw/datasette-upload-csvs/workflows/Test/badge.svg" alt="Tests" style="max-width: 100%;"></a>
  <a href="https://github.com/simonw/datasette-upload-csvs/blob/main/LICENSE"><img src="https://img.shields.io/badge/license-Apache%202.0-blue.svg" alt="License" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/c355f200ea90fddaa407b6eaab303663a669248ea3ca7b1fcf77dbe04ff5f48c/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6c6963656e73652d417061636865253230322e302d626c75652e737667"></a></p>
  <p dir="auto">Datasette plugin for uploading CSV files and converting them to database tables</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Installation</h2><a id="user-content-installation" class="anchor" aria-label="Permalink: Installation" href="#installation"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="datasette install datasette-upload-csvs"><pre class="notranslate"><code>datasette install datasette-upload-csvs
  </code></pre></div>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Usage</h2><a id="user-content-usage" class="anchor" aria-label="Permalink: Usage" href="#usage"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">The plugin adds an interface at <code>/-/upload-csvs</code> for uploading a CSV file and using it to create a new database table.</p>
  <p dir="auto">By default only <a href="https://datasette.readthedocs.io/en/stable/authentication.html#using-the-root-actor" rel="nofollow">the root actor</a> can access the page - so you'll need to run Datasette with the <code>--root</code> option and click on the link shown in the terminal to sign in and access the page.</p>
  <p dir="auto">The <code>upload-csvs</code> permission governs access. You can use permission plugins such as <a href="https://github.com/simonw/datasette-permissions-sql">datasette-permissions-sql</a> to grant additional access to the write interface.</p>
  </article></div>


