# repos

node_id: R_kgDOIwP5mQ
name: json-to-files
full_name: simonw/json-to-files
html_url: https://github.com/simonw/json-to-files
description: Create separate files on disk based on a JSON object
created_at: 2023-01-10T20:09:34Z
updated_at: 2025-02-21T04:18:57Z
pushed_at: 2023-01-10T20:15:20Z
language: Python
license: apache-2.0
topics: []
visibility: public
default_branch: main
permissions: {"admin": false, "maintain": false, "push": false, "triage": false, "pull": true}
readme:
  # json-to-files

  [![PyPI](https://img.shields.io/pypi/v/json-to-files.svg)](https://pypi.org/project/json-to-files/)
  [![Changelog](https://img.shields.io/github/v/release/simonw/json-to-files?include_prereleases&label=changelog)](https://github.com/simonw/json-to-files/releases)
  [![Tests](https://github.com/simonw/json-to-files/workflows/Test/badge.svg)](https://github.com/simonw/json-to-files/actions?query=workflow%3ATest)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/json-to-files/blob/master/LICENSE)

  Create separate files on disk based on a JSON object

  ## Installation

  Install this tool using `pip`:

      pip install json-to-files

  ## Usage

  This tool takes a JSON file that looks like this:

  ```json
  {
      "foo.txt": "The contents of foo.txt",
      "bar/baz.txt": "The contents of baz.txt"
  }
  ```
  And uses it to write out the following files on disk:

  - `foo.txt` containing "The contents of foo.txt"
  - `bar/baz.txt` containing "The contents of baz.txt"

  You can run it like this:

      json-to-files bundle.json

  Or you can specify a directory to write those files to:

      json-to-files bundle.json -d /tmp/other-directory

  The `bundles.json` file name is optional - if omitted, this tool will read from standard input:

      cat bundle.json | json-to-files

  ## Development

  To contribute to this tool, first checkout the code. Then create a new virtual environment:

      cd json-to-files
      python -m venv venv
      source venv/bin/activate

  Now install the dependencies and test dependencies:

      pip install -e '.[test]'

  To run the tests:

      pytest

readme_html:
  <div id="readme" class="md" data-path="README.md"><article class="markdown-body entry-content container-lg" itemprop="text"><div class="markdown-heading" dir="auto"><h1 class="heading-element" dir="auto">json-to-files</h1><a id="user-content-json-to-files" class="anchor" aria-label="Permalink: json-to-files" href="#json-to-files"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto"><a href="https://pypi.org/project/json-to-files/" rel="nofollow"><img src="https://img.shields.io/pypi/v/json-to-files.svg" alt="PyPI" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/1b26c61e354a0cf80a952fcb384bdfc365c004b0037bce1f4161f8275a52dfc0/68747470733a2f2f696d672e736869656c64732e696f2f707970692f762f6a736f6e2d746f2d66696c65732e737667"></a>
  <a href="https://github.com/simonw/json-to-files/releases"><img src="https://img.shields.io/github/v/release/simonw/json-to-files?include_prereleases&label=changelog" alt="Changelog" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/8ef2c3908f822cb26169e24d3f8c5ce218f44e94c473d3edab1e8686b27025ca/68747470733a2f2f696d672e736869656c64732e696f2f6769746875622f762f72656c656173652f73696d6f6e772f6a736f6e2d746f2d66696c65733f696e636c7564655f70726572656c6561736573266c6162656c3d6368616e67656c6f67"></a>
  <a href="https://github.com/simonw/json-to-files/actions?query=workflow%3ATest"><img src="https://github.com/simonw/json-to-files/workflows/Test/badge.svg" alt="Tests" style="max-width: 100%;"></a>
  <a href="https://github.com/simonw/json-to-files/blob/master/LICENSE"><img src="https://img.shields.io/badge/license-Apache%202.0-blue.svg" alt="License" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/c355f200ea90fddaa407b6eaab303663a669248ea3ca7b1fcf77dbe04ff5f48c/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6c6963656e73652d417061636865253230322e302d626c75652e737667"></a></p>
  <p dir="auto">Create separate files on disk based on a JSON object</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Installation</h2><a id="user-content-installation" class="anchor" aria-label="Permalink: Installation" href="#installation"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">Install this tool using <code>pip</code>:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="pip install json-to-files"><pre class="notranslate"><code>pip install json-to-files
  </code></pre></div>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Usage</h2><a id="user-content-usage" class="anchor" aria-label="Permalink: Usage" href="#usage"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">This tool takes a JSON file that looks like this:</p>
  <div class="highlight highlight-source-json notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="{
      "foo.txt": "The contents of foo.txt",
      "bar/baz.txt": "The contents of baz.txt"
  }"><pre>{
      <span class="pl-ent">"foo.txt"</span>: <span class="pl-s"><span class="pl-pds">"</span>The contents of foo.txt<span class="pl-pds">"</span></span>,
      <span class="pl-ent">"bar/baz.txt"</span>: <span class="pl-s"><span class="pl-pds">"</span>The contents of baz.txt<span class="pl-pds">"</span></span>
  }</pre></div>
  <p dir="auto">And uses it to write out the following files on disk:</p>
  <ul dir="auto">
  <li><code>foo.txt</code> containing "The contents of foo.txt"</li>
  <li><code>bar/baz.txt</code> containing "The contents of baz.txt"</li>
  </ul>
  <p dir="auto">You can run it like this:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="json-to-files bundle.json"><pre class="notranslate"><code>json-to-files bundle.json
  </code></pre></div>
  <p dir="auto">Or you can specify a directory to write those files to:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="json-to-files bundle.json -d /tmp/other-directory"><pre class="notranslate"><code>json-to-files bundle.json -d /tmp/other-directory
  </code></pre></div>
  <p dir="auto">The <code>bundles.json</code> file name is optional - if omitted, this tool will read from standard input:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="cat bundle.json | json-to-files"><pre class="notranslate"><code>cat bundle.json | json-to-files
  </code></pre></div>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Development</h2><a id="user-content-development" class="anchor" aria-label="Permalink: Development" href="#development"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">To contribute to this tool, first checkout the code. Then create a new virtual environment:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="cd json-to-files
  python -m venv venv
  source venv/bin/activate"><pre class="notranslate"><code>cd json-to-files
  python -m venv venv
  source venv/bin/activate
  </code></pre></div>
  <p dir="auto">Now install the dependencies and test dependencies:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="pip install -e '.[test]'"><pre class="notranslate"><code>pip install -e '.[test]'
  </code></pre></div>
  <p dir="auto">To run the tests:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="pytest"><pre class="notranslate"><code>pytest
  </code></pre></div>
  </article></div>


node_id: R_kgDOIwW61g
name: datasette-faiss
full_name: simonw/datasette-faiss
html_url: https://github.com/simonw/datasette-faiss
description: Maintain a FAISS index for specified Datasette tables
created_at: 2023-01-11T04:28:21Z
updated_at: 2025-03-28T00:36:26Z
pushed_at: 2024-06-17T18:12:08Z
language: Python
license: apache-2.0
topics: []
visibility: public
default_branch: main
permissions: {"admin": false, "maintain": false, "push": false, "triage": false, "pull": true}
readme:
  # datasette-faiss

  [![PyPI](https://img.shields.io/pypi/v/datasette-faiss.svg)](https://pypi.org/project/datasette-faiss/)
  [![Changelog](https://img.shields.io/github/v/release/simonw/datasette-faiss?include_prereleases&label=changelog)](https://github.com/simonw/datasette-faiss/releases)
  [![Tests](https://github.com/simonw/datasette-faiss/workflows/Test/badge.svg)](https://github.com/simonw/datasette-faiss/actions?query=workflow%3ATest)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/datasette-faiss/blob/main/LICENSE)

  Maintain a [FAISS index](https://github.com/facebookresearch/faiss) for specified Datasette tables

  See [Semantic search answers: Q&A against documentation with GPT3 + OpenAI embeddings](https://simonwillison.net/2023/Jan/13/semantic-search-answers/) for background on this project.

  ## Installation

  Install this plugin in the same environment as Datasette.
  ```bash
  datasette install datasette-faiss
  ```
  ## Usage

  This plugin creates in-memory FAISS indexes for specified tables on startup, using an `IndexFlatL2` [FAISS index type](https://github.com/facebookresearch/faiss/wiki/Faiss-indexes).

  If the tables are modified after the server has started the indexes will not (yet) pick up those changes.

  ### Configuration

  The tables to be indexed must have `id` and `embedding` columns. The `embedding` column must be a `blob` containing embeddings that are arrays of floating point numbers that have been encoded using the following Python function:

  ```python
  def encode(vector):
      return struct.pack("f" * len(vector), *vector)
  ```
  You can import that function from this package like so:
  ```python
  from datasette_faiss import encode
  ```
  You can specify which tables should have indexes created for them by adding this to `metadata.json`:
  ```json
  {
      "plugins": {
          "datasette-faiss": {
              "tables": [
                  ["blog", "embeddings"]
              ]
          }
      }
  }
  ```
  Each table is an array listing the database name and the table name.

  If you are using `metadata.yml` the configuration should look like this:

  ```yaml
  plugins:
    datasette-faiss:
      tables:
      - ["blog", "embeddings"]
  ```

  ### SQL functions

  The plugin makes four new SQL functions available within Datasette:

  #### faiss_search(database, table, embedding, k)
    
  Returns the `k` nearest neighbors to the `embedding` found in the specified database and table. For example:
  ```sql
  select faiss_search('blog', 'embeddings', (select embedding from embeddings where id = 3), 5)
  ```
  This will return a JSON array of the five IDs of the records in the `embeddings` table in the `blog` database that are closest to the specified embedding. The returned value looks like this:

  ```json
  ["1", "1249", "1011", "5", "10"]
  ```
  You can use the SQLite `json_each()` function to turn that into a table-like sequence that you can join against.

  Here's an example query that does that:

  ```sql
  with related as (
    select value from json_each(
      faiss_search(
        'blog',
        'embeddings',
        (select embedding from embeddings limit 1),
        5
      )
    )
  )
  select * from blog_entry, related
  where id = value
  ```
  #### faiss_search_with_scores(database, table, embedding, k)

  Takes the same arguments as above, but the return value is a JSON array of pairs, each with an ID and a score - something like this:

  ```json
  [
      ["1", 0.0],
      ["1249", 0.21042244136333466],
      ["1011", 0.29391372203826904],
      ["5", 0.29505783319473267],
      ["10", 0.31554925441741943]
  ]
  ```

  #### faiss_encode(json_vector)

  Given a JSON array of floats, returns the binary embedding blob that can be used with the other functions:

  ```sql
  select faiss_encode('[2.4, 4.1, 1.8]')
  -- Returns a 12 byte blob
  select hex(faiss_encode('[2.4, 4.1, 1.8]'))
  -- Returns 9A991940333383406666E63F
  ```

  #### faiss_decode(vector_blob)

  The opposite of `faiss_encode()`.

  ```sql
  select faiss_decode(X'9A991940333383406666E63F')
  ```
  Returns:
  ```json
  [2.4000000953674316, 4.099999904632568, 1.7999999523162842]
  ```
  Note that floating point arithmetic results in numbers that don't quite round-trip to the exact same expected value.

  #### faiss_agg(id, embedding, compare_embedding, k)

  This aggregate function can be used to find the `k` nearest neighbors to `compare_embedding` for each unique value of `id` in the table. For example:

  ```sql
  select faiss_agg(
      id, embedding, (select embedding from embeddings where id = 3), 5
  ) from embeddings
  ```
  Unlike the `faiss_search()` function, this does not depend on the per-table index that the plugin creates when it first starts running. Instead, an index is built every time the aggregation function is run.

  This means that it should only be used on smaller sets of values - once you get above 10,000 or so the performance from this function is likely to become prohibitively expensive.

  The function returns a JSON array of IDs representing the `k` rows with the closest distance scores, like this:

  ```json
  [1324, 344, 5562, 553, 2534]
  ```
  You can use the `json_each()` function to turn that into a table-like sequence that you can join against.

  [Try an example fais_agg() query](https://datasette.simonwillison.net/simonwillisonblog?sql=with+last_500+as+%28%0D%0A++select%0D%0A++++id%2C%0D%0A++++embedding%0D%0A++from%0D%0A++++blog_entry_embeddings%0D%0A++order+by%0D%0A++++id+desc%0D%0A++limit%0D%0A++++500%0D%0A%29%2C+faiss+as+%28%0D%0A++select%0D%0A++++faiss_agg%28%0D%0A++++++id%2C%0D%0A++++++embedding%2C%0D%0A++++++%28%0D%0A++++++++select%0D%0A++++++++++embedding%0D%0A++++++++from%0D%0A++++++++++blog_entry_embeddings%0D%0A++++++++where%0D%0A++++++++++id+%3D+%3Aid%0D%0A++++++%29%2C%0D%0A++++++10%0D%0A++++%29+as+results%0D%0A++from%0D%0A++++last_500%0D%0A%29%2C%0D%0Aids+as+%28%0D%0A++select%0D%0A++++value+as+id%0D%0A++from%0D%0A++++json_each%28faiss.results%29%2C%0D%0A++++faiss%0D%0A%29%0D%0Aselect%0D%0A++blog_entry.id%2C%0D%0A++blog_entry.title%2C%0D%0A++blog_entry.created%0D%0Afrom%0D%0A++ids%0D%0A++join+blog_entry+on+ids.id+%3D+blog_entry.id&id=8214).

  #### faiss_agg_with_scores(id, embedding, compare_embedding, k)

  This is similar to the `faiss_agg()` aggregate function but it returns a list of pairs, each with an ID and the corresponding score - something that looks like this (if `k` was 2):

  ```json
  [[2412, 0.25], [1245, 24.25]]
  ```
  [Try an example fais_agg_with_scores() query](https://datasette.simonwillison.net/simonwillisonblog?sql=with+last_500+as+%28%0D%0A++select%0D%0A++++id%2C%0D%0A++++embedding%0D%0A++from%0D%0A++++blog_entry_embeddings%0D%0A++order+by%0D%0A++++id+desc%0D%0A++limit%0D%0A++++500%0D%0A%29%2C+ids_and_scores+as+%28%0D%0A++select%0D%0A++++faiss_agg_with_scores%28%0D%0A++++++id%2C%0D%0A++++++embedding%2C%0D%0A++++++%28%0D%0A++++++++select%0D%0A++++++++++embedding%0D%0A++++++++from%0D%0A++++++++++blog_entry_embeddings%0D%0A++++++++where%0D%0A++++++++++id+%3D+%3Aid%0D%0A++++++%29%2C+10%0D%0A++++%29+as+s%0D%0A++from%0D%0A++++last_500%0D%0A%29%2C%0D%0Aresults+as+%28%0D%0A++select%0D%0A++++json_extract%28value%2C+%27%24%5B0%5D%27%29+as+id%2C%0D%0A++++json_extract%28value%2C+%27%24%5B1%5D%27%29+as+score%0D%0A++from%0D%0A++++json_each%28ids_and_scores.s%29%2C%0D%0A++++ids_and_scores%0D%0A%29%0D%0Aselect%0D%0A++results.score%2C%0D%0A++blog_entry.id%2C%0D%0A++blog_entry.title%2C%0D%0A++blog_entry.created%0D%0Afrom%0D%0A++results%0D%0A++join+blog_entry+on+results.id+%3D+blog_entry.id&id=8214).

  ## Development

  To set up this plugin locally, first checkout the code. Then create a new virtual environment:
  ```bash
  cd datasette-faiss
  python3 -m venv venv
  source venv/bin/activate
  ```
  Now install the dependencies and test dependencies:
  ```bash
  pip install -e '.[test]'
  ```
  To run the tests:
  ```bash
  pytest
  ```

readme_html:
  <div id="readme" class="md" data-path="README.md"><article class="markdown-body entry-content container-lg" itemprop="text"><div class="markdown-heading" dir="auto"><h1 class="heading-element" dir="auto">datasette-faiss</h1><a id="user-content-datasette-faiss" class="anchor" aria-label="Permalink: datasette-faiss" href="#datasette-faiss"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto"><a href="https://pypi.org/project/datasette-faiss/" rel="nofollow"><img src="https://img.shields.io/pypi/v/datasette-faiss.svg" alt="PyPI" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/d3c54d64c62cac2f841065f27ae3add463120bc568f60d8a8b55f551befd8a94/68747470733a2f2f696d672e736869656c64732e696f2f707970692f762f6461746173657474652d66616973732e737667"></a>
  <a href="https://github.com/simonw/datasette-faiss/releases"><img src="https://img.shields.io/github/v/release/simonw/datasette-faiss?include_prereleases&label=changelog" alt="Changelog" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/3570cd1979f075f4f1104a2e7a29e9a6fb4b91ac51f9a2be28c2645b0fce4301/68747470733a2f2f696d672e736869656c64732e696f2f6769746875622f762f72656c656173652f73696d6f6e772f6461746173657474652d66616973733f696e636c7564655f70726572656c6561736573266c6162656c3d6368616e67656c6f67"></a>
  <a href="https://github.com/simonw/datasette-faiss/actions?query=workflow%3ATest"><img src="https://github.com/simonw/datasette-faiss/workflows/Test/badge.svg" alt="Tests" style="max-width: 100%;"></a>
  <a href="https://github.com/simonw/datasette-faiss/blob/main/LICENSE"><img src="https://img.shields.io/badge/license-Apache%202.0-blue.svg" alt="License" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/c355f200ea90fddaa407b6eaab303663a669248ea3ca7b1fcf77dbe04ff5f48c/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6c6963656e73652d417061636865253230322e302d626c75652e737667"></a></p>
  <p dir="auto">Maintain a <a href="https://github.com/facebookresearch/faiss">FAISS index</a> for specified Datasette tables</p>
  <p dir="auto">See <a href="https://simonwillison.net/2023/Jan/13/semantic-search-answers/" rel="nofollow">Semantic search answers: Q&A against documentation with GPT3 + OpenAI embeddings</a> for background on this project.</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Installation</h2><a id="user-content-installation" class="anchor" aria-label="Permalink: Installation" href="#installation"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">Install this plugin in the same environment as Datasette.</p>
  <div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="datasette install datasette-faiss"><pre>datasette install datasette-faiss</pre></div>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Usage</h2><a id="user-content-usage" class="anchor" aria-label="Permalink: Usage" href="#usage"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">This plugin creates in-memory FAISS indexes for specified tables on startup, using an <code>IndexFlatL2</code> <a href="https://github.com/facebookresearch/faiss/wiki/Faiss-indexes">FAISS index type</a>.</p>
  <p dir="auto">If the tables are modified after the server has started the indexes will not (yet) pick up those changes.</p>
  <div class="markdown-heading" dir="auto"><h3 class="heading-element" dir="auto">Configuration</h3><a id="user-content-configuration" class="anchor" aria-label="Permalink: Configuration" href="#configuration"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">The tables to be indexed must have <code>id</code> and <code>embedding</code> columns. The <code>embedding</code> column must be a <code>blob</code> containing embeddings that are arrays of floating point numbers that have been encoded using the following Python function:</p>
  <div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="def encode(vector):
      return struct.pack("f" * len(vector), *vector)"><pre><span class="pl-k">def</span> <span class="pl-en">encode</span>(<span class="pl-s1">vector</span>):
      <span class="pl-k">return</span> <span class="pl-s1">struct</span>.<span class="pl-c1">pack</span>(<span class="pl-s">"f"</span> <span class="pl-c1">*</span> <span class="pl-en">len</span>(<span class="pl-s1">vector</span>), <span class="pl-c1">*</span><span class="pl-s1">vector</span>)</pre></div>
  <p dir="auto">You can import that function from this package like so:</p>
  <div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="from datasette_faiss import encode"><pre><span class="pl-k">from</span> <span class="pl-s1">datasette_faiss</span> <span class="pl-k">import</span> <span class="pl-s1">encode</span></pre></div>
  <p dir="auto">You can specify which tables should have indexes created for them by adding this to <code>metadata.json</code>:</p>
  <div class="highlight highlight-source-json notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="{
      "plugins": {
          "datasette-faiss": {
              "tables": [
                  ["blog", "embeddings"]
              ]
          }
      }
  }"><pre>{
      <span class="pl-ent">"plugins"</span>: {
          <span class="pl-ent">"datasette-faiss"</span>: {
              <span class="pl-ent">"tables"</span>: [
                  [<span class="pl-s"><span class="pl-pds">"</span>blog<span class="pl-pds">"</span></span>, <span class="pl-s"><span class="pl-pds">"</span>embeddings<span class="pl-pds">"</span></span>]
              ]
          }
      }
  }</pre></div>
  <p dir="auto">Each table is an array listing the database name and the table name.</p>
  <p dir="auto">If you are using <code>metadata.yml</code> the configuration should look like this:</p>
  <div class="highlight highlight-source-yaml notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="plugins:
    datasette-faiss:
      tables:
      - ["blog", "embeddings"]"><pre><span class="pl-ent">plugins</span>:
    <span class="pl-ent">datasette-faiss</span>:
      <span class="pl-ent">tables</span>:
      - <span class="pl-s">["blog", "embeddings"]</span></pre></div>
  <div class="markdown-heading" dir="auto"><h3 class="heading-element" dir="auto">SQL functions</h3><a id="user-content-sql-functions" class="anchor" aria-label="Permalink: SQL functions" href="#sql-functions"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">The plugin makes four new SQL functions available within Datasette:</p>
  <div class="markdown-heading" dir="auto"><h4 class="heading-element" dir="auto">faiss_search(database, table, embedding, k)</h4><a id="user-content-faiss_searchdatabase-table-embedding-k" class="anchor" aria-label="Permalink: faiss_search(database, table, embedding, k)" href="#faiss_searchdatabase-table-embedding-k"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">Returns the <code>k</code> nearest neighbors to the <code>embedding</code> found in the specified database and table. For example:</p>
  <div class="highlight highlight-source-sql notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="select faiss_search('blog', 'embeddings', (select embedding from embeddings where id = 3), 5)"><pre><span class="pl-k">select</span> faiss_search(<span class="pl-s"><span class="pl-pds">'</span>blog<span class="pl-pds">'</span></span>, <span class="pl-s"><span class="pl-pds">'</span>embeddings<span class="pl-pds">'</span></span>, (<span class="pl-k">select</span> embedding <span class="pl-k">from</span> embeddings <span class="pl-k">where</span> id <span class="pl-k">=</span> <span class="pl-c1">3</span>), <span class="pl-c1">5</span>)</pre></div>
  <p dir="auto">This will return a JSON array of the five IDs of the records in the <code>embeddings</code> table in the <code>blog</code> database that are closest to the specified embedding. The returned value looks like this:</p>
  <div class="highlight highlight-source-json notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="["1", "1249", "1011", "5", "10"]"><pre>[<span class="pl-s"><span class="pl-pds">"</span>1<span class="pl-pds">"</span></span>, <span class="pl-s"><span class="pl-pds">"</span>1249<span class="pl-pds">"</span></span>, <span class="pl-s"><span class="pl-pds">"</span>1011<span class="pl-pds">"</span></span>, <span class="pl-s"><span class="pl-pds">"</span>5<span class="pl-pds">"</span></span>, <span class="pl-s"><span class="pl-pds">"</span>10<span class="pl-pds">"</span></span>]</pre></div>
  <p dir="auto">You can use the SQLite <code>json_each()</code> function to turn that into a table-like sequence that you can join against.</p>
  <p dir="auto">Here's an example query that does that:</p>
  <div class="highlight highlight-source-sql notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="with related as (
    select value from json_each(
      faiss_search(
        'blog',
        'embeddings',
        (select embedding from embeddings limit 1),
        5
      )
    )
  )
  select * from blog_entry, related
  where id = value"><pre>with related <span class="pl-k">as</span> (
    <span class="pl-k">select</span> value <span class="pl-k">from</span> json_each(
      faiss_search(
        <span class="pl-s"><span class="pl-pds">'</span>blog<span class="pl-pds">'</span></span>,
        <span class="pl-s"><span class="pl-pds">'</span>embeddings<span class="pl-pds">'</span></span>,
        (<span class="pl-k">select</span> embedding <span class="pl-k">from</span> embeddings <span class="pl-k">limit</span> <span class="pl-c1">1</span>),
        <span class="pl-c1">5</span>
      )
    )
  )
  <span class="pl-k">select</span> <span class="pl-k">*</span> <span class="pl-k">from</span> blog_entry, related
  <span class="pl-k">where</span> id <span class="pl-k">=</span> value</pre></div>
  <div class="markdown-heading" dir="auto"><h4 class="heading-element" dir="auto">faiss_search_with_scores(database, table, embedding, k)</h4><a id="user-content-faiss_search_with_scoresdatabase-table-embedding-k" class="anchor" aria-label="Permalink: faiss_search_with_scores(database, table, embedding, k)" href="#faiss_search_with_scoresdatabase-table-embedding-k"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">Takes the same arguments as above, but the return value is a JSON array of pairs, each with an ID and a score - something like this:</p>
  <div class="highlight highlight-source-json notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="[
      ["1", 0.0],
      ["1249", 0.21042244136333466],
      ["1011", 0.29391372203826904],
      ["5", 0.29505783319473267],
      ["10", 0.31554925441741943]
  ]"><pre>[
      [<span class="pl-s"><span class="pl-pds">"</span>1<span class="pl-pds">"</span></span>, <span class="pl-c1">0.0</span>],
      [<span class="pl-s"><span class="pl-pds">"</span>1249<span class="pl-pds">"</span></span>, <span class="pl-c1">0.21042244136333466</span>],
      [<span class="pl-s"><span class="pl-pds">"</span>1011<span class="pl-pds">"</span></span>, <span class="pl-c1">0.29391372203826904</span>],
      [<span class="pl-s"><span class="pl-pds">"</span>5<span class="pl-pds">"</span></span>, <span class="pl-c1">0.29505783319473267</span>],
      [<span class="pl-s"><span class="pl-pds">"</span>10<span class="pl-pds">"</span></span>, <span class="pl-c1">0.31554925441741943</span>]
  ]</pre></div>
  <div class="markdown-heading" dir="auto"><h4 class="heading-element" dir="auto">faiss_encode(json_vector)</h4><a id="user-content-faiss_encodejson_vector" class="anchor" aria-label="Permalink: faiss_encode(json_vector)" href="#faiss_encodejson_vector"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">Given a JSON array of floats, returns the binary embedding blob that can be used with the other functions:</p>
  <div class="highlight highlight-source-sql notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="select faiss_encode('[2.4, 4.1, 1.8]')
  -- Returns a 12 byte blob
  select hex(faiss_encode('[2.4, 4.1, 1.8]'))
  -- Returns 9A991940333383406666E63F"><pre><span class="pl-k">select</span> faiss_encode(<span class="pl-s"><span class="pl-pds">'</span>[2.4, 4.1, 1.8]<span class="pl-pds">'</span></span>)
  <span class="pl-c"><span class="pl-c">--</span> Returns a 12 byte blob</span>
  <span class="pl-k">select</span> hex(faiss_encode(<span class="pl-s"><span class="pl-pds">'</span>[2.4, 4.1, 1.8]<span class="pl-pds">'</span></span>))
  <span class="pl-c"><span class="pl-c">--</span> Returns 9A991940333383406666E63F</span></pre></div>
  <div class="markdown-heading" dir="auto"><h4 class="heading-element" dir="auto">faiss_decode(vector_blob)</h4><a id="user-content-faiss_decodevector_blob" class="anchor" aria-label="Permalink: faiss_decode(vector_blob)" href="#faiss_decodevector_blob"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">The opposite of <code>faiss_encode()</code>.</p>
  <div class="highlight highlight-source-sql notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="select faiss_decode(X'9A991940333383406666E63F')"><pre><span class="pl-k">select</span> faiss_decode(X<span class="pl-s"><span class="pl-pds">'</span>9A991940333383406666E63F<span class="pl-pds">'</span></span>)</pre></div>
  <p dir="auto">Returns:</p>
  <div class="highlight highlight-source-json notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="[2.4000000953674316, 4.099999904632568, 1.7999999523162842]"><pre>[<span class="pl-c1">2.4000000953674316</span>, <span class="pl-c1">4.099999904632568</span>, <span class="pl-c1">1.7999999523162842</span>]</pre></div>
  <p dir="auto">Note that floating point arithmetic results in numbers that don't quite round-trip to the exact same expected value.</p>
  <div class="markdown-heading" dir="auto"><h4 class="heading-element" dir="auto">faiss_agg(id, embedding, compare_embedding, k)</h4><a id="user-content-faiss_aggid-embedding-compare_embedding-k" class="anchor" aria-label="Permalink: faiss_agg(id, embedding, compare_embedding, k)" href="#faiss_aggid-embedding-compare_embedding-k"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">This aggregate function can be used to find the <code>k</code> nearest neighbors to <code>compare_embedding</code> for each unique value of <code>id</code> in the table. For example:</p>
  <div class="highlight highlight-source-sql notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="select faiss_agg(
      id, embedding, (select embedding from embeddings where id = 3), 5
  ) from embeddings"><pre><span class="pl-k">select</span> faiss_agg(
      id, embedding, (<span class="pl-k">select</span> embedding <span class="pl-k">from</span> embeddings <span class="pl-k">where</span> id <span class="pl-k">=</span> <span class="pl-c1">3</span>), <span class="pl-c1">5</span>
  ) <span class="pl-k">from</span> embeddings</pre></div>
  <p dir="auto">Unlike the <code>faiss_search()</code> function, this does not depend on the per-table index that the plugin creates when it first starts running. Instead, an index is built every time the aggregation function is run.</p>
  <p dir="auto">This means that it should only be used on smaller sets of values - once you get above 10,000 or so the performance from this function is likely to become prohibitively expensive.</p>
  <p dir="auto">The function returns a JSON array of IDs representing the <code>k</code> rows with the closest distance scores, like this:</p>
  <div class="highlight highlight-source-json notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="[1324, 344, 5562, 553, 2534]"><pre>[<span class="pl-c1">1324</span>, <span class="pl-c1">344</span>, <span class="pl-c1">5562</span>, <span class="pl-c1">553</span>, <span class="pl-c1">2534</span>]</pre></div>
  <p dir="auto">You can use the <code>json_each()</code> function to turn that into a table-like sequence that you can join against.</p>
  <p dir="auto"><a href="https://datasette.simonwillison.net/simonwillisonblog?sql=with+last_500+as+%28%0D%0A++select%0D%0A++++id%2C%0D%0A++++embedding%0D%0A++from%0D%0A++++blog_entry_embeddings%0D%0A++order+by%0D%0A++++id+desc%0D%0A++limit%0D%0A++++500%0D%0A%29%2C+faiss+as+%28%0D%0A++select%0D%0A++++faiss_agg%28%0D%0A++++++id%2C%0D%0A++++++embedding%2C%0D%0A++++++%28%0D%0A++++++++select%0D%0A++++++++++embedding%0D%0A++++++++from%0D%0A++++++++++blog_entry_embeddings%0D%0A++++++++where%0D%0A++++++++++id+%3D+%3Aid%0D%0A++++++%29%2C%0D%0A++++++10%0D%0A++++%29+as+results%0D%0A++from%0D%0A++++last_500%0D%0A%29%2C%0D%0Aids+as+%28%0D%0A++select%0D%0A++++value+as+id%0D%0A++from%0D%0A++++json_each%28faiss.results%29%2C%0D%0A++++faiss%0D%0A%29%0D%0Aselect%0D%0A++blog_entry.id%2C%0D%0A++blog_entry.title%2C%0D%0A++blog_entry.created%0D%0Afrom%0D%0A++ids%0D%0A++join+blog_entry+on+ids.id+%3D+blog_entry.id&id=8214" rel="nofollow">Try an example fais_agg() query</a>.</p>
  <div class="markdown-heading" dir="auto"><h4 class="heading-element" dir="auto">faiss_agg_with_scores(id, embedding, compare_embedding, k)</h4><a id="user-content-faiss_agg_with_scoresid-embedding-compare_embedding-k" class="anchor" aria-label="Permalink: faiss_agg_with_scores(id, embedding, compare_embedding, k)" href="#faiss_agg_with_scoresid-embedding-compare_embedding-k"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">This is similar to the <code>faiss_agg()</code> aggregate function but it returns a list of pairs, each with an ID and the corresponding score - something that looks like this (if <code>k</code> was 2):</p>
  <div class="highlight highlight-source-json notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="[[2412, 0.25], [1245, 24.25]]"><pre>[[<span class="pl-c1">2412</span>, <span class="pl-c1">0.25</span>], [<span class="pl-c1">1245</span>, <span class="pl-c1">24.25</span>]]</pre></div>
  <p dir="auto"><a href="https://datasette.simonwillison.net/simonwillisonblog?sql=with+last_500+as+%28%0D%0A++select%0D%0A++++id%2C%0D%0A++++embedding%0D%0A++from%0D%0A++++blog_entry_embeddings%0D%0A++order+by%0D%0A++++id+desc%0D%0A++limit%0D%0A++++500%0D%0A%29%2C+ids_and_scores+as+%28%0D%0A++select%0D%0A++++faiss_agg_with_scores%28%0D%0A++++++id%2C%0D%0A++++++embedding%2C%0D%0A++++++%28%0D%0A++++++++select%0D%0A++++++++++embedding%0D%0A++++++++from%0D%0A++++++++++blog_entry_embeddings%0D%0A++++++++where%0D%0A++++++++++id+%3D+%3Aid%0D%0A++++++%29%2C+10%0D%0A++++%29+as+s%0D%0A++from%0D%0A++++last_500%0D%0A%29%2C%0D%0Aresults+as+%28%0D%0A++select%0D%0A++++json_extract%28value%2C+%27%24%5B0%5D%27%29+as+id%2C%0D%0A++++json_extract%28value%2C+%27%24%5B1%5D%27%29+as+score%0D%0A++from%0D%0A++++json_each%28ids_and_scores.s%29%2C%0D%0A++++ids_and_scores%0D%0A%29%0D%0Aselect%0D%0A++results.score%2C%0D%0A++blog_entry.id%2C%0D%0A++blog_entry.title%2C%0D%0A++blog_entry.created%0D%0Afrom%0D%0A++results%0D%0A++join+blog_entry+on+results.id+%3D+blog_entry.id&id=8214" rel="nofollow">Try an example fais_agg_with_scores() query</a>.</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Development</h2><a id="user-content-development" class="anchor" aria-label="Permalink: Development" href="#development"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">To set up this plugin locally, first checkout the code. Then create a new virtual environment:</p>
  <div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="cd datasette-faiss
  python3 -m venv venv
  source venv/bin/activate"><pre><span class="pl-c1">cd</span> datasette-faiss
  python3 -m venv venv
  <span class="pl-c1">source</span> venv/bin/activate</pre></div>
  <p dir="auto">Now install the dependencies and test dependencies:</p>
  <div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="pip install -e '.[test]'"><pre>pip install -e <span class="pl-s"><span class="pl-pds">'</span>.[test]<span class="pl-pds">'</span></span></pre></div>
  <p dir="auto">To run the tests:</p>
  <div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="pytest"><pre>pytest</pre></div>
  </article></div>


node_id: R_kgDOIwsDBg
name: datasette-cookies-for-magic-parameters
full_name: simonw/datasette-cookies-for-magic-parameters
html_url: https://github.com/simonw/datasette-cookies-for-magic-parameters
description: UI for setting cookies to populate magic parameters
created_at: 2023-01-11T22:47:31Z
updated_at: 2023-01-16T19:11:46Z
pushed_at: 2023-01-12T19:04:37Z
homepage: https://datasette.io/plugins/datasette-cookies-for-magic-parameters
language: Python
license: apache-2.0
topics: ["datasette", "datasette-plugin"]
visibility: public
default_branch: main
permissions: {"admin": false, "maintain": false, "push": false, "triage": false, "pull": true}
readme:
  # datasette-cookies-for-magic-parameters

  [![PyPI](https://img.shields.io/pypi/v/datasette-cookies-for-magic-parameters.svg)](https://pypi.org/project/datasette-cookies-for-magic-parameters/)
  [![Changelog](https://img.shields.io/github/v/release/simonw/datasette-cookies-for-magic-parameters?include_prereleases&label=changelog)](https://github.com/simonw/datasette-cookies-for-magic-parameters/releases)
  [![Tests](https://github.com/simonw/datasette-cookies-for-magic-parameters/workflows/Test/badge.svg)](https://github.com/simonw/datasette-cookies-for-magic-parameters/actions?query=workflow%3ATest)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/datasette-cookies-for-magic-parameters/blob/main/LICENSE)

  UI for setting cookies to populate magic parameters

  ## Installation

  Install this plugin in the same environment as Datasette.

      datasette install datasette-cookies-for-magic-parameters

  ## Usage

  This plugin only affects canned queries. If you have a canned query configured like this:

  ```yaml
  databases:
    mydatabase:
      queries:
        api_query:
          sql: select 'Your API key is' || :_cookie_openai_api_token;
  ```
  Then the `/mydatabase/api_query` page will include a new interface for setting the `openapi_api_token` cookie, or unsetting it if it has already been set.

  ## Why use this?

  This plugin was initially developed to use with [datasette-openai](https://github.com/simonw/datasette-openai) - a plugin that provides custom SQL functions that take an API token as one of their parameters.

  Passing these API tokens in a GET query string is unsafe, as they may leak through referrers or other log files.

  Instead, this plugin enables them to be set and passed using a cookie, which is much less likely to be logged.

  This pattern takes advantage of the `:_cookie_x` feature of Datasette's [magic parameters](https://docs.datasette.io/en/stable/sql_queries.html#magic-parameters) mechanism.

  ## Development

  To set up this plugin locally, first checkout the code. Then create a new virtual environment:

      cd datasette-cookies-for-magic-parameters
      python3 -m venv venv
      source venv/bin/activate

  Now install the dependencies and test dependencies:

      pip install -e '.[test]'

  To run the tests:

      pytest

readme_html:
  <div id="readme" class="md" data-path="README.md"><article class="markdown-body entry-content container-lg" itemprop="text"><div class="markdown-heading" dir="auto"><h1 class="heading-element" dir="auto">datasette-cookies-for-magic-parameters</h1><a id="user-content-datasette-cookies-for-magic-parameters" class="anchor" aria-label="Permalink: datasette-cookies-for-magic-parameters" href="#datasette-cookies-for-magic-parameters"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto"><a href="https://pypi.org/project/datasette-cookies-for-magic-parameters/" rel="nofollow"><img src="https://img.shields.io/pypi/v/datasette-cookies-for-magic-parameters.svg" alt="PyPI" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/db04ae598a82e1008d0f8ba21a56820474f960ea2ea7f0fbc8365edb6ac83da6/68747470733a2f2f696d672e736869656c64732e696f2f707970692f762f6461746173657474652d636f6f6b6965732d666f722d6d616769632d706172616d65746572732e737667"></a>
  <a href="https://github.com/simonw/datasette-cookies-for-magic-parameters/releases"><img src="https://img.shields.io/github/v/release/simonw/datasette-cookies-for-magic-parameters?include_prereleases&label=changelog" alt="Changelog" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/891b5ecad0484ef6ff41c491a776ccb5c244eb35a22d9379e564f3f2e677b61b/68747470733a2f2f696d672e736869656c64732e696f2f6769746875622f762f72656c656173652f73696d6f6e772f6461746173657474652d636f6f6b6965732d666f722d6d616769632d706172616d65746572733f696e636c7564655f70726572656c6561736573266c6162656c3d6368616e67656c6f67"></a>
  <a href="https://github.com/simonw/datasette-cookies-for-magic-parameters/actions?query=workflow%3ATest"><img src="https://github.com/simonw/datasette-cookies-for-magic-parameters/workflows/Test/badge.svg" alt="Tests" style="max-width: 100%;"></a>
  <a href="https://github.com/simonw/datasette-cookies-for-magic-parameters/blob/main/LICENSE"><img src="https://img.shields.io/badge/license-Apache%202.0-blue.svg" alt="License" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/c355f200ea90fddaa407b6eaab303663a669248ea3ca7b1fcf77dbe04ff5f48c/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6c6963656e73652d417061636865253230322e302d626c75652e737667"></a></p>
  <p dir="auto">UI for setting cookies to populate magic parameters</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Installation</h2><a id="user-content-installation" class="anchor" aria-label="Permalink: Installation" href="#installation"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">Install this plugin in the same environment as Datasette.</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="datasette install datasette-cookies-for-magic-parameters"><pre class="notranslate"><code>datasette install datasette-cookies-for-magic-parameters
  </code></pre></div>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Usage</h2><a id="user-content-usage" class="anchor" aria-label="Permalink: Usage" href="#usage"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">This plugin only affects canned queries. If you have a canned query configured like this:</p>
  <div class="highlight highlight-source-yaml notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="databases:
    mydatabase:
      queries:
        api_query:
          sql: select 'Your API key is' || :_cookie_openai_api_token;"><pre><span class="pl-ent">databases</span>:
    <span class="pl-ent">mydatabase</span>:
      <span class="pl-ent">queries</span>:
        <span class="pl-ent">api_query</span>:
          <span class="pl-ent">sql</span>: <span class="pl-s">select 'Your API key is' || :_cookie_openai_api_token;</span></pre></div>
  <p dir="auto">Then the <code>/mydatabase/api_query</code> page will include a new interface for setting the <code>openapi_api_token</code> cookie, or unsetting it if it has already been set.</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Why use this?</h2><a id="user-content-why-use-this" class="anchor" aria-label="Permalink: Why use this?" href="#why-use-this"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">This plugin was initially developed to use with <a href="https://github.com/simonw/datasette-openai">datasette-openai</a> - a plugin that provides custom SQL functions that take an API token as one of their parameters.</p>
  <p dir="auto">Passing these API tokens in a GET query string is unsafe, as they may leak through referrers or other log files.</p>
  <p dir="auto">Instead, this plugin enables them to be set and passed using a cookie, which is much less likely to be logged.</p>
  <p dir="auto">This pattern takes advantage of the <code>:_cookie_x</code> feature of Datasette's <a href="https://docs.datasette.io/en/stable/sql_queries.html#magic-parameters" rel="nofollow">magic parameters</a> mechanism.</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Development</h2><a id="user-content-development" class="anchor" aria-label="Permalink: Development" href="#development"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">To set up this plugin locally, first checkout the code. Then create a new virtual environment:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="cd datasette-cookies-for-magic-parameters
  python3 -m venv venv
  source venv/bin/activate"><pre class="notranslate"><code>cd datasette-cookies-for-magic-parameters
  python3 -m venv venv
  source venv/bin/activate
  </code></pre></div>
  <p dir="auto">Now install the dependencies and test dependencies:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="pip install -e '.[test]'"><pre class="notranslate"><code>pip install -e '.[test]'
  </code></pre></div>
  <p dir="auto">To run the tests:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="pytest"><pre class="notranslate"><code>pytest
  </code></pre></div>
  </article></div>


node_id: R_kgDOI1rqCA
name: datasette-matomo
full_name: CentreForDigitalHumanities/datasette-matomo
html_url: https://github.com/CentreForDigitalHumanities/datasette-matomo
description: Add Matomo Web Analytics JavaScript tracking code to Datasette
created_at: 2023-01-25T11:35:22Z
updated_at: 2023-09-28T12:12:20Z
pushed_at: 2023-12-20T17:03:13Z
language: Python
license: apache-2.0
topics: ["datasette", "datasette-plugin", "matomo", "web-analytics"]
visibility: public
default_branch: develop
permissions: {"admin": false, "maintain": false, "push": false, "triage": false, "pull": true}
custom_properties: {"Allow-ruleset-bypass": "false", "Team": ["Research Software Lab"]}
readme:
  # datasette-matomo

  [![PyPI](https://img.shields.io/pypi/v/datasette-matomo.svg)](https://pypi.org/project/datasette-matomo/)
  [![Changelog](https://img.shields.io/github/v/release/UUDigitalHumanitieslab/datasette-matomo?include_prereleases&label=changelog)](https://github.com/UUDigitalHumanitieslab/datasette-matomo/releases)
  [![Tests](https://github.com/UUDigitalHumanitieslab/datasette-matomo/workflows/Test/badge.svg)](https://github.com/UUDigitalHumanitieslab/datasette-matomo/actions?query=workflow%3ATest)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/UUDigitalHumanitieslab/datasette-matomo/blob/main/LICENSE)

  [Datasette plugin](https://datasette.io/) that adds [Matomo Web Analytics](https://matomo.org/) JavaScript tracking code to a Datasette application.

  ## Installation

  Install this plugin in the same environment as Datasette.

      datasette install datasette-matomo

  ## Usage

  Make your configuration available to the Datasette process as environment variables:

      DATASETTE_MATOMO_SERVER_URL=https://example.com/
      DATASETTE_MATOMO_SITE_ID=1

  The server URL must include a final slash.

  If you use `datasette publish`:

      datasette publish heroku example.db --install datasette-matomo \
          --plugin-secret datasette-matomo server_url http://example.com/ \
          --plugin-secret datasette-matomo site_id 1

  Note: not all `datasette publish` providers currently support the `--plugin-secret` option.

  ## Development

  To set up this plugin locally, first checkout the code. Then create a new virtual environment:

      cd datasette-matomo
      python3 -m venv venv
      source venv/bin/activate

  Now install the dependencies and test dependencies:

      pip install -e '.[test]'

  To run the tests:

      pytest

readme_html:
  <div id="readme" class="md" data-path="README.md"><article class="markdown-body entry-content container-lg" itemprop="text"><div class="markdown-heading" dir="auto"><h1 class="heading-element" dir="auto">datasette-matomo</h1><a id="user-content-datasette-matomo" class="anchor" aria-label="Permalink: datasette-matomo" href="#datasette-matomo"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto"><a href="https://pypi.org/project/datasette-matomo/" rel="nofollow"><img src="https://img.shields.io/pypi/v/datasette-matomo.svg" alt="PyPI" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/558e01601a3abe179c64534bdc97ebc149127a52bbbef708c7f87f0353d07e4c/68747470733a2f2f696d672e736869656c64732e696f2f707970692f762f6461746173657474652d6d61746f6d6f2e737667"></a>
  <a href="https://github.com/UUDigitalHumanitieslab/datasette-matomo/releases"><img src="https://img.shields.io/github/v/release/UUDigitalHumanitieslab/datasette-matomo?include_prereleases&label=changelog" alt="Changelog" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/58ac8c2cbde68763c9ee88ab6e73b3749cfb47eb5f20213c844d1da86e8f7ded/68747470733a2f2f696d672e736869656c64732e696f2f6769746875622f762f72656c656173652f55554469676974616c48756d616e69746965736c61622f6461746173657474652d6d61746f6d6f3f696e636c7564655f70726572656c6561736573266c6162656c3d6368616e67656c6f67"></a>
  <a href="https://github.com/UUDigitalHumanitieslab/datasette-matomo/actions?query=workflow%3ATest"><img src="https://github.com/UUDigitalHumanitieslab/datasette-matomo/workflows/Test/badge.svg" alt="Tests" style="max-width: 100%;"></a>
  <a href="https://github.com/UUDigitalHumanitieslab/datasette-matomo/blob/main/LICENSE"><img src="https://img.shields.io/badge/license-Apache%202.0-blue.svg" alt="License" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/c355f200ea90fddaa407b6eaab303663a669248ea3ca7b1fcf77dbe04ff5f48c/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6c6963656e73652d417061636865253230322e302d626c75652e737667"></a></p>
  <p dir="auto"><a href="https://datasette.io/" rel="nofollow">Datasette plugin</a> that adds <a href="https://matomo.org/" rel="nofollow">Matomo Web Analytics</a> JavaScript tracking code to a Datasette application.</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Installation</h2><a id="user-content-installation" class="anchor" aria-label="Permalink: Installation" href="#installation"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">Install this plugin in the same environment as Datasette.</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="datasette install datasette-matomo"><pre class="notranslate"><code>datasette install datasette-matomo
  </code></pre></div>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Usage</h2><a id="user-content-usage" class="anchor" aria-label="Permalink: Usage" href="#usage"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">Make your configuration available to the Datasette process as environment variables:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="DATASETTE_MATOMO_SERVER_URL=https://example.com/
  DATASETTE_MATOMO_SITE_ID=1"><pre class="notranslate"><code>DATASETTE_MATOMO_SERVER_URL=https://example.com/
  DATASETTE_MATOMO_SITE_ID=1
  </code></pre></div>
  <p dir="auto">The server URL must include a final slash.</p>
  <p dir="auto">If you use <code>datasette publish</code>:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="datasette publish heroku example.db --install datasette-matomo \
      --plugin-secret datasette-matomo server_url http://example.com/ \
      --plugin-secret datasette-matomo site_id 1"><pre class="notranslate"><code>datasette publish heroku example.db --install datasette-matomo \
      --plugin-secret datasette-matomo server_url http://example.com/ \
      --plugin-secret datasette-matomo site_id 1
  </code></pre></div>
  <p dir="auto">Note: not all <code>datasette publish</code> providers currently support the <code>--plugin-secret</code> option.</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Development</h2><a id="user-content-development" class="anchor" aria-label="Permalink: Development" href="#development"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">To set up this plugin locally, first checkout the code. Then create a new virtual environment:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="cd datasette-matomo
  python3 -m venv venv
  source venv/bin/activate"><pre class="notranslate"><code>cd datasette-matomo
  python3 -m venv venv
  source venv/bin/activate
  </code></pre></div>
  <p dir="auto">Now install the dependencies and test dependencies:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="pip install -e '.[test]'"><pre class="notranslate"><code>pip install -e '.[test]'
  </code></pre></div>
  <p dir="auto">To run the tests:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="pytest"><pre class="notranslate"><code>pytest
  </code></pre></div>
  </article></div>


node_id: R_kgDOI2nvlA
name: datasette-youtube-embed
full_name: simonw/datasette-youtube-embed
html_url: https://github.com/simonw/datasette-youtube-embed
description: Turn YouTube URLs into embedded players in Datasette
created_at: 2023-01-27T17:52:05Z
updated_at: 2026-02-04T16:47:28Z
pushed_at: 2026-02-04T16:47:07Z
language: JavaScript
license: apache-2.0
topics: []
visibility: public
default_branch: main
permissions: {"admin": false, "maintain": false, "push": false, "triage": false, "pull": false}
pull_request_creation_policy: all
readme:
  # datasette-youtube-embed

  [![PyPI](https://img.shields.io/pypi/v/datasette-youtube-embed.svg)](https://pypi.org/project/datasette-youtube-embed/)
  [![Changelog](https://img.shields.io/github/v/release/simonw/datasette-youtube-embed?include_prereleases&label=changelog)](https://github.com/simonw/datasette-youtube-embed/releases)
  [![Tests](https://github.com/simonw/datasette-youtube-embed/workflows/Test/badge.svg)](https://github.com/simonw/datasette-youtube-embed/actions?query=workflow%3ATest)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/datasette-youtube-embed/blob/main/LICENSE)

  Turn YouTube URLs into embedded players in Datasette

  ## Installation

  Install this plugin in the same environment as Datasette.

      datasette install datasette-youtube-embed

  ## Usage

  Once installed, any columns containing URLs that look like one of these:

  - `https://www.youtube.com/watch?v=xyz`
  - `https://www.youtube.com/watch?v=xyz&start=30`
  - `https://www.youtube.com/watch?v=xyz&start=30&end=50`

  Will be turned into YouTube embeds like this:

  ```html
  <lite-youtube videoid="xyz" params="start=30" style="min-width: 200px"></lite-youtube>
  ```
  These will then be rendered using [Lite YouTube Embed](https://github.com/paulirish/lite-youtube-embed), which avoids loading the full YouTube embed until someone interacts with it.

  ## Development

  To set up this plugin locally, first checkout the code. Then create a new virtual environment:

      cd datasette-youtube-embed
      python3 -m venv venv
      source venv/bin/activate

  Now install the dependencies and test dependencies:

      pip install -e '.[test]'

  To run the tests:

      pytest

readme_html:
  <div id="readme" class="md" data-path="README.md"><article class="markdown-body entry-content container-lg" itemprop="text"><div class="markdown-heading" dir="auto"><h1 class="heading-element" dir="auto">datasette-youtube-embed</h1><a id="user-content-datasette-youtube-embed" class="anchor" aria-label="Permalink: datasette-youtube-embed" href="#datasette-youtube-embed"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto"><a href="https://pypi.org/project/datasette-youtube-embed/" rel="nofollow"><img src="https://img.shields.io/pypi/v/datasette-youtube-embed.svg" alt="PyPI" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/ae6787d7aea761f637f2c9a18279fb4fabf7eba26a3cfd73a594c5db573e75a1/68747470733a2f2f696d672e736869656c64732e696f2f707970692f762f6461746173657474652d796f75747562652d656d6265642e737667"></a>
  <a href="https://github.com/simonw/datasette-youtube-embed/releases"><img src="https://img.shields.io/github/v/release/simonw/datasette-youtube-embed?include_prereleases&label=changelog" alt="Changelog" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/6186fb01fd90df9e4e06f2ce4fda647dc9724f5a6d5163ff1ecb6bf0a5d3be93/68747470733a2f2f696d672e736869656c64732e696f2f6769746875622f762f72656c656173652f73696d6f6e772f6461746173657474652d796f75747562652d656d6265643f696e636c7564655f70726572656c6561736573266c6162656c3d6368616e67656c6f67"></a>
  <a href="https://github.com/simonw/datasette-youtube-embed/actions?query=workflow%3ATest"><img src="https://github.com/simonw/datasette-youtube-embed/workflows/Test/badge.svg" alt="Tests" style="max-width: 100%;"></a>
  <a href="https://github.com/simonw/datasette-youtube-embed/blob/main/LICENSE"><img src="https://img.shields.io/badge/license-Apache%202.0-blue.svg" alt="License" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/c355f200ea90fddaa407b6eaab303663a669248ea3ca7b1fcf77dbe04ff5f48c/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6c6963656e73652d417061636865253230322e302d626c75652e737667"></a></p>
  <p dir="auto">Turn YouTube URLs into embedded players in Datasette</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Installation</h2><a id="user-content-installation" class="anchor" aria-label="Permalink: Installation" href="#installation"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">Install this plugin in the same environment as Datasette.</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="datasette install datasette-youtube-embed"><pre class="notranslate"><code>datasette install datasette-youtube-embed
  </code></pre></div>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Usage</h2><a id="user-content-usage" class="anchor" aria-label="Permalink: Usage" href="#usage"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">Once installed, any columns containing URLs that look like one of these:</p>
  <ul dir="auto">
  <li><code>https://www.youtube.com/watch?v=xyz</code></li>
  <li><code>https://www.youtube.com/watch?v=xyz&start=30</code></li>
  <li><code>https://www.youtube.com/watch?v=xyz&start=30&end=50</code></li>
  </ul>
  <p dir="auto">Will be turned into YouTube embeds like this:</p>
  <div class="highlight highlight-text-html-basic notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="<lite-youtube videoid="xyz" params="start=30" style="min-width: 200px"></lite-youtube>"><pre><span class="pl-kos"><</span><span class="pl-ent">lite-youtube</span> <span class="pl-c1">videoid</span>="<span class="pl-s">xyz</span>" <span class="pl-c1">params</span>="<span class="pl-s">start=30</span>" <span class="pl-c1">style</span>="<span class="pl-s">min-width: 200px</span>"<span class="pl-kos">></span><span class="pl-kos"></</span><span class="pl-ent">lite-youtube</span><span class="pl-kos">></span></pre></div>
  <p dir="auto">These will then be rendered using <a href="https://github.com/paulirish/lite-youtube-embed">Lite YouTube Embed</a>, which avoids loading the full YouTube embed until someone interacts with it.</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Development</h2><a id="user-content-development" class="anchor" aria-label="Permalink: Development" href="#development"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">To set up this plugin locally, first checkout the code. Then create a new virtual environment:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="cd datasette-youtube-embed
  python3 -m venv venv
  source venv/bin/activate"><pre class="notranslate"><code>cd datasette-youtube-embed
  python3 -m venv venv
  source venv/bin/activate
  </code></pre></div>
  <p dir="auto">Now install the dependencies and test dependencies:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="pip install -e '.[test]'"><pre class="notranslate"><code>pip install -e '.[test]'
  </code></pre></div>
  <p dir="auto">To run the tests:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="pytest"><pre class="notranslate"><code>pytest
  </code></pre></div>
  </article></div>


node_id: R_kgDOI7mujg
name: datasette-explain
full_name: simonw/datasette-explain
html_url: https://github.com/simonw/datasette-explain
description: Explain and validate SQL queries as you type them into Datasette
created_at: 2023-02-09T01:51:11Z
updated_at: 2025-03-11T05:28:40Z
pushed_at: 2024-02-28T16:55:57Z
language: Python
license: apache-2.0
topics: ["datasette", "datasette-plugin"]
visibility: public
default_branch: main
permissions: {"admin": false, "maintain": false, "push": false, "triage": false, "pull": true}
readme:
  # datasette-explain

  [![PyPI](https://img.shields.io/pypi/v/datasette-explain.svg)](https://pypi.org/project/datasette-explain/)
  [![Changelog](https://img.shields.io/github/v/release/simonw/datasette-explain?include_prereleases&label=changelog)](https://github.com/simonw/datasette-explain/releases)
  [![Tests](https://github.com/simonw/datasette-explain/workflows/Test/badge.svg)](https://github.com/simonw/datasette-explain/actions?query=workflow%3ATest)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/datasette-explain/blob/main/LICENSE)

  Explain SQL queries executed using Datasette

  ## Installation

  Install this plugin in the same environment as Datasette.
  ```bash
  datasette install datasette-explain
  ```
  ## Usage

  The plugin adds JavaScript to the query editor page which will constantly update the page with information gained from running EXPLAIN QUERY PLAN queries against the entered SQL.

  This may result in an error message, or it may show the query plan along with any tables used by the query.

  ## Demo

  You can see this plugin in action on [datasette.simonwillison.net](https://datasette.simonwillison.net/simonwillisonblog?sql=select%0D%0A++blog_tag.tag%2C%0D%0A++blog_blogmark.link_title%2C%0D%0A++blog_blogmark.link_url%0D%0Afrom%0D%0A++blog_blogmark_tags%0D%0A++join+blog_tag+on+tag_id+%3D+blog_tag.id%0D%0A++join+blog_blogmark+on+blog_blogmark_tags.blogmark_id+%3D+blog_blogmark.id%0D%0Aorder+by+blog_blogmark.id+desc).

  <img src="https://raw.githubusercontent.com/simonw/datasette-explain/main/datasette-explain-screenshot.jpg" alt="Demo of this plugin - a SQL query shows the explain plan as well as the three tables that were used in the query (their names and columns)" width="832">

  ## Development

  To set up this plugin locally, first checkout the code. Then create a new virtual environment:
  ```bash
  cd datasette-explain
  python3 -m venv venv
  source venv/bin/activate
  ```
  Now install the dependencies and test dependencies:
  ```bash
  pip install -e '.[test]'
  ```
  To run the tests:
  ```bash
  pytest
  ```

readme_html:
  <div id="readme" class="md" data-path="README.md"><article class="markdown-body entry-content container-lg" itemprop="text"><div class="markdown-heading" dir="auto"><h1 class="heading-element" dir="auto">datasette-explain</h1><a id="user-content-datasette-explain" class="anchor" aria-label="Permalink: datasette-explain" href="#datasette-explain"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto"><a href="https://pypi.org/project/datasette-explain/" rel="nofollow"><img src="https://img.shields.io/pypi/v/datasette-explain.svg" alt="PyPI" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/83f94b9a62bebc6dde4cd11dafcb62ac2395d5b8c6a9a121e74dca2ea1ff8b65/68747470733a2f2f696d672e736869656c64732e696f2f707970692f762f6461746173657474652d6578706c61696e2e737667"></a>
  <a href="https://github.com/simonw/datasette-explain/releases"><img src="https://img.shields.io/github/v/release/simonw/datasette-explain?include_prereleases&label=changelog" alt="Changelog" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/fff768e9c4ac68b3eaae3dd05d8e557793832ac1969315d3f8de2bd7f187d9e0/68747470733a2f2f696d672e736869656c64732e696f2f6769746875622f762f72656c656173652f73696d6f6e772f6461746173657474652d6578706c61696e3f696e636c7564655f70726572656c6561736573266c6162656c3d6368616e67656c6f67"></a>
  <a href="https://github.com/simonw/datasette-explain/actions?query=workflow%3ATest"><img src="https://github.com/simonw/datasette-explain/workflows/Test/badge.svg" alt="Tests" style="max-width: 100%;"></a>
  <a href="https://github.com/simonw/datasette-explain/blob/main/LICENSE"><img src="https://img.shields.io/badge/license-Apache%202.0-blue.svg" alt="License" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/c355f200ea90fddaa407b6eaab303663a669248ea3ca7b1fcf77dbe04ff5f48c/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6c6963656e73652d417061636865253230322e302d626c75652e737667"></a></p>
  <p dir="auto">Explain SQL queries executed using Datasette</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Installation</h2><a id="user-content-installation" class="anchor" aria-label="Permalink: Installation" href="#installation"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">Install this plugin in the same environment as Datasette.</p>
  <div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="datasette install datasette-explain"><pre>datasette install datasette-explain</pre></div>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Usage</h2><a id="user-content-usage" class="anchor" aria-label="Permalink: Usage" href="#usage"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">The plugin adds JavaScript to the query editor page which will constantly update the page with information gained from running EXPLAIN QUERY PLAN queries against the entered SQL.</p>
  <p dir="auto">This may result in an error message, or it may show the query plan along with any tables used by the query.</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Demo</h2><a id="user-content-demo" class="anchor" aria-label="Permalink: Demo" href="#demo"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">You can see this plugin in action on <a href="https://datasette.simonwillison.net/simonwillisonblog?sql=select%0D%0A++blog_tag.tag%2C%0D%0A++blog_blogmark.link_title%2C%0D%0A++blog_blogmark.link_url%0D%0Afrom%0D%0A++blog_blogmark_tags%0D%0A++join+blog_tag+on+tag_id+%3D+blog_tag.id%0D%0A++join+blog_blogmark+on+blog_blogmark_tags.blogmark_id+%3D+blog_blogmark.id%0D%0Aorder+by+blog_blogmark.id+desc" rel="nofollow">datasette.simonwillison.net</a>.</p>
  <p dir="auto"><a target="_blank" rel="noopener noreferrer nofollow" href="https://raw.githubusercontent.com/simonw/datasette-explain/main/datasette-explain-screenshot.jpg"><img src="https://raw.githubusercontent.com/simonw/datasette-explain/main/datasette-explain-screenshot.jpg" alt="Demo of this plugin - a SQL query shows the explain plan as well as the three tables that were used in the query (their names and columns)" width="832" style="max-width: 100%;"></a></p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Development</h2><a id="user-content-development" class="anchor" aria-label="Permalink: Development" href="#development"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">To set up this plugin locally, first checkout the code. Then create a new virtual environment:</p>
  <div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="cd datasette-explain
  python3 -m venv venv
  source venv/bin/activate"><pre><span class="pl-c1">cd</span> datasette-explain
  python3 -m venv venv
  <span class="pl-c1">source</span> venv/bin/activate</pre></div>
  <p dir="auto">Now install the dependencies and test dependencies:</p>
  <div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="pip install -e '.[test]'"><pre>pip install -e <span class="pl-s"><span class="pl-pds">'</span>.[test]<span class="pl-pds">'</span></span></pre></div>
  <p dir="auto">To run the tests:</p>
  <div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="pytest"><pre>pytest</pre></div>
  </article></div>


node_id: R_kgDOJBM_kA
name: datasette-codespaces
full_name: simonw/datasette-codespaces
html_url: https://github.com/simonw/datasette-codespaces
description: Conveniences for running Datasette on GitHub Codespaces
created_at: 2023-02-22T18:43:18Z
updated_at: 2025-05-05T14:59:31Z
pushed_at: 2023-02-24T01:05:32Z
language: Python
license: apache-2.0
topics: []
visibility: public
default_branch: main
permissions: {"admin": false, "maintain": false, "push": false, "triage": false, "pull": true}
template_repository:
[cell repeats an earlier one in this file: 6377 characters]

readme:
  # datasette-codespaces

  [![PyPI](https://img.shields.io/pypi/v/datasette-codespaces.svg)](https://pypi.org/project/datasette-codespaces/)
  [![Changelog](https://img.shields.io/github/v/release/simonw/datasette-codespaces?include_prereleases&label=changelog)](https://github.com/simonw/datasette-codespaces/releases)
  [![Tests](https://github.com/simonw/datasette-codespaces/workflows/Test/badge.svg)](https://github.com/simonw/datasette-codespaces/actions?query=workflow%3ATest)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/datasette-codespaces/blob/main/LICENSE)

   Conveniences for running [Datasette](https://datasette.io/) on [GitHub Codespaces](https://github.com/features/codespaces)

  ## Installation

  Install this plugin in the same environment as Datasette.

      datasette install datasette-codespaces

  ## Tutorial

  [Using Datasette in GitHub Codespaces](https://datasette.io/tutorials/codespaces) provides a detailed tutorial introduction to this plugin.

  ## Usage

  Install this when you are using Datasette inside of GitHub Codespaces. It makes the following changes:

  - All requests will identified as representing the `root` actor - provided Datasette is running inside Codespaces. This is detected through the presence of the `CODESPACE_NAME` environment variable. GitHub restricts access to servers running in the Codespaces environment based on GitHub authentication, so treating all requests as root should be acceptably secure.
  - The [datasette-x-forwarded-host](https://datasette.io/plugins/datasette-x-forwarded-host) plugin will be installed, ensuring links to other pages within Datasette such as facet navigation work correctly.

  ## Development

  To set up this plugin locally, first checkout the code. Then create a new virtual environment:

      cd datasette-codespaces
      python3 -m venv venv
      source venv/bin/activate

  Now install the dependencies and test dependencies:

      pip install -e '.[test]'

  To run the tests:

      pytest

readme_html:
  <div id="readme" class="md" data-path="README.md"><article class="markdown-body entry-content container-lg" itemprop="text"><div class="markdown-heading" dir="auto"><h1 class="heading-element" dir="auto">datasette-codespaces</h1><a id="user-content-datasette-codespaces" class="anchor" aria-label="Permalink: datasette-codespaces" href="#datasette-codespaces"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto"><a href="https://pypi.org/project/datasette-codespaces/" rel="nofollow"><img src="https://img.shields.io/pypi/v/datasette-codespaces.svg" alt="PyPI" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/6acf9e338804a0028d1b32e5429d4b36e7edfd9e28333a784ed175e0ae3269f6/68747470733a2f2f696d672e736869656c64732e696f2f707970692f762f6461746173657474652d636f64657370616365732e737667"></a>
  <a href="https://github.com/simonw/datasette-codespaces/releases"><img src="https://img.shields.io/github/v/release/simonw/datasette-codespaces?include_prereleases&label=changelog" alt="Changelog" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/d926ea7a829a10582400ddde772a10ca0e1f9c9d74056b29c648ccef0009569b/68747470733a2f2f696d672e736869656c64732e696f2f6769746875622f762f72656c656173652f73696d6f6e772f6461746173657474652d636f64657370616365733f696e636c7564655f70726572656c6561736573266c6162656c3d6368616e67656c6f67"></a>
  <a href="https://github.com/simonw/datasette-codespaces/actions?query=workflow%3ATest"><img src="https://github.com/simonw/datasette-codespaces/workflows/Test/badge.svg" alt="Tests" style="max-width: 100%;"></a>
  <a href="https://github.com/simonw/datasette-codespaces/blob/main/LICENSE"><img src="https://img.shields.io/badge/license-Apache%202.0-blue.svg" alt="License" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/c355f200ea90fddaa407b6eaab303663a669248ea3ca7b1fcf77dbe04ff5f48c/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6c6963656e73652d417061636865253230322e302d626c75652e737667"></a></p>
  <p dir="auto">Conveniences for running <a href="https://datasette.io/" rel="nofollow">Datasette</a> on <a href="https://github.com/features/codespaces">GitHub Codespaces</a></p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Installation</h2><a id="user-content-installation" class="anchor" aria-label="Permalink: Installation" href="#installation"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">Install this plugin in the same environment as Datasette.</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="datasette install datasette-codespaces"><pre class="notranslate"><code>datasette install datasette-codespaces
  </code></pre></div>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Tutorial</h2><a id="user-content-tutorial" class="anchor" aria-label="Permalink: Tutorial" href="#tutorial"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto"><a href="https://datasette.io/tutorials/codespaces" rel="nofollow">Using Datasette in GitHub Codespaces</a> provides a detailed tutorial introduction to this plugin.</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Usage</h2><a id="user-content-usage" class="anchor" aria-label="Permalink: Usage" href="#usage"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">Install this when you are using Datasette inside of GitHub Codespaces. It makes the following changes:</p>
  <ul dir="auto">
  <li>All requests will identified as representing the <code>root</code> actor - provided Datasette is running inside Codespaces. This is detected through the presence of the <code>CODESPACE_NAME</code> environment variable. GitHub restricts access to servers running in the Codespaces environment based on GitHub authentication, so treating all requests as root should be acceptably secure.</li>
  <li>The <a href="https://datasette.io/plugins/datasette-x-forwarded-host" rel="nofollow">datasette-x-forwarded-host</a> plugin will be installed, ensuring links to other pages within Datasette such as facet navigation work correctly.</li>
  </ul>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Development</h2><a id="user-content-development" class="anchor" aria-label="Permalink: Development" href="#development"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">To set up this plugin locally, first checkout the code. Then create a new virtual environment:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="cd datasette-codespaces
  python3 -m venv venv
  source venv/bin/activate"><pre class="notranslate"><code>cd datasette-codespaces
  python3 -m venv venv
  source venv/bin/activate
  </code></pre></div>
  <p dir="auto">Now install the dependencies and test dependencies:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="pip install -e '.[test]'"><pre class="notranslate"><code>pip install -e '.[test]'
  </code></pre></div>
  <p dir="auto">To run the tests:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="pytest"><pre class="notranslate"><code>pytest
  </code></pre></div>
  </article></div>


node_id: R_kgDOJDyulQ
name: datasette-simple-html
full_name: simonw/datasette-simple-html
html_url: https://github.com/simonw/datasette-simple-html
description: Datasette SQL functions for very simple HTML operations
created_at: 2023-03-01T02:29:26Z
updated_at: 2023-03-01T02:36:47Z
pushed_at: 2023-03-12T17:30:47Z
language: Python
license: apache-2.0
topics: ["datasette-plugin"]
visibility: public
default_branch: main
permissions: {"admin": false, "maintain": false, "push": false, "triage": false, "pull": true}
readme:
  # datasette-simple-html

  [![PyPI](https://img.shields.io/pypi/v/datasette-simple-html.svg)](https://pypi.org/project/datasette-simple-html/)
  [![Changelog](https://img.shields.io/github/v/release/simonw/datasette-simple-html?include_prereleases&label=changelog)](https://github.com/simonw/datasette-simple-html/releases)
  [![Tests](https://github.com/simonw/datasette-simple-html/workflows/Test/badge.svg)](https://github.com/simonw/datasette-simple-html/actions?query=workflow%3ATest)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/datasette-simple-html/blob/main/LICENSE)

  Datasette SQL functions for very simple HTML operations

  ## Installation

  Install this plugin in the same environment as Datasette.

      datasette install datasette-simple-html

  ## Usage

  This plugin provides the following SQL functions:

  ### html_strip_tags(text)

  Returns the text with any `<...>` tags removed.

  ### html_escape(text)

  Escapes any HTML special characters in the text, e.g. `>` becomes `&gt;`. Uses [html.escape(text, quote=True)](https://docs.python.org/3/library/html.html#html.escape) from the Python standard library.

  ### html_unescape(text)

  Unescapes any HTML special characters, so `&gt;` becomes `>`. Also handles numeric entities, so `&#x27;` becomes `'`. Uses [html.unescape(text)](https://docs.python.org/3/library/html.html#html.unescape) from the Python standard library.

  ## Development

  To set up this plugin locally, first checkout the code. Then create a new virtual environment:

      cd datasette-simple-html
      python3 -m venv venv
      source venv/bin/activate

  Now install the dependencies and test dependencies:

      pip install -e '.[test]'

  To run the tests:

      pytest

readme_html:
  <div id="readme" class="md" data-path="README.md"><article class="markdown-body entry-content container-lg" itemprop="text"><div class="markdown-heading" dir="auto"><h1 class="heading-element" dir="auto">datasette-simple-html</h1><a id="user-content-datasette-simple-html" class="anchor" aria-label="Permalink: datasette-simple-html" href="#datasette-simple-html"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto"><a href="https://pypi.org/project/datasette-simple-html/" rel="nofollow"><img src="https://img.shields.io/pypi/v/datasette-simple-html.svg" alt="PyPI" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/829e4d0c9a2f5c736a5785b9afb292fc9b752be4ee0a7887458cc130d07c55ba/68747470733a2f2f696d672e736869656c64732e696f2f707970692f762f6461746173657474652d73696d706c652d68746d6c2e737667"></a>
  <a href="https://github.com/simonw/datasette-simple-html/releases"><img src="https://img.shields.io/github/v/release/simonw/datasette-simple-html?include_prereleases&label=changelog" alt="Changelog" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/284ab3b199714c52a90aba4813089e96754c7f91246d7c4c87777ef9d15a8396/68747470733a2f2f696d672e736869656c64732e696f2f6769746875622f762f72656c656173652f73696d6f6e772f6461746173657474652d73696d706c652d68746d6c3f696e636c7564655f70726572656c6561736573266c6162656c3d6368616e67656c6f67"></a>
  <a href="https://github.com/simonw/datasette-simple-html/actions?query=workflow%3ATest"><img src="https://github.com/simonw/datasette-simple-html/workflows/Test/badge.svg" alt="Tests" style="max-width: 100%;"></a>
  <a href="https://github.com/simonw/datasette-simple-html/blob/main/LICENSE"><img src="https://img.shields.io/badge/license-Apache%202.0-blue.svg" alt="License" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/c355f200ea90fddaa407b6eaab303663a669248ea3ca7b1fcf77dbe04ff5f48c/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6c6963656e73652d417061636865253230322e302d626c75652e737667"></a></p>
  <p dir="auto">Datasette SQL functions for very simple HTML operations</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Installation</h2><a id="user-content-installation" class="anchor" aria-label="Permalink: Installation" href="#installation"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">Install this plugin in the same environment as Datasette.</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="datasette install datasette-simple-html"><pre class="notranslate"><code>datasette install datasette-simple-html
  </code></pre></div>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Usage</h2><a id="user-content-usage" class="anchor" aria-label="Permalink: Usage" href="#usage"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">This plugin provides the following SQL functions:</p>
  <div class="markdown-heading" dir="auto"><h3 class="heading-element" dir="auto">html_strip_tags(text)</h3><a id="user-content-html_strip_tagstext" class="anchor" aria-label="Permalink: html_strip_tags(text)" href="#html_strip_tagstext"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">Returns the text with any <code><...></code> tags removed.</p>
  <div class="markdown-heading" dir="auto"><h3 class="heading-element" dir="auto">html_escape(text)</h3><a id="user-content-html_escapetext" class="anchor" aria-label="Permalink: html_escape(text)" href="#html_escapetext"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">Escapes any HTML special characters in the text, e.g. <code>></code> becomes <code>&gt;</code>. Uses <a href="https://docs.python.org/3/library/html.html#html.escape" rel="nofollow">html.escape(text, quote=True)</a> from the Python standard library.</p>
  <div class="markdown-heading" dir="auto"><h3 class="heading-element" dir="auto">html_unescape(text)</h3><a id="user-content-html_unescapetext" class="anchor" aria-label="Permalink: html_unescape(text)" href="#html_unescapetext"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">Unescapes any HTML special characters, so <code>&gt;</code> becomes <code>></code>. Also handles numeric entities, so <code>&#x27;</code> becomes <code>'</code>. Uses <a href="https://docs.python.org/3/library/html.html#html.unescape" rel="nofollow">html.unescape(text)</a> from the Python standard library.</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Development</h2><a id="user-content-development" class="anchor" aria-label="Permalink: Development" href="#development"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">To set up this plugin locally, first checkout the code. Then create a new virtual environment:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="cd datasette-simple-html
  python3 -m venv venv
  source venv/bin/activate"><pre class="notranslate"><code>cd datasette-simple-html
  python3 -m venv venv
  source venv/bin/activate
  </code></pre></div>
  <p dir="auto">Now install the dependencies and test dependencies:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="pip install -e '.[test]'"><pre class="notranslate"><code>pip install -e '.[test]'
  </code></pre></div>
  <p dir="auto">To run the tests:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="pytest"><pre class="notranslate"><code>pytest
  </code></pre></div>
  </article></div>


node_id: R_kgDOJHON9g
name: apple-notes-to-sqlite
full_name: dogsheep/apple-notes-to-sqlite
html_url: https://github.com/dogsheep/apple-notes-to-sqlite
description: Export Apple Notes to SQLite
created_at: 2023-03-09T03:44:10Z
updated_at: 2025-05-04T02:54:46Z
pushed_at: 2023-09-04T23:48:28Z
language: Python
license: apache-2.0
topics: ["apple-notes", "datasette", "datasette-tool", "dogsheep", "sqlite"]
visibility: public
default_branch: main
permissions: {"admin": false, "maintain": false, "push": false, "triage": false, "pull": true}
custom_properties: {}
readme:
  # apple-notes-to-sqlite

  [![PyPI](https://img.shields.io/pypi/v/apple-notes-to-sqlite.svg)](https://pypi.org/project/apple-notes-to-sqlite/)
  [![Changelog](https://img.shields.io/github/v/release/dogsheep/apple-notes-to-sqlite?include_prereleases&label=changelog)](https://github.com/dogsheep/apple-notes-to-sqlite/releases)
  [![Tests](https://github.com/dogsheep/apple-notes-to-sqlite/workflows/Test/badge.svg)](https://github.com/dogsheep/apple-notes-to-sqlite/actions?query=workflow%3ATest)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/dogsheep/apple-notes-to-sqlite/blob/master/LICENSE)

  Export Apple Notes to SQLite

  ## Installation

  Install this tool using `pip`:

      pip install apple-notes-to-sqlite

  ## Usage

  For help, run:

      apple-notes-to-sqlite --help

  You can also use:

      python -m apple_notes_to_sqlite --help

  To save your notes to a SQLite database called `notes.db` run:

      apple-notes-to-sqlite notes.db

  A progress bar will be shown.

  You can stop it after a specified number of notes using `--stop-after`:

      apple-notes-to-sqlite notes.db --stop-after 10

  To dump the notes to standard output out as newline-delimited JSON, rather than saving them to a database, use the `--dump` option:

      apple-notes-to-sqlite --dump

  ## Schema

  The database schema generated by this tool looks like this:

  <!-- [[[cog
  import cog, json
  from apple_notes_to_sqlite import cli
  from click.testing import CliRunner
  import sqlite_utils
  import tempfile, pathlib
  tmpdir = pathlib.Path(tempfile.mkdtemp())
  db_path = str(tmpdir / "temp.db")
  runner = CliRunner()
  result = runner.invoke(cli.cli, [db_path, "--schema"])
  cog.out("```sql\n")
  cog.out(sqlite_utils.Database(db_path).schema)
  cog.out("\n```")
  ]]] -->
  ```sql
  CREATE TABLE [folders] (
     [id] INTEGER PRIMARY KEY,
     [long_id] TEXT,
     [name] TEXT,
     [parent] INTEGER,
     FOREIGN KEY([parent]) REFERENCES [folders]([id])
  );
  CREATE TABLE [notes] (
     [id] TEXT PRIMARY KEY,
     [created] TEXT,
     [updated] TEXT,
     [folder] INTEGER,
     [title] TEXT,
     [body] TEXT,
     FOREIGN KEY([folder]) REFERENCES [folders]([id])
  );
  CREATE UNIQUE INDEX [idx_folders_long_id]
      ON [folders] ([long_id]);
  ```
  <!-- [[[end]]] -->

  ### apple-notes-to-sqlite --help

  <!-- [[[cog
  import cog
  from apple_notes_to_sqlite import cli
  from click.testing import CliRunner
  runner = CliRunner()
  result = runner.invoke(cli.cli, ["--help"])
  help = result.output.replace("Usage: cli", "Usage: apple-notes-to-sqlite")
  cog.out(
      "```\n{}\n```".format(help)
  )
  ]]] -->
  ```
  Usage: apple-notes-to-sqlite [OPTIONS] [DB_PATH]

    Export Apple Notes to SQLite

    Example usage:

        apple-notes-to-sqlite notes.db

    This will populate notes.db with 'notes' and 'folders' tables containing all
    of your notes.

  Options:
    --version             Show the version and exit.
    --stop-after INTEGER  Stop after this many notes
    --dump                Output notes to standard output
    --schema              Create database schema and exit
    --help                Show this message and exit.

  ```
  <!-- [[[end]]] -->


  ## Development

  To contribute to this tool, first checkout the code. Then create a new virtual environment:

      cd apple-notes-to-sqlite
      python -m venv venv
      source venv/bin/activate

  Now install the dependencies and test dependencies:

      pip install -e '.[test]'

  To run the tests:

      pytest

readme_html:
  <div id="readme" class="md" data-path="README.md"><article class="markdown-body entry-content container-lg" itemprop="text"><div class="markdown-heading" dir="auto"><h1 class="heading-element" dir="auto">apple-notes-to-sqlite</h1><a id="user-content-apple-notes-to-sqlite" class="anchor" aria-label="Permalink: apple-notes-to-sqlite" href="#apple-notes-to-sqlite"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto"><a href="https://pypi.org/project/apple-notes-to-sqlite/" rel="nofollow"><img src="https://img.shields.io/pypi/v/apple-notes-to-sqlite.svg" alt="PyPI" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/5faddb7c90f9c2487843077fa354c9a48365b7dd7ec6e6050404660b59d11afe/68747470733a2f2f696d672e736869656c64732e696f2f707970692f762f6170706c652d6e6f7465732d746f2d73716c6974652e737667"></a>
  <a href="https://github.com/dogsheep/apple-notes-to-sqlite/releases"><img src="https://img.shields.io/github/v/release/dogsheep/apple-notes-to-sqlite?include_prereleases&label=changelog" alt="Changelog" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/44895c747bf460b32e36ec855f5d2dd0995e3924defd7a45c69372d93ab5b97c/68747470733a2f2f696d672e736869656c64732e696f2f6769746875622f762f72656c656173652f646f6773686565702f6170706c652d6e6f7465732d746f2d73716c6974653f696e636c7564655f70726572656c6561736573266c6162656c3d6368616e67656c6f67"></a>
  <a href="https://github.com/dogsheep/apple-notes-to-sqlite/actions?query=workflow%3ATest"><img src="https://github.com/dogsheep/apple-notes-to-sqlite/workflows/Test/badge.svg" alt="Tests" style="max-width: 100%;"></a>
  <a href="https://github.com/dogsheep/apple-notes-to-sqlite/blob/master/LICENSE"><img src="https://img.shields.io/badge/license-Apache%202.0-blue.svg" alt="License" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/c355f200ea90fddaa407b6eaab303663a669248ea3ca7b1fcf77dbe04ff5f48c/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6c6963656e73652d417061636865253230322e302d626c75652e737667"></a></p>
  <p dir="auto">Export Apple Notes to SQLite</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Installation</h2><a id="user-content-installation" class="anchor" aria-label="Permalink: Installation" href="#installation"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">Install this tool using <code>pip</code>:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="pip install apple-notes-to-sqlite"><pre class="notranslate"><code>pip install apple-notes-to-sqlite
  </code></pre></div>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Usage</h2><a id="user-content-usage" class="anchor" aria-label="Permalink: Usage" href="#usage"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">For help, run:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="apple-notes-to-sqlite --help"><pre class="notranslate"><code>apple-notes-to-sqlite --help
  </code></pre></div>
  <p dir="auto">You can also use:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="python -m apple_notes_to_sqlite --help"><pre class="notranslate"><code>python -m apple_notes_to_sqlite --help
  </code></pre></div>
  <p dir="auto">To save your notes to a SQLite database called <code>notes.db</code> run:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="apple-notes-to-sqlite notes.db"><pre class="notranslate"><code>apple-notes-to-sqlite notes.db
  </code></pre></div>
  <p dir="auto">A progress bar will be shown.</p>
  <p dir="auto">You can stop it after a specified number of notes using <code>--stop-after</code>:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="apple-notes-to-sqlite notes.db --stop-after 10"><pre class="notranslate"><code>apple-notes-to-sqlite notes.db --stop-after 10
  </code></pre></div>
  <p dir="auto">To dump the notes to standard output out as newline-delimited JSON, rather than saving them to a database, use the <code>--dump</code> option:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="apple-notes-to-sqlite --dump"><pre class="notranslate"><code>apple-notes-to-sqlite --dump
  </code></pre></div>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Schema</h2><a id="user-content-schema" class="anchor" aria-label="Permalink: Schema" href="#schema"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">The database schema generated by this tool looks like this:</p>

  <div class="highlight highlight-source-sql notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="CREATE TABLE [folders] (
     [id] INTEGER PRIMARY KEY,
     [long_id] TEXT,
     [name] TEXT,
     [parent] INTEGER,
     FOREIGN KEY([parent]) REFERENCES [folders]([id])
  );
  CREATE TABLE [notes] (
     [id] TEXT PRIMARY KEY,
     [created] TEXT,
     [updated] TEXT,
     [folder] INTEGER,
     [title] TEXT,
     [body] TEXT,
     FOREIGN KEY([folder]) REFERENCES [folders]([id])
  );
  CREATE UNIQUE INDEX [idx_folders_long_id]
      ON [folders] ([long_id]);"><pre>CREATE TABLE [folders] (
     [id] <span class="pl-k">INTEGER</span> <span class="pl-k">PRIMARY KEY</span>,
     [long_id] <span class="pl-k">TEXT</span>,
     [name] <span class="pl-k">TEXT</span>,
     [parent] <span class="pl-k">INTEGER</span>,
     <span class="pl-k">FOREIGN KEY</span>([parent]) <span class="pl-k">REFERENCES</span> [folders]([id])
  );
  CREATE TABLE [notes] (
     [id] <span class="pl-k">TEXT</span> <span class="pl-k">PRIMARY KEY</span>,
     [created] <span class="pl-k">TEXT</span>,
     [updated] <span class="pl-k">TEXT</span>,
     [folder] <span class="pl-k">INTEGER</span>,
     [title] <span class="pl-k">TEXT</span>,
     [body] <span class="pl-k">TEXT</span>,
     <span class="pl-k">FOREIGN KEY</span>([folder]) <span class="pl-k">REFERENCES</span> [folders]([id])
  );
  CREATE UNIQUE INDEX [idx_folders_long_id]
      <span class="pl-k">ON</span> [folders] ([long_id]);</pre></div>

  <div class="markdown-heading" dir="auto"><h3 class="heading-element" dir="auto">apple-notes-to-sqlite --help</h3><a id="user-content-apple-notes-to-sqlite---help" class="anchor" aria-label="Permalink: apple-notes-to-sqlite --help" href="#apple-notes-to-sqlite---help"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>

  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="Usage: apple-notes-to-sqlite [OPTIONS] [DB_PATH]

    Export Apple Notes to SQLite

    Example usage:

        apple-notes-to-sqlite notes.db

    This will populate notes.db with 'notes' and 'folders' tables containing all
    of your notes.

  Options:
    --version             Show the version and exit.
    --stop-after INTEGER  Stop after this many notes
    --dump                Output notes to standard output
    --schema              Create database schema and exit
    --help                Show this message and exit.
  "><pre class="notranslate"><code>Usage: apple-notes-to-sqlite [OPTIONS] [DB_PATH]

    Export Apple Notes to SQLite

    Example usage:

        apple-notes-to-sqlite notes.db

    This will populate notes.db with 'notes' and 'folders' tables containing all
    of your notes.

  Options:
    --version             Show the version and exit.
    --stop-after INTEGER  Stop after this many notes
    --dump                Output notes to standard output
    --schema              Create database schema and exit
    --help                Show this message and exit.

  </code></pre></div>

  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Development</h2><a id="user-content-development" class="anchor" aria-label="Permalink: Development" href="#development"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">To contribute to this tool, first checkout the code. Then create a new virtual environment:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="cd apple-notes-to-sqlite
  python -m venv venv
  source venv/bin/activate"><pre class="notranslate"><code>cd apple-notes-to-sqlite
  python -m venv venv
  source venv/bin/activate
  </code></pre></div>
  <p dir="auto">Now install the dependencies and test dependencies:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="pip install -e '.[test]'"><pre class="notranslate"><code>pip install -e '.[test]'
  </code></pre></div>
  <p dir="auto">To run the tests:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="pytest"><pre class="notranslate"><code>pytest
  </code></pre></div>
  </article></div>


node_id: R_kgDOJNnn3g
name: datasette-chatgpt-plugin
full_name: simonw/datasette-chatgpt-plugin
html_url: https://github.com/simonw/datasette-chatgpt-plugin
description: A Datasette plugin that turns a Datasette instance into a ChatGPT plugin
created_at: 2023-03-24T04:46:24Z
updated_at: 2024-09-27T12:30:02Z
pushed_at: 2024-02-28T03:16:01Z
homepage: https://datasette.io/plugins/datasette-chatgpt-plugin
language: Python
license: apache-2.0
topics: ["chatgpt", "chatgpt-plugin", "datasette", "datasette-plugin"]
visibility: public
default_branch: main
permissions: {"admin": false, "maintain": false, "push": false, "triage": false, "pull": true}
template_repository:
[cell repeats an earlier one in this file: 6377 characters]

readme:
  # datasette-chatgpt-plugin

  [![PyPI](https://img.shields.io/pypi/v/datasette-chatgpt-plugin.svg)](https://pypi.org/project/datasette-chatgpt-plugin/)
  [![Changelog](https://img.shields.io/github/v/release/simonw/datasette-chatgpt-plugin?include_prereleases&label=changelog)](https://github.com/simonw/datasette-chatgpt-plugin/releases)
  [![Tests](https://github.com/simonw/datasette-chatgpt-plugin/workflows/Test/badge.svg)](https://github.com/simonw/datasette-chatgpt-plugin/actions?query=workflow%3ATest)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/datasette-chatgpt-plugin/blob/main/LICENSE)

  A Datasette plugin that turns a Datasette instance into a ChatGPT plugin - so you can use ChatGPT to ask questions of your data.

  For more on this project, see [Building a ChatGPT plugin to ask questions of data hosted in Datasette](https://simonwillison.net/2023/Mar/24/datasette-chatgpt-plugin/).

  ⚠️ **Warning**: ChatGPT can still hallucinate results using this plugin! See [this issue](https://github.com/simonw/datasette-chatgpt-plugin/issues/2) for more details of this problem.

  ## Installation

  Install this plugin in the same environment as Datasette, on a deployed instance.

      datasette install datasette-chatgpt-plugin

  Or if you are using `datasette publish` to deploy Datasette:

      datasette publish cloudrun/vercel/fly/heroku data.db \
        --install datasette-chatgpt-plugin

  ## Usage

  Once installed, your Datasette instance will work with the new [ChatGPT plugins](https://openai.com/blog/chatgpt-plugins) system - provided you have access to that preview.

  Click `Plugins -> Plugin store -> Install an unverified plugin` and enter the URL of your Datasette instance.

  If this doesn't work, try `Develop my own plugin -> My manifest is ready` instead and then paste in your URL.

  ChatGPT will discover the plugin by hitting the `/.well-known/ai-plugin.json` endpoint.

  You can then ask it questions! Some starting examples:

  - Show a list of tables (this is always good to start with as it ensures ChatGPT knows which tables are available)
  - Show the first 10 rows of the `mytable` table

  This plugin currently exposes a single database - the first database attached to your instance.

  ## Development

  To set up this plugin locally, first checkout the code. Then create a new virtual environment:

      cd datasette-chatgpt-plugin
      python3 -m venv venv
      source venv/bin/activate

  Now install the dependencies and test dependencies:

      pip install -e '.[test]'

  To run the tests:

      pytest

readme_html:
  <div id="readme" class="md" data-path="README.md"><article class="markdown-body entry-content container-lg" itemprop="text"><div class="markdown-heading" dir="auto"><h1 class="heading-element" dir="auto">datasette-chatgpt-plugin</h1><a id="user-content-datasette-chatgpt-plugin" class="anchor" aria-label="Permalink: datasette-chatgpt-plugin" href="#datasette-chatgpt-plugin"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto"><a href="https://pypi.org/project/datasette-chatgpt-plugin/" rel="nofollow"><img src="https://img.shields.io/pypi/v/datasette-chatgpt-plugin.svg" alt="PyPI" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/ab5063a97c91e067a3b78836553511a60a6a579410060a6702f052819246c002/68747470733a2f2f696d672e736869656c64732e696f2f707970692f762f6461746173657474652d636861746770742d706c7567696e2e737667"></a>
  <a href="https://github.com/simonw/datasette-chatgpt-plugin/releases"><img src="https://img.shields.io/github/v/release/simonw/datasette-chatgpt-plugin?include_prereleases&label=changelog" alt="Changelog" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/f28e2057f93f4ced0bc18be1e38fbcb7178f140ef102f462c60aaece3bc4dc19/68747470733a2f2f696d672e736869656c64732e696f2f6769746875622f762f72656c656173652f73696d6f6e772f6461746173657474652d636861746770742d706c7567696e3f696e636c7564655f70726572656c6561736573266c6162656c3d6368616e67656c6f67"></a>
  <a href="https://github.com/simonw/datasette-chatgpt-plugin/actions?query=workflow%3ATest"><img src="https://github.com/simonw/datasette-chatgpt-plugin/workflows/Test/badge.svg" alt="Tests" style="max-width: 100%;"></a>
  <a href="https://github.com/simonw/datasette-chatgpt-plugin/blob/main/LICENSE"><img src="https://img.shields.io/badge/license-Apache%202.0-blue.svg" alt="License" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/c355f200ea90fddaa407b6eaab303663a669248ea3ca7b1fcf77dbe04ff5f48c/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6c6963656e73652d417061636865253230322e302d626c75652e737667"></a></p>
  <p dir="auto">A Datasette plugin that turns a Datasette instance into a ChatGPT plugin - so you can use ChatGPT to ask questions of your data.</p>
  <p dir="auto">For more on this project, see <a href="https://simonwillison.net/2023/Mar/24/datasette-chatgpt-plugin/" rel="nofollow">Building a ChatGPT plugin to ask questions of data hosted in Datasette</a>.</p>
  <p dir="auto"><g-emoji class="g-emoji" alias="warning">⚠️</g-emoji> <strong>Warning</strong>: ChatGPT can still hallucinate results using this plugin! See <a href="https://github.com/simonw/datasette-chatgpt-plugin/issues/2" data-hovercard-type="issue" data-hovercard-url="/simonw/datasette-chatgpt-plugin/issues/2/hovercard">this issue</a> for more details of this problem.</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Installation</h2><a id="user-content-installation" class="anchor" aria-label="Permalink: Installation" href="#installation"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">Install this plugin in the same environment as Datasette, on a deployed instance.</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="datasette install datasette-chatgpt-plugin"><pre class="notranslate"><code>datasette install datasette-chatgpt-plugin
  </code></pre></div>
  <p dir="auto">Or if you are using <code>datasette publish</code> to deploy Datasette:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="datasette publish cloudrun/vercel/fly/heroku data.db \
    --install datasette-chatgpt-plugin"><pre class="notranslate"><code>datasette publish cloudrun/vercel/fly/heroku data.db \
    --install datasette-chatgpt-plugin
  </code></pre></div>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Usage</h2><a id="user-content-usage" class="anchor" aria-label="Permalink: Usage" href="#usage"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">Once installed, your Datasette instance will work with the new <a href="https://openai.com/blog/chatgpt-plugins" rel="nofollow">ChatGPT plugins</a> system - provided you have access to that preview.</p>
  <p dir="auto">Click <code>Plugins -> Plugin store -> Install an unverified plugin</code> and enter the URL of your Datasette instance.</p>
  <p dir="auto">If this doesn't work, try <code>Develop my own plugin -> My manifest is ready</code> instead and then paste in your URL.</p>
  <p dir="auto">ChatGPT will discover the plugin by hitting the <code>/.well-known/ai-plugin.json</code> endpoint.</p>
  <p dir="auto">You can then ask it questions! Some starting examples:</p>
  <ul dir="auto">
  <li>Show a list of tables (this is always good to start with as it ensures ChatGPT knows which tables are available)</li>
  <li>Show the first 10 rows of the <code>mytable</code> table</li>
  </ul>
  <p dir="auto">This plugin currently exposes a single database - the first database attached to your instance.</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Development</h2><a id="user-content-development" class="anchor" aria-label="Permalink: Development" href="#development"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">To set up this plugin locally, first checkout the code. Then create a new virtual environment:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="cd datasette-chatgpt-plugin
  python3 -m venv venv
  source venv/bin/activate"><pre class="notranslate"><code>cd datasette-chatgpt-plugin
  python3 -m venv venv
  source venv/bin/activate
  </code></pre></div>
  <p dir="auto">Now install the dependencies and test dependencies:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="pip install -e '.[test]'"><pre class="notranslate"><code>pip install -e '.[test]'
  </code></pre></div>
  <p dir="auto">To run the tests:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="pytest"><pre class="notranslate"><code>pytest
  </code></pre></div>
  </article></div>


node_id: R_kgDOJRhX7A
name: llm
full_name: simonw/llm
html_url: https://github.com/simonw/llm
description: Access large language models from the command-line
created_at: 2023-04-01T21:16:57Z
updated_at: 2026-06-11T11:14:02Z
pushed_at: 2026-06-11T08:42:28Z
homepage: https://llm.datasette.io
language: Python
license: apache-2.0
topics: ["ai", "llms", "openai"]
visibility: public
default_branch: main
permissions: {"admin": false, "maintain": false, "push": false, "triage": false, "pull": false}
pull_request_creation_policy: all
readme:
  <!-- [[[cog
  # README.md is generated from docs/index.md using sphinx_markdown_builder
  import tempfile
  import subprocess
  from pathlib import Path

  readme_markdown = ''

  with tempfile.TemporaryDirectory() as tmpdir:
      tmp_path = Path(tmpdir)
      # Run: sphinx-build -M markdown ./docs ./tmpdir
      subprocess.run([
          "sphinx-build",
          "-M", "markdown",
          "./docs",
          str(tmp_path)
      ], check=True)
      index_file = tmp_path / "markdown" / "index.md"
      readme_markdown = index_file.read_text(encoding="utf-8")

  cog.out(readme_markdown)
  ]]] -->
  # LLM

  [![GitHub repo](https://img.shields.io/badge/github-repo-green)](https://github.com/simonw/llm)
  [![PyPI](https://img.shields.io/pypi/v/llm.svg)](https://pypi.org/project/llm/)
  [![Changelog](https://img.shields.io/github/v/release/simonw/llm?include_prereleases&label=changelog)](https://llm.datasette.io/en/stable/changelog.html)
  [![Tests](https://github.com/simonw/llm/workflows/Test/badge.svg)](https://github.com/simonw/llm/actions?query=workflow%3ATest)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/llm/blob/main/LICENSE)
  [![Discord](https://img.shields.io/discord/823971286308356157?label=discord)](https://datasette.io/discord-llm)
  [![Homebrew](https://img.shields.io/homebrew/installs/dy/llm?color=yellow&label=homebrew&logo=homebrew)](https://formulae.brew.sh/formula/llm)

  A CLI tool and Python library for interacting with **OpenAI**, **Anthropic’s Claude**, **Google’s Gemini**, **Meta’s Llama** and dozens of other Large Language Models, both via remote APIs and with models that can be installed and run on your own machine.

  Watch **[Language models on the command-line](https://www.youtube.com/watch?v=QUXQNi6jQ30)** on YouTube for a demo or [read the accompanying detailed notes](https://simonwillison.net/2024/Jun/17/cli-language-models/).

  With LLM you can:

  - [Run prompts from the command-line](https://llm.datasette.io/en/stable/usage.html#usage-executing-prompts)
  - [Store prompts and responses in SQLite](https://llm.datasette.io/en/stable/logging.html#logging)
  - [Generate and store embeddings](https://llm.datasette.io/en/stable/embeddings/index.html#embeddings)
  - [Extract structured content from text and images](https://llm.datasette.io/en/stable/schemas.html#schemas)
  - … and much, much more

  ## Quick start

  First, install LLM using `pip` or Homebrew or `pipx` or `uv`:

  ```bash
  pip install llm
  ```

  Or with Homebrew (see [warning note](https://llm.datasette.io/en/stable/setup.html#homebrew-warning)):

  ```bash
  brew install llm
  ```

  Or with [pipx](https://pypa.github.io/pipx/):

  ```bash
  pipx install llm
  ```

  Or with [uv](https://docs.astral.sh/uv/guides/tools/)

  ```bash
  uv tool install llm
  ```

  If you have an [OpenAI API key](https://platform.openai.com/api-keys) key you can run this:

  ```bash
  # Paste your OpenAI API key into this
  llm keys set openai

  # Run a prompt (with the default gpt-4o-mini model)
  llm "Ten fun names for a pet pelican"

  # Extract text from an image
  llm "extract text" -a scanned-document.jpg

  # Use a system prompt against a file
  cat myfile.py | llm -s "Explain this code"
  ```

  Run prompts against [Gemini](https://aistudio.google.com/apikey) or [Anthropic](https://console.anthropic.com/) with their respective plugins:

  ```bash
  llm install llm-gemini
  llm keys set gemini
  # Paste Gemini API key here
  llm -m gemini-2.0-flash 'Tell me fun facts about Mountain View'

  llm install llm-anthropic
  llm keys set anthropic
  # Paste Anthropic API key here
  llm -m claude-4-opus 'Impress me with wild facts about turnips'
  ```

  You can also [install a plugin](https://llm.datasette.io/en/stable/plugins/installing-plugins.html#installing-plugins) to access models that can run on your local device. If you use [Ollama](https://ollama.com/):

  ```bash
  # Install the plugin
  llm install llm-ollama

  # Download and run a prompt against the Orca Mini 7B model
  ollama pull llama3.2:latest
  llm -m llama3.2:latest 'What is the capital of France?'
  ```

  To start [an interactive chat](https://llm.datasette.io/en/stable/usage.html#usage-chat) with a model, use `llm chat`:

  ```bash
  llm chat -m gpt-4.1
  ```

  ```default
  Chatting with gpt-4.1
  Type 'exit' or 'quit' to exit
  Type '!multi' to enter multiple lines, then '!end' to finish
  Type '!edit' to open your default editor and modify the prompt.
  Type '!fragment <my_fragment> [<another_fragment> ...]' to insert one or more fragments
  > Tell me a joke about a pelican
  Why don't pelicans like to tip waiters?

  Because they always have a big bill!
  ```

  More background on this project:

  - [llm, ttok and strip-tags—CLI tools for working with ChatGPT and other LLMs](https://simonwillison.net/2023/May/18/cli-tools-for-llms/)
  - [The LLM CLI tool now supports self-hosted language models via plugins](https://simonwillison.net/2023/Jul/12/llm/)
  - [LLM now provides tools for working with embeddings](https://simonwillison.net/2023/Sep/4/llm-embeddings/)
  - [Build an image search engine with llm-clip, chat with models with llm chat](https://simonwillison.net/2023/Sep/12/llm-clip-and-chat/)
  - [You can now run prompts against images, audio and video in your terminal using LLM](https://simonwillison.net/2024/Oct/29/llm-multi-modal/)
  - [Structured data extraction from unstructured content using LLM schemas](https://simonwillison.net/2025/Feb/28/llm-schemas/)
  - [Long context support in LLM 0.24 using fragments and template plugins](https://simonwillison.net/2025/Apr/7/long-context-llm/)

  See also [the llm tag](https://simonwillison.net/tags/llm/) on my blog.

  ## Contents

  * [Setup](https://llm.datasette.io/en/stable/setup.html)
    * [Installation](https://llm.datasette.io/en/stable/setup.html#installation)
    * [Upgrading to the latest version](https://llm.datasette.io/en/stable/setup.html#upgrading-to-the-latest-version)
    * [Using uvx](https://llm.datasette.io/en/stable/setup.html#using-uvx)
    * [A note about Homebrew and PyTorch](https://llm.datasette.io/en/stable/setup.html#a-note-about-homebrew-and-pytorch)
    * [Installing plugins](https://llm.datasette.io/en/stable/setup.html#installing-plugins)
    * [API key management](https://llm.datasette.io/en/stable/setup.html#api-key-management)
      * [Saving and using stored keys](https://llm.datasette.io/en/stable/setup.html#saving-and-using-stored-keys)
      * [Passing keys using the –key option](https://llm.datasette.io/en/stable/setup.html#passing-keys-using-the-key-option)
      * [Keys in environment variables](https://llm.datasette.io/en/stable/setup.html#keys-in-environment-variables)
    * [Configuration](https://llm.datasette.io/en/stable/setup.html#configuration)
      * [Setting a custom default model](https://llm.datasette.io/en/stable/setup.html#setting-a-custom-default-model)
      * [Setting a custom directory location](https://llm.datasette.io/en/stable/setup.html#setting-a-custom-directory-location)
      * [Turning SQLite logging on and off](https://llm.datasette.io/en/stable/setup.html#turning-sqlite-logging-on-and-off)
  * [Usage](https://llm.datasette.io/en/stable/usage.html)
    * [Executing a prompt](https://llm.datasette.io/en/stable/usage.html#executing-a-prompt)
      * [Model options](https://llm.datasette.io/en/stable/usage.html#model-options)
      * [Attachments](https://llm.datasette.io/en/stable/usage.html#attachments)
      * [System prompts](https://llm.datasette.io/en/stable/usage.html#system-prompts)
      * [Tools](https://llm.datasette.io/en/stable/usage.html#tools)
      * [Extracting fenced code blocks](https://llm.datasette.io/en/stable/usage.html#extracting-fenced-code-blocks)
      * [Schemas](https://llm.datasette.io/en/stable/usage.html#schemas)
      * [Fragments](https://llm.datasette.io/en/stable/usage.html#fragments)
      * [Continuing a conversation](https://llm.datasette.io/en/stable/usage.html#continuing-a-conversation)
      * [Tips for using LLM with Bash or Zsh](https://llm.datasette.io/en/stable/usage.html#tips-for-using-llm-with-bash-or-zsh)
      * [Completion prompts](https://llm.datasette.io/en/stable/usage.html#completion-prompts)
    * [Starting an interactive chat](https://llm.datasette.io/en/stable/usage.html#starting-an-interactive-chat)
    * [Listing available models](https://llm.datasette.io/en/stable/usage.html#listing-available-models)
    * [Setting default options for models](https://llm.datasette.io/en/stable/usage.html#setting-default-options-for-models)
  * [OpenAI models](https://llm.datasette.io/en/stable/openai-models.html)
    * [Configuration](https://llm.datasette.io/en/stable/openai-models.html#configuration)
    * [OpenAI language models](https://llm.datasette.io/en/stable/openai-models.html#openai-language-models)
    * [Model features](https://llm.datasette.io/en/stable/openai-models.html#model-features)
    * [OpenAI embedding models](https://llm.datasette.io/en/stable/openai-models.html#openai-embedding-models)
    * [OpenAI completion models](https://llm.datasette.io/en/stable/openai-models.html#openai-completion-models)
    * [Adding more OpenAI models](https://llm.datasette.io/en/stable/openai-models.html#adding-more-openai-models)
  * [Other models](https://llm.datasette.io/en/stable/other-models.html)
    * [Installing and using a local model](https://llm.datasette.io/en/stable/other-models.html#installing-and-using-a-local-model)
    * [OpenAI-compatible models](https://llm.datasette.io/en/stable/other-models.html#openai-compatible-models)
      * [Extra HTTP headers](https://llm.datasette.io/en/stable/other-models.html#extra-http-headers)
  * [Tools](https://llm.datasette.io/en/stable/tools.html)
    * [How tools work](https://llm.datasette.io/en/stable/tools.html#how-tools-work)
    * [Trying out tools](https://llm.datasette.io/en/stable/tools.html#trying-out-tools)
    * [LLM’s implementation of tools](https://llm.datasette.io/en/stable/tools.html#llm-s-implementation-of-tools)
    * [Default tools](https://llm.datasette.io/en/stable/tools.html#default-tools)
    * [Tips for implementing tools](https://llm.datasette.io/en/stable/tools.html#tips-for-implementing-tools)
  * [Schemas](https://llm.datasette.io/en/stable/schemas.html)
    * [Schemas tutorial](https://llm.datasette.io/en/stable/schemas.html#schemas-tutorial)
      * [Getting started with dogs](https://llm.datasette.io/en/stable/schemas.html#getting-started-with-dogs)
      * [Extracting people from a news articles](https://llm.datasette.io/en/stable/schemas.html#extracting-people-from-a-news-articles)
    * [Using JSON schemas](https://llm.datasette.io/en/stable/schemas.html#using-json-schemas)
    * [Ways to specify a schema](https://llm.datasette.io/en/stable/schemas.html#ways-to-specify-a-schema)
    * [Concise LLM schema syntax](https://llm.datasette.io/en/stable/schemas.html#concise-llm-schema-syntax)
    * [Saving reusable schemas in templates](https://llm.datasette.io/en/stable/schemas.html#saving-reusable-schemas-in-templates)
    * [Browsing logged JSON objects created using schemas](https://llm.datasette.io/en/stable/schemas.html#browsing-logged-json-objects-created-using-schemas)
  * [Templates](https://llm.datasette.io/en/stable/templates.html)
    * [Getting started with <code>–save</code>](https://llm.datasette.io/en/stable/templates.html#getting-started-with-save)
    * [Using a template](https://llm.datasette.io/en/stable/templates.html#using-a-template)
    * [Listing available templates](https://llm.datasette.io/en/stable/templates.html#listing-available-templates)
    * [Templates as YAML files](https://llm.datasette.io/en/stable/templates.html#templates-as-yaml-files)
      * [System prompts](https://llm.datasette.io/en/stable/templates.html#system-prompts)
      * [Fragments](https://llm.datasette.io/en/stable/templates.html#fragments)
      * [Options](https://llm.datasette.io/en/stable/templates.html#options)
      * [Schemas](https://llm.datasette.io/en/stable/templates.html#schemas)
      * [Additional template variables](https://llm.datasette.io/en/stable/templates.html#additional-template-variables)
      * [Specifying default parameters](https://llm.datasette.io/en/stable/templates.html#specifying-default-parameters)
      * [Configuring code extraction](https://llm.datasette.io/en/stable/templates.html#configuring-code-extraction)
      * [Setting a default model for a template](https://llm.datasette.io/en/stable/templates.html#setting-a-default-model-for-a-template)
    * [Template loaders from plugins](https://llm.datasette.io/en/stable/templates.html#template-loaders-from-plugins)
  * [Fragments](https://llm.datasette.io/en/stable/fragments.html)
    * [Using fragments in a prompt](https://llm.datasette.io/en/stable/fragments.html#using-fragments-in-a-prompt)
    * [Using fragments in chat](https://llm.datasette.io/en/stable/fragments.html#using-fragments-in-chat)
    * [Browsing fragments](https://llm.datasette.io/en/stable/fragments.html#browsing-fragments)
    * [Setting aliases for fragments](https://llm.datasette.io/en/stable/fragments.html#setting-aliases-for-fragments)
    * [Viewing fragments in your logs](https://llm.datasette.io/en/stable/fragments.html#viewing-fragments-in-your-logs)
    * [Using fragments from plugins](https://llm.datasette.io/en/stable/fragments.html#using-fragments-from-plugins)
    * [Listing available fragment prefixes](https://llm.datasette.io/en/stable/fragments.html#listing-available-fragment-prefixes)
  * [Model aliases](https://llm.datasette.io/en/stable/aliases.html)
    * [Listing aliases](https://llm.datasette.io/en/stable/aliases.html#listing-aliases)
    * [Adding a new alias](https://llm.datasette.io/en/stable/aliases.html#adding-a-new-alias)
    * [Removing an alias](https://llm.datasette.io/en/stable/aliases.html#removing-an-alias)
    * [Viewing the aliases file](https://llm.datasette.io/en/stable/aliases.html#viewing-the-aliases-file)
  * [Embeddings](https://llm.datasette.io/en/stable/embeddings/index.html)
    * [Embedding with the CLI](https://llm.datasette.io/en/stable/embeddings/cli.html)
      * [llm embed](https://llm.datasette.io/en/stable/embeddings/cli.html#llm-embed)
      * [llm embed-multi](https://llm.datasette.io/en/stable/embeddings/cli.html#llm-embed-multi)
      * [llm similar](https://llm.datasette.io/en/stable/embeddings/cli.html#llm-similar)
      * [llm embed-models](https://llm.datasette.io/en/stable/embeddings/cli.html#llm-embed-models)
      * [llm collections list](https://llm.datasette.io/en/stable/embeddings/cli.html#llm-collections-list)
      * [llm collections delete](https://llm.datasette.io/en/stable/embeddings/cli.html#llm-collections-delete)
    * [Using embeddings from Python](https://llm.datasette.io/en/stable/embeddings/python-api.html)
      * [Working with collections](https://llm.datasette.io/en/stable/embeddings/python-api.html#working-with-collections)
      * [Retrieving similar items](https://llm.datasette.io/en/stable/embeddings/python-api.html#retrieving-similar-items)
      * [SQL schema](https://llm.datasette.io/en/stable/embeddings/python-api.html#sql-schema)
    * [Writing plugins to add new embedding models](https://llm.datasette.io/en/stable/embeddings/writing-plugins.html)
      * [Embedding binary content](https://llm.datasette.io/en/stable/embeddings/writing-plugins.html#embedding-binary-content)
    * [Embedding storage format](https://llm.datasette.io/en/stable/embeddings/storage.html)
  * [Plugins](https://llm.datasette.io/en/stable/plugins/index.html)
    * [Installing plugins](https://llm.datasette.io/en/stable/plugins/installing-plugins.html)
      * [Listing installed plugins](https://llm.datasette.io/en/stable/plugins/installing-plugins.html#listing-installed-plugins)
      * [Running with a subset of plugins](https://llm.datasette.io/en/stable/plugins/installing-plugins.html#running-with-a-subset-of-plugins)
    * [Plugin directory](https://llm.datasette.io/en/stable/plugins/directory.html)
      * [Local models](https://llm.datasette.io/en/stable/plugins/directory.html#local-models)
      * [Remote APIs](https://llm.datasette.io/en/stable/plugins/directory.html#remote-apis)
      * [Tools](https://llm.datasette.io/en/stable/plugins/directory.html#tools)
      * [Fragments and template loaders](https://llm.datasette.io/en/stable/plugins/directory.html#fragments-and-template-loaders)
      * [Embedding models](https://llm.datasette.io/en/stable/plugins/directory.html#embedding-models)
      * [Extra commands](https://llm.datasette.io/en/stable/plugins/directory.html#extra-commands)
      * [Just for fun](https://llm.datasette.io/en/stable/plugins/directory.html#just-for-fun)
    * [Plugin hooks](https://llm.datasette.io/en/stable/plugins/plugin-hooks.html)
      * [register_commands(cli)](https://llm.datasette.io/en/stable/plugins/plugin-hooks.html#register-commands-cli)
      * [register_models(register)](https://llm.datasette.io/en/stable/plugins/plugin-hooks.html#register-models-register)
      * [register_embedding_models(register)](https://llm.datasette.io/en/stable/plugins/plugin-hooks.html#register-embedding-models-register)
      * [register_tools(register)](https://llm.datasette.io/en/stable/plugins/plugin-hooks.html#register-tools-register)
      * [register_template_loaders(register)](https://llm.datasette.io/en/stable/plugins/plugin-hooks.html#register-template-loaders-register)
      * [register_fragment_loaders(register)](https://llm.datasette.io/en/stable/plugins/plugin-hooks.html#register-fragment-loaders-register)
    * [Developing a model plugin](https://llm.datasette.io/en/stable/plugins/tutorial-model-plugin.html)
      * [The initial structure of the plugin](https://llm.datasette.io/en/stable/plugins/tutorial-model-plugin.html#the-initial-structure-of-the-plugin)
      * [Installing your plugin to try it out](https://llm.datasette.io/en/stable/plugins/tutorial-model-plugin.html#installing-your-plugin-to-try-it-out)
      * [Building the Markov chain](https://llm.datasette.io/en/stable/plugins/tutorial-model-plugin.html#building-the-markov-chain)
      * [Executing the Markov chain](https://llm.datasette.io/en/stable/plugins/tutorial-model-plugin.html#executing-the-markov-chain)
      * [Adding that to the plugin](https://llm.datasette.io/en/stable/plugins/tutorial-model-plugin.html#adding-that-to-the-plugin)
      * [Understanding execute()](https://llm.datasette.io/en/stable/plugins/tutorial-model-plugin.html#understanding-execute)
      * [Prompts and responses are logged to the database](https://llm.datasette.io/en/stable/plugins/tutorial-model-plugin.html#prompts-and-responses-are-logged-to-the-database)
      * [Adding options](https://llm.datasette.io/en/stable/plugins/tutorial-model-plugin.html#adding-options)
      * [Distributing your plugin](https://llm.datasette.io/en/stable/plugins/tutorial-model-plugin.html#distributing-your-plugin)
      * [GitHub repositories](https://llm.datasette.io/en/stable/plugins/tutorial-model-plugin.html#github-repositories)
      * [Publishing plugins to PyPI](https://llm.datasette.io/en/stable/plugins/tutorial-model-plugin.html#publishing-plugins-to-pypi)
      * [Adding metadata](https://llm.datasette.io/en/stable/plugins/tutorial-model-plugin.html#adding-metadata)
      * [What to do if it breaks](https://llm.datasette.io/en/stable/plugins/tutorial-model-plugin.html#what-to-do-if-it-breaks)
    * [Advanced model plugins](https://llm.datasette.io/en/stable/plugins/advanced-model-plugins.html)
      * [Tip: lazily load expensive dependencies](https://llm.datasette.io/en/stable/plugins/advanced-model-plugins.html#tip-lazily-load-expensive-dependencies)
      * [Models that accept API keys](https://llm.datasette.io/en/stable/plugins/advanced-model-plugins.html#models-that-accept-api-keys)
      * [Async models](https://llm.datasette.io/en/stable/plugins/advanced-model-plugins.html#async-models)
      * [Supporting schemas](https://llm.datasette.io/en/stable/plugins/advanced-model-plugins.html#supporting-schemas)
      * [Supporting tools](https://llm.datasette.io/en/stable/plugins/advanced-model-plugins.html#supporting-tools)
      * [Attachments for multi-modal models](https://llm.datasette.io/en/stable/plugins/advanced-model-plugins.html#attachments-for-multi-modal-models)
      * [Tracking token usage](https://llm.datasette.io/en/stable/plugins/advanced-model-plugins.html#tracking-token-usage)
      * [Tracking resolved model names](https://llm.datasette.io/en/stable/plugins/advanced-model-plugins.html#tracking-resolved-model-names)
      * [LLM_RAISE_ERRORS](https://llm.datasette.io/en/stable/plugins/advanced-model-plugins.html#llm-raise-errors)
    * [Utility functions for plugins](https://llm.datasette.io/en/stable/plugins/plugin-utilities.html)
      * [llm.get_key()](https://llm.datasette.io/en/stable/plugins/plugin-utilities.html#llm-get-key)
      * [llm.user_dir()](https://llm.datasette.io/en/stable/plugins/plugin-utilities.html#llm-user-dir)
      * [llm.ModelError](https://llm.datasette.io/en/stable/plugins/plugin-utilities.html#llm-modelerror)
      * [Response.fake()](https://llm.datasette.io/en/stable/plugins/plugin-utilities.html#response-fake)
  * [Python API](https://llm.datasette.io/en/stable/python-api.html)
    * [Basic prompt execution](https://llm.datasette.io/en/stable/python-api.html#basic-prompt-execution)
      * [System prompts](https://llm.datasette.io/en/stable/python-api.html#system-prompts)
      * [Attachments](https://llm.datasette.io/en/stable/python-api.html#attachments)
      * [Tools](https://llm.datasette.io/en/stable/python-api.html#tools)
      * [Schemas](https://llm.datasette.io/en/stable/python-api.html#schemas)
      * [Fragments](https://llm.datasette.io/en/stable/python-api.html#fragments)
      * [Model options](https://llm.datasette.io/en/stable/python-api.html#model-options)
      * [Passing an API key](https://llm.datasette.io/en/stable/python-api.html#passing-an-api-key)
      * [Models from plugins](https://llm.datasette.io/en/stable/python-api.html#models-from-plugins)
      * [Accessing the underlying JSON](https://llm.datasette.io/en/stable/python-api.html#accessing-the-underlying-json)
      * [Token usage](https://llm.datasette.io/en/stable/python-api.html#token-usage)
      * [Streaming responses](https://llm.datasette.io/en/stable/python-api.html#streaming-responses)
    * [Async models](https://llm.datasette.io/en/stable/python-api.html#async-models)
      * [Tool functions can be sync or async](https://llm.datasette.io/en/stable/python-api.html#tool-functions-can-be-sync-or-async)
      * [Tool use for async models](https://llm.datasette.io/en/stable/python-api.html#tool-use-for-async-models)
    * [Conversations](https://llm.datasette.io/en/stable/python-api.html#conversations)
      * [Conversations using tools](https://llm.datasette.io/en/stable/python-api.html#conversations-using-tools)
    * [Listing models](https://llm.datasette.io/en/stable/python-api.html#listing-models)
    * [Running code when a response has completed](https://llm.datasette.io/en/stable/python-api.html#running-code-when-a-response-has-completed)
    * [Other functions](https://llm.datasette.io/en/stable/python-api.html#other-functions)
      * [set_alias(alias, model_id)](https://llm.datasette.io/en/stable/python-api.html#set-alias-alias-model-id)
      * [remove_alias(alias)](https://llm.datasette.io/en/stable/python-api.html#remove-alias-alias)
      * [set_default_model(alias)](https://llm.datasette.io/en/stable/python-api.html#set-default-model-alias)
      * [get_default_model()](https://llm.datasette.io/en/stable/python-api.html#get-default-model)
      * [set_default_embedding_model(alias) and get_default_embedding_model()](https://llm.datasette.io/en/stable/python-api.html#set-default-embedding-model-alias-and-get-default-embedding-model)
  * [Logging to SQLite](https://llm.datasette.io/en/stable/logging.html)
    * [Viewing the logs](https://llm.datasette.io/en/stable/logging.html#viewing-the-logs)
      * [-s/–short mode](https://llm.datasette.io/en/stable/logging.html#s-short-mode)
      * [Logs for a conversation](https://llm.datasette.io/en/stable/logging.html#logs-for-a-conversation)
      * [Searching the logs](https://llm.datasette.io/en/stable/logging.html#searching-the-logs)
      * [Filtering past a specific ID](https://llm.datasette.io/en/stable/logging.html#filtering-past-a-specific-id)
      * [Filtering by model](https://llm.datasette.io/en/stable/logging.html#filtering-by-model)
      * [Filtering by prompts that used specific fragments](https://llm.datasette.io/en/stable/logging.html#filtering-by-prompts-that-used-specific-fragments)
      * [Filtering by prompts that used specific tools](https://llm.datasette.io/en/stable/logging.html#filtering-by-prompts-that-used-specific-tools)
      * [Browsing data collected using schemas](https://llm.datasette.io/en/stable/logging.html#browsing-data-collected-using-schemas)
    * [Browsing logs using Datasette](https://llm.datasette.io/en/stable/logging.html#browsing-logs-using-datasette)
    * [Backing up your database](https://llm.datasette.io/en/stable/logging.html#backing-up-your-database)
    * [SQL schema](https://llm.datasette.io/en/stable/logging.html#sql-schema)
  * [Related tools](https://llm.datasette.io/en/stable/related-tools.html)
    * [strip-tags](https://llm.datasette.io/en/stable/related-tools.html#strip-tags)
    * [ttok](https://llm.datasette.io/en/stable/related-tools.html#ttok)
    * [Symbex](https://llm.datasette.io/en/stable/related-tools.html#symbex)
  * [CLI reference](https://llm.datasette.io/en/stable/help.html)
    * [llm  –help](https://llm.datasette.io/en/stable/help.html#llm-help)
      * [llm prompt –help](https://llm.datasette.io/en/stable/help.html#llm-prompt-help)
      * [llm chat –help](https://llm.datasette.io/en/stable/help.html#llm-chat-help)
      * [llm keys –help](https://llm.datasette.io/en/stable/help.html#llm-keys-help)
      * [llm logs –help](https://llm.datasette.io/en/stable/help.html#llm-logs-help)
      * [llm models –help](https://llm.datasette.io/en/stable/help.html#llm-models-help)
      * [llm templates –help](https://llm.datasette.io/en/stable/help.html#llm-templates-help)
      * [llm schemas –help](https://llm.datasette.io/en/stable/help.html#llm-schemas-help)
      * [llm tools –help](https://llm.datasette.io/en/stable/help.html#llm-tools-help)
      * [llm aliases –help](https://llm.datasette.io/en/stable/help.html#llm-aliases-help)
      * [llm fragments –help](https://llm.datasette.io/en/stable/help.html#llm-fragments-help)
      * [llm plugins –help](https://llm.datasette.io/en/stable/help.html#llm-plugins-help)
      * [llm install –help](https://llm.datasette.io/en/stable/help.html#llm-install-help)
      * [llm uninstall –help](https://llm.datasette.io/en/stable/help.html#llm-uninstall-help)
      * [llm embed –help](https://llm.datasette.io/en/stable/help.html#llm-embed-help)
      * [llm embed-multi –help](https://llm.datasette.io/en/stable/help.html#llm-embed-multi-help)
      * [llm similar –help](https://llm.datasette.io/en/stable/help.html#llm-similar-help)
      * [llm embed-models –help](https://llm.datasette.io/en/stable/help.html#llm-embed-models-help)
      * [llm collections –help](https://llm.datasette.io/en/stable/help.html#llm-collections-help)
      * [llm openai –help](https://llm.datasette.io/en/stable/help.html#llm-openai-help)
  * [Contributing](https://llm.datasette.io/en/stable/contributing.html)
    * [Updating recorded HTTP API interactions and associated snapshots](https://llm.datasette.io/en/stable/contributing.html#updating-recorded-http-api-interactions-and-associated-snapshots)
    * [Debugging tricks](https://llm.datasette.io/en/stable/contributing.html#debugging-tricks)
    * [Documentation](https://llm.datasette.io/en/stable/contributing.html#documentation)
    * [Release process](https://llm.datasette.io/en/stable/contributing.html#release-process)

  * [Changelog](https://llm.datasette.io/en/stable/changelog.html)
  <!-- [[[end]]] -->

readme_html:
  <div id="readme" class="md" data-path="README.md"><article class="markdown-body entry-content container-lg" itemprop="text">
  <div class="markdown-heading" dir="auto"><h1 class="heading-element" dir="auto">LLM</h1><a id="user-content-llm" class="anchor" aria-label="Permalink: LLM" href="#llm"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto"><a href="https://github.com/simonw/llm"><img src="https://img.shields.io/badge/github-repo-green" alt="GitHub repo" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/6c41bd81c08d25790a5a7f18f08e98763aa0dafcadf62c8f8a56ba9b48c8d1ba/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6769746875622d7265706f2d677265656e"></a>
  <a href="https://pypi.org/project/llm/" rel="nofollow"><img src="https://img.shields.io/pypi/v/llm.svg" alt="PyPI" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/6c9ef699a3c2efe71383382274e70fc5e84a3911297fc536ea67c29579ddb3d9/68747470733a2f2f696d672e736869656c64732e696f2f707970692f762f6c6c6d2e737667"></a>
  <a href="https://llm.datasette.io/en/stable/changelog.html" rel="nofollow"><img src="https://img.shields.io/github/v/release/simonw/llm?include_prereleases&label=changelog" alt="Changelog" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/4a64d6e4de982cad92be96faf5efbd77a505bdb8e594a6bc303416efdfdc78a1/68747470733a2f2f696d672e736869656c64732e696f2f6769746875622f762f72656c656173652f73696d6f6e772f6c6c6d3f696e636c7564655f70726572656c6561736573266c6162656c3d6368616e67656c6f67"></a>
  <a href="https://github.com/simonw/llm/actions?query=workflow%3ATest"><img src="https://github.com/simonw/llm/workflows/Test/badge.svg" alt="Tests" style="max-width: 100%;"></a>
  <a href="https://github.com/simonw/llm/blob/main/LICENSE"><img src="https://img.shields.io/badge/license-Apache%202.0-blue.svg" alt="License" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/c355f200ea90fddaa407b6eaab303663a669248ea3ca7b1fcf77dbe04ff5f48c/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6c6963656e73652d417061636865253230322e302d626c75652e737667"></a>
  <a href="https://datasette.io/discord-llm" rel="nofollow"><img src="https://img.shields.io/discord/823971286308356157?label=discord" alt="Discord" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/94e396359356f1b1dd8fed242a2f694cb88a3b94a9eef104404225094357e912/68747470733a2f2f696d672e736869656c64732e696f2f646973636f72642f3832333937313238363330383335363135373f6c6162656c3d646973636f7264"></a>
  <a href="https://formulae.brew.sh/formula/llm" rel="nofollow"><img src="https://img.shields.io/homebrew/installs/dy/llm?color=yellow&label=homebrew&logo=homebrew" alt="Homebrew" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/dbee90cd77c9cfffdc83b95c51d566d88c97f699c965f64f2dbad11acc74582f/68747470733a2f2f696d672e736869656c64732e696f2f686f6d65627265772f696e7374616c6c732f64792f6c6c6d3f636f6c6f723d79656c6c6f77266c6162656c3d686f6d6562726577266c6f676f3d686f6d6562726577"></a></p>
  <p dir="auto">A CLI tool and Python library for interacting with <strong>OpenAI</strong>, <strong>Anthropic’s Claude</strong>, <strong>Google’s Gemini</strong>, <strong>Meta’s Llama</strong> and dozens of other Large Language Models, both via remote APIs and with models that can be installed and run on your own machine.</p>
  <p dir="auto">Watch <strong><a href="https://www.youtube.com/watch?v=QUXQNi6jQ30" rel="nofollow">Language models on the command-line</a></strong> on YouTube for a demo or <a href="https://simonwillison.net/2024/Jun/17/cli-language-models/" rel="nofollow">read the accompanying detailed notes</a>.</p>
  <p dir="auto">With LLM you can:</p>
  <ul dir="auto">
  <li><a href="https://llm.datasette.io/en/stable/usage.html#usage-executing-prompts" rel="nofollow">Run prompts from the command-line</a></li>
  <li><a href="https://llm.datasette.io/en/stable/logging.html#logging" rel="nofollow">Store prompts and responses in SQLite</a></li>
  <li><a href="https://llm.datasette.io/en/stable/embeddings/index.html#embeddings" rel="nofollow">Generate and store embeddings</a></li>
  <li><a href="https://llm.datasette.io/en/stable/schemas.html#schemas" rel="nofollow">Extract structured content from text and images</a></li>
  <li>… and much, much more</li>
  </ul>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Quick start</h2><a id="user-content-quick-start" class="anchor" aria-label="Permalink: Quick start" href="#quick-start"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">First, install LLM using <code>pip</code> or Homebrew or <code>pipx</code> or <code>uv</code>:</p>
  <div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="pip install llm"><pre>pip install llm</pre></div>
  <p dir="auto">Or with Homebrew (see <a href="https://llm.datasette.io/en/stable/setup.html#homebrew-warning" rel="nofollow">warning note</a>):</p>
  <div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="brew install llm"><pre>brew install llm</pre></div>
  <p dir="auto">Or with <a href="https://pypa.github.io/pipx/" rel="nofollow">pipx</a>:</p>
  <div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="pipx install llm"><pre>pipx install llm</pre></div>
  <p dir="auto">Or with <a href="https://docs.astral.sh/uv/guides/tools/" rel="nofollow">uv</a></p>
  <div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="uv tool install llm"><pre>uv tool install llm</pre></div>
  <p dir="auto">If you have an <a href="https://platform.openai.com/api-keys" rel="nofollow">OpenAI API key</a> key you can run this:</p>
  <div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="# Paste your OpenAI API key into this
  llm keys set openai

  # Run a prompt (with the default gpt-4o-mini model)
  llm "Ten fun names for a pet pelican"

  # Extract text from an image
  llm "extract text" -a scanned-document.jpg

  # Use a system prompt against a file
  cat myfile.py | llm -s "Explain this code""><pre><span class="pl-c"><span class="pl-c">#</span> Paste your OpenAI API key into this</span>
  llm keys <span class="pl-c1">set</span> openai

  <span class="pl-c"><span class="pl-c">#</span> Run a prompt (with the default gpt-4o-mini model)</span>
  llm <span class="pl-s"><span class="pl-pds">"</span>Ten fun names for a pet pelican<span class="pl-pds">"</span></span>

  <span class="pl-c"><span class="pl-c">#</span> Extract text from an image</span>
  llm <span class="pl-s"><span class="pl-pds">"</span>extract text<span class="pl-pds">"</span></span> -a scanned-document.jpg

  <span class="pl-c"><span class="pl-c">#</span> Use a system prompt against a file</span>
  cat myfile.py <span class="pl-k">|</span> llm -s <span class="pl-s"><span class="pl-pds">"</span>Explain this code<span class="pl-pds">"</span></span></pre></div>
  <p dir="auto">Run prompts against <a href="https://aistudio.google.com/apikey" rel="nofollow">Gemini</a> or <a href="https://console.anthropic.com/" rel="nofollow">Anthropic</a> with their respective plugins:</p>
  <div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="llm install llm-gemini
  llm keys set gemini
  # Paste Gemini API key here
  llm -m gemini-2.0-flash 'Tell me fun facts about Mountain View'

  llm install llm-anthropic
  llm keys set anthropic
  # Paste Anthropic API key here
  llm -m claude-4-opus 'Impress me with wild facts about turnips'"><pre>llm install llm-gemini
  llm keys <span class="pl-c1">set</span> gemini
  <span class="pl-c"><span class="pl-c">#</span> Paste Gemini API key here</span>
  llm -m gemini-2.0-flash <span class="pl-s"><span class="pl-pds">'</span>Tell me fun facts about Mountain View<span class="pl-pds">'</span></span>

  llm install llm-anthropic
  llm keys <span class="pl-c1">set</span> anthropic
  <span class="pl-c"><span class="pl-c">#</span> Paste Anthropic API key here</span>
  llm -m claude-4-opus <span class="pl-s"><span class="pl-pds">'</span>Impress me with wild facts about turnips<span class="pl-pds">'</span></span></pre></div>
  <p dir="auto">You can also <a href="https://llm.datasette.io/en/stable/plugins/installing-plugins.html#installing-plugins" rel="nofollow">install a plugin</a> to access models that can run on your local device. If you use <a href="https://ollama.com/" rel="nofollow">Ollama</a>:</p>
  <div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="# Install the plugin
  llm install llm-ollama

  # Download and run a prompt against the Orca Mini 7B model
  ollama pull llama3.2:latest
  llm -m llama3.2:latest 'What is the capital of France?'"><pre><span class="pl-c"><span class="pl-c">#</span> Install the plugin</span>
  llm install llm-ollama

  <span class="pl-c"><span class="pl-c">#</span> Download and run a prompt against the Orca Mini 7B model</span>
  ollama pull llama3.2:latest
  llm -m llama3.2:latest <span class="pl-s"><span class="pl-pds">'</span>What is the capital of France?<span class="pl-pds">'</span></span></pre></div>
  <p dir="auto">To start <a href="https://llm.datasette.io/en/stable/usage.html#usage-chat" rel="nofollow">an interactive chat</a> with a model, use <code>llm chat</code>:</p>
  <div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="llm chat -m gpt-4.1"><pre>llm chat -m gpt-4.1</pre></div>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="Chatting with gpt-4.1
  Type 'exit' or 'quit' to exit
  Type '!multi' to enter multiple lines, then '!end' to finish
  Type '!edit' to open your default editor and modify the prompt.
  Type '!fragment <my_fragment> [<another_fragment> ...]' to insert one or more fragments
  > Tell me a joke about a pelican
  Why don't pelicans like to tip waiters?

  Because they always have a big bill!"><pre lang="default" class="notranslate"><code>Chatting with gpt-4.1
  Type 'exit' or 'quit' to exit
  Type '!multi' to enter multiple lines, then '!end' to finish
  Type '!edit' to open your default editor and modify the prompt.
  Type '!fragment <my_fragment> [<another_fragment> ...]' to insert one or more fragments
  > Tell me a joke about a pelican
  Why don't pelicans like to tip waiters?

  Because they always have a big bill!
  </code></pre></div>
  <p dir="auto">More background on this project:</p>
  <ul dir="auto">
  <li><a href="https://simonwillison.net/2023/May/18/cli-tools-for-llms/" rel="nofollow">llm, ttok and strip-tags—CLI tools for working with ChatGPT and other LLMs</a></li>
  <li><a href="https://simonwillison.net/2023/Jul/12/llm/" rel="nofollow">The LLM CLI tool now supports self-hosted language models via plugins</a></li>
  <li><a href="https://simonwillison.net/2023/Sep/4/llm-embeddings/" rel="nofollow">LLM now provides tools for working with embeddings</a></li>
  <li><a href="https://simonwillison.net/2023/Sep/12/llm-clip-and-chat/" rel="nofollow">Build an image search engine with llm-clip, chat with models with llm chat</a></li>
  <li><a href="https://simonwillison.net/2024/Oct/29/llm-multi-modal/" rel="nofollow">You can now run prompts against images, audio and video in your terminal using LLM</a></li>
  <li><a href="https://simonwillison.net/2025/Feb/28/llm-schemas/" rel="nofollow">Structured data extraction from unstructured content using LLM schemas</a></li>
  <li><a href="https://simonwillison.net/2025/Apr/7/long-context-llm/" rel="nofollow">Long context support in LLM 0.24 using fragments and template plugins</a></li>
  </ul>
  <p dir="auto">See also <a href="https://simonwillison.net/tags/llm/" rel="nofollow">the llm tag</a> on my blog.</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Contents</h2><a id="user-content-contents" class="anchor" aria-label="Permalink: Contents" href="#contents"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <ul dir="auto">
  <li>
  <p dir="auto"><a href="https://llm.datasette.io/en/stable/setup.html" rel="nofollow">Setup</a></p>
  <ul dir="auto">
  <li><a href="https://llm.datasette.io/en/stable/setup.html#installation" rel="nofollow">Installation</a></li>
  <li><a href="https://llm.datasette.io/en/stable/setup.html#upgrading-to-the-latest-version" rel="nofollow">Upgrading to the latest version</a></li>
  <li><a href="https://llm.datasette.io/en/stable/setup.html#using-uvx" rel="nofollow">Using uvx</a></li>
  <li><a href="https://llm.datasette.io/en/stable/setup.html#a-note-about-homebrew-and-pytorch" rel="nofollow">A note about Homebrew and PyTorch</a></li>
  <li><a href="https://llm.datasette.io/en/stable/setup.html#installing-plugins" rel="nofollow">Installing plugins</a></li>
  <li><a href="https://llm.datasette.io/en/stable/setup.html#api-key-management" rel="nofollow">API key management</a>
  <ul dir="auto">
  <li><a href="https://llm.datasette.io/en/stable/setup.html#saving-and-using-stored-keys" rel="nofollow">Saving and using stored keys</a></li>
  <li><a href="https://llm.datasette.io/en/stable/setup.html#passing-keys-using-the-key-option" rel="nofollow">Passing keys using the –key option</a></li>
  <li><a href="https://llm.datasette.io/en/stable/setup.html#keys-in-environment-variables" rel="nofollow">Keys in environment variables</a></li>
  </ul>
  </li>
  <li><a href="https://llm.datasette.io/en/stable/setup.html#configuration" rel="nofollow">Configuration</a>
  <ul dir="auto">
  <li><a href="https://llm.datasette.io/en/stable/setup.html#setting-a-custom-default-model" rel="nofollow">Setting a custom default model</a></li>
  <li><a href="https://llm.datasette.io/en/stable/setup.html#setting-a-custom-directory-location" rel="nofollow">Setting a custom directory location</a></li>
  <li><a href="https://llm.datasette.io/en/stable/setup.html#turning-sqlite-logging-on-and-off" rel="nofollow">Turning SQLite logging on and off</a></li>
  </ul>
  </li>
  </ul>
  </li>
  <li>
  <p dir="auto"><a href="https://llm.datasette.io/en/stable/usage.html" rel="nofollow">Usage</a></p>
  <ul dir="auto">
  <li><a href="https://llm.datasette.io/en/stable/usage.html#executing-a-prompt" rel="nofollow">Executing a prompt</a>
  <ul dir="auto">
  <li><a href="https://llm.datasette.io/en/stable/usage.html#model-options" rel="nofollow">Model options</a></li>
  <li><a href="https://llm.datasette.io/en/stable/usage.html#attachments" rel="nofollow">Attachments</a></li>
  <li><a href="https://llm.datasette.io/en/stable/usage.html#system-prompts" rel="nofollow">System prompts</a></li>
  <li><a href="https://llm.datasette.io/en/stable/usage.html#tools" rel="nofollow">Tools</a></li>
  <li><a href="https://llm.datasette.io/en/stable/usage.html#extracting-fenced-code-blocks" rel="nofollow">Extracting fenced code blocks</a></li>
  <li><a href="https://llm.datasette.io/en/stable/usage.html#schemas" rel="nofollow">Schemas</a></li>
  <li><a href="https://llm.datasette.io/en/stable/usage.html#fragments" rel="nofollow">Fragments</a></li>
  <li><a href="https://llm.datasette.io/en/stable/usage.html#continuing-a-conversation" rel="nofollow">Continuing a conversation</a></li>
  <li><a href="https://llm.datasette.io/en/stable/usage.html#tips-for-using-llm-with-bash-or-zsh" rel="nofollow">Tips for using LLM with Bash or Zsh</a></li>
  <li><a href="https://llm.datasette.io/en/stable/usage.html#completion-prompts" rel="nofollow">Completion prompts</a></li>
  </ul>
  </li>
  <li><a href="https://llm.datasette.io/en/stable/usage.html#starting-an-interactive-chat" rel="nofollow">Starting an interactive chat</a></li>
  <li><a href="https://llm.datasette.io/en/stable/usage.html#listing-available-models" rel="nofollow">Listing available models</a></li>
  <li><a href="https://llm.datasette.io/en/stable/usage.html#setting-default-options-for-models" rel="nofollow">Setting default options for models</a></li>
  </ul>
  </li>
  <li>
  <p dir="auto"><a href="https://llm.datasette.io/en/stable/openai-models.html" rel="nofollow">OpenAI models</a></p>
  <ul dir="auto">
  <li><a href="https://llm.datasette.io/en/stable/openai-models.html#configuration" rel="nofollow">Configuration</a></li>
  <li><a href="https://llm.datasette.io/en/stable/openai-models.html#openai-language-models" rel="nofollow">OpenAI language models</a></li>
  <li><a href="https://llm.datasette.io/en/stable/openai-models.html#model-features" rel="nofollow">Model features</a></li>
  <li><a href="https://llm.datasette.io/en/stable/openai-models.html#openai-embedding-models" rel="nofollow">OpenAI embedding models</a></li>
  <li><a href="https://llm.datasette.io/en/stable/openai-models.html#openai-completion-models" rel="nofollow">OpenAI completion models</a></li>
  <li><a href="https://llm.datasette.io/en/stable/openai-models.html#adding-more-openai-models" rel="nofollow">Adding more OpenAI models</a></li>
  </ul>
  </li>
  <li>
  <p dir="auto"><a href="https://llm.datasette.io/en/stable/other-models.html" rel="nofollow">Other models</a></p>
  <ul dir="auto">
  <li><a href="https://llm.datasette.io/en/stable/other-models.html#installing-and-using-a-local-model" rel="nofollow">Installing and using a local model</a></li>
  <li><a href="https://llm.datasette.io/en/stable/other-models.html#openai-compatible-models" rel="nofollow">OpenAI-compatible models</a>
  <ul dir="auto">
  <li><a href="https://llm.datasette.io/en/stable/other-models.html#extra-http-headers" rel="nofollow">Extra HTTP headers</a></li>
  </ul>
  </li>
  </ul>
  </li>
  <li>
  <p dir="auto"><a href="https://llm.datasette.io/en/stable/tools.html" rel="nofollow">Tools</a></p>
  <ul dir="auto">
  <li><a href="https://llm.datasette.io/en/stable/tools.html#how-tools-work" rel="nofollow">How tools work</a></li>
  <li><a href="https://llm.datasette.io/en/stable/tools.html#trying-out-tools" rel="nofollow">Trying out tools</a></li>
  <li><a href="https://llm.datasette.io/en/stable/tools.html#llm-s-implementation-of-tools" rel="nofollow">LLM’s implementation of tools</a></li>
  <li><a href="https://llm.datasette.io/en/stable/tools.html#default-tools" rel="nofollow">Default tools</a></li>
  <li><a href="https://llm.datasette.io/en/stable/tools.html#tips-for-implementing-tools" rel="nofollow">Tips for implementing tools</a></li>
  </ul>
  </li>
  <li>
  <p dir="auto"><a href="https://llm.datasette.io/en/stable/schemas.html" rel="nofollow">Schemas</a></p>
  <ul dir="auto">
  <li><a href="https://llm.datasette.io/en/stable/schemas.html#schemas-tutorial" rel="nofollow">Schemas tutorial</a>
  <ul dir="auto">
  <li><a href="https://llm.datasette.io/en/stable/schemas.html#getting-started-with-dogs" rel="nofollow">Getting started with dogs</a></li>
  <li><a href="https://llm.datasette.io/en/stable/schemas.html#extracting-people-from-a-news-articles" rel="nofollow">Extracting people from a news articles</a></li>
  </ul>
  </li>
  <li><a href="https://llm.datasette.io/en/stable/schemas.html#using-json-schemas" rel="nofollow">Using JSON schemas</a></li>
  <li><a href="https://llm.datasette.io/en/stable/schemas.html#ways-to-specify-a-schema" rel="nofollow">Ways to specify a schema</a></li>
  <li><a href="https://llm.datasette.io/en/stable/schemas.html#concise-llm-schema-syntax" rel="nofollow">Concise LLM schema syntax</a></li>
  <li><a href="https://llm.datasette.io/en/stable/schemas.html#saving-reusable-schemas-in-templates" rel="nofollow">Saving reusable schemas in templates</a></li>
  <li><a href="https://llm.datasette.io/en/stable/schemas.html#browsing-logged-json-objects-created-using-schemas" rel="nofollow">Browsing logged JSON objects created using schemas</a></li>
  </ul>
  </li>
  <li>
  <p dir="auto"><a href="https://llm.datasette.io/en/stable/templates.html" rel="nofollow">Templates</a></p>
  <ul dir="auto">
  <li><a href="https://llm.datasette.io/en/stable/templates.html#getting-started-with-save" rel="nofollow">Getting started with <code>–save</code></a></li>
  <li><a href="https://llm.datasette.io/en/stable/templates.html#using-a-template" rel="nofollow">Using a template</a></li>
  <li><a href="https://llm.datasette.io/en/stable/templates.html#listing-available-templates" rel="nofollow">Listing available templates</a></li>
  <li><a href="https://llm.datasette.io/en/stable/templates.html#templates-as-yaml-files" rel="nofollow">Templates as YAML files</a>
  <ul dir="auto">
  <li><a href="https://llm.datasette.io/en/stable/templates.html#system-prompts" rel="nofollow">System prompts</a></li>
  <li><a href="https://llm.datasette.io/en/stable/templates.html#fragments" rel="nofollow">Fragments</a></li>
  <li><a href="https://llm.datasette.io/en/stable/templates.html#options" rel="nofollow">Options</a></li>
  <li><a href="https://llm.datasette.io/en/stable/templates.html#schemas" rel="nofollow">Schemas</a></li>
  <li><a href="https://llm.datasette.io/en/stable/templates.html#additional-template-variables" rel="nofollow">Additional template variables</a></li>
  <li><a href="https://llm.datasette.io/en/stable/templates.html#specifying-default-parameters" rel="nofollow">Specifying default parameters</a></li>
  <li><a href="https://llm.datasette.io/en/stable/templates.html#configuring-code-extraction" rel="nofollow">Configuring code extraction</a></li>
  <li><a href="https://llm.datasette.io/en/stable/templates.html#setting-a-default-model-for-a-template" rel="nofollow">Setting a default model for a template</a></li>
  </ul>
  </li>
  <li><a href="https://llm.datasette.io/en/stable/templates.html#template-loaders-from-plugins" rel="nofollow">Template loaders from plugins</a></li>
  </ul>
  </li>
  <li>
  <p dir="auto"><a href="https://llm.datasette.io/en/stable/fragments.html" rel="nofollow">Fragments</a></p>
  <ul dir="auto">
  <li><a href="https://llm.datasette.io/en/stable/fragments.html#using-fragments-in-a-prompt" rel="nofollow">Using fragments in a prompt</a></li>
  <li><a href="https://llm.datasette.io/en/stable/fragments.html#using-fragments-in-chat" rel="nofollow">Using fragments in chat</a></li>
  <li><a href="https://llm.datasette.io/en/stable/fragments.html#browsing-fragments" rel="nofollow">Browsing fragments</a></li>
  <li><a href="https://llm.datasette.io/en/stable/fragments.html#setting-aliases-for-fragments" rel="nofollow">Setting aliases for fragments</a></li>
  <li><a href="https://llm.datasette.io/en/stable/fragments.html#viewing-fragments-in-your-logs" rel="nofollow">Viewing fragments in your logs</a></li>
  <li><a href="https://llm.datasette.io/en/stable/fragments.html#using-fragments-from-plugins" rel="nofollow">Using fragments from plugins</a></li>
  <li><a href="https://llm.datasette.io/en/stable/fragments.html#listing-available-fragment-prefixes" rel="nofollow">Listing available fragment prefixes</a></li>
  </ul>
  </li>
  <li>
  <p dir="auto"><a href="https://llm.datasette.io/en/stable/aliases.html" rel="nofollow">Model aliases</a></p>
  <ul dir="auto">
  <li><a href="https://llm.datasette.io/en/stable/aliases.html#listing-aliases" rel="nofollow">Listing aliases</a></li>
  <li><a href="https://llm.datasette.io/en/stable/aliases.html#adding-a-new-alias" rel="nofollow">Adding a new alias</a></li>
  <li><a href="https://llm.datasette.io/en/stable/aliases.html#removing-an-alias" rel="nofollow">Removing an alias</a></li>
  <li><a href="https://llm.datasette.io/en/stable/aliases.html#viewing-the-aliases-file" rel="nofollow">Viewing the aliases file</a></li>
  </ul>
  </li>
  <li>
  <p dir="auto"><a href="https://llm.datasette.io/en/stable/embeddings/index.html" rel="nofollow">Embeddings</a></p>
  <ul dir="auto">
  <li><a href="https://llm.datasette.io/en/stable/embeddings/cli.html" rel="nofollow">Embedding with the CLI</a>
  <ul dir="auto">
  <li><a href="https://llm.datasette.io/en/stable/embeddings/cli.html#llm-embed" rel="nofollow">llm embed</a></li>
  <li><a href="https://llm.datasette.io/en/stable/embeddings/cli.html#llm-embed-multi" rel="nofollow">llm embed-multi</a></li>
  <li><a href="https://llm.datasette.io/en/stable/embeddings/cli.html#llm-similar" rel="nofollow">llm similar</a></li>
  <li><a href="https://llm.datasette.io/en/stable/embeddings/cli.html#llm-embed-models" rel="nofollow">llm embed-models</a></li>
  <li><a href="https://llm.datasette.io/en/stable/embeddings/cli.html#llm-collections-list" rel="nofollow">llm collections list</a></li>
  <li><a href="https://llm.datasette.io/en/stable/embeddings/cli.html#llm-collections-delete" rel="nofollow">llm collections delete</a></li>
  </ul>
  </li>
  <li><a href="https://llm.datasette.io/en/stable/embeddings/python-api.html" rel="nofollow">Using embeddings from Python</a>
  <ul dir="auto">
  <li><a href="https://llm.datasette.io/en/stable/embeddings/python-api.html#working-with-collections" rel="nofollow">Working with collections</a></li>
  <li><a href="https://llm.datasette.io/en/stable/embeddings/python-api.html#retrieving-similar-items" rel="nofollow">Retrieving similar items</a></li>
  <li><a href="https://llm.datasette.io/en/stable/embeddings/python-api.html#sql-schema" rel="nofollow">SQL schema</a></li>
  </ul>
  </li>
  <li><a href="https://llm.datasette.io/en/stable/embeddings/writing-plugins.html" rel="nofollow">Writing plugins to add new embedding models</a>
  <ul dir="auto">
  <li><a href="https://llm.datasette.io/en/stable/embeddings/writing-plugins.html#embedding-binary-content" rel="nofollow">Embedding binary content</a></li>
  </ul>
  </li>
  <li><a href="https://llm.datasette.io/en/stable/embeddings/storage.html" rel="nofollow">Embedding storage format</a></li>
  </ul>
  </li>
  <li>
  <p dir="auto"><a href="https://llm.datasette.io/en/stable/plugins/index.html" rel="nofollow">Plugins</a></p>
  <ul dir="auto">
  <li><a href="https://llm.datasette.io/en/stable/plugins/installing-plugins.html" rel="nofollow">Installing plugins</a>
  <ul dir="auto">
  <li><a href="https://llm.datasette.io/en/stable/plugins/installing-plugins.html#listing-installed-plugins" rel="nofollow">Listing installed plugins</a></li>
  <li><a href="https://llm.datasette.io/en/stable/plugins/installing-plugins.html#running-with-a-subset-of-plugins" rel="nofollow">Running with a subset of plugins</a></li>
  </ul>
  </li>
  <li><a href="https://llm.datasette.io/en/stable/plugins/directory.html" rel="nofollow">Plugin directory</a>
  <ul dir="auto">
  <li><a href="https://llm.datasette.io/en/stable/plugins/directory.html#local-models" rel="nofollow">Local models</a></li>
  <li><a href="https://llm.datasette.io/en/stable/plugins/directory.html#remote-apis" rel="nofollow">Remote APIs</a></li>
  <li><a href="https://llm.datasette.io/en/stable/plugins/directory.html#tools" rel="nofollow">Tools</a></li>
  <li><a href="https://llm.datasette.io/en/stable/plugins/directory.html#fragments-and-template-loaders" rel="nofollow">Fragments and template loaders</a></li>
  <li><a href="https://llm.datasette.io/en/stable/plugins/directory.html#embedding-models" rel="nofollow">Embedding models</a></li>
  <li><a href="https://llm.datasette.io/en/stable/plugins/directory.html#extra-commands" rel="nofollow">Extra commands</a></li>
  <li><a href="https://llm.datasette.io/en/stable/plugins/directory.html#just-for-fun" rel="nofollow">Just for fun</a></li>
  </ul>
  </li>
  <li><a href="https://llm.datasette.io/en/stable/plugins/plugin-hooks.html" rel="nofollow">Plugin hooks</a>
  <ul dir="auto">
  <li><a href="https://llm.datasette.io/en/stable/plugins/plugin-hooks.html#register-commands-cli" rel="nofollow">register_commands(cli)</a></li>
  <li><a href="https://llm.datasette.io/en/stable/plugins/plugin-hooks.html#register-models-register" rel="nofollow">register_models(register)</a></li>
  <li><a href="https://llm.datasette.io/en/stable/plugins/plugin-hooks.html#register-embedding-models-register" rel="nofollow">register_embedding_models(register)</a></li>
  <li><a href="https://llm.datasette.io/en/stable/plugins/plugin-hooks.html#register-tools-register" rel="nofollow">register_tools(register)</a></li>
  <li><a href="https://llm.datasette.io/en/stable/plugins/plugin-hooks.html#register-template-loaders-register" rel="nofollow">register_template_loaders(register)</a></li>
  <li><a href="https://llm.datasette.io/en/stable/plugins/plugin-hooks.html#register-fragment-loaders-register" rel="nofollow">register_fragment_loaders(register)</a></li>
  </ul>
  </li>
  <li><a href="https://llm.datasette.io/en/stable/plugins/tutorial-model-plugin.html" rel="nofollow">Developing a model plugin</a>
  <ul dir="auto">
  <li><a href="https://llm.datasette.io/en/stable/plugins/tutorial-model-plugin.html#the-initial-structure-of-the-plugin" rel="nofollow">The initial structure of the plugin</a></li>
  <li><a href="https://llm.datasette.io/en/stable/plugins/tutorial-model-plugin.html#installing-your-plugin-to-try-it-out" rel="nofollow">Installing your plugin to try it out</a></li>
  <li><a href="https://llm.datasette.io/en/stable/plugins/tutorial-model-plugin.html#building-the-markov-chain" rel="nofollow">Building the Markov chain</a></li>
  <li><a href="https://llm.datasette.io/en/stable/plugins/tutorial-model-plugin.html#executing-the-markov-chain" rel="nofollow">Executing the Markov chain</a></li>
  <li><a href="https://llm.datasette.io/en/stable/plugins/tutorial-model-plugin.html#adding-that-to-the-plugin" rel="nofollow">Adding that to the plugin</a></li>
  <li><a href="https://llm.datasette.io/en/stable/plugins/tutorial-model-plugin.html#understanding-execute" rel="nofollow">Understanding execute()</a></li>
  <li><a href="https://llm.datasette.io/en/stable/plugins/tutorial-model-plugin.html#prompts-and-responses-are-logged-to-the-database" rel="nofollow">Prompts and responses are logged to the database</a></li>
  <li><a href="https://llm.datasette.io/en/stable/plugins/tutorial-model-plugin.html#adding-options" rel="nofollow">Adding options</a></li>
  <li><a href="https://llm.datasette.io/en/stable/plugins/tutorial-model-plugin.html#distributing-your-plugin" rel="nofollow">Distributing your plugin</a></li>
  <li><a href="https://llm.datasette.io/en/stable/plugins/tutorial-model-plugin.html#github-repositories" rel="nofollow">GitHub repositories</a></li>
  <li><a href="https://llm.datasette.io/en/stable/plugins/tutorial-model-plugin.html#publishing-plugins-to-pypi" rel="nofollow">Publishing plugins to PyPI</a></li>
  <li><a href="https://llm.datasette.io/en/stable/plugins/tutorial-model-plugin.html#adding-metadata" rel="nofollow">Adding metadata</a></li>
  <li><a href="https://llm.datasette.io/en/stable/plugins/tutorial-model-plugin.html#what-to-do-if-it-breaks" rel="nofollow">What to do if it breaks</a></li>
  </ul>
  </li>
  <li><a href="https://llm.datasette.io/en/stable/plugins/advanced-model-plugins.html" rel="nofollow">Advanced model plugins</a>
  <ul dir="auto">
  <li><a href="https://llm.datasette.io/en/stable/plugins/advanced-model-plugins.html#tip-lazily-load-expensive-dependencies" rel="nofollow">Tip: lazily load expensive dependencies</a></li>
  <li><a href="https://llm.datasette.io/en/stable/plugins/advanced-model-plugins.html#models-that-accept-api-keys" rel="nofollow">Models that accept API keys</a></li>
  <li><a href="https://llm.datasette.io/en/stable/plugins/advanced-model-plugins.html#async-models" rel="nofollow">Async models</a></li>
  <li><a href="https://llm.datasette.io/en/stable/plugins/advanced-model-plugins.html#supporting-schemas" rel="nofollow">Supporting schemas</a></li>
  <li><a href="https://llm.datasette.io/en/stable/plugins/advanced-model-plugins.html#supporting-tools" rel="nofollow">Supporting tools</a></li>
  <li><a href="https://llm.datasette.io/en/stable/plugins/advanced-model-plugins.html#attachments-for-multi-modal-models" rel="nofollow">Attachments for multi-modal models</a></li>
  <li><a href="https://llm.datasette.io/en/stable/plugins/advanced-model-plugins.html#tracking-token-usage" rel="nofollow">Tracking token usage</a></li>
  <li><a href="https://llm.datasette.io/en/stable/plugins/advanced-model-plugins.html#tracking-resolved-model-names" rel="nofollow">Tracking resolved model names</a></li>
  <li><a href="https://llm.datasette.io/en/stable/plugins/advanced-model-plugins.html#llm-raise-errors" rel="nofollow">LLM_RAISE_ERRORS</a></li>
  </ul>
  </li>
  <li><a href="https://llm.datasette.io/en/stable/plugins/plugin-utilities.html" rel="nofollow">Utility functions for plugins</a>
  <ul dir="auto">
  <li><a href="https://llm.datasette.io/en/stable/plugins/plugin-utilities.html#llm-get-key" rel="nofollow">llm.get_key()</a></li>
  <li><a href="https://llm.datasette.io/en/stable/plugins/plugin-utilities.html#llm-user-dir" rel="nofollow">llm.user_dir()</a></li>
  <li><a href="https://llm.datasette.io/en/stable/plugins/plugin-utilities.html#llm-modelerror" rel="nofollow">llm.ModelError</a></li>
  <li><a href="https://llm.datasette.io/en/stable/plugins/plugin-utilities.html#response-fake" rel="nofollow">Response.fake()</a></li>
  </ul>
  </li>
  </ul>
  </li>
  <li>
  <p dir="auto"><a href="https://llm.datasette.io/en/stable/python-api.html" rel="nofollow">Python API</a></p>
  <ul dir="auto">
  <li><a href="https://llm.datasette.io/en/stable/python-api.html#basic-prompt-execution" rel="nofollow">Basic prompt execution</a>
  <ul dir="auto">
  <li><a href="https://llm.datasette.io/en/stable/python-api.html#system-prompts" rel="nofollow">System prompts</a></li>
  <li><a href="https://llm.datasette.io/en/stable/python-api.html#attachments" rel="nofollow">Attachments</a></li>
  <li><a href="https://llm.datasette.io/en/stable/python-api.html#tools" rel="nofollow">Tools</a></li>
  <li><a href="https://llm.datasette.io/en/stable/python-api.html#schemas" rel="nofollow">Schemas</a></li>
  <li><a href="https://llm.datasette.io/en/stable/python-api.html#fragments" rel="nofollow">Fragments</a></li>
  <li><a href="https://llm.datasette.io/en/stable/python-api.html#model-options" rel="nofollow">Model options</a></li>
  <li><a href="https://llm.datasette.io/en/stable/python-api.html#passing-an-api-key" rel="nofollow">Passing an API key</a></li>
  <li><a href="https://llm.datasette.io/en/stable/python-api.html#models-from-plugins" rel="nofollow">Models from plugins</a></li>
  <li><a href="https://llm.datasette.io/en/stable/python-api.html#accessing-the-underlying-json" rel="nofollow">Accessing the underlying JSON</a></li>
  <li><a href="https://llm.datasette.io/en/stable/python-api.html#token-usage" rel="nofollow">Token usage</a></li>
  <li><a href="https://llm.datasette.io/en/stable/python-api.html#streaming-responses" rel="nofollow">Streaming responses</a></li>
  </ul>
  </li>
  <li><a href="https://llm.datasette.io/en/stable/python-api.html#async-models" rel="nofollow">Async models</a>
  <ul dir="auto">
  <li><a href="https://llm.datasette.io/en/stable/python-api.html#tool-functions-can-be-sync-or-async" rel="nofollow">Tool functions can be sync or async</a></li>
  <li><a href="https://llm.datasette.io/en/stable/python-api.html#tool-use-for-async-models" rel="nofollow">Tool use for async models</a></li>
  </ul>
  </li>
  <li><a href="https://llm.datasette.io/en/stable/python-api.html#conversations" rel="nofollow">Conversations</a>
  <ul dir="auto">
  <li><a href="https://llm.datasette.io/en/stable/python-api.html#conversations-using-tools" rel="nofollow">Conversations using tools</a></li>
  </ul>
  </li>
  <li><a href="https://llm.datasette.io/en/stable/python-api.html#listing-models" rel="nofollow">Listing models</a></li>
  <li><a href="https://llm.datasette.io/en/stable/python-api.html#running-code-when-a-response-has-completed" rel="nofollow">Running code when a response has completed</a></li>
  <li><a href="https://llm.datasette.io/en/stable/python-api.html#other-functions" rel="nofollow">Other functions</a>
  <ul dir="auto">
  <li><a href="https://llm.datasette.io/en/stable/python-api.html#set-alias-alias-model-id" rel="nofollow">set_alias(alias, model_id)</a></li>
  <li><a href="https://llm.datasette.io/en/stable/python-api.html#remove-alias-alias" rel="nofollow">remove_alias(alias)</a></li>
  <li><a href="https://llm.datasette.io/en/stable/python-api.html#set-default-model-alias" rel="nofollow">set_default_model(alias)</a></li>
  <li><a href="https://llm.datasette.io/en/stable/python-api.html#get-default-model" rel="nofollow">get_default_model()</a></li>
  <li><a href="https://llm.datasette.io/en/stable/python-api.html#set-default-embedding-model-alias-and-get-default-embedding-model" rel="nofollow">set_default_embedding_model(alias) and get_default_embedding_model()</a></li>
  </ul>
  </li>
  </ul>
  </li>
  <li>
  <p dir="auto"><a href="https://llm.datasette.io/en/stable/logging.html" rel="nofollow">Logging to SQLite</a></p>
  <ul dir="auto">
  <li><a href="https://llm.datasette.io/en/stable/logging.html#viewing-the-logs" rel="nofollow">Viewing the logs</a>
  <ul dir="auto">
  <li><a href="https://llm.datasette.io/en/stable/logging.html#s-short-mode" rel="nofollow">-s/–short mode</a></li>
  <li><a href="https://llm.datasette.io/en/stable/logging.html#logs-for-a-conversation" rel="nofollow">Logs for a conversation</a></li>
  <li><a href="https://llm.datasette.io/en/stable/logging.html#searching-the-logs" rel="nofollow">Searching the logs</a></li>
  <li><a href="https://llm.datasette.io/en/stable/logging.html#filtering-past-a-specific-id" rel="nofollow">Filtering past a specific ID</a></li>
  <li><a href="https://llm.datasette.io/en/stable/logging.html#filtering-by-model" rel="nofollow">Filtering by model</a></li>
  <li><a href="https://llm.datasette.io/en/stable/logging.html#filtering-by-prompts-that-used-specific-fragments" rel="nofollow">Filtering by prompts that used specific fragments</a></li>
  <li><a href="https://llm.datasette.io/en/stable/logging.html#filtering-by-prompts-that-used-specific-tools" rel="nofollow">Filtering by prompts that used specific tools</a></li>
  <li><a href="https://llm.datasette.io/en/stable/logging.html#browsing-data-collected-using-schemas" rel="nofollow">Browsing data collected using schemas</a></li>
  </ul>
  </li>
  <li><a href="https://llm.datasette.io/en/stable/logging.html#browsing-logs-using-datasette" rel="nofollow">Browsing logs using Datasette</a></li>
  <li><a href="https://llm.datasette.io/en/stable/logging.html#backing-up-your-database" rel="nofollow">Backing up your database</a></li>
  <li><a href="https://llm.datasette.io/en/stable/logging.html#sql-schema" rel="nofollow">SQL schema</a></li>
  </ul>
  </li>
  <li>
  <p dir="auto"><a href="https://llm.datasette.io/en/stable/related-tools.html" rel="nofollow">Related tools</a></p>
  <ul dir="auto">
  <li><a href="https://llm.datasette.io/en/stable/related-tools.html#strip-tags" rel="nofollow">strip-tags</a></li>
  <li><a href="https://llm.datasette.io/en/stable/related-tools.html#ttok" rel="nofollow">ttok</a></li>
  <li><a href="https://llm.datasette.io/en/stable/related-tools.html#symbex" rel="nofollow">Symbex</a></li>
  </ul>
  </li>
  <li>
  <p dir="auto"><a href="https://llm.datasette.io/en/stable/help.html" rel="nofollow">CLI reference</a></p>
  <ul dir="auto">
  <li><a href="https://llm.datasette.io/en/stable/help.html#llm-help" rel="nofollow">llm  –help</a>
  <ul dir="auto">
  <li><a href="https://llm.datasette.io/en/stable/help.html#llm-prompt-help" rel="nofollow">llm prompt –help</a></li>
  <li><a href="https://llm.datasette.io/en/stable/help.html#llm-chat-help" rel="nofollow">llm chat –help</a></li>
  <li><a href="https://llm.datasette.io/en/stable/help.html#llm-keys-help" rel="nofollow">llm keys –help</a></li>
  <li><a href="https://llm.datasette.io/en/stable/help.html#llm-logs-help" rel="nofollow">llm logs –help</a></li>
  <li><a href="https://llm.datasette.io/en/stable/help.html#llm-models-help" rel="nofollow">llm models –help</a></li>
  <li><a href="https://llm.datasette.io/en/stable/help.html#llm-templates-help" rel="nofollow">llm templates –help</a></li>
  <li><a href="https://llm.datasette.io/en/stable/help.html#llm-schemas-help" rel="nofollow">llm schemas –help</a></li>
  <li><a href="https://llm.datasette.io/en/stable/help.html#llm-tools-help" rel="nofollow">llm tools –help</a></li>
  <li><a href="https://llm.datasette.io/en/stable/help.html#llm-aliases-help" rel="nofollow">llm aliases –help</a></li>
  <li><a href="https://llm.datasette.io/en/stable/help.html#llm-fragments-help" rel="nofollow">llm fragments –help</a></li>
  <li><a href="https://llm.datasette.io/en/stable/help.html#llm-plugins-help" rel="nofollow">llm plugins –help</a></li>
  <li><a href="https://llm.datasette.io/en/stable/help.html#llm-install-help" rel="nofollow">llm install –help</a></li>
  <li><a href="https://llm.datasette.io/en/stable/help.html#llm-uninstall-help" rel="nofollow">llm uninstall –help</a></li>
  <li><a href="https://llm.datasette.io/en/stable/help.html#llm-embed-help" rel="nofollow">llm embed –help</a></li>
  <li><a href="https://llm.datasette.io/en/stable/help.html#llm-embed-multi-help" rel="nofollow">llm embed-multi –help</a></li>
  <li><a href="https://llm.datasette.io/en/stable/help.html#llm-similar-help" rel="nofollow">llm similar –help</a></li>
  <li><a href="https://llm.datasette.io/en/stable/help.html#llm-embed-models-help" rel="nofollow">llm embed-models –help</a></li>
  <li><a href="https://llm.datasette.io/en/stable/help.html#llm-collections-help" rel="nofollow">llm collections –help</a></li>
  <li><a href="https://llm.datasette.io/en/stable/help.html#llm-openai-help" rel="nofollow">llm openai –help</a></li>
  </ul>
  </li>
  </ul>
  </li>
  <li>
  <p dir="auto"><a href="https://llm.datasette.io/en/stable/contributing.html" rel="nofollow">Contributing</a></p>
  <ul dir="auto">
  <li><a href="https://llm.datasette.io/en/stable/contributing.html#updating-recorded-http-api-interactions-and-associated-snapshots" rel="nofollow">Updating recorded HTTP API interactions and associated snapshots</a></li>
  <li><a href="https://llm.datasette.io/en/stable/contributing.html#debugging-tricks" rel="nofollow">Debugging tricks</a></li>
  <li><a href="https://llm.datasette.io/en/stable/contributing.html#documentation" rel="nofollow">Documentation</a></li>
  <li><a href="https://llm.datasette.io/en/stable/contributing.html#release-process" rel="nofollow">Release process</a></li>
  </ul>
  </li>
  <li>
  <p dir="auto"><a href="https://llm.datasette.io/en/stable/changelog.html" rel="nofollow">Changelog</a></p>
  </li>
  </ul>

  </article></div>


node_id: R_kgDOJksOkw
name: strip-tags
full_name: simonw/strip-tags
html_url: https://github.com/simonw/strip-tags
description: CLI tool for stripping tags from HTML
created_at: 2023-05-18T15:44:34Z
updated_at: 2025-05-07T03:01:51Z
pushed_at: 2025-02-28T21:53:34Z
language: Python
license: apache-2.0
topics: []
visibility: public
default_branch: main
permissions: {"admin": false, "maintain": false, "push": false, "triage": false, "pull": true}
readme:
  # strip-tags

  [![PyPI](https://img.shields.io/pypi/v/strip-tags.svg)](https://pypi.org/project/strip-tags/)
  [![Changelog](https://img.shields.io/github/v/release/simonw/strip-tags?include_prereleases&label=changelog)](https://github.com/simonw/strip-tags/releases)
  [![Tests](https://github.com/simonw/strip-tags/workflows/Test/badge.svg)](https://github.com/simonw/strip-tags/actions?query=workflow%3ATest)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/strip-tags/blob/master/LICENSE)

  Strip tags from HTML, optionally from areas identified by CSS selectors

  See [llm, ttok and strip-tags—CLI tools for working with ChatGPT and other LLMs](https://simonwillison.net/2023/May/18/cli-tools-for-llms/) for more on this project.

  ## Installation

  Install this tool using `pip`:
  ```bash
  pip install strip-tags
  ```
  ## Usage

  Pipe content into this tool to strip tags from it:
  ```bash
  cat input.html | strip-tags > output.txt
  ````
  Or pass a filename:
  ```bash
  strip-tags -i input.html > output.txt
  ```
  To run against just specific areas identified by CSS selectors:
  ```bash
  strip-tags '.content' -i input.html > output.txt
  ```
  This can be called with multiple selectors:
  ```bash
  cat input.html | strip-tags '.content' '.sidebar' > output.txt
  ```
  To return just the first element on the page that matches one of the selectors, use `--first`:
  ```bash
  cat input.html | strip-tags .content --first > output.txt
  ```
  To remove content contained by specific selectors - e.g. the `<nav>` section of a page, use `-r` or `--remove`:
  ```bash
  cat input.html | strip-tags -r nav > output.txt
  ```
  To minify whitespace - reducing multiple space and tab characters to a single space, removing any remaining blank lines - add `-m` or `--minify`:
  ```bash
  cat input.html | strip-tags -m > output.txt
  ```
  You can also run this command using `python -m` like this:
  ```bash
  python -m strip_tags --help
  ```
  ### Keeping the markup for specified tags

  When passing content to a language model, it can sometimes be useful to leave in a subset of HTML tags - `<h1>This is the heading</h1>` for example - to provide extra hints to the model.

  The `-t/--keep-tag` option can be passed multiple times to specify tags that should be kept.

  This example looks at the `<header>` section of https://datasette.io/ and keeps the tags around the list items and `<h1>` elements:

  ```
  curl -s https://datasette.io/ | strip-tags header -t h1 -t li
  ```
  ```html
  <li>Uses</li>
  <li>Documentation Docs</li>
  <li>Tutorials</li>
  <li>Examples</li>
  <li>Plugins</li>
  <li>Tools</li>
  <li>News</li>
  <h1>
      Datasette
  </h1>
  Find stories in data
  ```
  All attributes will be removed from the tags, except for the `id=` and `class=` attribute since those may provide further useful hints to the language model.

  The `href` attribute on links, the `alt` attribute on images and the `name` and `value` attributes on `meta` tags are kept as well.

  You can also specify a bundle of tags. For example, `strip-tags -t hs` will keep the tag markup for all levels of headings.

  The following bundles can be used:

  <!-- [[[cog
  import cog
  from strip_tags.lib import BUNDLES
  lines = []
  for name, tags in BUNDLES.items():
      lines.append("- `-t {}`: {}".format(name, ", ".join("`<{}>`".format(tag) for tag in tags)))
  cog.out("\n".join(lines))
  ]]] -->
  - `-t hs`: `<h1>`, `<h2>`, `<h3>`, `<h4>`, `<h5>`, `<h6>`
  - `-t metadata`: `<title>`, `<meta>`
  - `-t structure`: `<header>`, `<nav>`, `<main>`, `<article>`, `<section>`, `<aside>`, `<footer>`
  - `-t tables`: `<table>`, `<tr>`, `<td>`, `<th>`, `<thead>`, `<tbody>`, `<tfoot>`, `<caption>`, `<colgroup>`, `<col>`
  - `-t lists`: `<ul>`, `<ol>`, `<li>`, `<dl>`, `<dd>`, `<dt>`
  <!-- [[[end]]] -->

  ## As a Python library

  You can use `strip-tags` from Python code too. The function signature looks like this:

  <!-- [[[cog
  import ast
  module = ast.parse(open("strip_tags/lib.py").read())
  strip_tags = [
      fn for fn in module.body
      if getattr(fn, 'name', None) == 'strip_tags'
  ][0]
  code = ast.unparse(strip_tags)
  defline = code.split("\n")[0]
  code = (
      ',\n    '.join(defline.split(', ')).replace(") ->", "\n) ->").replace("strip_tags(", "strip_tags(\n    ")
  )
  cog.out("```python\n{}\n```".format(code))
  ]]] -->
  ```python
  def strip_tags(
      input: str,
      selectors: Optional[Iterable[str]]=None,
      *,
      removes: Optional[Iterable[str]]=None,
      minify: bool=False,
      remove_blank_lines: bool=False,
      first: bool=False,
      keep_tags: Optional[Iterable[str]]=None,
      all_attrs: bool=False
  ) -> str:
  ```
  <!-- [[[end]]] -->

  Here's an example:
  ```python
  from strip_tags import strip_tags

  html = """
  <div>
  <h1>This has tags</h1>

  <p>And whitespace too</p>
  </div>
  Ignore this bit.
  """
  stripped = strip_tags(html, ["div"], minify=True, keep_tags=["h1"])
  print(stripped)
  ```
  Output:
  ```
  <h1>This has tags</h1>

  And whitespace too
  ```
  Use `remove_blank_lines=True` to remove any remaining blank lines from the output.

  ## strip-tags --help

  <!-- [[[cog
  import cog
  from strip_tags import cli
  from click.testing import CliRunner
  runner = CliRunner()
  result = runner.invoke(cli.cli, ["--help"])
  help = result.output.replace("Usage: cli", "Usage: strip-tags")
  cog.out(
      "```\n{}\n```".format(help)
  )
  ]]] -->
  ```
  Usage: strip-tags [OPTIONS] [SELECTORS]...

    Strip tags from HTML, optionally from areas identified by CSS selectors

    Example usage:

        cat input.html | strip-tags > output.txt

    To run against just specific areas identified by CSS selectors:

        cat input.html | strip-tags .entry .footer > output.txt

  Options:
    --version             Show the version and exit.
    -r, --remove TEXT     Remove content in these selectors
    -i, --input FILENAME  Input file
    -m, --minify          Minify whitespace
    -t, --keep-tag TEXT   Keep these <tags>
    --all-attrs           Include all attributes on kept tags
    --first               First element matching the selectors
    --help                Show this message and exit.

  ```
  <!-- [[[end]]] -->

  ## Development

  To contribute to this tool, first checkout the code. Then create a new virtual environment:
  ```bash
  cd strip-tags
  python -m venv venv
  source venv/bin/activate
  ```
  Now install the dependencies and test dependencies:
  ```bash
  pip install -e '.[test]'
  ```
  To run the tests:
  ```bash
  pytest
  ```

readme_html:
  <div id="readme" class="md" data-path="README.md"><article class="markdown-body entry-content container-lg" itemprop="text"><div class="markdown-heading" dir="auto"><h1 class="heading-element" dir="auto">strip-tags</h1><a id="user-content-strip-tags" class="anchor" aria-label="Permalink: strip-tags" href="#strip-tags"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto"><a href="https://pypi.org/project/strip-tags/" rel="nofollow"><img src="https://img.shields.io/pypi/v/strip-tags.svg" alt="PyPI" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/7d1a969afbeaf91a1c6f25492b2cc5d476f18b9a583c8f4987a59589697fc5d2/68747470733a2f2f696d672e736869656c64732e696f2f707970692f762f73747269702d746167732e737667"></a>
  <a href="https://github.com/simonw/strip-tags/releases"><img src="https://img.shields.io/github/v/release/simonw/strip-tags?include_prereleases&label=changelog" alt="Changelog" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/d97dc3eb63b5efc35d2b296da1183af4b2bdb622a5050856b4be5f63c1cceb09/68747470733a2f2f696d672e736869656c64732e696f2f6769746875622f762f72656c656173652f73696d6f6e772f73747269702d746167733f696e636c7564655f70726572656c6561736573266c6162656c3d6368616e67656c6f67"></a>
  <a href="https://github.com/simonw/strip-tags/actions?query=workflow%3ATest"><img src="https://github.com/simonw/strip-tags/workflows/Test/badge.svg" alt="Tests" style="max-width: 100%;"></a>
  <a href="https://github.com/simonw/strip-tags/blob/master/LICENSE"><img src="https://img.shields.io/badge/license-Apache%202.0-blue.svg" alt="License" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/c355f200ea90fddaa407b6eaab303663a669248ea3ca7b1fcf77dbe04ff5f48c/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6c6963656e73652d417061636865253230322e302d626c75652e737667"></a></p>
  <p dir="auto">Strip tags from HTML, optionally from areas identified by CSS selectors</p>
  <p dir="auto">See <a href="https://simonwillison.net/2023/May/18/cli-tools-for-llms/" rel="nofollow">llm, ttok and strip-tags—CLI tools for working with ChatGPT and other LLMs</a> for more on this project.</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Installation</h2><a id="user-content-installation" class="anchor" aria-label="Permalink: Installation" href="#installation"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">Install this tool using <code>pip</code>:</p>
  <div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="pip install strip-tags"><pre>pip install strip-tags</pre></div>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Usage</h2><a id="user-content-usage" class="anchor" aria-label="Permalink: Usage" href="#usage"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">Pipe content into this tool to strip tags from it:</p>
  <div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="cat input.html | strip-tags > output.txt"><pre>cat input.html <span class="pl-k">|</span> strip-tags <span class="pl-k">></span> output.txt</pre></div>
  <p dir="auto">Or pass a filename:</p>
  <div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="strip-tags -i input.html > output.txt"><pre>strip-tags -i input.html <span class="pl-k">></span> output.txt</pre></div>
  <p dir="auto">To run against just specific areas identified by CSS selectors:</p>
  <div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="strip-tags '.content' -i input.html > output.txt"><pre>strip-tags <span class="pl-s"><span class="pl-pds">'</span>.content<span class="pl-pds">'</span></span> -i input.html <span class="pl-k">></span> output.txt</pre></div>
  <p dir="auto">This can be called with multiple selectors:</p>
  <div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="cat input.html | strip-tags '.content' '.sidebar' > output.txt"><pre>cat input.html <span class="pl-k">|</span> strip-tags <span class="pl-s"><span class="pl-pds">'</span>.content<span class="pl-pds">'</span></span> <span class="pl-s"><span class="pl-pds">'</span>.sidebar<span class="pl-pds">'</span></span> <span class="pl-k">></span> output.txt</pre></div>
  <p dir="auto">To return just the first element on the page that matches one of the selectors, use <code>--first</code>:</p>
  <div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="cat input.html | strip-tags .content --first > output.txt"><pre>cat input.html <span class="pl-k">|</span> strip-tags .content --first <span class="pl-k">></span> output.txt</pre></div>
  <p dir="auto">To remove content contained by specific selectors - e.g. the <code><nav></code> section of a page, use <code>-r</code> or <code>--remove</code>:</p>
  <div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="cat input.html | strip-tags -r nav > output.txt"><pre>cat input.html <span class="pl-k">|</span> strip-tags -r nav <span class="pl-k">></span> output.txt</pre></div>
  <p dir="auto">To minify whitespace - reducing multiple space and tab characters to a single space, removing any remaining blank lines - add <code>-m</code> or <code>--minify</code>:</p>
  <div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="cat input.html | strip-tags -m > output.txt"><pre>cat input.html <span class="pl-k">|</span> strip-tags -m <span class="pl-k">></span> output.txt</pre></div>
  <p dir="auto">You can also run this command using <code>python -m</code> like this:</p>
  <div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="python -m strip_tags --help"><pre>python -m strip_tags --help</pre></div>
  <div class="markdown-heading" dir="auto"><h3 class="heading-element" dir="auto">Keeping the markup for specified tags</h3><a id="user-content-keeping-the-markup-for-specified-tags" class="anchor" aria-label="Permalink: Keeping the markup for specified tags" href="#keeping-the-markup-for-specified-tags"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">When passing content to a language model, it can sometimes be useful to leave in a subset of HTML tags - <code><h1>This is the heading</h1></code> for example - to provide extra hints to the model.</p>
  <p dir="auto">The <code>-t/--keep-tag</code> option can be passed multiple times to specify tags that should be kept.</p>
  <p dir="auto">This example looks at the <code><header></code> section of <a href="https://datasette.io/" rel="nofollow">https://datasette.io/</a> and keeps the tags around the list items and <code><h1></code> elements:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="curl -s https://datasette.io/ | strip-tags header -t h1 -t li"><pre class="notranslate"><code>curl -s https://datasette.io/ | strip-tags header -t h1 -t li
  </code></pre></div>
  <div class="highlight highlight-text-html-basic notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="<li>Uses</li>
  <li>Documentation Docs</li>
  <li>Tutorials</li>
  <li>Examples</li>
  <li>Plugins</li>
  <li>Tools</li>
  <li>News</li>
  <h1>
      Datasette
  </h1>
  Find stories in data"><pre><span class="pl-kos"><</span><span class="pl-ent">li</span><span class="pl-kos">></span>Uses<span class="pl-kos"></</span><span class="pl-ent">li</span><span class="pl-kos">></span>
  <span class="pl-kos"><</span><span class="pl-ent">li</span><span class="pl-kos">></span>Documentation Docs<span class="pl-kos"></</span><span class="pl-ent">li</span><span class="pl-kos">></span>
  <span class="pl-kos"><</span><span class="pl-ent">li</span><span class="pl-kos">></span>Tutorials<span class="pl-kos"></</span><span class="pl-ent">li</span><span class="pl-kos">></span>
  <span class="pl-kos"><</span><span class="pl-ent">li</span><span class="pl-kos">></span>Examples<span class="pl-kos"></</span><span class="pl-ent">li</span><span class="pl-kos">></span>
  <span class="pl-kos"><</span><span class="pl-ent">li</span><span class="pl-kos">></span>Plugins<span class="pl-kos"></</span><span class="pl-ent">li</span><span class="pl-kos">></span>
  <span class="pl-kos"><</span><span class="pl-ent">li</span><span class="pl-kos">></span>Tools<span class="pl-kos"></</span><span class="pl-ent">li</span><span class="pl-kos">></span>
  <span class="pl-kos"><</span><span class="pl-ent">li</span><span class="pl-kos">></span>News<span class="pl-kos"></</span><span class="pl-ent">li</span><span class="pl-kos">></span>
  <span class="pl-kos"><</span><span class="pl-ent">h1</span><span class="pl-kos">></span>
      Datasette
  <span class="pl-kos"></</span><span class="pl-ent">h1</span><span class="pl-kos">></span>
  Find stories in data</pre></div>
  <p dir="auto">All attributes will be removed from the tags, except for the <code>id=</code> and <code>class=</code> attribute since those may provide further useful hints to the language model.</p>
  <p dir="auto">The <code>href</code> attribute on links, the <code>alt</code> attribute on images and the <code>name</code> and <code>value</code> attributes on <code>meta</code> tags are kept as well.</p>
  <p dir="auto">You can also specify a bundle of tags. For example, <code>strip-tags -t hs</code> will keep the tag markup for all levels of headings.</p>
  <p dir="auto">The following bundles can be used:</p>

  <ul dir="auto">
  <li><code>-t hs</code>: <code><h1></code>, <code><h2></code>, <code><h3></code>, <code><h4></code>, <code><h5></code>, <code><h6></code></li>
  <li><code>-t metadata</code>: <code><title></code>, <code><meta></code></li>
  <li><code>-t structure</code>: <code><header></code>, <code><nav></code>, <code><main></code>, <code><article></code>, <code><section></code>, <code><aside></code>, <code><footer></code></li>
  <li><code>-t tables</code>: <code><table></code>, <code><tr></code>, <code><td></code>, <code><th></code>, <code><thead></code>, <code><tbody></code>, <code><tfoot></code>, <code><caption></code>, <code><colgroup></code>, <code><col></code></li>
  <li><code>-t lists</code>: <code><ul></code>, <code><ol></code>, <code><li></code>, <code><dl></code>, <code><dd></code>, <code><dt></code></li>
  </ul>

  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">As a Python library</h2><a id="user-content-as-a-python-library" class="anchor" aria-label="Permalink: As a Python library" href="#as-a-python-library"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">You can use <code>strip-tags</code> from Python code too. The function signature looks like this:</p>

  <div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="def strip_tags(
      input: str,
      selectors: Optional[Iterable[str]]=None,
      *,
      removes: Optional[Iterable[str]]=None,
      minify: bool=False,
      remove_blank_lines: bool=False,
      first: bool=False,
      keep_tags: Optional[Iterable[str]]=None,
      all_attrs: bool=False
  ) -> str:"><pre><span class="pl-k">def</span> <span class="pl-en">strip_tags</span>(
      <span class="pl-s1">input</span>: <span class="pl-smi">str</span>,
      <span class="pl-s1">selectors</span>: <span class="pl-v">Optional</span>[<span class="pl-v">Iterable</span>[<span class="pl-smi">str</span>]]<span class="pl-c1">=</span><span class="pl-c1">None</span>,
      <span class="pl-c1">*</span>,
      <span class="pl-s1">removes</span>: <span class="pl-v">Optional</span>[<span class="pl-v">Iterable</span>[<span class="pl-smi">str</span>]]<span class="pl-c1">=</span><span class="pl-c1">None</span>,
      <span class="pl-s1">minify</span>: <span class="pl-smi">bool</span><span class="pl-c1">=</span><span class="pl-c1">False</span>,
      <span class="pl-s1">remove_blank_lines</span>: <span class="pl-smi">bool</span><span class="pl-c1">=</span><span class="pl-c1">False</span>,
      <span class="pl-s1">first</span>: <span class="pl-smi">bool</span><span class="pl-c1">=</span><span class="pl-c1">False</span>,
      <span class="pl-s1">keep_tags</span>: <span class="pl-v">Optional</span>[<span class="pl-v">Iterable</span>[<span class="pl-smi">str</span>]]<span class="pl-c1">=</span><span class="pl-c1">None</span>,
      <span class="pl-s1">all_attrs</span>: <span class="pl-smi">bool</span><span class="pl-c1">=</span><span class="pl-c1">False</span>
  ) <span class="pl-c1">-></span> <span class="pl-smi">str</span>:</pre></div>

  <p dir="auto">Here's an example:</p>
  <div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="from strip_tags import strip_tags

  html = """
  <div>
  <h1>This has tags</h1>

  <p>And whitespace too</p>
  </div>
  Ignore this bit.
  """
  stripped = strip_tags(html, ["div"], minify=True, keep_tags=["h1"])
  print(stripped)"><pre><span class="pl-k">from</span> <span class="pl-s1">strip_tags</span> <span class="pl-k">import</span> <span class="pl-s1">strip_tags</span>

  <span class="pl-s1">html</span> <span class="pl-c1">=</span> <span class="pl-s">"""</span>
  <span class="pl-s"><div></span>
  <span class="pl-s"><h1>This has tags</h1></span>
  <span class="pl-s"></span>
  <span class="pl-s"><p>And whitespace too</p></span>
  <span class="pl-s"></div></span>
  <span class="pl-s">Ignore this bit.</span>
  <span class="pl-s">"""</span>
  <span class="pl-s1">stripped</span> <span class="pl-c1">=</span> <span class="pl-en">strip_tags</span>(<span class="pl-s1">html</span>, [<span class="pl-s">"div"</span>], <span class="pl-s1">minify</span><span class="pl-c1">=</span><span class="pl-c1">True</span>, <span class="pl-s1">keep_tags</span><span class="pl-c1">=</span>[<span class="pl-s">"h1"</span>])
  <span class="pl-en">print</span>(<span class="pl-s1">stripped</span>)</pre></div>
  <p dir="auto">Output:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="<h1>This has tags</h1>

  And whitespace too"><pre class="notranslate"><code><h1>This has tags</h1>

  And whitespace too
  </code></pre></div>
  <p dir="auto">Use <code>remove_blank_lines=True</code> to remove any remaining blank lines from the output.</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">strip-tags --help</h2><a id="user-content-strip-tags---help" class="anchor" aria-label="Permalink: strip-tags --help" href="#strip-tags---help"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>

  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="Usage: strip-tags [OPTIONS] [SELECTORS]...

    Strip tags from HTML, optionally from areas identified by CSS selectors

    Example usage:

        cat input.html | strip-tags > output.txt

    To run against just specific areas identified by CSS selectors:

        cat input.html | strip-tags .entry .footer > output.txt

  Options:
    --version             Show the version and exit.
    -r, --remove TEXT     Remove content in these selectors
    -i, --input FILENAME  Input file
    -m, --minify          Minify whitespace
    -t, --keep-tag TEXT   Keep these <tags>
    --all-attrs           Include all attributes on kept tags
    --first               First element matching the selectors
    --help                Show this message and exit.
  "><pre class="notranslate"><code>Usage: strip-tags [OPTIONS] [SELECTORS]...

    Strip tags from HTML, optionally from areas identified by CSS selectors

    Example usage:

        cat input.html | strip-tags > output.txt

    To run against just specific areas identified by CSS selectors:

        cat input.html | strip-tags .entry .footer > output.txt

  Options:
    --version             Show the version and exit.
    -r, --remove TEXT     Remove content in these selectors
    -i, --input FILENAME  Input file
    -m, --minify          Minify whitespace
    -t, --keep-tag TEXT   Keep these <tags>
    --all-attrs           Include all attributes on kept tags
    --first               First element matching the selectors
    --help                Show this message and exit.

  </code></pre></div>

  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Development</h2><a id="user-content-development" class="anchor" aria-label="Permalink: Development" href="#development"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">To contribute to this tool, first checkout the code. Then create a new virtual environment:</p>
  <div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="cd strip-tags
  python -m venv venv
  source venv/bin/activate"><pre><span class="pl-c1">cd</span> strip-tags
  python -m venv venv
  <span class="pl-c1">source</span> venv/bin/activate</pre></div>
  <p dir="auto">Now install the dependencies and test dependencies:</p>
  <div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="pip install -e '.[test]'"><pre>pip install -e <span class="pl-s"><span class="pl-pds">'</span>.[test]<span class="pl-pds">'</span></span></pre></div>
  <p dir="auto">To run the tests:</p>
  <div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="pytest"><pre>pytest</pre></div>
  </article></div>


node_id: R_kgDOJkvrZQ
name: ttok
full_name: simonw/ttok
html_url: https://github.com/simonw/ttok
description: Count and truncate text based on tokens
created_at: 2023-05-18T18:22:59Z
updated_at: 2025-05-13T11:28:20Z
pushed_at: 2024-05-02T23:37:54Z
language: Python
license: apache-2.0
topics: []
visibility: public
default_branch: main
permissions: {"admin": false, "maintain": false, "push": false, "triage": false, "pull": true}
readme:
  # ttok

  [![PyPI](https://img.shields.io/pypi/v/ttok.svg)](https://pypi.org/project/ttok/)
  [![Changelog](https://img.shields.io/github/v/release/simonw/ttok?include_prereleases&label=changelog)](https://github.com/simonw/ttok/releases)
  [![Tests](https://github.com/simonw/ttok/workflows/Test/badge.svg)](https://github.com/simonw/ttok/actions?query=workflow%3ATest)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/ttok/blob/master/LICENSE)

  Count and truncate text based on tokens

  ## Background

  Large language models such as GPT-3.5 and GPT-4 work in terms of tokens.

  This tool can count tokens, using OpenAI's [tiktoken](https://github.com/openai/tiktoken) library.

  It can also truncate text to a specified number of tokens.

  See [llm, ttok and strip-tags—CLI tools for working with ChatGPT and other LLMs](https://simonwillison.net/2023/May/18/cli-tools-for-llms/) for more on this project.

  ## Installation

  Install this tool using `pip`:
  ```bash
  pip install ttok
  ```
  Or using Homebrew:
  ```bash
  brew install simonw/llm/ttok
  ```

  ## Counting tokens

  Provide text as arguments to this tool to count tokens:

  ```bash
  ttok Hello world
  ```
  ```
  2
  ```
  You can also pipe text into the tool:
  ```bash
  echo -n "Hello world" | ttok
  ```
  ```
  2
  ```
  Here the `echo -n` option prevents echo from adding a newline - without that you would get a token count of 3.

  To pipe in text and then append extra tokens from arguments, use the `-i -` option:

  ```bash
  echo -n "Hello world" | ttok more text -i -
  ```
  ```
  6
  ```
  ## Different models

  By default, the tokenizer model for GPT-3.5 and GPT-4 is used.

  To use the model for GPT-2 and GPT-3, add `--model gpt2`:

  ```bash
  ttok boo Hello there this is -m gpt2
  ```
  ```
  6
  ```
  Compared to GPT-3.5:
  ```bash
  ttok boo Hello there this is
  ```
  ```
  5
  ```
  Further model options are [documented here](https://github.com/openai/openai-cookbook/blob/main/examples/How_to_count_tokens_with_tiktoken.ipynb).

  ## Truncating text

  Use the `-t 10` or `--truncate 10` option to truncate text to a specified number of tokens:

  ```bash
  ttok This is too many tokens -t 3
  ```
  ```
  This is too
  ```

  ## Viewing tokens

  The `--encode` option can be used to view the integer token IDs for the incoming text:

  ```bash
  ttok Hello world --encode
  ```
  ```
  9906 1917
  ```
  The `--decode` method reverses this process:

  ```bash
  ttok 9906 1917 --decode
  ```
  ```
  Hello world
  ```
  Add `--tokens` to either of these options to see a detailed breakdown of the tokens:

  ```bash
  ttok Hello world --encode --tokens
  ```
  ```
  [b'Hello', b' world']
  ```

  ## Available models

  This is the full list of available models and their corresponding encodings. Model names and encoding names are valid for the `-m/--model` option.

  <!-- [[[cog
  import cog
  import tiktoken
  output = []
  for key, value in tiktoken.model.MODEL_TO_ENCODING.items():
      output.append("- `{}` (`{}`)".format(key, value))
  cog.out("\n".join(output))
  ]]] -->
  - `gpt-4` (`cl100k_base`)
  - `gpt-3.5-turbo` (`cl100k_base`)
  - `gpt-3.5` (`cl100k_base`)
  - `gpt-35-turbo` (`cl100k_base`)
  - `davinci-002` (`cl100k_base`)
  - `babbage-002` (`cl100k_base`)
  - `text-embedding-ada-002` (`cl100k_base`)
  - `text-embedding-3-small` (`cl100k_base`)
  - `text-embedding-3-large` (`cl100k_base`)
  - `text-davinci-003` (`p50k_base`)
  - `text-davinci-002` (`p50k_base`)
  - `text-davinci-001` (`r50k_base`)
  - `text-curie-001` (`r50k_base`)
  - `text-babbage-001` (`r50k_base`)
  - `text-ada-001` (`r50k_base`)
  - `davinci` (`r50k_base`)
  - `curie` (`r50k_base`)
  - `babbage` (`r50k_base`)
  - `ada` (`r50k_base`)
  - `code-davinci-002` (`p50k_base`)
  - `code-davinci-001` (`p50k_base`)
  - `code-cushman-002` (`p50k_base`)
  - `code-cushman-001` (`p50k_base`)
  - `davinci-codex` (`p50k_base`)
  - `cushman-codex` (`p50k_base`)
  - `text-davinci-edit-001` (`p50k_edit`)
  - `code-davinci-edit-001` (`p50k_edit`)
  - `text-similarity-davinci-001` (`r50k_base`)
  - `text-similarity-curie-001` (`r50k_base`)
  - `text-similarity-babbage-001` (`r50k_base`)
  - `text-similarity-ada-001` (`r50k_base`)
  - `text-search-davinci-doc-001` (`r50k_base`)
  - `text-search-curie-doc-001` (`r50k_base`)
  - `text-search-babbage-doc-001` (`r50k_base`)
  - `text-search-ada-doc-001` (`r50k_base`)
  - `code-search-babbage-code-001` (`r50k_base`)
  - `code-search-ada-code-001` (`r50k_base`)
  - `gpt2` (`gpt2`)
  - `gpt-2` (`gpt2`)
  <!-- [[[end]]] -->

  ## ttok --help

  <!-- [[[cog
  from ttok import cli
  from click.testing import CliRunner
  runner = CliRunner()
  result = runner.invoke(cli.cli, ["--help"])
  help = result.output.replace("Usage: cli", "Usage: ttok")
  cog.out(
      "```\n{}\n```".format(help)
  )
  ]]] -->
  ```
  Usage: ttok [OPTIONS] [PROMPT]...

    Count and truncate text based on tokens

    To count tokens for text passed as arguments:

        ttok one two three

    To count tokens from stdin:

        cat input.txt | ttok

    To truncate to 100 tokens:

        cat input.txt | ttok -t 100

    To truncate to 100 tokens using the gpt2 model:

        cat input.txt | ttok -t 100 -m gpt2

    To view token integers:

        cat input.txt | ttok --encode

    To convert tokens back to text:

        ttok 9906 1917 --decode

    To see the details of the tokens:

        ttok "hello world" --tokens

    Outputs:

        [b'hello', b' world']

  Options:
    --version               Show the version and exit.
    -i, --input FILENAME
    -t, --truncate INTEGER  Truncate to this many tokens
    -m, --model TEXT        Which model to use
    --encode, --tokens      Output token integers
    --decode                Convert token integers to text
    --tokens                Output full tokens
    --allow-special         Do not error on special tokens
    --help                  Show this message and exit.

  ```
  <!-- [[[end]]] -->

  You can also run this command using:

  ```bash
  python -m ttok --help
  ```

  ## Development

  To contribute to this tool, first checkout the code. Then create a new virtual environment:

  ```bash
  cd ttok
  python -m venv venv
  source venv/bin/activate
  ```

  Now install the dependencies and test dependencies:

  ```bash
  pip install -e '.[test]'
  ```

  To run the tests:

  ```bash
  pytest
  ```

readme_html:
  <div id="readme" class="md" data-path="README.md"><article class="markdown-body entry-content container-lg" itemprop="text"><div class="markdown-heading" dir="auto"><h1 class="heading-element" dir="auto">ttok</h1><a id="user-content-ttok" class="anchor" aria-label="Permalink: ttok" href="#ttok"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto"><a href="https://pypi.org/project/ttok/" rel="nofollow"><img src="https://img.shields.io/pypi/v/ttok.svg" alt="PyPI" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/7d87e9f1867c0f6892ee93994da7f30a421fdc5c9fa707d23da5e14db6580d5e/68747470733a2f2f696d672e736869656c64732e696f2f707970692f762f74746f6b2e737667"></a>
  <a href="https://github.com/simonw/ttok/releases"><img src="https://img.shields.io/github/v/release/simonw/ttok?include_prereleases&label=changelog" alt="Changelog" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/bb85ac35bf6f7c4b7656ff69370a17885b49b2dae1eab796da4b1672e192939a/68747470733a2f2f696d672e736869656c64732e696f2f6769746875622f762f72656c656173652f73696d6f6e772f74746f6b3f696e636c7564655f70726572656c6561736573266c6162656c3d6368616e67656c6f67"></a>
  <a href="https://github.com/simonw/ttok/actions?query=workflow%3ATest"><img src="https://github.com/simonw/ttok/workflows/Test/badge.svg" alt="Tests" style="max-width: 100%;"></a>
  <a href="https://github.com/simonw/ttok/blob/master/LICENSE"><img src="https://img.shields.io/badge/license-Apache%202.0-blue.svg" alt="License" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/c355f200ea90fddaa407b6eaab303663a669248ea3ca7b1fcf77dbe04ff5f48c/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6c6963656e73652d417061636865253230322e302d626c75652e737667"></a></p>
  <p dir="auto">Count and truncate text based on tokens</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Background</h2><a id="user-content-background" class="anchor" aria-label="Permalink: Background" href="#background"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">Large language models such as GPT-3.5 and GPT-4 work in terms of tokens.</p>
  <p dir="auto">This tool can count tokens, using OpenAI's <a href="https://github.com/openai/tiktoken">tiktoken</a> library.</p>
  <p dir="auto">It can also truncate text to a specified number of tokens.</p>
  <p dir="auto">See <a href="https://simonwillison.net/2023/May/18/cli-tools-for-llms/" rel="nofollow">llm, ttok and strip-tags—CLI tools for working with ChatGPT and other LLMs</a> for more on this project.</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Installation</h2><a id="user-content-installation" class="anchor" aria-label="Permalink: Installation" href="#installation"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">Install this tool using <code>pip</code>:</p>
  <div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="pip install ttok"><pre>pip install ttok</pre></div>
  <p dir="auto">Or using Homebrew:</p>
  <div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="brew install simonw/llm/ttok"><pre>brew install simonw/llm/ttok</pre></div>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Counting tokens</h2><a id="user-content-counting-tokens" class="anchor" aria-label="Permalink: Counting tokens" href="#counting-tokens"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">Provide text as arguments to this tool to count tokens:</p>
  <div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="ttok Hello world"><pre>ttok Hello world</pre></div>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="2"><pre class="notranslate"><code>2
  </code></pre></div>
  <p dir="auto">You can also pipe text into the tool:</p>
  <div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="echo -n "Hello world" | ttok"><pre><span class="pl-c1">echo</span> -n <span class="pl-s"><span class="pl-pds">"</span>Hello world<span class="pl-pds">"</span></span> <span class="pl-k">|</span> ttok</pre></div>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="2"><pre class="notranslate"><code>2
  </code></pre></div>
  <p dir="auto">Here the <code>echo -n</code> option prevents echo from adding a newline - without that you would get a token count of 3.</p>
  <p dir="auto">To pipe in text and then append extra tokens from arguments, use the <code>-i -</code> option:</p>
  <div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="echo -n "Hello world" | ttok more text -i -"><pre><span class="pl-c1">echo</span> -n <span class="pl-s"><span class="pl-pds">"</span>Hello world<span class="pl-pds">"</span></span> <span class="pl-k">|</span> ttok more text -i -</pre></div>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="6"><pre class="notranslate"><code>6
  </code></pre></div>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Different models</h2><a id="user-content-different-models" class="anchor" aria-label="Permalink: Different models" href="#different-models"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">By default, the tokenizer model for GPT-3.5 and GPT-4 is used.</p>
  <p dir="auto">To use the model for GPT-2 and GPT-3, add <code>--model gpt2</code>:</p>
  <div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="ttok boo Hello there this is -m gpt2"><pre>ttok boo Hello there this is -m gpt2</pre></div>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="6"><pre class="notranslate"><code>6
  </code></pre></div>
  <p dir="auto">Compared to GPT-3.5:</p>
  <div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="ttok boo Hello there this is"><pre>ttok boo Hello there this is</pre></div>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="5"><pre class="notranslate"><code>5
  </code></pre></div>
  <p dir="auto">Further model options are <a href="https://github.com/openai/openai-cookbook/blob/main/examples/How_to_count_tokens_with_tiktoken.ipynb">documented here</a>.</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Truncating text</h2><a id="user-content-truncating-text" class="anchor" aria-label="Permalink: Truncating text" href="#truncating-text"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">Use the <code>-t 10</code> or <code>--truncate 10</code> option to truncate text to a specified number of tokens:</p>
  <div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="ttok This is too many tokens -t 3"><pre>ttok This is too many tokens -t 3</pre></div>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="This is too"><pre class="notranslate"><code>This is too
  </code></pre></div>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Viewing tokens</h2><a id="user-content-viewing-tokens" class="anchor" aria-label="Permalink: Viewing tokens" href="#viewing-tokens"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">The <code>--encode</code> option can be used to view the integer token IDs for the incoming text:</p>
  <div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="ttok Hello world --encode"><pre>ttok Hello world --encode</pre></div>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="9906 1917"><pre class="notranslate"><code>9906 1917
  </code></pre></div>
  <p dir="auto">The <code>--decode</code> method reverses this process:</p>
  <div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="ttok 9906 1917 --decode"><pre>ttok 9906 1917 --decode</pre></div>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="Hello world"><pre class="notranslate"><code>Hello world
  </code></pre></div>
  <p dir="auto">Add <code>--tokens</code> to either of these options to see a detailed breakdown of the tokens:</p>
  <div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="ttok Hello world --encode --tokens"><pre>ttok Hello world --encode --tokens</pre></div>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="[b'Hello', b' world']"><pre class="notranslate"><code>[b'Hello', b' world']
  </code></pre></div>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Available models</h2><a id="user-content-available-models" class="anchor" aria-label="Permalink: Available models" href="#available-models"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">This is the full list of available models and their corresponding encodings. Model names and encoding names are valid for the <code>-m/--model</code> option.</p>

  <ul dir="auto">
  <li><code>gpt-4</code> (<code>cl100k_base</code>)</li>
  <li><code>gpt-3.5-turbo</code> (<code>cl100k_base</code>)</li>
  <li><code>gpt-3.5</code> (<code>cl100k_base</code>)</li>
  <li><code>gpt-35-turbo</code> (<code>cl100k_base</code>)</li>
  <li><code>davinci-002</code> (<code>cl100k_base</code>)</li>
  <li><code>babbage-002</code> (<code>cl100k_base</code>)</li>
  <li><code>text-embedding-ada-002</code> (<code>cl100k_base</code>)</li>
  <li><code>text-embedding-3-small</code> (<code>cl100k_base</code>)</li>
  <li><code>text-embedding-3-large</code> (<code>cl100k_base</code>)</li>
  <li><code>text-davinci-003</code> (<code>p50k_base</code>)</li>
  <li><code>text-davinci-002</code> (<code>p50k_base</code>)</li>
  <li><code>text-davinci-001</code> (<code>r50k_base</code>)</li>
  <li><code>text-curie-001</code> (<code>r50k_base</code>)</li>
  <li><code>text-babbage-001</code> (<code>r50k_base</code>)</li>
  <li><code>text-ada-001</code> (<code>r50k_base</code>)</li>
  <li><code>davinci</code> (<code>r50k_base</code>)</li>
  <li><code>curie</code> (<code>r50k_base</code>)</li>
  <li><code>babbage</code> (<code>r50k_base</code>)</li>
  <li><code>ada</code> (<code>r50k_base</code>)</li>
  <li><code>code-davinci-002</code> (<code>p50k_base</code>)</li>
  <li><code>code-davinci-001</code> (<code>p50k_base</code>)</li>
  <li><code>code-cushman-002</code> (<code>p50k_base</code>)</li>
  <li><code>code-cushman-001</code> (<code>p50k_base</code>)</li>
  <li><code>davinci-codex</code> (<code>p50k_base</code>)</li>
  <li><code>cushman-codex</code> (<code>p50k_base</code>)</li>
  <li><code>text-davinci-edit-001</code> (<code>p50k_edit</code>)</li>
  <li><code>code-davinci-edit-001</code> (<code>p50k_edit</code>)</li>
  <li><code>text-similarity-davinci-001</code> (<code>r50k_base</code>)</li>
  <li><code>text-similarity-curie-001</code> (<code>r50k_base</code>)</li>
  <li><code>text-similarity-babbage-001</code> (<code>r50k_base</code>)</li>
  <li><code>text-similarity-ada-001</code> (<code>r50k_base</code>)</li>
  <li><code>text-search-davinci-doc-001</code> (<code>r50k_base</code>)</li>
  <li><code>text-search-curie-doc-001</code> (<code>r50k_base</code>)</li>
  <li><code>text-search-babbage-doc-001</code> (<code>r50k_base</code>)</li>
  <li><code>text-search-ada-doc-001</code> (<code>r50k_base</code>)</li>
  <li><code>code-search-babbage-code-001</code> (<code>r50k_base</code>)</li>
  <li><code>code-search-ada-code-001</code> (<code>r50k_base</code>)</li>
  <li><code>gpt2</code> (<code>gpt2</code>)</li>
  <li><code>gpt-2</code> (<code>gpt2</code>)</li>
  </ul>

  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">ttok --help</h2><a id="user-content-ttok---help" class="anchor" aria-label="Permalink: ttok --help" href="#ttok---help"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>

  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="Usage: ttok [OPTIONS] [PROMPT]...

    Count and truncate text based on tokens

    To count tokens for text passed as arguments:

        ttok one two three

    To count tokens from stdin:

        cat input.txt | ttok

    To truncate to 100 tokens:

        cat input.txt | ttok -t 100

    To truncate to 100 tokens using the gpt2 model:

        cat input.txt | ttok -t 100 -m gpt2

    To view token integers:

        cat input.txt | ttok --encode

    To convert tokens back to text:

        ttok 9906 1917 --decode

    To see the details of the tokens:

        ttok "hello world" --tokens

    Outputs:

        [b'hello', b' world']

  Options:
    --version               Show the version and exit.
    -i, --input FILENAME
    -t, --truncate INTEGER  Truncate to this many tokens
    -m, --model TEXT        Which model to use
    --encode, --tokens      Output token integers
    --decode                Convert token integers to text
    --tokens                Output full tokens
    --allow-special         Do not error on special tokens
    --help                  Show this message and exit.
  "><pre class="notranslate"><code>Usage: ttok [OPTIONS] [PROMPT]...

    Count and truncate text based on tokens

    To count tokens for text passed as arguments:

        ttok one two three

    To count tokens from stdin:

        cat input.txt | ttok

    To truncate to 100 tokens:

        cat input.txt | ttok -t 100

    To truncate to 100 tokens using the gpt2 model:

        cat input.txt | ttok -t 100 -m gpt2

    To view token integers:

        cat input.txt | ttok --encode

    To convert tokens back to text:

        ttok 9906 1917 --decode

    To see the details of the tokens:

        ttok "hello world" --tokens

    Outputs:

        [b'hello', b' world']

  Options:
    --version               Show the version and exit.
    -i, --input FILENAME
    -t, --truncate INTEGER  Truncate to this many tokens
    -m, --model TEXT        Which model to use
    --encode, --tokens      Output token integers
    --decode                Convert token integers to text
    --tokens                Output full tokens
    --allow-special         Do not error on special tokens
    --help                  Show this message and exit.

  </code></pre></div>

  <p dir="auto">You can also run this command using:</p>
  <div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="python -m ttok --help"><pre>python -m ttok --help</pre></div>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Development</h2><a id="user-content-development" class="anchor" aria-label="Permalink: Development" href="#development"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">To contribute to this tool, first checkout the code. Then create a new virtual environment:</p>
  <div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="cd ttok
  python -m venv venv
  source venv/bin/activate"><pre><span class="pl-c1">cd</span> ttok
  python -m venv venv
  <span class="pl-c1">source</span> venv/bin/activate</pre></div>
  <p dir="auto">Now install the dependencies and test dependencies:</p>
  <div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="pip install -e '.[test]'"><pre>pip install -e <span class="pl-s"><span class="pl-pds">'</span>.[test]<span class="pl-pds">'</span></span></pre></div>
  <p dir="auto">To run the tests:</p>
  <div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="pytest"><pre>pytest</pre></div>
  </article></div>


node_id: R_kgDOJ6yOgw
name: datasette-write-ui
full_name: datasette/datasette-write-ui
html_url: https://github.com/datasette/datasette-write-ui
description: A Datasette plugin that adds UI elements to edit, insert, or delete rows in SQLite tables
created_at: 2023-07-12T15:51:50Z
updated_at: 2026-03-15T18:57:11Z
pushed_at: 2026-03-15T18:57:08Z
language: Python
license: apache-2.0
topics: []
visibility: public
default_branch: main
permissions: {"admin": false, "maintain": false, "push": false, "triage": false, "pull": false}
custom_properties: {}
pull_request_creation_policy: all
readme:
  # datasette-write-ui

  [![PyPI](https://img.shields.io/pypi/v/datasette-write-ui.svg)](https://pypi.org/project/datasette-write-ui/)
  [![Changelog](https://img.shields.io/github/v/release/datasette/datasette-write-ui?include_prereleases&label=changelog)](https://github.com/datasette/datasette-write-ui/releases)
  [![Tests](https://github.com/datasette/datasette-write-ui/workflows/Test/badge.svg)](https://github.com/datasette/datasette-write-ui/actions?query=workflow%3ATest)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/datasette/datasette-write-ui/blob/main/LICENSE)

  A Datasette plugin that adds UI elements to edit, insert, or delete rows in SQLite tables.

  ## Installation

  This plugin requires the alpha version of Datasette 1.0. You can install it with:

  ```bash
  pip install datasette==1.0a3
  ```

  After that, you can install this plugin in the same environment as Datasette.

  ```bash
  datasette install datasette-write-ui
  ```

  ## Usage

  Once installed, the new UI elements will appear on the table page. You'll need special permission in order to see them. The easiest way is to [use the root user](https://docs.datasette.io/en/latest/authentication.html#using-the-root-actor), or otherwise use a [custom actor](https://docs.datasette.io/en/latest/authentication.html#actors).

  <img width=600 src="https://camo.githubusercontent.com/f02dbf756d3ba7ed148a9e7263eb045fafcbb549733f37cd1350aafe54ff2c9f/68747470733a2f2f6461746173657474652d636c6f75642d6173736574732e73332e616d617a6f6e6177732e636f6d2f626c6f672f323032332f696e74726f647563696e672d6461746173657474652d77726974652d75692f75692d7461626c652d73637265656e73686f742e6a7067"/>

  If you actor has the `insert-row` permissions, you'll see the "Insert new row" button at the bottom of the page.

  <video src="https://datasette-cloud-assets.s3.amazonaws.com/blog/2023/introducing-datasette-write-ui/insert-demo.mp4" width=600 controls autoplay type="video/mp4" ></video>

  If your actor has the `update-row` permissions, you'll see a new "Edit row" option in a new menu on each row.

  <video src="https://datasette-cloud-assets.s3.amazonaws.com/blog/2023/introducing-datasette-write-ui/edit-demo.mp4" width=600 controls autoplay type="video/mp4" ></video>

  If your actor has the `delete-row` permissions, you'll see a new "Delete row" option in a new menu on each row.

  <video src="https://datasette-cloud-assets.s3.amazonaws.com/blog/2023/introducing-datasette-write-ui/delete-demo.mp4" width=600 controls autoplay type="video/mp4" ></video>

  ## Development

  To set up this plugin locally, first checkout the code. Then create a new virtual environment:
  ```bash
  cd datasette-write-ui
  python3 -m venv venv
  source venv/bin/activate
  ```
  Now install the dependencies and test dependencies:
  ```
  pip install -e '.[test]'
  ```
  To install the JavaScript build dependencies, run this:
  ```bash
  npm install
  ```
  You can use the [Just](https://github.com/casey/just) command runner to build the TypeScript to minified JavaScript like this:
  ```bash
  just js
  ```
  To run the tests:
  ```bash
  pytest
  ```

readme_html:
  <div id="readme" class="md" data-path="README.md"><article class="markdown-body entry-content container-lg" itemprop="text"><div class="markdown-heading" dir="auto"><h1 class="heading-element" dir="auto">datasette-write-ui</h1><a id="user-content-datasette-write-ui" class="anchor" aria-label="Permalink: datasette-write-ui" href="#datasette-write-ui"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto"><a href="https://pypi.org/project/datasette-write-ui/" rel="nofollow"><img src="https://img.shields.io/pypi/v/datasette-write-ui.svg" alt="PyPI" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/173118b6efe6bfd5a0a6431eb64557f90f9ec53950c6b6e4c33d7cfde5e3f5a1/68747470733a2f2f696d672e736869656c64732e696f2f707970692f762f6461746173657474652d77726974652d75692e737667"></a>
  <a href="https://github.com/datasette/datasette-write-ui/releases"><img src="https://img.shields.io/github/v/release/datasette/datasette-write-ui?include_prereleases&label=changelog" alt="Changelog" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/f1472a0ea9d68c1969848ab078cc9ae8ccb5a7224bd6bfff6e67dd5e0dcf3a7c/68747470733a2f2f696d672e736869656c64732e696f2f6769746875622f762f72656c656173652f6461746173657474652f6461746173657474652d77726974652d75693f696e636c7564655f70726572656c6561736573266c6162656c3d6368616e67656c6f67"></a>
  <a href="https://github.com/datasette/datasette-write-ui/actions?query=workflow%3ATest"><img src="https://github.com/datasette/datasette-write-ui/workflows/Test/badge.svg" alt="Tests" style="max-width: 100%;"></a>
  <a href="https://github.com/datasette/datasette-write-ui/blob/main/LICENSE"><img src="https://img.shields.io/badge/license-Apache%202.0-blue.svg" alt="License" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/c355f200ea90fddaa407b6eaab303663a669248ea3ca7b1fcf77dbe04ff5f48c/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6c6963656e73652d417061636865253230322e302d626c75652e737667"></a></p>
  <p dir="auto">A Datasette plugin that adds UI elements to edit, insert, or delete rows in SQLite tables.</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Installation</h2><a id="user-content-installation" class="anchor" aria-label="Permalink: Installation" href="#installation"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">This plugin requires the alpha version of Datasette 1.0. You can install it with:</p>
  <div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="pip install datasette==1.0a3"><pre>pip install datasette==1.0a3</pre></div>
  <p dir="auto">After that, you can install this plugin in the same environment as Datasette.</p>
  <div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="datasette install datasette-write-ui"><pre>datasette install datasette-write-ui</pre></div>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Usage</h2><a id="user-content-usage" class="anchor" aria-label="Permalink: Usage" href="#usage"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">Once installed, the new UI elements will appear on the table page. You'll need special permission in order to see them. The easiest way is to <a href="https://docs.datasette.io/en/latest/authentication.html#using-the-root-actor" rel="nofollow">use the root user</a>, or otherwise use a <a href="https://docs.datasette.io/en/latest/authentication.html#actors" rel="nofollow">custom actor</a>.</p>
  <p dir="auto"><a target="_blank" rel="noopener noreferrer nofollow" href="https://camo.githubusercontent.com/f02dbf756d3ba7ed148a9e7263eb045fafcbb549733f37cd1350aafe54ff2c9f/68747470733a2f2f6461746173657474652d636c6f75642d6173736574732e73332e616d617a6f6e6177732e636f6d2f626c6f672f323032332f696e74726f647563696e672d6461746173657474652d77726974652d75692f75692d7461626c652d73637265656e73686f742e6a7067"><img width="600" src="https://camo.githubusercontent.com/f02dbf756d3ba7ed148a9e7263eb045fafcbb549733f37cd1350aafe54ff2c9f/68747470733a2f2f6461746173657474652d636c6f75642d6173736574732e73332e616d617a6f6e6177732e636f6d2f626c6f672f323032332f696e74726f647563696e672d6461746173657474652d77726974652d75692f75692d7461626c652d73637265656e73686f742e6a7067" style="max-width: 100%;"></a></p>
  <p dir="auto">If you actor has the <code>insert-row</code> permissions, you'll see the "Insert new row" button at the bottom of the page.</p>
  <p dir="auto"></p>
  <p dir="auto">If your actor has the <code>update-row</code> permissions, you'll see a new "Edit row" option in a new menu on each row.</p>
  <p dir="auto"></p>
  <p dir="auto">If your actor has the <code>delete-row</code> permissions, you'll see a new "Delete row" option in a new menu on each row.</p>
  <p dir="auto"></p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Development</h2><a id="user-content-development" class="anchor" aria-label="Permalink: Development" href="#development"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">To set up this plugin locally, first checkout the code. Then create a new virtual environment:</p>
  <div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="cd datasette-write-ui
  python3 -m venv venv
  source venv/bin/activate"><pre><span class="pl-c1">cd</span> datasette-write-ui
  python3 -m venv venv
  <span class="pl-c1">source</span> venv/bin/activate</pre></div>
  <p dir="auto">Now install the dependencies and test dependencies:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="pip install -e '.[test]'"><pre class="notranslate"><code>pip install -e '.[test]'
  </code></pre></div>
  <p dir="auto">To install the JavaScript build dependencies, run this:</p>
  <div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="npm install"><pre>npm install</pre></div>
  <p dir="auto">You can use the <a href="https://github.com/casey/just">Just</a> command runner to build the TypeScript to minified JavaScript like this:</p>
  <div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="just js"><pre>just js</pre></div>
  <p dir="auto">To run the tests:</p>
  <div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="pytest"><pre>pytest</pre></div>
  </article></div>


node_id: R_kgDOKHgxRA
name: datasette-extract
full_name: datasette/datasette-extract
html_url: https://github.com/datasette/datasette-extract
description: Import unstructured data (text and images) into structured tables
created_at: 2023-08-15T19:57:04Z
updated_at: 2026-04-01T19:33:03Z
pushed_at: 2026-04-01T03:32:16Z
language: Python
license: apache-2.0
topics: []
visibility: public
default_branch: main
permissions: {"admin": false, "maintain": false, "push": false, "triage": false, "pull": false}
custom_properties: {}
pull_request_creation_policy: all
readme:
  # datasette-extract

  [![PyPI](https://img.shields.io/pypi/v/datasette-extract.svg)](https://pypi.org/project/datasette-extract/)
  [![Changelog](https://img.shields.io/github/v/release/datasette/datasette-extract?include_prereleases&label=changelog)](https://github.com/datasette/datasette-extract/releases)
  [![Tests](https://github.com/datasette/datasette-extract/workflows/Test/badge.svg)](https://github.com/datasette/datasette-extract/actions?query=workflow%3ATest)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/datasette/datasette-extract/blob/main/LICENSE)

  Import unstructured data (text and images) into structured tables

  ## Installation

  Install this plugin in the same environment as [Datasette](https://datasette.io/).
  ```bash
  datasette install datasette-extract
  ```

  ## Configuration

  This plugin requires an [OpenAI API key](https://platform.openai.com/api-keys).

  You can set this using the `DATASETTE_SECRETS_OPENAI_API_KEY` environment variable, or you can configure the [datasette-secrets](https://github.com/datasette/datasette-secrets) plugin to allow users to enter their own plugin and save it, encrypted, in their database.

  Here's how to start using Datasette with that environment variable:

  ```bash
  DATASETTE_SECRETS_OPENAI_API_KEY="xxx" datasette data.db --root --create
  # Now click or command-click the URL containing .../-/auth-token?token=...
  ```
  - Replace `xxx` with your OpenAI API key
  - The `--root` flag causes Datasette to output a link that will sign you in as root
  - The `--create` flag will create the `data.db` SQLite database file if it does not exist

  If you are using other models from plugins you should consult those LLM plugins for documentation on how to configure their API keys, if they need one.

  By default all asyncio and schema supporting LLM models will be provided as options for the user. You can restrict that to a subset of models using the `models` setting:

  ```yaml
  plugins:
   datasette-extract:
    models:
    - openai/gpt-4.1-nano
  ```
  If you only list a single model users will not get an option to select the model when they use the extraction tool.

  ## Usage

  This plugin provides the following features:

  - In the database action cog menu for a database select "Create table with extracted data" to create a new table with data extracted from text or an image
  - In the table action cog menu select "Extract data into this table" to extract data into an existing table

  When creating a table you can specify the column names, types and provide an optional hint (like "YYYY-MM-DD" for dates) to influence how the data should be extracted.

  When populating an existing table you can provide hints and select which columns should be populated.

  Text input can be pasted directly into the textarea.

  Drag and drop a PDF or text file onto the textarea to populate it with the contents of that file. PDF files will have their text extracted, but only if the file contains text as opposed to scanned images.

  Drag and drop a single image onto the textarea - or select it with the image file input box - to process an image.

  ## Permissions

  Users must have the `datasette-extract` permission to use this tool.

  In order to create tables they also need the `create-table` permission.

  To insert rows into an existing table they need `insert-row`.

  Run this to grant those permissions to the root user:
  ```bash
  datasette . --root \
    -s permissions.insert-row.id root \
    -s permissions.create-table.id root \
    -s permissions.datasette-extract.id root \
  ```

  ## Development

  To set up this plugin locally, first checkout the code. Then create a new virtual environment:
  ```bash
  cd datasette-extract
  python3 -m venv venv
  source venv/bin/activate
  ```
  Now install the dependencies and test dependencies:
  ```bash
  pip install -e '.[test]'
  ```
  To run the tests:
  ```bash
  pytest
  ```
  One option to run this in development is to use this recipe:
  ```bash
  DATASETTE_SECRETS_OPENAI_API_KEY="$(llm keys get openai)" \
    datasette . --root --secret 1 \
    -s permissions.insert-row.id root \
    -s permissions.create-table.id root \
    -s permissions.datasette-extract.id root \
    -s plugins.datasette-extract.models '["openai/gpt-4.1-nano", "openai/gpt-4.1-mini"]' \
    --internal internal.db --reload
  ```

readme_html:
  <div id="readme" class="md" data-path="README.md"><article class="markdown-body entry-content container-lg" itemprop="text"><div class="markdown-heading" dir="auto"><h1 class="heading-element" dir="auto">datasette-extract</h1><a id="user-content-datasette-extract" class="anchor" aria-label="Permalink: datasette-extract" href="#datasette-extract"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto"><a href="https://pypi.org/project/datasette-extract/" rel="nofollow"><img src="https://img.shields.io/pypi/v/datasette-extract.svg" alt="PyPI" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/3264fccfc83d0cf2e7a0dc024aed75994f487a647c792af58bab5cf3cc31915c/68747470733a2f2f696d672e736869656c64732e696f2f707970692f762f6461746173657474652d657874726163742e737667"></a>
  <a href="https://github.com/datasette/datasette-extract/releases"><img src="https://img.shields.io/github/v/release/datasette/datasette-extract?include_prereleases&label=changelog" alt="Changelog" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/c319012b9de441a037cb8b4f7ee5d3405cff9118e265e74dd681f4df77a9a4c4/68747470733a2f2f696d672e736869656c64732e696f2f6769746875622f762f72656c656173652f6461746173657474652f6461746173657474652d657874726163743f696e636c7564655f70726572656c6561736573266c6162656c3d6368616e67656c6f67"></a>
  <a href="https://github.com/datasette/datasette-extract/actions?query=workflow%3ATest"><img src="https://github.com/datasette/datasette-extract/workflows/Test/badge.svg" alt="Tests" style="max-width: 100%;"></a>
  <a href="https://github.com/datasette/datasette-extract/blob/main/LICENSE"><img src="https://img.shields.io/badge/license-Apache%202.0-blue.svg" alt="License" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/c355f200ea90fddaa407b6eaab303663a669248ea3ca7b1fcf77dbe04ff5f48c/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6c6963656e73652d417061636865253230322e302d626c75652e737667"></a></p>
  <p dir="auto">Import unstructured data (text and images) into structured tables</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Installation</h2><a id="user-content-installation" class="anchor" aria-label="Permalink: Installation" href="#installation"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">Install this plugin in the same environment as <a href="https://datasette.io/" rel="nofollow">Datasette</a>.</p>
  <div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="datasette install datasette-extract"><pre>datasette install datasette-extract</pre></div>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Configuration</h2><a id="user-content-configuration" class="anchor" aria-label="Permalink: Configuration" href="#configuration"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">This plugin requires an <a href="https://platform.openai.com/api-keys" rel="nofollow">OpenAI API key</a>.</p>
  <p dir="auto">You can set this using the <code>DATASETTE_SECRETS_OPENAI_API_KEY</code> environment variable, or you can configure the <a href="https://github.com/datasette/datasette-secrets">datasette-secrets</a> plugin to allow users to enter their own plugin and save it, encrypted, in their database.</p>
  <p dir="auto">Here's how to start using Datasette with that environment variable:</p>
  <div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="DATASETTE_SECRETS_OPENAI_API_KEY="xxx" datasette data.db --root --create
  # Now click or command-click the URL containing .../-/auth-token?token=..."><pre>DATASETTE_SECRETS_OPENAI_API_KEY=<span class="pl-s"><span class="pl-pds">"</span>xxx<span class="pl-pds">"</span></span> datasette data.db --root --create
  <span class="pl-c"><span class="pl-c">#</span> Now click or command-click the URL containing .../-/auth-token?token=...</span></pre></div>
  <ul dir="auto">
  <li>Replace <code>xxx</code> with your OpenAI API key</li>
  <li>The <code>--root</code> flag causes Datasette to output a link that will sign you in as root</li>
  <li>The <code>--create</code> flag will create the <code>data.db</code> SQLite database file if it does not exist</li>
  </ul>
  <p dir="auto">If you are using other models from plugins you should consult those LLM plugins for documentation on how to configure their API keys, if they need one.</p>
  <p dir="auto">By default all asyncio and schema supporting LLM models will be provided as options for the user. You can restrict that to a subset of models using the <code>models</code> setting:</p>
  <div class="highlight highlight-source-yaml notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="plugins:
   datasette-extract:
    models:
    - openai/gpt-4.1-nano"><pre><span class="pl-ent">plugins</span>:
   <span class="pl-ent">datasette-extract</span>:
    <span class="pl-ent">models</span>:
    - <span class="pl-s">openai/gpt-4.1-nano</span></pre></div>
  <p dir="auto">If you only list a single model users will not get an option to select the model when they use the extraction tool.</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Usage</h2><a id="user-content-usage" class="anchor" aria-label="Permalink: Usage" href="#usage"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">This plugin provides the following features:</p>
  <ul dir="auto">
  <li>In the database action cog menu for a database select "Create table with extracted data" to create a new table with data extracted from text or an image</li>
  <li>In the table action cog menu select "Extract data into this table" to extract data into an existing table</li>
  </ul>
  <p dir="auto">When creating a table you can specify the column names, types and provide an optional hint (like "YYYY-MM-DD" for dates) to influence how the data should be extracted.</p>
  <p dir="auto">When populating an existing table you can provide hints and select which columns should be populated.</p>
  <p dir="auto">Text input can be pasted directly into the textarea.</p>
  <p dir="auto">Drag and drop a PDF or text file onto the textarea to populate it with the contents of that file. PDF files will have their text extracted, but only if the file contains text as opposed to scanned images.</p>
  <p dir="auto">Drag and drop a single image onto the textarea - or select it with the image file input box - to process an image.</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Permissions</h2><a id="user-content-permissions" class="anchor" aria-label="Permalink: Permissions" href="#permissions"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">Users must have the <code>datasette-extract</code> permission to use this tool.</p>
  <p dir="auto">In order to create tables they also need the <code>create-table</code> permission.</p>
  <p dir="auto">To insert rows into an existing table they need <code>insert-row</code>.</p>
  <p dir="auto">Run this to grant those permissions to the root user:</p>
  <div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="datasette . --root \
    -s permissions.insert-row.id root \
    -s permissions.create-table.id root \
    -s permissions.datasette-extract.id root \"><pre>datasette <span class="pl-c1">.</span> --root \
    -s permissions.insert-row.id root \
    -s permissions.create-table.id root \
    -s permissions.datasette-extract.id root \</pre></div>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Development</h2><a id="user-content-development" class="anchor" aria-label="Permalink: Development" href="#development"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">To set up this plugin locally, first checkout the code. Then create a new virtual environment:</p>
  <div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="cd datasette-extract
  python3 -m venv venv
  source venv/bin/activate"><pre><span class="pl-c1">cd</span> datasette-extract
  python3 -m venv venv
  <span class="pl-c1">source</span> venv/bin/activate</pre></div>
  <p dir="auto">Now install the dependencies and test dependencies:</p>
  <div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="pip install -e '.[test]'"><pre>pip install -e <span class="pl-s"><span class="pl-pds">'</span>.[test]<span class="pl-pds">'</span></span></pre></div>
  <p dir="auto">To run the tests:</p>
  <div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="pytest"><pre>pytest</pre></div>
  <p dir="auto">One option to run this in development is to use this recipe:</p>
  <div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="DATASETTE_SECRETS_OPENAI_API_KEY="$(llm keys get openai)" \
    datasette . --root --secret 1 \
    -s permissions.insert-row.id root \
    -s permissions.create-table.id root \
    -s permissions.datasette-extract.id root \
    -s plugins.datasette-extract.models '["openai/gpt-4.1-nano", "openai/gpt-4.1-mini"]' \
    --internal internal.db --reload"><pre>DATASETTE_SECRETS_OPENAI_API_KEY=<span class="pl-s"><span class="pl-pds">"</span><span class="pl-s"><span class="pl-pds">$(</span>llm keys get openai<span class="pl-pds">)</span></span><span class="pl-pds">"</span></span> \
    datasette <span class="pl-c1">.</span> --root --secret 1 \
    -s permissions.insert-row.id root \
    -s permissions.create-table.id root \
    -s permissions.datasette-extract.id root \
    -s plugins.datasette-extract.models <span class="pl-s"><span class="pl-pds">'</span>["openai/gpt-4.1-nano", "openai/gpt-4.1-mini"]<span class="pl-pds">'</span></span> \
    --internal internal.db --reload</pre></div>
  </article></div>


node_id: R_kgDOKNibWA
name: datasette-visible-internal-db
full_name: datasette/datasette-visible-internal-db
html_url: https://github.com/datasette/datasette-visible-internal-db
description: Datasette plugin for making the internal database visible for debugging
created_at: 2023-08-30T22:38:55Z
updated_at: 2026-01-26T03:39:06Z
pushed_at: 2026-01-26T03:40:04Z
language: Python
topics: []
visibility: public
default_branch: main
permissions: {"admin": false, "maintain": false, "push": false, "triage": false, "pull": false}
custom_properties: {}
pull_request_creation_policy: all
readme:
  # datasette-visible-internal-db

  [![PyPI](https://img.shields.io/pypi/v/datasette-visible-internal-db.svg)](https://pypi.org/project/datasette-visible-internal-db/)
  [![Changelog](https://img.shields.io/github/v/release/datasette/datasette-visible-internal-db?include_prereleases&label=changelog)](https://github.com/datasette/datasette-visible-internal-db/releases)
  [![Tests](https://github.com/datasette/datasette-visible-internal-db/workflows/Test/badge.svg)](https://github.com/datasette/datasette-visible-internal-db/actions?query=workflow%3ATest)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/datasette/datasette-visible-internal-db/blob/main/LICENSE)

  Datasette plugin for making the internal database visible for debugging

  Datasette 1.0a5 made [the internal database](https://docs.datasette.io/en/1.0a5/internals.html#datasette-s-internal-database) no longer visible through the Datasette interface.

  This plugin brings it back, for debugging purposes.

  ## Installation

  Install this plugin in the same environment as Datasette.
  ```bash
  datasette install datasette-visible-internal-db
  ```

  The internal database will now be publicly visible at `/_internal` in your Datasette instance.

  ## Controlling access to the internal database

  You can use Datasette's [permissions mechanism](https://docs.datasette.io/en/stable/authentication.html#permissions) to control who is allowed to interact with the internal database.

  To restrict access to just the `root` actor, drop this into your configuration:

  ```yaml
  databases:
    _internal:
      allow:
        id: root
  ```

readme_html:
  <div id="readme" class="md" data-path="README.md"><article class="markdown-body entry-content container-lg" itemprop="text"><div class="markdown-heading" dir="auto"><h1 class="heading-element" dir="auto">datasette-visible-internal-db</h1><a id="user-content-datasette-visible-internal-db" class="anchor" aria-label="Permalink: datasette-visible-internal-db" href="#datasette-visible-internal-db"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto"><a href="https://pypi.org/project/datasette-visible-internal-db/" rel="nofollow"><img src="https://img.shields.io/pypi/v/datasette-visible-internal-db.svg" alt="PyPI" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/9d71f90286f27b01978020bf9ae48625da8c669b3bd903292f075f326d19810a/68747470733a2f2f696d672e736869656c64732e696f2f707970692f762f6461746173657474652d76697369626c652d696e7465726e616c2d64622e737667"></a>
  <a href="https://github.com/datasette/datasette-visible-internal-db/releases"><img src="https://img.shields.io/github/v/release/datasette/datasette-visible-internal-db?include_prereleases&label=changelog" alt="Changelog" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/383323ea65aedaead14462f67dcc6b7d6024553d9d26a2b075885015f07b6991/68747470733a2f2f696d672e736869656c64732e696f2f6769746875622f762f72656c656173652f6461746173657474652f6461746173657474652d76697369626c652d696e7465726e616c2d64623f696e636c7564655f70726572656c6561736573266c6162656c3d6368616e67656c6f67"></a>
  <a href="https://github.com/datasette/datasette-visible-internal-db/actions?query=workflow%3ATest"><img src="https://github.com/datasette/datasette-visible-internal-db/workflows/Test/badge.svg" alt="Tests" style="max-width: 100%;"></a>
  <a href="https://github.com/datasette/datasette-visible-internal-db/blob/main/LICENSE"><img src="https://img.shields.io/badge/license-Apache%202.0-blue.svg" alt="License" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/c355f200ea90fddaa407b6eaab303663a669248ea3ca7b1fcf77dbe04ff5f48c/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6c6963656e73652d417061636865253230322e302d626c75652e737667"></a></p>
  <p dir="auto">Datasette plugin for making the internal database visible for debugging</p>
  <p dir="auto">Datasette 1.0a5 made <a href="https://docs.datasette.io/en/1.0a5/internals.html#datasette-s-internal-database" rel="nofollow">the internal database</a> no longer visible through the Datasette interface.</p>
  <p dir="auto">This plugin brings it back, for debugging purposes.</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Installation</h2><a id="user-content-installation" class="anchor" aria-label="Permalink: Installation" href="#installation"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">Install this plugin in the same environment as Datasette.</p>
  <div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="datasette install datasette-visible-internal-db"><pre>datasette install datasette-visible-internal-db</pre></div>
  <p dir="auto">The internal database will now be publicly visible at <code>/_internal</code> in your Datasette instance.</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Controlling access to the internal database</h2><a id="user-content-controlling-access-to-the-internal-database" class="anchor" aria-label="Permalink: Controlling access to the internal database" href="#controlling-access-to-the-internal-database"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">You can use Datasette's <a href="https://docs.datasette.io/en/stable/authentication.html#permissions" rel="nofollow">permissions mechanism</a> to control who is allowed to interact with the internal database.</p>
  <p dir="auto">To restrict access to just the <code>root</code> actor, drop this into your configuration:</p>
  <div class="highlight highlight-source-yaml notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="databases:
    _internal:
      allow:
        id: root"><pre><span class="pl-ent">databases</span>:
    <span class="pl-ent">_internal</span>:
      <span class="pl-ent">allow</span>:
        <span class="pl-ent">id</span>: <span class="pl-s">root</span></pre></div>
  </article></div>


node_id: R_kgDOKRMeGQ
name: datasette-llm-embed
full_name: simonw/datasette-llm-embed
html_url: https://github.com/simonw/datasette-llm-embed
description: Datasette plugin adding a llm_embed(model_id, text) SQL function
created_at: 2023-09-08T21:09:40Z
updated_at: 2025-04-23T20:42:34Z
pushed_at: 2024-03-17T22:15:00Z
language: Python
license: apache-2.0
topics: []
visibility: public
default_branch: main
permissions: {"admin": false, "maintain": false, "push": false, "triage": false, "pull": true}
readme:
  # datasette-llm-embed

  [![PyPI](https://img.shields.io/pypi/v/datasette-llm-embed.svg)](https://pypi.org/project/datasette-llm-embed/)
  [![Changelog](https://img.shields.io/github/v/release/simonw/datasette-llm-embed?include_prereleases&label=changelog)](https://github.com/simonw/datasette-llm-embed/releases)
  [![Tests](https://github.com/simonw/datasette-llm-embed/workflows/Test/badge.svg)](https://github.com/simonw/datasette-llm-embed/actions?query=workflow%3ATest)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/datasette-llm-embed/blob/main/LICENSE)

  Datasette plugin adding a `llm_embed(model_id, text)` SQL function.

  ## Installation

  ```bash
  datasette install datasette-llm-embed
  ```

  ## Usage

  Adds a SQL function that can be called like this:
  ```sql
  select llm_embed('sentence-transformers/all-mpnet-base-v2', 'This is some text')
  ```
  This embeds the provided text using the specified embedding model and returns a binary blob, suitable for use with plugins such as [datasette-faiss](https://datasette.io/plugins/datasette-faiss).

  The models need to be installed using [LLM](https://llm.datasette.io/) plugins such as [llm-sentence-transformers](https://github.com/simonw/llm-sentence-transformers).

  Use `llm_embed_cosine(a, b)` to calculate cosine similarity between two vector blobs:

  ```sql
  select llm_embed_cosine(
      llm_embed('sentence-transformers/all-mpnet-base-v2', 'This is some text'),
      llm_embed('sentence-transformers/all-mpnet-base-v2', 'This is some other text')
  )
  ```

  The `llm_embed_decode()` function can be used to decode a binary BLOB into a JSON array of floats:

  ```sql
  select llm_embed_decode(
      llm_embed('sentence-transformers/all-mpnet-base-v2', 'This is some text')
  )
  ```

  ## Models that require API keys

  If your embedding model needs an API key - for example the `ada-002` model from OpenAI - you can configure that key in `metadata.yml` (or JSON) like this:

  ```yaml
  plugins:
    datasette-llm-embed:
      keys:
        ada-002:
          $env: OPENAI_API_KEY
  ```
  The key here should be the full model ID of the model - not an alias.

  You can then set the `OPENAI_API_KEY` environment variable to the key you want to use before starting Datasette:
  ```bash
  export OPENAI_API_KEY=sk-1234567890
  ```
  Once configured, calls like this will use the API key that has been provided:
  ```sql
  select llm_embed('ada-002', 'This is some text')
  ```

  ## Development

  To set up this plugin locally, first checkout the code. Then create a new virtual environment:
  ```bash
  cd datasette-llm-embed
  python3 -m venv venv
  source venv/bin/activate
  ```
  Now install the dependencies and test dependencies:
  ```
  pip install -e '.[test]'
  ```
  ```
  To run the tests:
  ```bash
  pytest
  ```

readme_html:
  <div id="readme" class="md" data-path="README.md"><article class="markdown-body entry-content container-lg" itemprop="text"><div class="markdown-heading" dir="auto"><h1 class="heading-element" dir="auto">datasette-llm-embed</h1><a id="user-content-datasette-llm-embed" class="anchor" aria-label="Permalink: datasette-llm-embed" href="#datasette-llm-embed"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto"><a href="https://pypi.org/project/datasette-llm-embed/" rel="nofollow"><img src="https://img.shields.io/pypi/v/datasette-llm-embed.svg" alt="PyPI" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/48e48cbdd7910718be4d14cd223a05558f4951dc9efaf91d3ee523c923620ae3/68747470733a2f2f696d672e736869656c64732e696f2f707970692f762f6461746173657474652d6c6c6d2d656d6265642e737667"></a>
  <a href="https://github.com/simonw/datasette-llm-embed/releases"><img src="https://img.shields.io/github/v/release/simonw/datasette-llm-embed?include_prereleases&label=changelog" alt="Changelog" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/bc7f16a995ce1d6da0bad0e55ed31d6ebd03862f22078c0ce8f800c0bfa20b11/68747470733a2f2f696d672e736869656c64732e696f2f6769746875622f762f72656c656173652f73696d6f6e772f6461746173657474652d6c6c6d2d656d6265643f696e636c7564655f70726572656c6561736573266c6162656c3d6368616e67656c6f67"></a>
  <a href="https://github.com/simonw/datasette-llm-embed/actions?query=workflow%3ATest"><img src="https://github.com/simonw/datasette-llm-embed/workflows/Test/badge.svg" alt="Tests" style="max-width: 100%;"></a>
  <a href="https://github.com/simonw/datasette-llm-embed/blob/main/LICENSE"><img src="https://img.shields.io/badge/license-Apache%202.0-blue.svg" alt="License" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/c355f200ea90fddaa407b6eaab303663a669248ea3ca7b1fcf77dbe04ff5f48c/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6c6963656e73652d417061636865253230322e302d626c75652e737667"></a></p>
  <p dir="auto">Datasette plugin adding a <code>llm_embed(model_id, text)</code> SQL function.</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Installation</h2><a id="user-content-installation" class="anchor" aria-label="Permalink: Installation" href="#installation"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="datasette install datasette-llm-embed"><pre>datasette install datasette-llm-embed</pre></div>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Usage</h2><a id="user-content-usage" class="anchor" aria-label="Permalink: Usage" href="#usage"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">Adds a SQL function that can be called like this:</p>
  <div class="highlight highlight-source-sql notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="select llm_embed('sentence-transformers/all-mpnet-base-v2', 'This is some text')"><pre><span class="pl-k">select</span> llm_embed(<span class="pl-s"><span class="pl-pds">'</span>sentence-transformers/all-mpnet-base-v2<span class="pl-pds">'</span></span>, <span class="pl-s"><span class="pl-pds">'</span>This is some text<span class="pl-pds">'</span></span>)</pre></div>
  <p dir="auto">This embeds the provided text using the specified embedding model and returns a binary blob, suitable for use with plugins such as <a href="https://datasette.io/plugins/datasette-faiss" rel="nofollow">datasette-faiss</a>.</p>
  <p dir="auto">The models need to be installed using <a href="https://llm.datasette.io/" rel="nofollow">LLM</a> plugins such as <a href="https://github.com/simonw/llm-sentence-transformers">llm-sentence-transformers</a>.</p>
  <p dir="auto">Use <code>llm_embed_cosine(a, b)</code> to calculate cosine similarity between two vector blobs:</p>
  <div class="highlight highlight-source-sql notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="select llm_embed_cosine(
      llm_embed('sentence-transformers/all-mpnet-base-v2', 'This is some text'),
      llm_embed('sentence-transformers/all-mpnet-base-v2', 'This is some other text')
  )"><pre><span class="pl-k">select</span> llm_embed_cosine(
      llm_embed(<span class="pl-s"><span class="pl-pds">'</span>sentence-transformers/all-mpnet-base-v2<span class="pl-pds">'</span></span>, <span class="pl-s"><span class="pl-pds">'</span>This is some text<span class="pl-pds">'</span></span>),
      llm_embed(<span class="pl-s"><span class="pl-pds">'</span>sentence-transformers/all-mpnet-base-v2<span class="pl-pds">'</span></span>, <span class="pl-s"><span class="pl-pds">'</span>This is some other text<span class="pl-pds">'</span></span>)
  )</pre></div>
  <p dir="auto">The <code>llm_embed_decode()</code> function can be used to decode a binary BLOB into a JSON array of floats:</p>
  <div class="highlight highlight-source-sql notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="select llm_embed_decode(
      llm_embed('sentence-transformers/all-mpnet-base-v2', 'This is some text')
  )"><pre><span class="pl-k">select</span> llm_embed_decode(
      llm_embed(<span class="pl-s"><span class="pl-pds">'</span>sentence-transformers/all-mpnet-base-v2<span class="pl-pds">'</span></span>, <span class="pl-s"><span class="pl-pds">'</span>This is some text<span class="pl-pds">'</span></span>)
  )</pre></div>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Models that require API keys</h2><a id="user-content-models-that-require-api-keys" class="anchor" aria-label="Permalink: Models that require API keys" href="#models-that-require-api-keys"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">If your embedding model needs an API key - for example the <code>ada-002</code> model from OpenAI - you can configure that key in <code>metadata.yml</code> (or JSON) like this:</p>
  <div class="highlight highlight-source-yaml notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="plugins:
    datasette-llm-embed:
      keys:
        ada-002:
          $env: OPENAI_API_KEY"><pre><span class="pl-ent">plugins</span>:
    <span class="pl-ent">datasette-llm-embed</span>:
      <span class="pl-ent">keys</span>:
        <span class="pl-ent">ada-002</span>:
          <span class="pl-ent">$env</span>: <span class="pl-s">OPENAI_API_KEY</span></pre></div>
  <p dir="auto">The key here should be the full model ID of the model - not an alias.</p>
  <p dir="auto">You can then set the <code>OPENAI_API_KEY</code> environment variable to the key you want to use before starting Datasette:</p>
  <div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="export OPENAI_API_KEY=sk-1234567890"><pre><span class="pl-k">export</span> OPENAI_API_KEY=sk-1234567890</pre></div>
  <p dir="auto">Once configured, calls like this will use the API key that has been provided:</p>
  <div class="highlight highlight-source-sql notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="select llm_embed('ada-002', 'This is some text')"><pre><span class="pl-k">select</span> llm_embed(<span class="pl-s"><span class="pl-pds">'</span>ada-002<span class="pl-pds">'</span></span>, <span class="pl-s"><span class="pl-pds">'</span>This is some text<span class="pl-pds">'</span></span>)</pre></div>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Development</h2><a id="user-content-development" class="anchor" aria-label="Permalink: Development" href="#development"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">To set up this plugin locally, first checkout the code. Then create a new virtual environment:</p>
  <div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="cd datasette-llm-embed
  python3 -m venv venv
  source venv/bin/activate"><pre><span class="pl-c1">cd</span> datasette-llm-embed
  python3 -m venv venv
  <span class="pl-c1">source</span> venv/bin/activate</pre></div>
  <p dir="auto">Now install the dependencies and test dependencies:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="pip install -e '.[test]'"><pre class="notranslate"><code>pip install -e '.[test]'
  </code></pre></div>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="To run the tests:
  ```bash
  pytest"><pre class="notranslate"><code>To run the tests:
  ```bash
  pytest
  </code></pre></div>
  </article></div>


node_id: R_kgDOKVzq2g
name: datasette-sqlite-debug-authorizer
full_name: datasette/datasette-sqlite-debug-authorizer
html_url: https://github.com/datasette/datasette-sqlite-debug-authorizer
description: Debug SQLite authorizer calls
created_at: 2023-09-20T03:30:17Z
updated_at: 2023-09-20T03:31:57Z
pushed_at: 2024-03-05T00:43:50Z
language: Python
license: apache-2.0
topics: []
visibility: public
default_branch: main
permissions: {"admin": false, "maintain": false, "push": false, "triage": false, "pull": true}
custom_properties: {}
readme:
  # datasette-sqlite-debug-authorizer

  [![PyPI](https://img.shields.io/pypi/v/datasette-sqlite-debug-authorizer.svg)](https://pypi.org/project/datasette-sqlite-debug-authorizer/)
  [![Changelog](https://img.shields.io/github/v/release/datasette/datasette-sqlite-debug-authorizer?include_prereleases&label=changelog)](https://github.com/datasette/datasette-sqlite-debug-authorizer/releases)
  [![Tests](https://github.com/datasette/datasette-sqlite-debug-authorizer/workflows/Test/badge.svg)](https://github.com/datasette/datasette-sqlite-debug-authorizer/actions?query=workflow%3ATest)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/datasette/datasette-sqlite-debug-authorizer/blob/main/LICENSE)

  Debug SQLite authorizer calls

  ## Installation

  Install this plugin in the same environment as Datasette.

      datasette install datasette-sqlite-debug-authorizer

  ## Usage

  Once installed, every SQLite permission check using the `conn.set_authorizer()` API will be logged to standard error.

  This is useful for if you want to use that authorizer API to implement your own custom permissions, and need to see what calls are being made to the authorizer.

  Example output (it's generally a lot noisier than this) for `select * from sqlite_master`:

  ```
  SQLITE_SELECT: 
  SQLITE_READ:  table="sqlite_master" column="type" db_name=main
  SQLITE_READ:  table="sqlite_master" column="name" db_name=main
  SQLITE_READ:  table="sqlite_master" column="tbl_name" db_name=main
  SQLITE_READ:  table="sqlite_master" column="rootpage" db_name=main
  SQLITE_READ:  table="sqlite_master" column="sql" db_name=main
  ```
  See [sqlite-authorizer-examples](https://github.com/simonw/sqlite-authorizer-examples) for more detailed documentation on what to expect from these calls.

  ## Development

  To set up this plugin locally, first checkout the code. Then create a new virtual environment:

      cd datasette-sqlite-debug-authorizer
      python3 -m venv venv
      source venv/bin/activate

  Now install the dependencies and test dependencies:

      pip install -e '.[test]'

  To run the tests:

      pytest

readme_html:
  <div id="readme" class="md" data-path="README.md"><article class="markdown-body entry-content container-lg" itemprop="text"><div class="markdown-heading" dir="auto"><h1 class="heading-element" dir="auto">datasette-sqlite-debug-authorizer</h1><a id="user-content-datasette-sqlite-debug-authorizer" class="anchor" aria-label="Permalink: datasette-sqlite-debug-authorizer" href="#datasette-sqlite-debug-authorizer"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto"><a href="https://pypi.org/project/datasette-sqlite-debug-authorizer/" rel="nofollow"><img src="https://img.shields.io/pypi/v/datasette-sqlite-debug-authorizer.svg" alt="PyPI" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/b7673cf98a11298d697b5c02151867763add4dc19124fddb397d5172f223cd8b/68747470733a2f2f696d672e736869656c64732e696f2f707970692f762f6461746173657474652d73716c6974652d64656275672d617574686f72697a65722e737667"></a>
  <a href="https://github.com/datasette/datasette-sqlite-debug-authorizer/releases"><img src="https://img.shields.io/github/v/release/datasette/datasette-sqlite-debug-authorizer?include_prereleases&label=changelog" alt="Changelog" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/e0d257d84321fa9cfee63f963be36aa3af30d146eb53a739545946a6920f52fc/68747470733a2f2f696d672e736869656c64732e696f2f6769746875622f762f72656c656173652f6461746173657474652f6461746173657474652d73716c6974652d64656275672d617574686f72697a65723f696e636c7564655f70726572656c6561736573266c6162656c3d6368616e67656c6f67"></a>
  <a href="https://github.com/datasette/datasette-sqlite-debug-authorizer/actions?query=workflow%3ATest"><img src="https://github.com/datasette/datasette-sqlite-debug-authorizer/workflows/Test/badge.svg" alt="Tests" style="max-width: 100%;"></a>
  <a href="https://github.com/datasette/datasette-sqlite-debug-authorizer/blob/main/LICENSE"><img src="https://img.shields.io/badge/license-Apache%202.0-blue.svg" alt="License" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/c355f200ea90fddaa407b6eaab303663a669248ea3ca7b1fcf77dbe04ff5f48c/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6c6963656e73652d417061636865253230322e302d626c75652e737667"></a></p>
  <p dir="auto">Debug SQLite authorizer calls</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Installation</h2><a id="user-content-installation" class="anchor" aria-label="Permalink: Installation" href="#installation"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">Install this plugin in the same environment as Datasette.</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="datasette install datasette-sqlite-debug-authorizer"><pre class="notranslate"><code>datasette install datasette-sqlite-debug-authorizer
  </code></pre></div>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Usage</h2><a id="user-content-usage" class="anchor" aria-label="Permalink: Usage" href="#usage"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">Once installed, every SQLite permission check using the <code>conn.set_authorizer()</code> API will be logged to standard error.</p>
  <p dir="auto">This is useful for if you want to use that authorizer API to implement your own custom permissions, and need to see what calls are being made to the authorizer.</p>
  <p dir="auto">Example output (it's generally a lot noisier than this) for <code>select * from sqlite_master</code>:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="SQLITE_SELECT: 
  SQLITE_READ:  table="sqlite_master" column="type" db_name=main
  SQLITE_READ:  table="sqlite_master" column="name" db_name=main
  SQLITE_READ:  table="sqlite_master" column="tbl_name" db_name=main
  SQLITE_READ:  table="sqlite_master" column="rootpage" db_name=main
  SQLITE_READ:  table="sqlite_master" column="sql" db_name=main"><pre class="notranslate"><code>SQLITE_SELECT: 
  SQLITE_READ:  table="sqlite_master" column="type" db_name=main
  SQLITE_READ:  table="sqlite_master" column="name" db_name=main
  SQLITE_READ:  table="sqlite_master" column="tbl_name" db_name=main
  SQLITE_READ:  table="sqlite_master" column="rootpage" db_name=main
  SQLITE_READ:  table="sqlite_master" column="sql" db_name=main
  </code></pre></div>
  <p dir="auto">See <a href="https://github.com/simonw/sqlite-authorizer-examples">sqlite-authorizer-examples</a> for more detailed documentation on what to expect from these calls.</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Development</h2><a id="user-content-development" class="anchor" aria-label="Permalink: Development" href="#development"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">To set up this plugin locally, first checkout the code. Then create a new virtual environment:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="cd datasette-sqlite-debug-authorizer
  python3 -m venv venv
  source venv/bin/activate"><pre class="notranslate"><code>cd datasette-sqlite-debug-authorizer
  python3 -m venv venv
  source venv/bin/activate
  </code></pre></div>
  <p dir="auto">Now install the dependencies and test dependencies:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="pip install -e '.[test]'"><pre class="notranslate"><code>pip install -e '.[test]'
  </code></pre></div>
  <p dir="auto">To run the tests:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="pytest"><pre class="notranslate"><code>pytest
  </code></pre></div>
  </article></div>


node_id: R_kgDOKWH3yw
name: datasette-sqlite-authorizer
full_name: datasette/datasette-sqlite-authorizer
html_url: https://github.com/datasette/datasette-sqlite-authorizer
description: Configure Datasette to block operations using the SQLIte set_authorizer mechanism
created_at: 2023-09-20T17:35:23Z
updated_at: 2024-01-04T10:57:11Z
pushed_at: 2024-03-05T01:09:05Z
language: Python
license: apache-2.0
topics: []
visibility: public
default_branch: main
permissions: {"admin": false, "maintain": false, "push": false, "triage": false, "pull": true}
custom_properties: {}
readme:
  # datasette-sqlite-authorizer

  [![PyPI](https://img.shields.io/pypi/v/datasette-sqlite-authorizer.svg)](https://pypi.org/project/datasette-sqlite-authorizer/)
  [![Changelog](https://img.shields.io/github/v/release/datasette/datasette-sqlite-authorizer?include_prereleases&label=changelog)](https://github.com/datasette/datasette-sqlite-authorizer/releases)
  [![Tests](https://github.com/datasette/datasette-sqlite-authorizer/workflows/Test/badge.svg)](https://github.com/datasette/datasette-sqlite-authorizer/actions?query=workflow%3ATest)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/datasette/datasette-sqlite-authorizer/blob/main/LICENSE)

  Configure Datasette to block operations using the SQLite set_authorizer mechanism

  ## Installation

  Install this plugin in the same environment as Datasette.
  ```bash
  datasette install datasette-sqlite-authorizer
  ```
  ## Usage

  This plugin currently offers a single configuration option: `read_only_tables`. You can use this to specify a list of tables that should be read-only.

  If a table is read-only, any attempt to write to it - `insert`, `update`, `delete`, `drop table`, `alter table` - will be denied with an error message.

  To configure read-only tables, add the following to your `metadata.yaml` file:

  ```yaml
  plugins:
    datasette-sqlite-authorizer:
      read_only_tables:
      - table: my_table
        database: my_database
  ```
  You can omit the `database` key if you want to apply the same rule to all databases.

  Here's how to use this plugin to make all tables relating to [Litestream](https://litestream.io/) synchronization read-only, across all attached databases:

  ```yaml
  plugins:
    datasette-sqlite-authorizer:
      read_only_tables:
      - table: _litestream_lock
      - table: _litestream_seq
  ```

  ## Logging actions

  To enable debug output (to standard error) for specific action checks, add them to a `debug_actions` list. For example, to log all `SQLITE_INSERT` actions:

  ```yaml
  plugins:
    datasette-sqlite-authorizer:
      debug_actions:
      - SQLITE_INSERT
      - SQLITE_DELETE
  ```
  Logged output will look something like this:
  ```
  authorizer: {"action": "SQLITE_INSERT", "arg1": "foo", "arg2": null, "db_name": "main", "trigger_name": null, "result": "OK"}
  authorizer: {"action": "SQLITE_DELETE", "arg1": "foo", "arg2": null, "db_name": "main", "trigger_name": null, "result": "OK"}
  ```

  ## Development

  To set up this plugin locally, first checkout the code. Then create a new virtual environment:

      cd datasette-sqlite-authorizer
      python3 -m venv venv
      source venv/bin/activate

  Now install the dependencies and test dependencies:

      pip install -e '.[test]'

  To run the tests:

      pytest

readme_html:
  <div id="readme" class="md" data-path="README.md"><article class="markdown-body entry-content container-lg" itemprop="text"><div class="markdown-heading" dir="auto"><h1 class="heading-element" dir="auto">datasette-sqlite-authorizer</h1><a id="user-content-datasette-sqlite-authorizer" class="anchor" aria-label="Permalink: datasette-sqlite-authorizer" href="#datasette-sqlite-authorizer"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto"><a href="https://pypi.org/project/datasette-sqlite-authorizer/" rel="nofollow"><img src="https://img.shields.io/pypi/v/datasette-sqlite-authorizer.svg" alt="PyPI" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/b309df3e4c0c629874efc0a13ae9e7af194d2ee5dc26853901eaa04b57846e43/68747470733a2f2f696d672e736869656c64732e696f2f707970692f762f6461746173657474652d73716c6974652d617574686f72697a65722e737667"></a>
  <a href="https://github.com/datasette/datasette-sqlite-authorizer/releases"><img src="https://img.shields.io/github/v/release/datasette/datasette-sqlite-authorizer?include_prereleases&label=changelog" alt="Changelog" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/b3ef4ccc17b998f4d2fddad4e6ec337e504f4f89b8ff0669c358d43d0a7fab1a/68747470733a2f2f696d672e736869656c64732e696f2f6769746875622f762f72656c656173652f6461746173657474652f6461746173657474652d73716c6974652d617574686f72697a65723f696e636c7564655f70726572656c6561736573266c6162656c3d6368616e67656c6f67"></a>
  <a href="https://github.com/datasette/datasette-sqlite-authorizer/actions?query=workflow%3ATest"><img src="https://github.com/datasette/datasette-sqlite-authorizer/workflows/Test/badge.svg" alt="Tests" style="max-width: 100%;"></a>
  <a href="https://github.com/datasette/datasette-sqlite-authorizer/blob/main/LICENSE"><img src="https://img.shields.io/badge/license-Apache%202.0-blue.svg" alt="License" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/c355f200ea90fddaa407b6eaab303663a669248ea3ca7b1fcf77dbe04ff5f48c/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6c6963656e73652d417061636865253230322e302d626c75652e737667"></a></p>
  <p dir="auto">Configure Datasette to block operations using the SQLite set_authorizer mechanism</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Installation</h2><a id="user-content-installation" class="anchor" aria-label="Permalink: Installation" href="#installation"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">Install this plugin in the same environment as Datasette.</p>
  <div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="datasette install datasette-sqlite-authorizer"><pre>datasette install datasette-sqlite-authorizer</pre></div>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Usage</h2><a id="user-content-usage" class="anchor" aria-label="Permalink: Usage" href="#usage"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">This plugin currently offers a single configuration option: <code>read_only_tables</code>. You can use this to specify a list of tables that should be read-only.</p>
  <p dir="auto">If a table is read-only, any attempt to write to it - <code>insert</code>, <code>update</code>, <code>delete</code>, <code>drop table</code>, <code>alter table</code> - will be denied with an error message.</p>
  <p dir="auto">To configure read-only tables, add the following to your <code>metadata.yaml</code> file:</p>
  <div class="highlight highlight-source-yaml notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="plugins:
    datasette-sqlite-authorizer:
      read_only_tables:
      - table: my_table
        database: my_database"><pre><span class="pl-ent">plugins</span>:
    <span class="pl-ent">datasette-sqlite-authorizer</span>:
      <span class="pl-ent">read_only_tables</span>:
      - <span class="pl-ent">table</span>: <span class="pl-s">my_table</span>
        <span class="pl-ent">database</span>: <span class="pl-s">my_database</span></pre></div>
  <p dir="auto">You can omit the <code>database</code> key if you want to apply the same rule to all databases.</p>
  <p dir="auto">Here's how to use this plugin to make all tables relating to <a href="https://litestream.io/" rel="nofollow">Litestream</a> synchronization read-only, across all attached databases:</p>
  <div class="highlight highlight-source-yaml notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="plugins:
    datasette-sqlite-authorizer:
      read_only_tables:
      - table: _litestream_lock
      - table: _litestream_seq"><pre><span class="pl-ent">plugins</span>:
    <span class="pl-ent">datasette-sqlite-authorizer</span>:
      <span class="pl-ent">read_only_tables</span>:
      - <span class="pl-ent">table</span>: <span class="pl-s">_litestream_lock</span>
      - <span class="pl-ent">table</span>: <span class="pl-s">_litestream_seq</span></pre></div>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Logging actions</h2><a id="user-content-logging-actions" class="anchor" aria-label="Permalink: Logging actions" href="#logging-actions"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">To enable debug output (to standard error) for specific action checks, add them to a <code>debug_actions</code> list. For example, to log all <code>SQLITE_INSERT</code> actions:</p>
  <div class="highlight highlight-source-yaml notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="plugins:
    datasette-sqlite-authorizer:
      debug_actions:
      - SQLITE_INSERT
      - SQLITE_DELETE"><pre><span class="pl-ent">plugins</span>:
    <span class="pl-ent">datasette-sqlite-authorizer</span>:
      <span class="pl-ent">debug_actions</span>:
      - <span class="pl-s">SQLITE_INSERT</span>
      - <span class="pl-s">SQLITE_DELETE</span></pre></div>
  <p dir="auto">Logged output will look something like this:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="authorizer: {"action": "SQLITE_INSERT", "arg1": "foo", "arg2": null, "db_name": "main", "trigger_name": null, "result": "OK"}
  authorizer: {"action": "SQLITE_DELETE", "arg1": "foo", "arg2": null, "db_name": "main", "trigger_name": null, "result": "OK"}"><pre class="notranslate"><code>authorizer: {"action": "SQLITE_INSERT", "arg1": "foo", "arg2": null, "db_name": "main", "trigger_name": null, "result": "OK"}
  authorizer: {"action": "SQLITE_DELETE", "arg1": "foo", "arg2": null, "db_name": "main", "trigger_name": null, "result": "OK"}
  </code></pre></div>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Development</h2><a id="user-content-development" class="anchor" aria-label="Permalink: Development" href="#development"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">To set up this plugin locally, first checkout the code. Then create a new virtual environment:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="cd datasette-sqlite-authorizer
  python3 -m venv venv
  source venv/bin/activate"><pre class="notranslate"><code>cd datasette-sqlite-authorizer
  python3 -m venv venv
  source venv/bin/activate
  </code></pre></div>
  <p dir="auto">Now install the dependencies and test dependencies:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="pip install -e '.[test]'"><pre class="notranslate"><code>pip install -e '.[test]'
  </code></pre></div>
  <p dir="auto">To run the tests:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="pytest"><pre class="notranslate"><code>pytest
  </code></pre></div>
  </article></div>


node_id: R_kgDOKWkLkQ
name: datasette-comments
full_name: datasette/datasette-comments
html_url: https://github.com/datasette/datasette-comments
description: A Datasette plugin for commenting on data in tables
created_at: 2023-09-21T16:03:52Z
updated_at: 2026-05-13T00:08:24Z
pushed_at: 2026-05-13T00:08:20Z
language: Python
license: apache-2.0
topics: ["datasette", "datasette-plugin"]
visibility: public
default_branch: main
permissions: {"admin": false, "maintain": false, "push": false, "triage": false, "pull": false}
custom_properties: {}
pull_request_creation_policy: all
readme:
  # datasette-comments

  [![PyPI](https://img.shields.io/pypi/v/datasette-comments.svg)](https://pypi.org/project/datasette-comments/)
  [![Changelog](https://img.shields.io/github/v/release/datasette/datasette-comments?include_prereleases&label=changelog)](https://github.com/datasette/datasette-comments/releases)
  [![Tests](https://github.com/datasette/datasette-comments/workflows/Test/badge.svg)](https://github.com/datasette/datasette-comments/actions?query=workflow%3ATest)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/datasette/datasette-comments/blob/main/LICENSE)

  A Datasette plugin for commenting on data in tables.

  Read [Annotate and explore your data with datasette-comments](https://www.datasette.cloud/blog/2023/datasette-comments/) for background on this project.

  <img src="https://datasette-cloud-assets.s3.amazonaws.com/blog/2023/datasette-comments/hero.jpg"/>

  ## Installation

  `datasette-comments` requires a recent 1.0 alpha version of Datasette to work.

  ```bash
  pip install datasette==1.0a7
  ```

  Afterwards, install this plugin in the same environment as Datasette.

  ```bash
  datasette install datasette-comments
  ```

  ## Usage

  `datasette-comments` store comments in [Datasette's internal database](https://docs.datasette.io/en/latest/internals.html#datasette-s-internal-database). So to persistent comments across multiple restarts, supply an database path on startup like so:

  ```bash
  datasette --internal internal.db my_data.db
  ```

  When comments are made on rows inside `my_data.db`, the comment themselves are stored separately in `internal.db`.

  The `datasette-comments-access` permission is required to be able to view and add comments. To give permissions to specfic users, set up your `metadata.yaml` like so:

  ```yaml
  permissions:
    datasette-comments-access:
      id: ["simonw", "asg017"]
  ```

  To provide actors and IDs, you'll need to setup a separate Datasette authentication plugin. Consider [datasette-auth-passwords](https://datasette.io/plugins/datasette-auth-passwords) for a simple username/password setup.

  ## Plugin hooks

  This plugin provies the following plugin hook which can be used to customize its behavior:

  ### datasette_comments_users(datasette)

  This hook should return a list of dictionaries, each representing a user that should be made available to the plugin. Each dictionary should have the following keys:

  - `id`: A unique ID of the user, same as the actor ID.
  - `username`: A unique string that is used in searches and @ mentions.
  - `name`: A string of the user's natural name.
  - `profile_photo_url`: Optional URL to the user's profile pic.
  - `email`: Optional email used for gravatar profile photo, if enabled.

  The plugin hook can return a list, or it can return an awaitable function that returns a list.

  ## Development

  To set up this plugin locally, first checkout the code. Then create a new virtual environment:

  ```bash
  cd datasette-comments
  python3 -m venv venv
  source venv/bin/activate
  ```

  Now install the dependencies and test dependencies:

  ```bash
  pip install -e '.[test]'
  ```

  And for the JavaScript dependencies (needed to run a JavaScript build):

  ```bash
  npm install
  ```

  To run the tests:

  ```bash
  pytest
  ```

  To rebuild the minified JavaScript after making a change to a `.ts` or `.tsx` file:

  ```bash
  just js
  ```

readme_html:
  <div id="readme" class="md" data-path="README.md"><article class="markdown-body entry-content container-lg" itemprop="text"><div class="markdown-heading" dir="auto"><h1 class="heading-element" dir="auto">datasette-comments</h1><a id="user-content-datasette-comments" class="anchor" aria-label="Permalink: datasette-comments" href="#datasette-comments"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto"><a href="https://pypi.org/project/datasette-comments/" rel="nofollow"><img src="https://img.shields.io/pypi/v/datasette-comments.svg" alt="PyPI" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/a788af8867a36787901f482360743190420aa8523b0d80971fda4726a4794946/68747470733a2f2f696d672e736869656c64732e696f2f707970692f762f6461746173657474652d636f6d6d656e74732e737667"></a>
  <a href="https://github.com/datasette/datasette-comments/releases"><img src="https://img.shields.io/github/v/release/datasette/datasette-comments?include_prereleases&label=changelog" alt="Changelog" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/9ed5a8b3a1cfdb50672baace9439763275802c6b4cd76fd82824f859d3b88c16/68747470733a2f2f696d672e736869656c64732e696f2f6769746875622f762f72656c656173652f6461746173657474652f6461746173657474652d636f6d6d656e74733f696e636c7564655f70726572656c6561736573266c6162656c3d6368616e67656c6f67"></a>
  <a href="https://github.com/datasette/datasette-comments/actions?query=workflow%3ATest"><img src="https://github.com/datasette/datasette-comments/workflows/Test/badge.svg" alt="Tests" style="max-width: 100%;"></a>
  <a href="https://github.com/datasette/datasette-comments/blob/main/LICENSE"><img src="https://img.shields.io/badge/license-Apache%202.0-blue.svg" alt="License" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/c355f200ea90fddaa407b6eaab303663a669248ea3ca7b1fcf77dbe04ff5f48c/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6c6963656e73652d417061636865253230322e302d626c75652e737667"></a></p>
  <p dir="auto">A Datasette plugin for commenting on data in tables.</p>
  <p dir="auto">Read <a href="https://www.datasette.cloud/blog/2023/datasette-comments/" rel="nofollow">Annotate and explore your data with datasette-comments</a> for background on this project.</p>
  <p dir="auto"><a target="_blank" rel="noopener noreferrer nofollow" href="https://camo.githubusercontent.com/da1bceba7993299022d683066c0d4c675ff59f2d105df3360ff105604ee87693/68747470733a2f2f6461746173657474652d636c6f75642d6173736574732e73332e616d617a6f6e6177732e636f6d2f626c6f672f323032332f6461746173657474652d636f6d6d656e74732f6865726f2e6a7067"><img src="https://datasette-cloud-assets.s3.amazonaws.com/blog/2023/datasette-comments/hero.jpg" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/da1bceba7993299022d683066c0d4c675ff59f2d105df3360ff105604ee87693/68747470733a2f2f6461746173657474652d636c6f75642d6173736574732e73332e616d617a6f6e6177732e636f6d2f626c6f672f323032332f6461746173657474652d636f6d6d656e74732f6865726f2e6a7067"></a></p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Installation</h2><a id="user-content-installation" class="anchor" aria-label="Permalink: Installation" href="#installation"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto"><code>datasette-comments</code> requires a recent 1.0 alpha version of Datasette to work.</p>
  <div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="pip install datasette==1.0a7"><pre>pip install datasette==1.0a7</pre></div>
  <p dir="auto">Afterwards, install this plugin in the same environment as Datasette.</p>
  <div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="datasette install datasette-comments"><pre>datasette install datasette-comments</pre></div>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Usage</h2><a id="user-content-usage" class="anchor" aria-label="Permalink: Usage" href="#usage"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto"><code>datasette-comments</code> store comments in <a href="https://docs.datasette.io/en/latest/internals.html#datasette-s-internal-database" rel="nofollow">Datasette's internal database</a>. So to persistent comments across multiple restarts, supply an database path on startup like so:</p>
  <div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="datasette --internal internal.db my_data.db"><pre>datasette --internal internal.db my_data.db</pre></div>
  <p dir="auto">When comments are made on rows inside <code>my_data.db</code>, the comment themselves are stored separately in <code>internal.db</code>.</p>
  <p dir="auto">The <code>datasette-comments-access</code> permission is required to be able to view and add comments. To give permissions to specfic users, set up your <code>metadata.yaml</code> like so:</p>
  <div class="highlight highlight-source-yaml notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="permissions:
    datasette-comments-access:
      id: ["simonw", "asg017"]"><pre><span class="pl-ent">permissions</span>:
    <span class="pl-ent">datasette-comments-access</span>:
      <span class="pl-ent">id</span>: <span class="pl-s">["simonw", "asg017"]</span></pre></div>
  <p dir="auto">To provide actors and IDs, you'll need to setup a separate Datasette authentication plugin. Consider <a href="https://datasette.io/plugins/datasette-auth-passwords" rel="nofollow">datasette-auth-passwords</a> for a simple username/password setup.</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Plugin hooks</h2><a id="user-content-plugin-hooks" class="anchor" aria-label="Permalink: Plugin hooks" href="#plugin-hooks"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">This plugin provies the following plugin hook which can be used to customize its behavior:</p>
  <div class="markdown-heading" dir="auto"><h3 class="heading-element" dir="auto">datasette_comments_users(datasette)</h3><a id="user-content-datasette_comments_usersdatasette" class="anchor" aria-label="Permalink: datasette_comments_users(datasette)" href="#datasette_comments_usersdatasette"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">This hook should return a list of dictionaries, each representing a user that should be made available to the plugin. Each dictionary should have the following keys:</p>
  <ul dir="auto">
  <li><code>id</code>: A unique ID of the user, same as the actor ID.</li>
  <li><code>username</code>: A unique string that is used in searches and @ mentions.</li>
  <li><code>name</code>: A string of the user's natural name.</li>
  <li><code>profile_photo_url</code>: Optional URL to the user's profile pic.</li>
  <li><code>email</code>: Optional email used for gravatar profile photo, if enabled.</li>
  </ul>
  <p dir="auto">The plugin hook can return a list, or it can return an awaitable function that returns a list.</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Development</h2><a id="user-content-development" class="anchor" aria-label="Permalink: Development" href="#development"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">To set up this plugin locally, first checkout the code. Then create a new virtual environment:</p>
  <div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="cd datasette-comments
  python3 -m venv venv
  source venv/bin/activate"><pre><span class="pl-c1">cd</span> datasette-comments
  python3 -m venv venv
  <span class="pl-c1">source</span> venv/bin/activate</pre></div>
  <p dir="auto">Now install the dependencies and test dependencies:</p>
  <div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="pip install -e '.[test]'"><pre>pip install -e <span class="pl-s"><span class="pl-pds">'</span>.[test]<span class="pl-pds">'</span></span></pre></div>
  <p dir="auto">And for the JavaScript dependencies (needed to run a JavaScript build):</p>
  <div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="npm install"><pre>npm install</pre></div>
  <p dir="auto">To run the tests:</p>
  <div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="pytest"><pre>pytest</pre></div>
  <p dir="auto">To rebuild the minified JavaScript after making a change to a <code>.ts</code> or <code>.tsx</code> file:</p>
  <div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="just js"><pre>just js</pre></div>
  </article></div>


node_id: R_kgDOKiitUw
name: datasette-plot
full_name: datasette/datasette-plot
html_url: https://github.com/datasette/datasette-plot
description: A Datasette plugin for making data visualizations with Observable Plot
created_at: 2023-10-19T16:22:23Z
updated_at: 2025-10-21T20:54:03Z
pushed_at: 2025-10-21T20:54:13Z
homepage: https://datasette.io/plugins/datasette-plot
language: TypeScript
license: apache-2.0
topics: ["datasette", "datasette-p", "observableplot"]
visibility: public
default_branch: main
permissions: {"admin": false, "maintain": false, "push": false, "triage": false, "pull": false}
custom_properties: {}
readme:
  # datasette-plot

  [![PyPI](https://img.shields.io/pypi/v/datasette-plot.svg)](https://pypi.org/project/datasette-plot/)
  [![Changelog](https://img.shields.io/github/v/release/datasette/datasette-plot?include_prereleases&label=changelog)](https://github.com/datasette/datasette-plot/releases)
  [![Tests](https://github.com/datasette/datasette-plot/workflows/Test/badge.svg)](https://github.com/datasette/datasette-plot/actions?query=workflow%3ATest)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/datasette/datasette-plot/blob/main/LICENSE)

  A Datasette plugin for making data visualizations with [Observable Plot](https://observablehq.com/plot/).

  <img width="600px" src="https://datasette-cloud-assets.s3.amazonaws.com/blog/2023/datasette-plot/hero.jpeg"/>

  ## Installation

  Install this plugin in the same environment as Datasette.

  ```bash
  datasette install datasette-plot
  ```

  ## Usage

  Once installed, table pages and SQL query pages will have a new "Show plot" button that will open the `datasette-plot`.

  <img width="600px" src="https://datasette-cloud-assets.s3.amazonaws.com/blog/2023/datasette-plot/table-2.jpeg"/>

  Currently, `datasette-plot` only supports a few [visualization marks](https://observablehq.com/plot/features/marks) from Plot, including dot, line, area, bar, and more.

  <video src="https://github.com/datasette/datasette-plot/assets/15178711/9d6f5f37-b827-44e2-a8f5-e3e552701771"></video>

  Use the `Link to this plot` URL to share visualization with others. The link will have your plot options encoded in the URL.

  ## Development

  To set up this plugin locally, first checkout the code. Then create a new virtual environment:

  ```bash
  cd datasette-plot
  python3 -m venv venv
  source venv/bin/activate
  ```

  Now install the dependencies and test dependencies:

  ```bash
  pip install -e '.[test]'
  ```

  To run the tests:

  ```bash
  pytest
  ```

readme_html:
  <div id="readme" class="md" data-path="README.md"><article class="markdown-body entry-content container-lg" itemprop="text"><div class="markdown-heading" dir="auto"><h1 class="heading-element" dir="auto">datasette-plot</h1><a id="user-content-datasette-plot" class="anchor" aria-label="Permalink: datasette-plot" href="#datasette-plot"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto"><a href="https://pypi.org/project/datasette-plot/" rel="nofollow"><img src="https://img.shields.io/pypi/v/datasette-plot.svg" alt="PyPI" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/df3da1ddd26136d5ebf9455b849a76692e80ba3a84e2a7016469b9f7fc100004/68747470733a2f2f696d672e736869656c64732e696f2f707970692f762f6461746173657474652d706c6f742e737667"></a>
  <a href="https://github.com/datasette/datasette-plot/releases"><img src="https://img.shields.io/github/v/release/datasette/datasette-plot?include_prereleases&label=changelog" alt="Changelog" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/0a4d24f46eea7c1b7eb55285ae5127af44e95c4c2c18544cbafa1e7c2ea0a71c/68747470733a2f2f696d672e736869656c64732e696f2f6769746875622f762f72656c656173652f6461746173657474652f6461746173657474652d706c6f743f696e636c7564655f70726572656c6561736573266c6162656c3d6368616e67656c6f67"></a>
  <a href="https://github.com/datasette/datasette-plot/actions?query=workflow%3ATest"><img src="https://github.com/datasette/datasette-plot/workflows/Test/badge.svg" alt="Tests" style="max-width: 100%;"></a>
  <a href="https://github.com/datasette/datasette-plot/blob/main/LICENSE"><img src="https://img.shields.io/badge/license-Apache%202.0-blue.svg" alt="License" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/c355f200ea90fddaa407b6eaab303663a669248ea3ca7b1fcf77dbe04ff5f48c/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6c6963656e73652d417061636865253230322e302d626c75652e737667"></a></p>
  <p dir="auto">A Datasette plugin for making data visualizations with <a href="https://observablehq.com/plot/" rel="nofollow">Observable Plot</a>.</p>
  <p dir="auto"><a target="_blank" rel="noopener noreferrer nofollow" href="https://camo.githubusercontent.com/9ea5684105b337499fbab8267a4ef72f529932b7f39a8329fbc0416dfd844d06/68747470733a2f2f6461746173657474652d636c6f75642d6173736574732e73332e616d617a6f6e6177732e636f6d2f626c6f672f323032332f6461746173657474652d706c6f742f6865726f2e6a706567"><img width="600px" src="https://datasette-cloud-assets.s3.amazonaws.com/blog/2023/datasette-plot/hero.jpeg" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/9ea5684105b337499fbab8267a4ef72f529932b7f39a8329fbc0416dfd844d06/68747470733a2f2f6461746173657474652d636c6f75642d6173736574732e73332e616d617a6f6e6177732e636f6d2f626c6f672f323032332f6461746173657474652d706c6f742f6865726f2e6a706567"></a></p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Installation</h2><a id="user-content-installation" class="anchor" aria-label="Permalink: Installation" href="#installation"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">Install this plugin in the same environment as Datasette.</p>
  <div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="datasette install datasette-plot"><pre>datasette install datasette-plot</pre></div>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Usage</h2><a id="user-content-usage" class="anchor" aria-label="Permalink: Usage" href="#usage"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">Once installed, table pages and SQL query pages will have a new "Show plot" button that will open the <code>datasette-plot</code>.</p>
  <p dir="auto"><a target="_blank" rel="noopener noreferrer nofollow" href="https://camo.githubusercontent.com/b8a65d4b8d391bdf1012c02febdbbcc28858c0eb2abe686a7a2a0c72b31f45d7/68747470733a2f2f6461746173657474652d636c6f75642d6173736574732e73332e616d617a6f6e6177732e636f6d2f626c6f672f323032332f6461746173657474652d706c6f742f7461626c652d322e6a706567"><img width="600px" src="https://datasette-cloud-assets.s3.amazonaws.com/blog/2023/datasette-plot/table-2.jpeg" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/b8a65d4b8d391bdf1012c02febdbbcc28858c0eb2abe686a7a2a0c72b31f45d7/68747470733a2f2f6461746173657474652d636c6f75642d6173736574732e73332e616d617a6f6e6177732e636f6d2f626c6f672f323032332f6461746173657474652d706c6f742f7461626c652d322e6a706567"></a></p>
  <p dir="auto">Currently, <code>datasette-plot</code> only supports a few <a href="https://observablehq.com/plot/features/marks" rel="nofollow">visualization marks</a> from Plot, including dot, line, area, bar, and more.</p>
  <p dir="auto"><gh:secured-asset-reference resource_type="UserAsset" resource_id="292297472"></gh:secured-asset-reference></p><details open="" class="details-reset border rounded-2">
    <summary class="px-3 py-2">
      <svg aria-hidden="true" height="16" viewbox="0 0 16 16" version="1.1" width="16" data-view-component="true" class="octicon octicon-device-camera-video">
      <path d="M16 3.75v8.5a.75.75 0 0 1-1.136.643L11 10.575v.675A1.75 1.75 0 0 1 9.25 13h-7.5A1.75 1.75 0 0 1 0 11.25v-6.5C0 3.784.784 3 1.75 3h7.5c.966 0 1.75.784 1.75 1.75v.675l3.864-2.318A.75.75 0 0 1 16 3.75Zm-6.5 1a.25.25 0 0 0-.25-.25h-7.5a.25.25 0 0 0-.25.25v6.5c0 .138.112.25.25.25h7.5a.25.25 0 0 0 .25-.25v-6.5ZM11 8.825l3.5 2.1v-5.85l-3.5 2.1Z"></path>
  </svg>
      <span aria-label="Video description dsc-plot-02.mp4" class="m-1">dsc-plot-02.mp4</span>
      <span class="dropdown-caret"></span>
    </summary>

    <video src="https://github.com/datasette/datasette-plot/assets/15178711/9d6f5f37-b827-44e2-a8f5-e3e552701771" data-canonical-src="https://github.com/datasette/datasette-plot/assets/15178711/9d6f5f37-b827-44e2-a8f5-e3e552701771" controls="controls" muted="muted" class="d-block rounded-bottom-2 border-top width-fit" style="max-height:640px; min-height: 200px">

    </video>
  </details>
  <p></p>
  <p dir="auto">Use the <code>Link to this plot</code> URL to share visualization with others. The link will have your plot options encoded in the URL.</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Development</h2><a id="user-content-development" class="anchor" aria-label="Permalink: Development" href="#development"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">To set up this plugin locally, first checkout the code. Then create a new virtual environment:</p>
  <div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="cd datasette-plot
  python3 -m venv venv
  source venv/bin/activate"><pre><span class="pl-c1">cd</span> datasette-plot
  python3 -m venv venv
  <span class="pl-c1">source</span> venv/bin/activate</pre></div>
  <p dir="auto">Now install the dependencies and test dependencies:</p>
  <div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="pip install -e '.[test]'"><pre>pip install -e <span class="pl-s"><span class="pl-pds">'</span>.[test]<span class="pl-pds">'</span></span></pre></div>
  <p dir="auto">To run the tests:</p>
  <div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="pytest"><pre>pytest</pre></div>
  </article></div>


node_id: R_kgDOKuZiqg
name: datasette-enrichments-jinja
full_name: datasette/datasette-enrichments-jinja
html_url: https://github.com/datasette/datasette-enrichments-jinja
description: Datasette enrichment for evaluating templates in a Jinja sandbox
created_at: 2023-11-16T20:03:52Z
updated_at: 2024-12-03T21:20:25Z
pushed_at: 2023-12-06T20:10:09Z
language: Python
license: apache-2.0
topics: []
visibility: public
default_branch: main
permissions: {"admin": false, "maintain": false, "push": false, "triage": false, "pull": true}
custom_properties: {}
readme:
  # datasette-enrichments-jinja

  [![PyPI](https://img.shields.io/pypi/v/datasette-enrichments-jinja.svg)](https://pypi.org/project/datasette-enrichments-jinja/)
  [![Changelog](https://img.shields.io/github/v/release/datasette/datasette-enrichments-jinja?include_prereleases&label=changelog)](https://github.com/datasette/datasette-enrichments-jinja/releases)
  [![Tests](https://github.com/datasette/datasette-enrichments-jinja/workflows/Test/badge.svg)](https://github.com/datasette/datasette-enrichments-jinja/actions?query=workflow%3ATest)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/datasette/datasette-enrichments-jinja/blob/main/LICENSE)

  [Datasette enrichment](https://github.com/simonw/datasette-enrichments) for evaluating templates in a Jinja sandbox

  ## Installation

  Install this plugin in the same environment as Datasette.
  ```bash
  datasette install datasette-enrichments-jinja
  ```
  ## Usage

  This enrichment allows you to select rows from a table and specify a Jinja template to use to generate new text for each of those rows.

  The text can then be saved to a new or existing column on the table.

  Code runs in a [Jinja sandbox](https://jinja.palletsprojects.com/en/3.1.x/sandbox/). This should protect against innocent mistakes, but may not be robust against malicious attackers - so only make this enrichment available to users who you trust not to abuse it.

  ## Development

  To set up this plugin locally, first checkout the code. Then create a new virtual environment:
  ```bash
  cd datasette-enrichments-jinja
  python3 -m venv venv
  source venv/bin/activate
  ```
  Now install the dependencies and test dependencies:
  ```bash
  pip install -e '.[test]'
  ```
  To run the tests:
  ```bash
  pytest
  ```

readme_html:
  <div id="readme" class="md" data-path="README.md"><article class="markdown-body entry-content container-lg" itemprop="text"><div class="markdown-heading" dir="auto"><h1 class="heading-element" dir="auto">datasette-enrichments-jinja</h1><a id="user-content-datasette-enrichments-jinja" class="anchor" aria-label="Permalink: datasette-enrichments-jinja" href="#datasette-enrichments-jinja"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto"><a href="https://pypi.org/project/datasette-enrichments-jinja/" rel="nofollow"><img src="https://img.shields.io/pypi/v/datasette-enrichments-jinja.svg" alt="PyPI" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/f21c01fbaa78ee6652023de002cc3841dc59d358b05d2c8953b6161c6392200b/68747470733a2f2f696d672e736869656c64732e696f2f707970692f762f6461746173657474652d656e726963686d656e74732d6a696e6a612e737667"></a>
  <a href="https://github.com/datasette/datasette-enrichments-jinja/releases"><img src="https://img.shields.io/github/v/release/datasette/datasette-enrichments-jinja?include_prereleases&label=changelog" alt="Changelog" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/1c1d7c9c1772fdf32b45e9bf5dfca8cd708cd4138edb19722470f75eadd45729/68747470733a2f2f696d672e736869656c64732e696f2f6769746875622f762f72656c656173652f6461746173657474652f6461746173657474652d656e726963686d656e74732d6a696e6a613f696e636c7564655f70726572656c6561736573266c6162656c3d6368616e67656c6f67"></a>
  <a href="https://github.com/datasette/datasette-enrichments-jinja/actions?query=workflow%3ATest"><img src="https://github.com/datasette/datasette-enrichments-jinja/workflows/Test/badge.svg" alt="Tests" style="max-width: 100%;"></a>
  <a href="https://github.com/datasette/datasette-enrichments-jinja/blob/main/LICENSE"><img src="https://img.shields.io/badge/license-Apache%202.0-blue.svg" alt="License" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/c355f200ea90fddaa407b6eaab303663a669248ea3ca7b1fcf77dbe04ff5f48c/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6c6963656e73652d417061636865253230322e302d626c75652e737667"></a></p>
  <p dir="auto"><a href="https://github.com/simonw/datasette-enrichments">Datasette enrichment</a> for evaluating templates in a Jinja sandbox</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Installation</h2><a id="user-content-installation" class="anchor" aria-label="Permalink: Installation" href="#installation"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">Install this plugin in the same environment as Datasette.</p>
  <div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="datasette install datasette-enrichments-jinja"><pre>datasette install datasette-enrichments-jinja</pre></div>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Usage</h2><a id="user-content-usage" class="anchor" aria-label="Permalink: Usage" href="#usage"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">This enrichment allows you to select rows from a table and specify a Jinja template to use to generate new text for each of those rows.</p>
  <p dir="auto">The text can then be saved to a new or existing column on the table.</p>
  <p dir="auto">Code runs in a <a href="https://jinja.palletsprojects.com/en/3.1.x/sandbox/" rel="nofollow">Jinja sandbox</a>. This should protect against innocent mistakes, but may not be robust against malicious attackers - so only make this enrichment available to users who you trust not to abuse it.</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Development</h2><a id="user-content-development" class="anchor" aria-label="Permalink: Development" href="#development"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">To set up this plugin locally, first checkout the code. Then create a new virtual environment:</p>
  <div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="cd datasette-enrichments-jinja
  python3 -m venv venv
  source venv/bin/activate"><pre><span class="pl-c1">cd</span> datasette-enrichments-jinja
  python3 -m venv venv
  <span class="pl-c1">source</span> venv/bin/activate</pre></div>
  <p dir="auto">Now install the dependencies and test dependencies:</p>
  <div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="pip install -e '.[test]'"><pre>pip install -e <span class="pl-s"><span class="pl-pds">'</span>.[test]<span class="pl-pds">'</span></span></pre></div>
  <p dir="auto">To run the tests:</p>
  <div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="pytest"><pre>pytest</pre></div>
  </article></div>


node_id: R_kgDOKugxLQ
name: datasette-enrichments-gpt
full_name: datasette/datasette-enrichments-gpt
html_url: https://github.com/datasette/datasette-enrichments-gpt
description: Datasette enrichment for analyzing row data using OpenAI's GPT models
created_at: 2023-11-17T03:41:59Z
updated_at: 2024-10-19T15:56:27Z
pushed_at: 2024-05-15T21:48:50Z
language: Python
license: apache-2.0
topics: []
visibility: public
default_branch: main
permissions: {"admin": false, "maintain": false, "push": false, "triage": false, "pull": true}
custom_properties: {}
readme:
  # datasette-enrichments-gpt

  [![PyPI](https://img.shields.io/pypi/v/datasette-enrichments-gpt.svg)](https://pypi.org/project/datasette-enrichments-gpt/)
  [![Changelog](https://img.shields.io/github/v/release/datasette/datasette-enrichments-gpt?include_prereleases&label=changelog)](https://github.com/datasette/datasette-enrichments-gpt/releases)
  [![Tests](https://github.com/datasette/datasette-enrichments-gpt/workflows/Test/badge.svg)](https://github.com/datasette/datasette-enrichments-gpt/actions?query=workflow%3ATest)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/datasette/datasette-enrichments-gpt/blob/main/LICENSE)

  Datasette enrichment for analyzing row data using OpenAI's GPT models

  ## Installation

  Install this plugin in the same environment as Datasette.
  ```bash
  datasette install datasette-enrichments-gpt
  ```
  ## Configuration

  This plugin can optionally be configured with an [OpenAI API key](https://platform.openai.com/api-keys). You can set this as an environment variable:
  ```bash
  export DATASETTE_SECRETS_OPENAPI_API_KEY=sk-..
  ```
  Or you can configure it using the [datasette-secrets](https://datasette.io/plugins/datasette-secrets) plugin.

  If you do not configure an OpenAI API key users will be asked to enter one any time they execute the enrichment. The key they provide will not be stored anywhere other than in-memory during the enrichment run.

  ## Usage

  Once installed, this plugin will allow users to select rows to enrich and run them through prompts using `gpt-3.5-turbo` or `gpt-4o`, saving the result of the prompt in the specified column.

  The plugin also provides `gpt-4o vision`, which can run prompts against an image identified by a URL.

  ## Development

  To set up this plugin locally, first checkout the code. Then create a new virtual environment:
  ```bash
  cd datasette-enrichments-gpt
  python3 -m venv venv
  source venv/bin/activate
  ```
  Now install the dependencies and test dependencies:
  ```bash
  pip install -e '.[test]'
  ```
  To run the tests:
  ```bash
  pytest
  ```

readme_html:
  <div id="readme" class="md" data-path="README.md"><article class="markdown-body entry-content container-lg" itemprop="text"><div class="markdown-heading" dir="auto"><h1 class="heading-element" dir="auto">datasette-enrichments-gpt</h1><a id="user-content-datasette-enrichments-gpt" class="anchor" aria-label="Permalink: datasette-enrichments-gpt" href="#datasette-enrichments-gpt"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto"><a href="https://pypi.org/project/datasette-enrichments-gpt/" rel="nofollow"><img src="https://img.shields.io/pypi/v/datasette-enrichments-gpt.svg" alt="PyPI" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/09596ce71e0a6f8be0f6c6966450c3046accf82afa1792d28424c4a22b2d9b3d/68747470733a2f2f696d672e736869656c64732e696f2f707970692f762f6461746173657474652d656e726963686d656e74732d6770742e737667"></a>
  <a href="https://github.com/datasette/datasette-enrichments-gpt/releases"><img src="https://img.shields.io/github/v/release/datasette/datasette-enrichments-gpt?include_prereleases&label=changelog" alt="Changelog" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/effce50743cf659f1d3488447ec64bef21f0500d10848b98ffbbb1c0ad4aa62c/68747470733a2f2f696d672e736869656c64732e696f2f6769746875622f762f72656c656173652f6461746173657474652f6461746173657474652d656e726963686d656e74732d6770743f696e636c7564655f70726572656c6561736573266c6162656c3d6368616e67656c6f67"></a>
  <a href="https://github.com/datasette/datasette-enrichments-gpt/actions?query=workflow%3ATest"><img src="https://github.com/datasette/datasette-enrichments-gpt/workflows/Test/badge.svg" alt="Tests" style="max-width: 100%;"></a>
  <a href="https://github.com/datasette/datasette-enrichments-gpt/blob/main/LICENSE"><img src="https://img.shields.io/badge/license-Apache%202.0-blue.svg" alt="License" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/c355f200ea90fddaa407b6eaab303663a669248ea3ca7b1fcf77dbe04ff5f48c/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6c6963656e73652d417061636865253230322e302d626c75652e737667"></a></p>
  <p dir="auto">Datasette enrichment for analyzing row data using OpenAI's GPT models</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Installation</h2><a id="user-content-installation" class="anchor" aria-label="Permalink: Installation" href="#installation"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">Install this plugin in the same environment as Datasette.</p>
  <div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="datasette install datasette-enrichments-gpt"><pre>datasette install datasette-enrichments-gpt</pre></div>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Configuration</h2><a id="user-content-configuration" class="anchor" aria-label="Permalink: Configuration" href="#configuration"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">This plugin can optionally be configured with an <a href="https://platform.openai.com/api-keys" rel="nofollow">OpenAI API key</a>. You can set this as an environment variable:</p>
  <div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="export DATASETTE_SECRETS_OPENAPI_API_KEY=sk-.."><pre><span class="pl-k">export</span> DATASETTE_SECRETS_OPENAPI_API_KEY=sk-..</pre></div>
  <p dir="auto">Or you can configure it using the <a href="https://datasette.io/plugins/datasette-secrets" rel="nofollow">datasette-secrets</a> plugin.</p>
  <p dir="auto">If you do not configure an OpenAI API key users will be asked to enter one any time they execute the enrichment. The key they provide will not be stored anywhere other than in-memory during the enrichment run.</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Usage</h2><a id="user-content-usage" class="anchor" aria-label="Permalink: Usage" href="#usage"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">Once installed, this plugin will allow users to select rows to enrich and run them through prompts using <code>gpt-3.5-turbo</code> or <code>gpt-4o</code>, saving the result of the prompt in the specified column.</p>
  <p dir="auto">The plugin also provides <code>gpt-4o vision</code>, which can run prompts against an image identified by a URL.</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Development</h2><a id="user-content-development" class="anchor" aria-label="Permalink: Development" href="#development"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">To set up this plugin locally, first checkout the code. Then create a new virtual environment:</p>
  <div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="cd datasette-enrichments-gpt
  python3 -m venv venv
  source venv/bin/activate"><pre><span class="pl-c1">cd</span> datasette-enrichments-gpt
  python3 -m venv venv
  <span class="pl-c1">source</span> venv/bin/activate</pre></div>
  <p dir="auto">Now install the dependencies and test dependencies:</p>
  <div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="pip install -e '.[test]'"><pre>pip install -e <span class="pl-s"><span class="pl-pds">'</span>.[test]<span class="pl-pds">'</span></span></pre></div>
  <p dir="auto">To run the tests:</p>
  <div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="pytest"><pre>pytest</pre></div>
  </article></div>


