# uses

_id: 61a01ca9a0e3de0a0f0af9691661af44ea6e149d
_path: for/data-journalism.md
title: Datasette for data journalism
text:
  Datasette was originally designed as a tool for data journalists, to help report on data-driven stories, crunch through large datasets and [publish the results](/for/publishing-data).

  [This demo](https://vimeo.com/436903714) from [SRCCON 2020](https://2020.srccon.org/) shows how Datasette can be used to analyze PPP loan data released by the Small Business Administration. 

  <iframe style="max-width: 100%" src="https://player.vimeo.com/video/436903714" width="640" height="400" frameborder="0" allow="autoplay; fullscreen" allowfullscreen></iframe>

  You can try the demo [here](https://sba-loans-covid-19.datasettes.com/), or read more about it in [this article](https://simonwillison.net/2020/Jul/9/sba-covid-19-ppp-loans/).

html:
  <p>Datasette was originally designed as a tool for data journalists, to help report on data-driven stories, crunch through large datasets and <a href="/for/publishing-data">publish the results</a>.</p>
  <p><a href="https://vimeo.com/436903714">This demo</a> from <a href="https://2020.srccon.org/">SRCCON 2020</a> shows how Datasette can be used to analyze PPP loan data released by the Small Business Administration. </p>
  <iframe style="max-width: 100%" src="https://player.vimeo.com/video/436903714" width="640" height="400" frameborder="0" allow="autoplay; fullscreen" allowfullscreen></iframe>

  <p>You can try the demo <a href="https://sba-loans-covid-19.datasettes.com/">here</a>, or read more about it in <a href="https://simonwillison.net/2020/Jul/9/sba-covid-19-ppp-loans/">this article</a>.</p>

summary:
  Datasette was originally designed as a tool for data journalists, to help report on data-driven stories, crunch through large datasets and publish the results.


_id: ddae37b01edec73668ab75f7e9799f91b580a554
_path: for/exploratory-analysis.md
title: Datasette for exploratory analysis
text:
  Your first step with any new data set should be to start exploring it. Datasette provides multiple tools for doing this.

  ## Start browsing the data

  Datasette's table interface lets you start scrolling through data straight away.

  ![Screenshot of the table interface](/static/datasette-screenshot.png)

  ## Facets

  Datasette automatically identifies columns with less than twenty unique values and gives you the option to facet by them - seeing the most common values, and selecting those to filter the set. This provides a powerful way to start spotting patterns in the data.

  ![Screenshot of the facet interface](/static/screenshots/facets.png)

  ## Visualization

  Plugins such as [datasette-vega](https://github.com/simonw/datasette-vega) and [datasette-cluster-map](https://github.com/simonw/datasette-cluster-map) provide tools for interactively visualizing data directly within the Datasette interface.

  ![Screenshot of datasette-cluster-map](/static/screenshots/datasette-cluster-map.png)

html:
  <p>Your first step with any new data set should be to start exploring it. Datasette provides multiple tools for doing this.</p>
  <h2>Start browsing the data</h2>
  <p>Datasette's table interface lets you start scrolling through data straight away.</p>
  <p><img alt="Screenshot of the table interface" src="/static/datasette-screenshot.png" /></p>
  <h2>Facets</h2>
  <p>Datasette automatically identifies columns with less than twenty unique values and gives you the option to facet by them - seeing the most common values, and selecting those to filter the set. This provides a powerful way to start spotting patterns in the data.</p>
  <p><img alt="Screenshot of the facet interface" src="/static/screenshots/facets.png" /></p>
  <h2>Visualization</h2>
  <p>Plugins such as <a href="https://github.com/simonw/datasette-vega">datasette-vega</a> and <a href="https://github.com/simonw/datasette-cluster-map">datasette-cluster-map</a> provide tools for interactively visualizing data directly within the Datasette interface.</p>
  <p><img alt="Screenshot of datasette-cluster-map" src="/static/screenshots/datasette-cluster-map.png" /></p>

summary:
  Import data from CSVs, JSON, database connections and more. Datasette will automatically show you patterns in your data and help you share your findings with your colleagues.


_id: ce0b74d7bd16f128e2bd4664b3aa910162b5373f
_path: for/geospatial.md
title: Datasette for geospatial analysis
text:
  The [SpatiaLite](https://www.gaia-gis.it/fossil/libspatialite/index) extension for SQLite can be used with Datasette to enable a full suite of geospatial SQL functions.

  ![A screenshot of Datasette running some geospatial plugins](https://camo.githubusercontent.com/56a2ed7e65e28e726d4c3abfbe56bd7960524e6af6512fc2cee083bdd7fcd863/68747470733a2f2f7374617469632e73696d6f6e77696c6c69736f6e2e6e65742f7374617469632f323032312f6461746173657474652d6c6561666c65742d66726565647261772e706e67)

  Datasette has a number of plugins and tools that can be used to work with geospatial data.

  Geospatial plugins include:

  - [datasette-cluster-map](https://datasette.io/plugins/datasette-cluster-map) renders a map of markers for any table or query with `latitude` and `longitude` columns
  - [datasette-geojson](https://datasette.io/plugins/datasette-geojson) adds a `.geojson` extension which can export SpatiaLite geometries as [GeoJSON](https://geojson.org/)
  - [datasette-geojson-map](https://datasette.io/plugins/datasette-geojson-map) renders that GeoJSON output on a map
  - [datasette-leaflet-freedraw](https://datasette.io/plugins/datasette-leaflet-freedraw) adds an interface to filter geometries by drawing a shape on a map
  - [datasette-leaflet-geojson](https://datasette.io/plugins/datasette-leaflet-geojson) shows maps inline for any column that includes GeoJSON
  - [datasette-tiles](https://datasette.io/plugins/datasette-tiles) allows Datasette to serve map tile images stored using the [MBTiles](https://github.com/mapbox/mbtiles-spec) format

  Tools for working with geospatial data include:

  - [geojson-to-sqlite](https://datasette.io/tools/geojson-to-sqlite), a command line utility for loading GeoJSON data into a SQLite or SpatiaLite database
  - [shapefile-to-sqlite](https://datasette.io/tools/shapefile-to-sqlite), a command line utility for loading shapefiles into SQLite or SpatiaLite
  - [download-tiles](https://datasette.io/tools/download-tiles) can be used to download map tiles and store them in MBTiles, suitable for use with [datasette-tiles](https://datasette.io/plugins/datasette-tiles)

  The following tutorials provide more detail on how Datasette can be used for geospatial processing:

  - [GUnion to combine geometries in SpatiaLite](https://til.simonwillison.net/spatialite/gunion-to-combine-geometries) shows how to load Amtrak data using `geojson-to-sqlite` and render it using `datasette-geojson-map`
  - [KNN queries with SpatiaLite](https://til.simonwillison.net/spatialite/knn) shows how to use the KNN feature to run queries showing the nearest geometries to a point
  - [Natural Earth in SpatiaLite and Datasette](https://til.simonwillison.net/gis/natural-earth-in-spatialite-and-datasette) shows how to use the 791MB [Natural Earth](https://www.naturalearthdata.com/) SQLite vector database with Datasette
  - [Drawing shapes on a map to query a SpatiaLite database](https://simonwillison.net/2021/Jan/24/drawing-shapes-spatialite/) introduces the `datasette-leaflet-freedraw` plugin

html:
  <p>The <a href="https://www.gaia-gis.it/fossil/libspatialite/index">SpatiaLite</a> extension for SQLite can be used with Datasette to enable a full suite of geospatial SQL functions.</p>
  <p><img alt="A screenshot of Datasette running some geospatial plugins" src="https://camo.githubusercontent.com/56a2ed7e65e28e726d4c3abfbe56bd7960524e6af6512fc2cee083bdd7fcd863/68747470733a2f2f7374617469632e73696d6f6e77696c6c69736f6e2e6e65742f7374617469632f323032312f6461746173657474652d6c6561666c65742d66726565647261772e706e67" /></p>
  <p>Datasette has a number of plugins and tools that can be used to work with geospatial data.</p>
  <p>Geospatial plugins include:</p>
  <ul>
  <li><a href="https://datasette.io/plugins/datasette-cluster-map">datasette-cluster-map</a> renders a map of markers for any table or query with <code>latitude</code> and <code>longitude</code> columns</li>
  <li><a href="https://datasette.io/plugins/datasette-geojson">datasette-geojson</a> adds a <code>.geojson</code> extension which can export SpatiaLite geometries as <a href="https://geojson.org/">GeoJSON</a></li>
  <li><a href="https://datasette.io/plugins/datasette-geojson-map">datasette-geojson-map</a> renders that GeoJSON output on a map</li>
  <li><a href="https://datasette.io/plugins/datasette-leaflet-freedraw">datasette-leaflet-freedraw</a> adds an interface to filter geometries by drawing a shape on a map</li>
  <li><a href="https://datasette.io/plugins/datasette-leaflet-geojson">datasette-leaflet-geojson</a> shows maps inline for any column that includes GeoJSON</li>
  <li><a href="https://datasette.io/plugins/datasette-tiles">datasette-tiles</a> allows Datasette to serve map tile images stored using the <a href="https://github.com/mapbox/mbtiles-spec">MBTiles</a> format</li>
  </ul>
  <p>Tools for working with geospatial data include:</p>
  <ul>
  <li><a href="https://datasette.io/tools/geojson-to-sqlite">geojson-to-sqlite</a>, a command line utility for loading GeoJSON data into a SQLite or SpatiaLite database</li>
  <li><a href="https://datasette.io/tools/shapefile-to-sqlite">shapefile-to-sqlite</a>, a command line utility for loading shapefiles into SQLite or SpatiaLite</li>
  <li><a href="https://datasette.io/tools/download-tiles">download-tiles</a> can be used to download map tiles and store them in MBTiles, suitable for use with <a href="https://datasette.io/plugins/datasette-tiles">datasette-tiles</a></li>
  </ul>
  <p>The following tutorials provide more detail on how Datasette can be used for geospatial processing:</p>
  <ul>
  <li><a href="https://til.simonwillison.net/spatialite/gunion-to-combine-geometries">GUnion to combine geometries in SpatiaLite</a> shows how to load Amtrak data using <code>geojson-to-sqlite</code> and render it using <code>datasette-geojson-map</code></li>
  <li><a href="https://til.simonwillison.net/spatialite/knn">KNN queries with SpatiaLite</a> shows how to use the KNN feature to run queries showing the nearest geometries to a point</li>
  <li><a href="https://til.simonwillison.net/gis/natural-earth-in-spatialite-and-datasette">Natural Earth in SpatiaLite and Datasette</a> shows how to use the 791MB <a href="https://www.naturalearthdata.com/">Natural Earth</a> SQLite vector database with Datasette</li>
  <li><a href="https://simonwillison.net/2021/Jan/24/drawing-shapes-spatialite/">Drawing shapes on a map to query a SpatiaLite database</a> introduces the <code>datasette-leaflet-freedraw</code> plugin</li>
  </ul>

summary:
  Combining Datasette with SpatiaLite provides a powerful set of tools for analyzing and visualizing geospatial data.


_id: 1940a8c4df2a2e0e7c95a457ce8dc4e7033c24d9
_path: for/publishing-data.md
title: Datasette for publishing data
text:
  Datasette was [originally inspired](https://simonwillison.net/2018/Aug/19/instantly-publish-datasette/) by work on [the Guardian Data Blog](https://www.theguardian.com/news/datablog/2011/jan/27/data-store-office-for-national-statistics), an initiative to publish the raw data behind the stories in the news.

  Datasette encourages sharing data as tables that can be explored by users and exported out as JSON or CSV. It's like publishing a CSV file but with much deeper flexibility for visitors to remix, visualize and export data.

  Data published by Datasette can be deployed to both traditional and serverless hosting platforms.

  The `datasette publish` command, [documented here](https://docs.datasette.io/en/stable/publish.html), can deploy to [Heroku](https://heroku.com/) and [Google Cloud Run](https://cloud.google.com/run). Plugins can be used to deploy Datasette on [Vercel](https://vercel.com/) or [fly.io](https://fly.io/).

  When combined with GitHub Actions, Datasette can be used to automatically deploy new databases on a scheduled basis. See [Deploying a data API using GitHub Actions and Cloud Run](https://simonwillison.net/2020/Jan/21/github-actions-cloud-run/) for more details on implementing this pattern.

html:
  <p>Datasette was <a href="https://simonwillison.net/2018/Aug/19/instantly-publish-datasette/">originally inspired</a> by work on <a href="https://www.theguardian.com/news/datablog/2011/jan/27/data-store-office-for-national-statistics">the Guardian Data Blog</a>, an initiative to publish the raw data behind the stories in the news.</p>
  <p>Datasette encourages sharing data as tables that can be explored by users and exported out as JSON or CSV. It's like publishing a CSV file but with much deeper flexibility for visitors to remix, visualize and export data.</p>
  <p>Data published by Datasette can be deployed to both traditional and serverless hosting platforms.</p>
  <p>The <code>datasette publish</code> command, <a href="https://docs.datasette.io/en/stable/publish.html">documented here</a>, can deploy to <a href="https://heroku.com/">Heroku</a> and <a href="https://cloud.google.com/run">Google Cloud Run</a>. Plugins can be used to deploy Datasette on <a href="https://vercel.com/">Vercel</a> or <a href="https://fly.io/">fly.io</a>.</p>
  <p>When combined with GitHub Actions, Datasette can be used to automatically deploy new databases on a scheduled basis. See <a href="https://simonwillison.net/2020/Jan/21/github-actions-cloud-run/">Deploying a data API using GitHub Actions and Cloud Run</a> for more details on implementing this pattern.</p>

summary:
  The built-in "datasette publish" command lets you instantly publish your data to hosting providers like Google Cloud Run, Heroku or Vercel.


_id: 0574db8535765cbe166676a6b73fd0205331313b
_path: for/rapid-prototyping.md
title: Datasette for rapid prototyping
text:
  Want to build a prototype? Datasette lets you turn any source of data into a powerful JSON or [GraphQL](https://github.com/simonw/datasette-graphql) API, ideal for use as the backend for an interactive prototype.

  Use Datasette with **[Observable notebooks](https://observablehq.com/)** to quickly try out new visualizations and explore the Datasette API using JavaScript.

  Datasette works as a great data for **[Jupyter notebooks](https://jupyter.org/)** too.

  - **[Analyzing ScotRail audio announcements with Datasette—from prototype to production](https://simonwillison.net/2022/Aug/21/scotrail/)** talks through the full process of creating an initial rapid prototype using Datasette Lite and growing that into a full production Datasette instance.
  - **[Fast Autocomplete Search for Your Website](https://simonwillison.net/2018/Dec/19/fast-autocomplete-search/)** shows how to build a full autocomplete search engine for your website, powered by Datasette and a few dozen lines of custom JavaScript.

html:
  <p>Want to build a prototype? Datasette lets you turn any source of data into a powerful JSON or <a href="https://github.com/simonw/datasette-graphql">GraphQL</a> API, ideal for use as the backend for an interactive prototype.</p>
  <p>Use Datasette with <strong><a href="https://observablehq.com/">Observable notebooks</a></strong> to quickly try out new visualizations and explore the Datasette API using JavaScript.</p>
  <p>Datasette works as a great data for <strong><a href="https://jupyter.org/">Jupyter notebooks</a></strong> too.</p>
  <ul>
  <li><strong><a href="https://simonwillison.net/2022/Aug/21/scotrail/">Analyzing ScotRail audio announcements with Datasette—from prototype to production</a></strong> talks through the full process of creating an initial rapid prototype using Datasette Lite and growing that into a full production Datasette instance.</li>
  <li><strong><a href="https://simonwillison.net/2018/Dec/19/fast-autocomplete-search/">Fast Autocomplete Search for Your Website</a></strong> shows how to build a full autocomplete search engine for your website, powered by Datasette and a few dozen lines of custom JavaScript.</li>
  </ul>

summary:
  Spin up a JSON (or GraphQL) API for any data in minutes. Use it to prototype and prove your ideas without building a custom backend.


_id: 863b689b3625af13bbba9bb9077c997fe3a04d13
_path: for/search.md
title: Datasette for search
text:
  Datasette is built on top of [SQLite](https://sqlite.org/), which includes a robust, full-featured [full-text search](https://www.sqlite.org/fts5.html) implementation.

  Datasette automatically detects tables which have been configured for full-text search and adds a search box which can be used by human site users or accessed from the JSON or CSV APIs.

  You can try this out now by [searching FARA records](https://fara.datasettes.com/) or [global power plants](https://datasette.io/global-power-plants/global-power-plants).

  ## Enabling search for a table

  The [sqlite-utils command-line tool](https://sqlite-utils.datasette.io/en/stable/cli.html#configuring-full-text-search) can be used to enable full-text search on a specific set of columns for a table:

      % sqlite-utils enable-fts mydb.db documents title summary

  Alternatively, you can configure search directly within the Datasette interface by installing the [datasette-configure-fts](https://github.com/simonw/datasette-configure-fts) plugin.

  [Fast Autocomplete Search for Your Website](https://simonwillison.net/2018/Dec/19/fast-autocomplete-search/) is a tutorial that shows how to write a scraper that populates a full-text search index in Datasette, then build a JavaScript autocomplete search interface using the Datasette JSON API.

  ## Implementing search with a canned query

  Datasette [canned queries](https://docs.datasette.io/en/stable/sql_queries.html#canned-queries) can be used to configure a SQL query which returns search results for a table, ordered by relevance.

  The search on [www.niche-museums.com](https://www.niche-museums.com/) uses this approach. Here's an example search, for `bones`:

  - [www.niche-museums.com/browse/search?q=bones](https://www.niche-museums.com/browse/search?q=bones)

  This uses the `search` canned query, which is [defined here]([https://github.com/simonw/til/blob/8f961be162868c53b5c484272091bdab703a747a/metadata.yaml#L16-L32](https://github.com/simonw/museums/blob/74e999c0e82781302bf0346a761ee5d88e168863/metadata.yaml#L55-L69)) and looks like this:

      select
        museums_fts.rank,
        museums.*
      from
        museums
        join museums_fts on museums.id = museums_fts.rowid
      where
        museums_fts match case
          :q
          when '' then '*'
          else escape_fts_query(:q)
        end
      order by
        museums_fts.rank

  Try [that query here](https://www.niche-museums.com/browse?sql=select%0D%0A++museums_fts.rank%2C%0D%0A++museums.*%0D%0Afrom%0D%0A++museums%0D%0A++join+museums_fts+on+museums.id+%3D+museums_fts.rowid%0D%0Awhere%0D%0A++museums_fts+match+case%0D%0A++++%3Aq%0D%0A++++when+%27%27+then+%27*%27%0D%0A++++else+escape_fts_query%28%3Aq%29%0D%0A++end%0D%0Aorder+by%0D%0A++museums_fts.rank&q=bones).

  The results are then rendered by [this custom template](https://github.com/simonw/museums/blob/74e999c0e82781302bf0346a761ee5d88e168863/templates/query-browse-search.html).

  ## Plugins for search

  The **[datasette-search-all](https://datasette.io/plugins/datasette-search-all)** plugin adds a search box which runs searches in parallel against all of the FTS-configured tables in all of the databases attached to Datasette.

  You can see that in action here: [fara.datasettes.com/-/search?q=manafort](https://fara.datasettes.com/-/search?q=manafort)

  **[dogsheep-beta](https://datasette.io/plugins/dogsheep-beta)** is a plugin which adds a customizable search engine that can create an index to search multiple tables at once with combined relevance scoring. It is used for the search on this site, for example [datasette.io/-/beta?q=fts](https://datasette.io/-/beta?q=fts) - you can read more about how it works in [Building a search engine for datasette.io](https://simonwillison.net/2020/Dec/19/dogsheep-beta/).

html:
  <p>Datasette is built on top of <a href="https://sqlite.org/">SQLite</a>, which includes a robust, full-featured <a href="https://www.sqlite.org/fts5.html">full-text search</a> implementation.</p>
  <p>Datasette automatically detects tables which have been configured for full-text search and adds a search box which can be used by human site users or accessed from the JSON or CSV APIs.</p>
  <p>You can try this out now by <a href="https://fara.datasettes.com/">searching FARA records</a> or <a href="https://datasette.io/global-power-plants/global-power-plants">global power plants</a>.</p>
  <h2>Enabling search for a table</h2>
  <p>The <a href="https://sqlite-utils.datasette.io/en/stable/cli.html#configuring-full-text-search">sqlite-utils command-line tool</a> can be used to enable full-text search on a specific set of columns for a table:</p>
  <pre><code>% sqlite-utils enable-fts mydb.db documents title summary
  </code></pre>
  <p>Alternatively, you can configure search directly within the Datasette interface by installing the <a href="https://github.com/simonw/datasette-configure-fts">datasette-configure-fts</a> plugin.</p>
  <p><a href="https://simonwillison.net/2018/Dec/19/fast-autocomplete-search/">Fast Autocomplete Search for Your Website</a> is a tutorial that shows how to write a scraper that populates a full-text search index in Datasette, then build a JavaScript autocomplete search interface using the Datasette JSON API.</p>
  <h2>Implementing search with a canned query</h2>
  <p>Datasette <a href="https://docs.datasette.io/en/stable/sql_queries.html#canned-queries">canned queries</a> can be used to configure a SQL query which returns search results for a table, ordered by relevance.</p>
  <p>The search on <a href="https://www.niche-museums.com/">www.niche-museums.com</a> uses this approach. Here's an example search, for <code>bones</code>:</p>
  <ul>
  <li><a href="https://www.niche-museums.com/browse/search?q=bones">www.niche-museums.com/browse/search?q=bones</a></li>
  </ul>
  <p>This uses the <code>search</code> canned query, which is <a href="[https://github.com/simonw/til/blob/8f961be162868c53b5c484272091bdab703a747a/metadata.yaml#L16-L32](https://github.com/simonw/museums/blob/74e999c0e82781302bf0346a761ee5d88e168863/metadata.yaml#L55-L69)">defined here</a> and looks like this:</p>
  <pre><code>select
    museums_fts.rank,
    museums.*
  from
    museums
    join museums_fts on museums.id = museums_fts.rowid
  where
    museums_fts match case
      :q
      when '' then '*'
      else escape_fts_query(:q)
    end
  order by
    museums_fts.rank
  </code></pre>
  <p>Try <a href="https://www.niche-museums.com/browse?sql=select%0D%0A++museums_fts.rank%2C%0D%0A++museums.*%0D%0Afrom%0D%0A++museums%0D%0A++join+museums_fts+on+museums.id+%3D+museums_fts.rowid%0D%0Awhere%0D%0A++museums_fts+match+case%0D%0A++++%3Aq%0D%0A++++when+%27%27+then+%27*%27%0D%0A++++else+escape_fts_query%28%3Aq%29%0D%0A++end%0D%0Aorder+by%0D%0A++museums_fts.rank&amp;q=bones">that query here</a>.</p>
  <p>The results are then rendered by <a href="https://github.com/simonw/museums/blob/74e999c0e82781302bf0346a761ee5d88e168863/templates/query-browse-search.html">this custom template</a>.</p>
  <h2>Plugins for search</h2>
  <p>The <strong><a href="https://datasette.io/plugins/datasette-search-all">datasette-search-all</a></strong> plugin adds a search box which runs searches in parallel against all of the FTS-configured tables in all of the databases attached to Datasette.</p>
  <p>You can see that in action here: <a href="https://fara.datasettes.com/-/search?q=manafort">fara.datasettes.com/-/search?q=manafort</a></p>
  <p><strong><a href="https://datasette.io/plugins/dogsheep-beta">dogsheep-beta</a></strong> is a plugin which adds a customizable search engine that can create an index to search multiple tables at once with combined relevance scoring. It is used for the search on this site, for example <a href="https://datasette.io/-/beta?q=fts">datasette.io/-/beta?q=fts</a> - you can read more about how it works in <a href="https://simonwillison.net/2020/Dec/19/dogsheep-beta/">Building a search engine for datasette.io</a>.</p>

summary:
  Datasette uses SQLite's built-in full-text search feature to provide faceted search over any correctly configured collection of records.


_id: 2a478757dc7f379d19b6c77d1710024b8f02a1e1
_path: for/serverless.md
title: Serverless read-only databases with Datasette
text:
  Serverless providers such as [Google Cloud Run](https://cloud.google.com/run) and [Vercel](https://vercel.com/) provide robust, scalable, inexpensive hosting for stateless web applications.

  Databases are usually provided as separate add-ons to serverless hosting. If your data is static or changes infrequently, Datasette can be used to unlock the benefits of serverless hosting without needing a separate hosted database.

  Datasette can be used to implement the [<strong>Baked Data</strong>](https://simonwillison.net/2021/Jul/28/baked-data/) pattern, where any structured data needed by your application is bundled and deployed with the application code itself, as part of the same container.

  This site is an example of Baked Data in action: datasette.io is deployed to Google Cloud Run by build scripts running [on GitHub Actions](https://github.com/simonw/datasette.io/blob/main/.github/workflows/deploy.yml).

  Many of the [Datasette examples](/examples) listed on this site are also deployed in this manner.

html:
  <p>Serverless providers such as <a href="https://cloud.google.com/run">Google Cloud Run</a> and <a href="https://vercel.com/">Vercel</a> provide robust, scalable, inexpensive hosting for stateless web applications.</p>
  <p>Databases are usually provided as separate add-ons to serverless hosting. If your data is static or changes infrequently, Datasette can be used to unlock the benefits of serverless hosting without needing a separate hosted database.</p>
  <p>Datasette can be used to implement the <a href="https://simonwillison.net/2021/Jul/28/baked-data/"><strong>Baked Data</strong></a> pattern, where any structured data needed by your application is bundled and deployed with the application code itself, as part of the same container.</p>
  <p>This site is an example of Baked Data in action: datasette.io is deployed to Google Cloud Run by build scripts running <a href="https://github.com/simonw/datasette.io/blob/main/.github/workflows/deploy.yml">on GitHub Actions</a>.</p>
  <p>Many of the <a href="/examples">Datasette examples</a> listed on this site are also deployed in this manner.</p>

summary:
  Serverless hosting usually depends on external hosted databases, but Datasette lets you implement the Baked Data pattern to bundle your data with your application code and deploy it to serverless providers.


_id: 41624e65228293a1622b9eee3a435699801dead1
_path: for/websites.md
title: Datasette for websites
text:
  Datasette templates can be used to build custom websites with Datasette under the hood. Examples include the site you are browsing right now.

  ## [datasette.io](https://datasette.io/)

  The official Datasette website runs on top of a customized Datasette instance.

  - [datasette.io, an official project website for Datasette](https://simonwillison.net/2020/Dec/13/datasette-io/) talks through the basics of how the site works.
  - [Building a search engine for datasette.io](https://simonwillison.net/2020/Dec/19/dogsheep-beta/) explains how the [site search](https://datasette.io/-/beta) works, using the [dogsheep-beta](https://datasette.io/plugins/dogsheep-beta) search plugin.
  - [The Baked Data architectural pattern](https://simonwillison.net/2021/Jul/28/baked-data/) describes the Baked Data architecture pattern used by the site, and provides a detailed description of how the build scripts for the site work.
  - [github.com/simonw/datasette.io](https://github.com/simonw/datasette.io) has the full source code for the site, including the GitHub Actions workflows that build the databases and deploy them to Google Cloud Run.
  - [datasette.io/content](https://datasette.io/content) is the Datasette interface for the main database that powers the site.

  ## [til.simonwillison.net](https://til.simonwillison.net/)

  A blog sharing TILs - Today I Learned snippets.

  - [Using a self-rewriting README powered by GitHub Actions to track TILs](https://simonwillison.net/2020/Apr/20/self-rewriting-readme/) introduces the site.
  - The site uses the [datasette-atom](https://datasette.io/plugins/datasette-atom) plugin to provide an Atom feed of new entries.
  - [til.simonwillison.net/tils](https://til.simonwillison.net/tils) exposes the underlying database.
  - The site is deployed to [Vercel](https://vercel.com/) using the [datasette-publish-vercel](https://datasette.io/plugins/datasette-publish-vercel) plugin.

  ## [Niche Museums](https://www.niche-museums.com/)

  [www.niche-museums.com](https://www.niche-museums.com/) is a directory of small and niche museums, powered by Datasette. Read [niche-museums.com, powered by Datasette](https://simonwillison.net/2019/Nov/25/niche-museums/) for details, or browse [the code](https://github.com/simonw/museums) on GitHub.

  ## [Rocky Beaches](https://www.rockybeaches.com/us/pillar-point)

  [Rocky Beaches](https://www.rockybeaches.com/us/pillar-point) by Natalie Downe shows the best times to go tidepooling at Pillar Point near Half Moon Bay, California. It runs on Datasette, using SQL queries to suggest the best times to go based on low tide times and sunrise/sunset and display recent observations recorded on [iNaturalist](https://www.inaturalist.org/).

  The source code is available at [github.com/natbat/rockybeaches](https://github.com/natbat/rockybeaches). The site uses the [datasette-graphql](https://datasette.io/plugins/datasette-graphql) in some of the templates, for example [here](https://github.com/natbat/rockybeaches/blob/e072d48479f1404baa206ee660ad228d9f95ac7a/templates/row-data-places.html#L273-L290).

html:
  <p>Datasette templates can be used to build custom websites with Datasette under the hood. Examples include the site you are browsing right now.</p>
  <h2><a href="https://datasette.io/">datasette.io</a></h2>
  <p>The official Datasette website runs on top of a customized Datasette instance.</p>
  <ul>
  <li><a href="https://simonwillison.net/2020/Dec/13/datasette-io/">datasette.io, an official project website for Datasette</a> talks through the basics of how the site works.</li>
  <li><a href="https://simonwillison.net/2020/Dec/19/dogsheep-beta/">Building a search engine for datasette.io</a> explains how the <a href="https://datasette.io/-/beta">site search</a> works, using the <a href="https://datasette.io/plugins/dogsheep-beta">dogsheep-beta</a> search plugin.</li>
  <li><a href="https://simonwillison.net/2021/Jul/28/baked-data/">The Baked Data architectural pattern</a> describes the Baked Data architecture pattern used by the site, and provides a detailed description of how the build scripts for the site work.</li>
  <li><a href="https://github.com/simonw/datasette.io">github.com/simonw/datasette.io</a> has the full source code for the site, including the GitHub Actions workflows that build the databases and deploy them to Google Cloud Run.</li>
  <li><a href="https://datasette.io/content">datasette.io/content</a> is the Datasette interface for the main database that powers the site.</li>
  </ul>
  <h2><a href="https://til.simonwillison.net/">til.simonwillison.net</a></h2>
  <p>A blog sharing TILs - Today I Learned snippets.</p>
  <ul>
  <li><a href="https://simonwillison.net/2020/Apr/20/self-rewriting-readme/">Using a self-rewriting README powered by GitHub Actions to track TILs</a> introduces the site.</li>
  <li>The site uses the <a href="https://datasette.io/plugins/datasette-atom">datasette-atom</a> plugin to provide an Atom feed of new entries.</li>
  <li><a href="https://til.simonwillison.net/tils">til.simonwillison.net/tils</a> exposes the underlying database.</li>
  <li>The site is deployed to <a href="https://vercel.com/">Vercel</a> using the <a href="https://datasette.io/plugins/datasette-publish-vercel">datasette-publish-vercel</a> plugin.</li>
  </ul>
  <h2><a href="https://www.niche-museums.com/">Niche Museums</a></h2>
  <p><a href="https://www.niche-museums.com/">www.niche-museums.com</a> is a directory of small and niche museums, powered by Datasette. Read <a href="https://simonwillison.net/2019/Nov/25/niche-museums/">niche-museums.com, powered by Datasette</a> for details, or browse <a href="https://github.com/simonw/museums">the code</a> on GitHub.</p>
  <h2><a href="https://www.rockybeaches.com/us/pillar-point">Rocky Beaches</a></h2>
  <p><a href="https://www.rockybeaches.com/us/pillar-point">Rocky Beaches</a> by Natalie Downe shows the best times to go tidepooling at Pillar Point near Half Moon Bay, California. It runs on Datasette, using SQL queries to suggest the best times to go based on low tide times and sunrise/sunset and display recent observations recorded on <a href="https://www.inaturalist.org/">iNaturalist</a>.</p>
  <p>The source code is available at <a href="https://github.com/natbat/rockybeaches">github.com/natbat/rockybeaches</a>. The site uses the <a href="https://datasette.io/plugins/datasette-graphql">datasette-graphql</a> in some of the templates, for example <a href="https://github.com/natbat/rockybeaches/blob/e072d48479f1404baa206ee660ad228d9f95ac7a/templates/row-data-places.html#L273-L290">here</a>.</p>

summary:
  Datasette can be used to power dynamic-static websites: sites that run on serverless hosting while providing dynamic data-backed functionality.


# datasette_repos

id: R_kgDOI1rqCA
nameWithOwner: CentreForDigitalHumanities/datasette-matomo
createdAt: 2023-01-25T11:35:22Z
topics: ["datasette-plugin", "matomo", "web-analytics", "datasette"]
latest_commit: 6905164372d4514c9f7c277dcc0553c0aba58771
openGraphImageUrl:
  https://opengraph.githubassets.com/1267b8674ad8bff47a872457e0fe9dfd93c26987a7adcea0cbfc71a61b8f80f4/CentreForDigitalHumanities/datasette-matomo


id: R_kgDOIu6Scg
nameWithOwner: asg017/sqlite-fastrand
createdAt: 2023-01-06T20:46:06Z
topics: ["sqlite", "sqlite-extension", "datasette-plugin"]
latest_commit: 2fdf7ed5117bad3385d3b3c643bb0b36336cf12e
openGraphImageUrl:
  https://opengraph.githubassets.com/157b4bbe735fa1f3455b2c33dbc7cb0d482f032c9b3fab9707dfa67e2eabd752/asg017/sqlite-fastrand


id: R_kgDOGM0DcA
nameWithOwner: asg017/sqlite-html
createdAt: 2021-10-11T21:15:19Z
topics: ["sqlite", "sqlite-extension"]
latest_commit: d3b1d77f2a0c1383349babed26fb8a196beab445
openGraphImageUrl:
  https://opengraph.githubassets.com/0cba6bb56018cff9cbddad63ba9167bbe7b8c1d87653b88cae7937ae5573b0cb/asg017/sqlite-html


id: R_kgDOGM0Ptw
nameWithOwner: asg017/sqlite-http
createdAt: 2021-10-11T21:27:49Z
topics: ["sqlite", "sqlite-extension"]
latest_commit: cffd28c93fbb2e29b9040d3da4c13e3f404eacad
openGraphImageUrl:
  https://opengraph.githubassets.com/afe2ab9ca1fbebbb274ddac6c94f455828ce974c288f2b08350af8e5043c13cc/asg017/sqlite-http


id: R_kgDOIX3aMg
nameWithOwner: asg017/sqlite-jsonschema
createdAt: 2022-11-04T18:34:59Z
topics: ["sqlite", "sqlite-extension"]
latest_commit: fadd55ddc28537f5f8769205e2c0798dfb854b79
openGraphImageUrl:
  https://opengraph.githubassets.com/788160db8264db8d1b5b23426ffa7ac7849cdd37de7c32ac51e337be1c3c0a9a/asg017/sqlite-jsonschema


id: R_kgDOHPAsug
nameWithOwner: asg017/sqlite-lines
createdAt: 2022-04-25T19:11:46Z
topics: ["sqlite", "sqlite-extension"]
latest_commit: 7d21410a22e106e912563022803ee93922b9b916
openGraphImageUrl:
  https://opengraph.githubassets.com/c3e3206b1ffca95b1faff655fe8963e4e4c635330ce4c62741b6d2585a47c420/asg017/sqlite-lines


id: R_kgDOHwJJsQ
nameWithOwner: asg017/sqlite-path
createdAt: 2022-08-01T19:49:33Z
topics: ["sqlite", "sqlite-extension"]
latest_commit: 1f8d452aa0c6a8920863f3a8e1686c7667306ac7
openGraphImageUrl:
  https://opengraph.githubassets.com/93a9c18ed73d58467d980822093d1f4b5e3c8ddec32441e016505e18e2b4d473/asg017/sqlite-path


id: R_kgDOIX3UzQ
nameWithOwner: asg017/sqlite-regex
createdAt: 2022-11-04T18:31:01Z
topics: ["sqlite", "sqlite-extension"]
latest_commit: 5fe28e3c7a978c621082cfef0ae6dea9a71167ee
openGraphImageUrl:
  https://opengraph.githubassets.com/e37d1e895c437d6c009013a59c058bfd1458867d5a43288fac8587de376b0d9c/asg017/sqlite-regex


id: R_kgDOIpo-2g
nameWithOwner: asg017/sqlite-ulid
createdAt: 2022-12-20T19:58:44Z
topics: ["sqlite", "sqlite-extension"]
latest_commit: 942d62ca1ebc483df3cc8d31af71cfff4778ed7f
openGraphImageUrl:
  https://opengraph.githubassets.com/8103b6c6c347b2c77bd83d50e2dc565c8350ed3e8e975b37f94778a0b08dde80/asg017/sqlite-ulid


id: R_kgDOHwJOSQ
nameWithOwner: asg017/sqlite-url
createdAt: 2022-08-01T19:54:03Z
topics: ["sqlite", "sqlite-extension"]
latest_commit: 19cf842b1a5f44a9c23ad0d396d167f004c6eb7f
openGraphImageUrl:
  https://opengraph.githubassets.com/c78b815b0dcc2aec200ba9c3c8b5f6dd724fa5c407cc74deb7900ae38f3b4bbe/asg017/sqlite-url


id: R_kgDOIt2TLQ
nameWithOwner: asg017/sqlite-vss
createdAt: 2023-01-03T23:27:31Z
topics: ["sqlite", "sqlite-extension", "vector-search", "faiss"]
latest_commit: 8d3c6ff2fca651104563779a0d927ea4fba40347
openGraphImageUrl:
  https://opengraph.githubassets.com/ef65fd2decf618241a4aa2119f37a6c208f533b7fb6088097459ff3067d53471/asg017/sqlite-vss


id: MDEwOlJlcG9zaXRvcnkzMzAzMDQ2Mjg=
nameWithOwner: bretwalker/datasette-query-history
createdAt: 2021-01-17T03:13:34Z
topics: []
latest_commit: c4d4cceb2a0b6f69dd96db1d7c1bf97194ad841a
openGraphImageUrl:
  https://opengraph.githubassets.com/8cebbec8fe37e7d40a5a8439b4c17d9ce36933a32a9055a4a4c550d8b51c23bc/bretwalker/datasette-query-history


id: R_kgDOIhVCuw
nameWithOwner: cldellow/datasette-scraper
createdAt: 2022-11-29T00:19:56Z
topics: ["datasette", "datasette-plugin", "scraping"]
latest_commit: 25af45b9fd204e1068d82f5c04b4a14b9f4cbd5a
openGraphImageUrl:
  https://opengraph.githubassets.com/203f88823cbeae5095883661833d3173ccca38233d0d445d398c7803648b8c4e/cldellow/datasette-scraper


id: R_kgDOMkfZPg
nameWithOwner: datasette/datasette-checkbox
createdAt: 2024-08-16T20:08:22Z
topics: []
latest_commit: 6c20c317fb34da35405b7e01f668b62d9a42c6ec
openGraphImageUrl:
  https://opengraph.githubassets.com/42cdfcb7898b543d7a71deb52649d60fc8500470627ac3254dabf72c1e630886/datasette/datasette-checkbox


id: R_kgDOK2ToCw
nameWithOwner: datasette/datasette-chronicle
createdAt: 2023-12-06T04:46:49Z
topics: []
latest_commit: f78b761db8dd95aae1d4d9320e703bbc9cbc0c9b
openGraphImageUrl:
  https://opengraph.githubassets.com/2be591cf99085851528ac26c882e7205d6e405f134155e96fa38065c48b28562/datasette/datasette-chronicle


id: R_kgDOLZ-KRA
nameWithOwner: datasette/datasette-column-sum
createdAt: 2024-02-29T22:44:02Z
topics: []
latest_commit: c3cdb7697b0ab4278b3a977a5a9f0ce7c8444dc9
openGraphImageUrl:
  https://opengraph.githubassets.com/1c6ed25357c14f3795bae23518576b94dcea94c3a3ce3d509d6107e773c3297d/datasette/datasette-column-sum


id: R_kgDOLc0adA
nameWithOwner: datasette/datasette-create-view
createdAt: 2024-03-07T03:22:29Z
topics: []
latest_commit: ba3c3fe21eccdcb000df6f298a2ef7e080e39171
openGraphImageUrl:
  https://opengraph.githubassets.com/c14bf30e71014828373c9c548ef3680fe9f88f47f89a46de97e642b6df8cdb7d/datasette/datasette-create-view


id: R_kgDOLk9FxA
nameWithOwner: datasette/datasette-embeddings
createdAt: 2024-03-24T21:45:24Z
topics: []
latest_commit: 8dc282b3a5a78f0598a64dddef7dfdede45398b4
openGraphImageUrl:
  https://opengraph.githubassets.com/b2b5ba1010e62d621cbf59cc1ef315c49afb4128898a5176cd75ada5caba698e/datasette/datasette-embeddings


id: R_kgDOG0Ywag
nameWithOwner: datasette/datasette-enrichments
createdAt: 2022-02-10T01:13:30Z
topics: ["datasette", "datasette-plugin"]
latest_commit: a52106b18ad235f56e5e9bbf8e93bd924e6f386c
openGraphImageUrl:
  https://opengraph.githubassets.com/4626a1ddc28c58f84191b76ca5d49c0952784a66c9c8dfc5e1f33038a2025581/datasette/datasette-enrichments


id: R_kgDOKugxLQ
nameWithOwner: datasette/datasette-enrichments-gpt
createdAt: 2023-11-17T03:41:59Z
topics: []
latest_commit: 1bede50ad21d7ae9fa663fd2a199bcd46b52cf6e
openGraphImageUrl:
  https://opengraph.githubassets.com/541acd7dcc515af834e390441127052902b424bb09dc2601a26ef666100a88f3/datasette/datasette-enrichments-gpt


id: R_kgDOKuZiqg
nameWithOwner: datasette/datasette-enrichments-jinja
createdAt: 2023-11-16T20:03:52Z
topics: []
latest_commit: 9ebfde7be09dbcc22f87662294bbf9dae7e81b6d
openGraphImageUrl:
  https://opengraph.githubassets.com/5cbdb34d98c1c8698e57009ee09e0dcaf091b9b85c861ba56e597d0696ebad23/datasette/datasette-enrichments-jinja


id: R_kgDOKx8XZg
nameWithOwner: datasette/datasette-enrichments-opencage
createdAt: 2023-11-25T18:21:47Z
topics: []
latest_commit: d93f178bdc6ca2ee18638700428839c818a4ae74
openGraphImageUrl:
  https://opengraph.githubassets.com/f92bacace57b4b7a5acd23b7947f2b9c52d5d9effac717a7e1c6bed99bc5fe7f/datasette/datasette-enrichments-opencage


id: R_kgDOLQOkPw
nameWithOwner: datasette/datasette-enrichments-quickjs
createdAt: 2024-02-09T16:58:21Z
topics: []
latest_commit: 0301b78e8562ef41e8ad886a22287038b26058f5
openGraphImageUrl:
  https://opengraph.githubassets.com/f4c38a71055587923a13aadaa520c7a195465dbc0f6c7073803057c6444ce1a2/datasette/datasette-enrichments-quickjs


id: R_kgDOKyy0cA
nameWithOwner: datasette/datasette-enrichments-re2
createdAt: 2023-11-27T22:39:11Z
topics: []
latest_commit: 2d1e767b6024d78a558cb88c6e39af135d1f366a
openGraphImageUrl:
  https://opengraph.githubassets.com/945311a33fad535eeac9eb9a17e3b064d95970d81b2e2835d75230f070299748/datasette/datasette-enrichments-re2


id: R_kgDOLMquvQ
nameWithOwner: datasette/datasette-events-db
createdAt: 2024-02-01T17:36:58Z
topics: []
latest_commit: 6378e4b80ef87b96bcc97a66e59497241b3973a4
openGraphImageUrl:
  https://opengraph.githubassets.com/2271e7cf25836aab370bdb1988ef8c4a67bd33828d76cd7aa0a633b80e03810e/datasette/datasette-events-db


id: R_kgDOLU-JCQ
nameWithOwner: datasette/datasette-events-forward
createdAt: 2024-02-20T00:13:09Z
topics: ["datasette-plugin"]
latest_commit: 887199fa53b943092a47995df2f1d3a38df909b5
openGraphImageUrl:
  https://opengraph.githubassets.com/01ea7a64c2b68ac8ed6b2151ec464666317f0651b25679f22904757002431c1e/datasette/datasette-events-forward


id: R_kgDOLd-PcQ
nameWithOwner: datasette/datasette-export-database
createdAt: 2024-03-09T15:53:50Z
topics: []
latest_commit: 1a84effc2c29b4539364f0816852feb127ed0a93
openGraphImageUrl:
  https://opengraph.githubassets.com/9ac6b0917533fbe0e3c837abb583fa7e3c9c459fca1051b1e9eaf13e59c7cd50/datasette/datasette-export-database


id: R_kgDOKHgxRA
nameWithOwner: datasette/datasette-extract
createdAt: 2023-08-15T19:57:04Z
topics: []
latest_commit: 4456f54f0cebf6aee9d0d5f7ee06f5988b34402e
openGraphImageUrl:
  https://opengraph.githubassets.com/59e98743dee28e2c4e62113c0b59f8b214cb44fd1968193b29c831a5e25f6e67/datasette/datasette-extract


id: R_kgDOLL5HMg
nameWithOwner: datasette/datasette-homepage-table
createdAt: 2024-01-31T04:31:00Z
topics: []
latest_commit: b99a70780cb3483032357a9d5a8069b846e46fea
openGraphImageUrl:
  https://opengraph.githubassets.com/464e970c77b09519952527c7becd1d0e930fedb99ca8fd17511d210eb76016ba/datasette/datasette-homepage-table


id: R_kgDOLqtkrQ
nameWithOwner: datasette/datasette-import
createdAt: 2024-04-06T15:57:38Z
topics: []
latest_commit: 5178a7a57ad80b263c6d92ca5e010271ceae1e10
openGraphImageUrl:
  https://opengraph.githubassets.com/40dca7aef1a7caf5164c699b521a461e3298bf46a5119eb3d01cb91b39f05e11/datasette/datasette-import


id: R_kgDOLIwk4A
nameWithOwner: datasette/datasette-proxy-url
createdAt: 2024-01-23T20:19:30Z
topics: []
latest_commit: 6aa8c581f82a6c58622af330cfbb4ec518e8151e
openGraphImageUrl:
  https://opengraph.githubassets.com/ff29026463e40dea9befb9571591156dd9a0aac3d4d45bb8cd5bc0446f064ec5/datasette/datasette-proxy-url


id: R_kgDOLl8JfA
nameWithOwner: datasette/datasette-query-assistant
createdAt: 2024-03-26T21:23:42Z
topics: []
latest_commit: d5a4f46ffaf2e5d272101a6ae65ca4582bda173e
openGraphImageUrl:
  https://opengraph.githubassets.com/555fe7ed15a40e97533ed3d471c6c12ac5962d257efa167fa2b36a4bc25d7d8f/datasette/datasette-query-assistant


id: R_kgDOLxu_vQ
nameWithOwner: datasette/datasette-secrets
createdAt: 2024-04-22T18:03:04Z
topics: []
latest_commit: 7b0bfd6fb42aca3a73400242cd2d181388798fb5
openGraphImageUrl:
  https://opengraph.githubassets.com/93648a78cc38c8aead6b378d9b99f6d71c89f957871a299ebe19aac52cd90714/datasette/datasette-secrets


id: R_kgDOKWH3yw
nameWithOwner: datasette/datasette-sqlite-authorizer
createdAt: 2023-09-20T17:35:23Z
topics: []
latest_commit: f223e450a23d44e99ecf20b1c93bd98e29f6eec3
openGraphImageUrl:
  https://opengraph.githubassets.com/7e2a8b4553e5fde46a2234bcc3be1f5fb747c5ca45eabbc97649f19147461a7b/datasette/datasette-sqlite-authorizer


id: R_kgDOKVzq2g
nameWithOwner: datasette/datasette-sqlite-debug-authorizer
createdAt: 2023-09-20T03:30:17Z
topics: []
latest_commit: 2c37cd662f0828059d112b155e31e61ef3d17a9e
openGraphImageUrl:
  https://opengraph.githubassets.com/64e417fa249fddbe376254ea4ba9a5ef422c95c96463fbd3cfbd4732155e101a/datasette/datasette-sqlite-debug-authorizer


id: R_kgDOLaT5Gw
nameWithOwner: datasette/datasette-tail
createdAt: 2024-03-01T16:03:13Z
topics: []
latest_commit: 04d457bf206042848621a72a192041a301094420
openGraphImageUrl:
  https://opengraph.githubassets.com/79766587eb7c44d40cb4fcc3ead75cc348af1984285992845759dfc80bbf4bdb/datasette/datasette-tail


id: R_kgDOLUUhXg
nameWithOwner: datasette/datasette-unsafe-actor-debug
createdAt: 2024-02-18T19:15:02Z
topics: ["datasette-plugin"]
latest_commit: 0585ca559a3486c396c594a55ae1d9cdb98ed944
openGraphImageUrl:
  https://opengraph.githubassets.com/3ec508449bc1479fb61fffe3e7f87b5c442887e4d6470ecd7fd906a97a36ac19/datasette/datasette-unsafe-actor-debug


id: R_kgDOLcU1eA
nameWithOwner: datasette/datasette-uptime
createdAt: 2024-03-06T05:10:08Z
topics: []
latest_commit: 2e651d3850fb8ddc28649cf7d453481d744f2fad
openGraphImageUrl:
  https://opengraph.githubassets.com/67429b6bc7637021141e51b31bb813267ef5eb2925e56a320ca13d33a87dca1c/datasette/datasette-uptime


id: R_kgDOKNibWA
nameWithOwner: datasette/datasette-visible-internal-db
createdAt: 2023-08-30T22:38:55Z
topics: []
latest_commit: fb9c8aaa670be27128e52037aa48c70c16acf610
openGraphImageUrl:
  https://opengraph.githubassets.com/ab703c32e3b0ca90e4d7d978ba0eb3133fa507ce1efb2d7eef40a1fa4d22a78b/datasette/datasette-visible-internal-db


id: R_kgDOJ6yOgw
nameWithOwner: datasette/datasette-write-ui
createdAt: 2023-07-12T15:51:50Z
topics: []
latest_commit: 1fab67ab46f87ed54091764940a45f062fe5b281
openGraphImageUrl:
  https://opengraph.githubassets.com/86062a6a12b620d3673ec43321260106c960af2e7a1d731fce1c4d98be123fd5/datasette/datasette-write-ui


id: R_kgDOJHON9g
nameWithOwner: dogsheep/apple-notes-to-sqlite
createdAt: 2023-03-09T03:44:10Z
topics: ["apple-notes", "datasette", "datasette-tool", "sqlite", "dogsheep"]
latest_commit: e55a802d37a896475b6cf475c1ba947af63cca73
openGraphImageUrl:
  https://opengraph.githubassets.com/7ee9c1732b6de4ed03d2061ebdf384a4bea26dba3940d5b6cfcd4a5b64f0efd8/dogsheep/apple-notes-to-sqlite


id: MDEwOlJlcG9zaXRvcnkxOTc0MzExMDk=
nameWithOwner: dogsheep/dogsheep-beta
createdAt: 2019-07-17T17:07:26Z
topics: ["dogsheep", "search", "datasette-plugin", "datasette-io", "datasette", "datasette-tool"]
latest_commit: 56fb14f634b332de4b957498a6c9a6290dc39e10
openGraphImageUrl:
  https://opengraph.githubassets.com/263005b65b2c11bd1a5230545a2e5f6a24b23cd068210a0083f661fce379cf4c/dogsheep/dogsheep-beta


id: MDEwOlJlcG9zaXRvcnkyNTY4MzQ5MDc=
nameWithOwner: dogsheep/dogsheep-photos
createdAt: 2020-04-18T19:22:13Z
topics: ["dogsheep", "datasette-io", "datasette", "sqlite", "datasette-tool"]
latest_commit: 325aa38cb23d0757bb1335ee2ea94a082475a66e
openGraphImageUrl:
  https://opengraph.githubassets.com/f7b9e488e52242eb4812cd3c6cfccff11d0efe795883ba2e5ff92e0857da3441/dogsheep/dogsheep-photos


id: MDEwOlJlcG9zaXRvcnkzMDMyMTgzNjk=
nameWithOwner: dogsheep/evernote-to-sqlite
createdAt: 2020-10-11T21:45:49Z
topics: ["evernote", "dogsheep", "datasette-io", "sqlite", "datasette-tool"]
latest_commit: fff89772b4404995400e33fe1d269050717ff4cf
openGraphImageUrl:
  https://opengraph.githubassets.com/7a954fd7e64b5dbc2fa5e0c6f302b3c29a9ffe97d2d06066d9d42795fe105837/dogsheep/evernote-to-sqlite


id: MDEwOlJlcG9zaXRvcnkyMDk1OTAzNDU=
nameWithOwner: dogsheep/genome-to-sqlite
createdAt: 2019-09-19T15:38:39Z
latest_commit: 6894c8bc767b4c0375ddb8b2f0d2fed857667ff5
openGraphImageUrl:
  https://opengraph.githubassets.com/ab5138a73a03fc2565c42b3dacf1ef9e97da2545c872c5d897d2399c5439e879/dogsheep/genome-to-sqlite

topics:
  ["sqlite", "datasette", "dogsheep", "personal-analytics", "genetics", "23andme", "datasette-io", "datasette-tool"]


id: MDEwOlJlcG9zaXRvcnkyMDY2NDk3NzA=
nameWithOwner: dogsheep/google-takeout-to-sqlite
createdAt: 2019-09-05T20:15:15Z
topics: ["google", "sqlite", "datasette", "dogsheep", "datasette-io", "datasette-tool"]
latest_commit: e54e544427f1cc3ea8189f0e95f54046301a8645
openGraphImageUrl:
  https://opengraph.githubassets.com/3ef7ea910df3cec1a827937608dc99de4d7a7dd79cd89cc6a5109ea77320543e/dogsheep/google-takeout-to-sqlite


id: MDEwOlJlcG9zaXRvcnkyNDg5MDM1NDQ=
nameWithOwner: dogsheep/hacker-news-to-sqlite
createdAt: 2020-03-21T04:02:05Z
topics: ["datasette-io", "hacker-news", "dogsheep", "datasette", "datasette-tool"]
latest_commit: c5585c103d124b23ba1e163f8857d4ba49fe452a
openGraphImageUrl:
  https://opengraph.githubassets.com/1cae05886ec99cdf08d8cda1b5362904de382614d623ffcc90b75a7266448fb0/dogsheep/hacker-news-to-sqlite


id: MDEwOlJlcG9zaXRvcnkxOTc4ODIzODI=
nameWithOwner: dogsheep/healthkit-to-sqlite
createdAt: 2019-07-20T05:03:12Z
topics: ["healthkit", "sqlite", "datasette", "dogsheep", "datasette-io", "datasette-tool"]
latest_commit: 9fe3cb17e03d6c73222b63e643638cf951567c4c
openGraphImageUrl:
  https://opengraph.githubassets.com/eabb973a22b2928f5077275d7cf2faf420520397d9405fec363b6699179bb8c5/dogsheep/healthkit-to-sqlite


id: MDEwOlJlcG9zaXRvcnkyMDYyMDI4NjQ=
nameWithOwner: dogsheep/inaturalist-to-sqlite
createdAt: 2019-09-04T01:21:21Z
topics: ["inaturalist", "sqlite", "datasette", "dogsheep", "datasette-io", "datasette-tool"]
latest_commit: d888c7c2f02aa0dfb1559603f02357cd0089da11
openGraphImageUrl:
  https://opengraph.githubassets.com/0a73aaea042af966a2964fdaba6b8d5e9915de5fa8b42a3b3a23151a17113fd8/dogsheep/inaturalist-to-sqlite


id: MDEwOlJlcG9zaXRvcnkyMTMyODY3NTI=
nameWithOwner: dogsheep/pocket-to-sqlite
createdAt: 2019-10-07T03:24:14Z
topics: ["sqlite", "pocket-api", "pocket", "dogsheep", "datasette", "datasette-io", "datasette-tool"]
latest_commit: b382b9330fa8819b7d34ee401c99a2b47e9b8c01
openGraphImageUrl:
  https://opengraph.githubassets.com/869ef0a972a5826c6b4ec64aa8cea4da1e8b662049b17637485a4e07726f32fb/dogsheep/pocket-to-sqlite


id: MDEwOlJlcG9zaXRvcnkyMDU0MjkzNzU=
nameWithOwner: dogsheep/swarm-to-sqlite
createdAt: 2019-08-30T17:37:29Z
latest_commit: e6c0f4b08964ecdd77ed5afc77c18c9641f34c24
openGraphImageUrl:
  https://opengraph.githubassets.com/ff0f42500231be725d330f7c82d244d535074884323dacf68b8e6c13f2839b06/dogsheep/swarm-to-sqlite

topics:
  ["foursquare", "sqlite", "foursquare-api", "datasette", "dogsheep", "datasette-io", "swarm", "datasette-tool"]


id: MDEwOlJlcG9zaXRvcnkyMDYxNTY4NjY=
nameWithOwner: dogsheep/twitter-to-sqlite
createdAt: 2019-09-03T19:30:08Z
topics: ["sqlite", "twitter", "twitter-api", "datasette", "dogsheep", "datasette-io", "datasette-tool"]
latest_commit: f09d611782a8372cfb002792dfa727325afb4db6
openGraphImageUrl:
  https://opengraph.githubassets.com/476302375205e362f53f317f42362fdfdc8979773a1ca6f4c69798f99003453d/dogsheep/twitter-to-sqlite


id: MDEwOlJlcG9zaXRvcnkzMDY5NzY1MzA=
nameWithOwner: drkane/datasette-reconcile
createdAt: 2020-10-24T21:41:06Z
topics: ["datasette-plugin", "datasette", "python", "reconciliation-service", "reconciliation-api"]
latest_commit: a904cf94c168da3e932b1e2a36cc07c64731e888
openGraphImageUrl:
  https://opengraph.githubassets.com/a62af6ac7881eda78f837f94ede6b4cfb5d770dd15b42e2e052a79d7ccc88081/drkane/datasette-reconcile


id: MDEwOlJlcG9zaXRvcnkzNzQ4NDYzMTE=
nameWithOwner: eyeseast/datasette-geojson
createdAt: 2021-06-08T01:33:19Z
topics: ["geojson", "gis", "sqlite", "datasette-io", "datasette-plugin"]
latest_commit: 6f66024665e9d2c2dcbfd02edeeda704704baf11
openGraphImageUrl:
  https://opengraph.githubassets.com/ea9736aa52db6fe442e5b343601bb6cde6f4943c23fe2618ee6c1a200cce9adb/eyeseast/datasette-geojson


id: MDEwOlJlcG9zaXRvcnkzODI5ODY1NjQ=
nameWithOwner: eyeseast/datasette-geojson-map
createdAt: 2021-07-05T01:54:13Z
topics: ["geojson", "gis", "datasette-plugin", "leafletjs", "datasette-io", "mapping"]
latest_commit: 88b615cfd06471924ab454ff0246143c7fe060d4
openGraphImageUrl:
  https://opengraph.githubassets.com/23944a3bc42bd813c7b9acc6dfde6b0e81f46e69737adc5149954415f6adbea7/eyeseast/datasette-geojson-map


id: R_kgDOHcwLAg
nameWithOwner: eyeseast/datasette-query-files
createdAt: 2022-06-04T18:52:07Z
topics: ["datasette", "datasette-plugin", "python", "sql", "sqlite"]
latest_commit: ea4cf949b260fe5700de63cea0eaf4cd3821d767
openGraphImageUrl:
  https://opengraph.githubassets.com/474dc81af4f26176865e54c998fdc08cd2a1843ca9be3e5e80e1e1b81f7f9010/eyeseast/datasette-query-files


id: MDEwOlJlcG9zaXRvcnkyOTMzNjE1MTQ=
nameWithOwner: eyeseast/geocode-sqlite
createdAt: 2020-09-06T21:05:39Z
topics: []
latest_commit: 9ff6bdbe21a209d3023afee6cb8d85bd80725301
openGraphImageUrl:
  https://opengraph.githubassets.com/2761f8c456d6d0e98b6648a01036e52692fe7e836c3c8db66adb0594c6738292/eyeseast/geocode-sqlite


id: R_kgDOG383_g
nameWithOwner: eyeseast/sqlite-colorbrewer
createdAt: 2022-02-19T21:53:46Z
topics: []
latest_commit: ce9f1e0dda489e1a032d3579d45639ea7f0b597d
openGraphImageUrl:
  https://opengraph.githubassets.com/7534d37b2457914e611faaf7bd2c34ff6f18aa65a28f55c1d03b123cfa96d623/eyeseast/sqlite-colorbrewer


id: R_kgDOIJsqeg
nameWithOwner: ocdtrekkie/datasette-external-links-new-tabs
createdAt: 2022-10-07T03:34:09Z
topics: []
latest_commit: 57f8662156b6b85328cc3d0b86ad984ca8dacd0d
openGraphImageUrl:
  https://opengraph.githubassets.com/bf94b4455e3a90fd027aeac10df3e697895e3ba716b065556e5a03faef79b60e/ocdtrekkie/datasette-external-links-new-tabs


id: MDEwOlJlcG9zaXRvcnkzNjEwMTQyNzM=
nameWithOwner: rclement/datasette-dashboards
createdAt: 2021-04-23T21:56:48Z
openGraphImageUrl: https://repository-images.githubusercontent.com/361014273/4215992d-9283-4a28-bc98-5e1b2b75fef7
latest_commit: 007ce5bac07b117da524eed57322b35a688335f2
topics:
  ["datasette", "datasette-io", "datasette-plugin", "dashboards", "data-visualization", "sql", "vega-lite", "business-intelligence"]


id: R_kgDOJOIHjg
nameWithOwner: rclement/datasette-ml
createdAt: 2023-03-25T11:18:48Z
latest_commit: 6f0077e3c877e58736d5574def6ea9f3a4a56ca2
openGraphImageUrl:
  https://opengraph.githubassets.com/c7a6b33be5506d53ffe59c9d580dca565612f245f1ecd190c7560a1b9173bb5d/rclement/datasette-ml

topics:
  ["ai", "datasette", "datasette-plugin", "machine-learning", "python", "scikit-learn", "sql", "sqlite", "mlops"]


id: MDEwOlJlcG9zaXRvcnkyOTEzMzkwODY=
nameWithOwner: simonw/airtable-export
createdAt: 2020-08-29T19:51:37Z
topics: ["airtable", "airtable-api", "yaml", "datasette-io", "datasette-tool"]
latest_commit: 74ba9314bc66f6fa1740d45b696e67dfba6b825e
openGraphImageUrl:
  https://opengraph.githubassets.com/51103d07e66718326a6cf2a80690ba9047ec7bf1c677f00b378658a59c897432/simonw/airtable-export


id: MDEwOlJlcG9zaXRvcnkxNzUzMjE0OTc=
nameWithOwner: simonw/csv-diff
createdAt: 2019-03-13T01:11:26Z
topics: ["csv", "click", "diff", "git-scraping", "datasette-io", "datasette-tool", "csv-diff", "tsv-diff"]
latest_commit: 26903b74eefcd65be761810f51b0e55c033bde66
openGraphImageUrl:
  https://opengraph.githubassets.com/bcb2c0d6e3ddaa91900366aabfcbb6347f205cbfc4573bdeebab5d7a4c4cbe92/simonw/csv-diff


id: MDEwOlJlcG9zaXRvcnkxMTA1MDk4MTY=
nameWithOwner: simonw/csvs-to-sqlite
createdAt: 2017-11-13T06:38:21Z
topics: ["sqlite", "python", "csv", "pandas", "click", "datasette", "datasette-io", "datasette-tool"]
latest_commit: 33b79ed80a688ceae4486ebdcca28ae689f54c03
openGraphImageUrl:
  https://opengraph.githubassets.com/d6fa84d152196fc64409e14d2354222d812dcde911210bd103dc466a299e9783/simonw/csvs-to-sqlite


id: MDEwOlJlcG9zaXRvcnkyMDkwOTEyNTY=
nameWithOwner: simonw/datasette-atom
createdAt: 2019-09-17T15:31:01Z
topics: ["datasette", "datasette-plugin", "datasette-io"]
latest_commit: 96295d1071cdf9b028b527db1b9778099897de12
openGraphImageUrl:
  https://opengraph.githubassets.com/a65bdc18966086b0d3d0aa27711b699c847b51742b245ec45162bebcc79c5209/simonw/datasette-atom


id: MDEwOlJlcG9zaXRvcnkyMzYxMTA3NTk=
nameWithOwner: simonw/datasette-auth-existing-cookies
createdAt: 2020-01-25T01:20:31Z
topics: ["datasette", "datasette-plugin", "datasette-io"]
latest_commit: 297e5b4fe6d577a0749bdcc5d20ff58e4521fa6e
openGraphImageUrl:
  https://opengraph.githubassets.com/388054325dbc4074fffb7808a416a905eb9f2f36eea7ad1ba765c5e4aa0e37e6/simonw/datasette-auth-existing-cookies


id: MDEwOlJlcG9zaXRvcnkxOTUwODcxMzc=
nameWithOwner: simonw/datasette-auth-github
createdAt: 2019-07-03T16:02:53Z
topics: ["datasette", "datasette-plugin", "asgi", "datasette-io"]
latest_commit: 10547b832bedc5b851252b08b591685358180630
openGraphImageUrl:
  https://opengraph.githubassets.com/715c201488f71740e32e0023f1d70d67fc517dd7c2af5a116b54eeacdf8d0159/simonw/datasette-auth-github


id: MDEwOlJlcG9zaXRvcnkyNzkzNTcxMjM=
nameWithOwner: simonw/datasette-auth-passwords
createdAt: 2020-07-13T16:34:39Z
topics: ["datasette", "datasette-io", "datasette-plugin"]
latest_commit: bb96899c84395bb40e903b0da5e1e7cdb5d8c4dd
openGraphImageUrl:
  https://opengraph.githubassets.com/e4f9d5a2a9807fed9fe4c02335d10ce6b223a2ba4cb8fcee4900ac599b45f51b/simonw/datasette-auth-passwords


id: MDEwOlJlcG9zaXRvcnkyNzE2NjUzMzY=
nameWithOwner: simonw/datasette-auth-tokens
createdAt: 2020-06-11T23:23:30Z
topics: ["datasette", "datasette-io", "datasette-plugin"]
latest_commit: c11bd461830e3a7a64216f9d1ebdfe156f268b1b
openGraphImageUrl:
  https://opengraph.githubassets.com/9a0c8db724740d0d21143e4bd7110f665e4a8cd061c061fffbb1b7301c9905e1/simonw/datasette-auth-tokens


id: R_kgDOHEfRqA
nameWithOwner: simonw/datasette-auth0
createdAt: 2022-03-26T21:19:31Z
topics: ["datasette", "datasette-plugin", "auth0"]
latest_commit: 2ffd71812267b4d39c9654449721e1106e3c6d3e
openGraphImageUrl:
  https://opengraph.githubassets.com/cc458b2ee75a493c6a4bf283a78356c9a0818f3ca9447ede94838fc1da15579b/simonw/datasette-auth0


id: MDEwOlJlcG9zaXRvcnkyOTMxNjQ0NDc=
nameWithOwner: simonw/datasette-backup
createdAt: 2020-09-05T22:33:29Z
topics: ["datasette", "datasette-io", "datasette-plugin"]
latest_commit: b69eb953c480a0de189cb532aaf699d7cb831f47
openGraphImageUrl:
  https://opengraph.githubassets.com/f2b7f7a1395ab8ad1d43b9afe2759546f696cdf859d89b00019f29b47a3fde22/simonw/datasette-backup


id: MDEwOlJlcG9zaXRvcnkzNDIxMjY2MTA=
nameWithOwner: simonw/datasette-block
createdAt: 2021-02-25T04:51:08Z
topics: ["datasette", "datasette-io", "datasette-plugin"]
latest_commit: 2c071fe71a6ad2fc4d4d397aa69aceda35eb3758
openGraphImageUrl:
  https://opengraph.githubassets.com/0c271d78e92f61b6d1eafc499c39538c890ed16b3d8a5aead54a9f549a76f6db/simonw/datasette-block


id: MDEwOlJlcG9zaXRvcnkyNzQyOTM1OTc=
nameWithOwner: simonw/datasette-block-robots
createdAt: 2020-06-23T02:52:23Z
topics: ["datasette", "datasette-io", "datasette-plugin", "robots-txt"]
latest_commit: 503bb6e5bda413ac664d2e9c32badc277d1405a7
openGraphImageUrl:
  https://opengraph.githubassets.com/eff6c45cc4f61cbeb55933b1feff9bdcb4a7c5708253979f2d2022f9375cbd78/simonw/datasette-block-robots


id: MDEwOlJlcG9zaXRvcnkxOTA5NTA3ODE=
nameWithOwner: simonw/datasette-bplist
createdAt: 2019-06-09T01:15:01Z
topics: ["datasette", "datasette-plugin", "bplist", "datasette-io"]
latest_commit: 218ab42bbac06871cab4748ae73497f1c3e5fc6f
openGraphImageUrl:
  https://opengraph.githubassets.com/74a407b468ee3fae8dccabbc17d815dfc84bc0a059865a272a15f8a406dbd4fe/simonw/datasette-bplist


id: R_kgDOJNnn3g
nameWithOwner: simonw/datasette-chatgpt-plugin
createdAt: 2023-03-24T04:46:24Z
topics: ["chatgpt", "chatgpt-plugin", "datasette", "datasette-plugin"]
latest_commit: 6fea8859a29afd087f94e587da8cb7030c254da5
openGraphImageUrl:
  https://opengraph.githubassets.com/538b7441a728a923121ba5681dd7a524b76f3d6321e778c6fd2cf78af717a370/simonw/datasette-chatgpt-plugin


id: MDEwOlJlcG9zaXRvcnkyNTU0NjAzNDc=
nameWithOwner: simonw/datasette-clone
createdAt: 2020-04-13T23:05:41Z
topics: ["datasette", "datasette-io", "datasette-tool"]
latest_commit: 4ded5c22a5ef6ca83bfa718a9187cc8c592b6815
openGraphImageUrl:
  https://opengraph.githubassets.com/9a69dab7a89afa37a500b122817cf63ed862d62175de3b82da011f997d66a8c1/simonw/datasette-clone


id: R_kgDOJBM_kA
nameWithOwner: simonw/datasette-codespaces
createdAt: 2023-02-22T18:43:18Z
topics: []
latest_commit: cd5afd23c3ebf39e7e2ede00f873f3f3429e7450
openGraphImageUrl:
  https://opengraph.githubassets.com/70905b5919e3b1b74ecc7e3f870f389d4bac41db6f8b0081dce134ed84a11275/simonw/datasette-codespaces


id: MDEwOlJlcG9zaXRvcnkyNDYxMDg1NjE=
nameWithOwner: simonw/datasette-column-inspect
createdAt: 2020-03-09T18:11:00Z
topics: ["datasette", "datasette-io", "datasette-plugin"]
latest_commit: 58ad947f78d237673730d56f63dd74af293ca6e9
openGraphImageUrl:
  https://opengraph.githubassets.com/70591a4d2684b1e92665baaf880d1aaa479701246da9da754c95026668f0a3f3/simonw/datasette-column-inspect


id: MDEwOlJlcG9zaXRvcnkyMjg0ODU4MDY=
nameWithOwner: simonw/datasette-configure-asgi
createdAt: 2019-12-16T22:17:10Z
topics: ["datasette", "datasette-plugin", "asgi", "datasette-io"]
latest_commit: 82e8796745cc51dc8881e4089614fb05f0010a64
openGraphImageUrl:
  https://opengraph.githubassets.com/93f06191304b87026e0aadd214c87551f54a023ddff8d68997fd670987f4ae81/simonw/datasette-configure-asgi


id: MDEwOlJlcG9zaXRvcnkyNDM4ODcwMzY=
nameWithOwner: simonw/datasette-configure-fts
createdAt: 2020-02-29T01:50:57Z
topics: ["datasette", "datasette-plugin", "datasette-io"]
latest_commit: d0946f22d349ab8ebedd83f319ce51ff90061e3e
openGraphImageUrl:
  https://opengraph.githubassets.com/72240edc91d7930f36c2ff3bf0b9a3f41955fab0a38d019e92dbe04ccd79c9b5/simonw/datasette-configure-fts


id: R_kgDOIwsDBg
nameWithOwner: simonw/datasette-cookies-for-magic-parameters
createdAt: 2023-01-11T22:47:31Z
topics: ["datasette", "datasette-plugin"]
latest_commit: 2ee086c7805f6fe994df6d77a6da725bb5ce2cfd
openGraphImageUrl:
  https://opengraph.githubassets.com/46f03d6a61fbb4c9e07cbb3032d90c993927a6d1804671c425811621c5f0f56b/simonw/datasette-cookies-for-magic-parameters


id: R_kgDOHQLzfQ
nameWithOwner: simonw/datasette-copy-to-memory
createdAt: 2022-04-28T20:02:21Z
topics: ["datasette", "datasette-plugin"]
latest_commit: d541c18a78ae6f707a8f9b1e7fc4c020a9f68f2e
openGraphImageUrl:
  https://opengraph.githubassets.com/1036ae933303d1d61c7c566621b4f52ed6122db8cd98c0b6c9dd33d662a54d3a/simonw/datasette-copy-to-memory


id: MDEwOlJlcG9zaXRvcnkxOTU2OTY4MDQ=
nameWithOwner: simonw/datasette-cors
createdAt: 2019-07-07T21:03:11Z
topics: ["datasette", "datasette-plugin", "datasette-io"]
latest_commit: f22b38ff3c50cd395b95968960ef9a5bb49c5465
openGraphImageUrl:
  https://opengraph.githubassets.com/e6f6ffffd6ded39b22b2ee278e4e8939aac6addcc1cb6999de7ef95c2ee80889/simonw/datasette-cors


id: MDEwOlJlcG9zaXRvcnkzMjcwODcyMDc=
nameWithOwner: simonw/datasette-css-properties
createdAt: 2021-01-05T18:38:07Z
topics: ["datasette-io", "datasette-plugin"]
latest_commit: fa371481a637b975b1e548f89fc1c65fb8a7095c
openGraphImageUrl:
  https://opengraph.githubassets.com/bc1814fb7e4c170bedd3dbb4b6ca9c5e3eded24acb8ccb2a9b050a4f395699cd/simonw/datasette-css-properties


id: MDEwOlJlcG9zaXRvcnkyOTkxNDM4NDk=
nameWithOwner: simonw/datasette-dateutil
createdAt: 2020-09-28T00:14:20Z
topics: ["datasette", "datasette-io", "datasette-plugin", "dateutil"]
latest_commit: 590c4bafabe73333be1796dc551d503b68c3a527
openGraphImageUrl:
  https://opengraph.githubassets.com/ca4b1345b86143cfa951e3a60ad7177d87eec088b7b1570f2df1662fff4d612a/simonw/datasette-dateutil


id: MDEwOlJlcG9zaXRvcnkyMzgzMzk0MTI=
nameWithOwner: simonw/datasette-debug-asgi
createdAt: 2020-02-05T00:57:09Z
topics: ["datasette-plugin", "datasette-io", "asgi"]
latest_commit: df72e40d779fac56a96ba1d3d40abdecb545da37
openGraphImageUrl:
  https://opengraph.githubassets.com/9092416e19efae05aecfdb6245503869605ad1dd911c3e9a2b532f28188591a9/simonw/datasette-debug-asgi


id: MDEwOlJlcG9zaXRvcnkyNDc1Mjc0Mzg=
nameWithOwner: simonw/datasette-edit-schema
createdAt: 2020-03-15T18:34:06Z
topics: ["datasette", "datasette-io", "datasette-plugin"]
latest_commit: 4310a06339cd25e2c5022909ba53bff71a94fc12
openGraphImageUrl:
  https://opengraph.githubassets.com/9eb9d08d911c0fc2818af8fb4ec60f3ab3e9c91478299eb6e013b969886d9207/simonw/datasette-edit-schema


id: MDEwOlJlcG9zaXRvcnkzMDg5MzAxMTg=
nameWithOwner: simonw/datasette-edit-templates
createdAt: 2020-10-31T16:58:29Z
topics: []
latest_commit: 7d0f8769f0f20ff09a9ed8c383ff703f95ff67f5
openGraphImageUrl:
  https://opengraph.githubassets.com/5656b72e354f0eac9fde18d92bc9c82f827f052f9f1346fdc8a82910e049fb3a/simonw/datasette-edit-templates


id: R_kgDOIhvnWQ
nameWithOwner: simonw/datasette-ephemeral-tables
createdAt: 2022-11-29T21:54:16Z
topics: []
latest_commit: 0acdd4a83bb513e15d85af42441512c002d0d7f4
openGraphImageUrl:
  https://opengraph.githubassets.com/ea0abfd961515cf8cb1c52805371799b31befcd8b8a24bd50c756271e1e0497d/simonw/datasette-ephemeral-tables


id: R_kgDOI7mujg
nameWithOwner: simonw/datasette-explain
createdAt: 2023-02-09T01:51:11Z
topics: ["datasette", "datasette-plugin"]
latest_commit: 75201adf80b6a41c755e4fa24f7b0ac332d2f611
openGraphImageUrl:
  https://opengraph.githubassets.com/31f510e3d4681173908091a95aa0db7b9049f555fa07fc7193f2e797607ef4dc/simonw/datasette-explain


id: R_kgDOIgngcA
nameWithOwner: simonw/datasette-export
createdAt: 2022-11-27T04:03:08Z
topics: []
latest_commit: b6e6e1fc569515c54cf141ad93d49a6f8eb4b8de
openGraphImageUrl:
  https://opengraph.githubassets.com/18c88a3a8eed610cdf93e230da55dd93efc80cca42d7b78ec9bb3806de159183/simonw/datasette-export


id: R_kgDOHmgSAw
nameWithOwner: simonw/datasette-expose-env
createdAt: 2022-07-03T21:14:29Z
topics: ["datasette", "datasette-plugin"]
latest_commit: 6c14a5998aa98af298ddc24416d43d4393262b34
openGraphImageUrl:
  https://opengraph.githubassets.com/a33eae2b697c7b4d607b595e205d63d1f4d9feb7c04f0a525e28bd68447f008a/simonw/datasette-expose-env


id: R_kgDOIwW61g
nameWithOwner: simonw/datasette-faiss
createdAt: 2023-01-11T04:28:21Z
topics: []
latest_commit: 4fe395607bd6e5a7ec63b737bd787535501b589d
openGraphImageUrl:
  https://opengraph.githubassets.com/4c172c2856f7e5b36c0bf9ec54bb95639b55998abf7a54621cfc9d01612a716c/simonw/datasette-faiss


id: MDEwOlJlcG9zaXRvcnkyNzU2MTU5NDc=
nameWithOwner: simonw/datasette-glitch
createdAt: 2020-06-28T15:41:25Z
topics: ["glitch", "datasette", "datasette-io", "datasette-plugin"]
latest_commit: ab01c36c2ffc3a8cdbfa714f14bf3143d7f88d79
openGraphImageUrl:
  https://opengraph.githubassets.com/37ede74d7c053b89914df562ffdb1ed268c0b9a66ba27d309833796e62657e76/simonw/datasette-glitch


id: R_kgDOISPElA
nameWithOwner: simonw/datasette-gunicorn
createdAt: 2022-10-22T20:23:57Z
topics: ["datasette", "datasette-plugin"]
latest_commit: 5b2ca643eb174f19d664c5bf97dc839da64188ec
openGraphImageUrl:
  https://opengraph.githubassets.com/fe79d75aabb3436f136930237068e054a15084024541f9fd8a84d525bb04cd72/simonw/datasette-gunicorn


id: R_kgDOHAjKFQ
nameWithOwner: simonw/datasette-hashed-urls
createdAt: 2022-03-15T21:31:52Z
topics: []
latest_commit: 3b54a1a44eaff05df9eae2fdde944b20d3c62405
openGraphImageUrl:
  https://opengraph.githubassets.com/32157609573e4c3ed4e2321186a95cebfd1c5b3ae439492481d788407cff2367/simonw/datasette-hashed-urls


id: MDEwOlJlcG9zaXRvcnkyMTcyMTY3ODc=
nameWithOwner: simonw/datasette-haversine
createdAt: 2019-10-24T05:16:14Z
topics: ["datasette", "datasette-plugin", "datasette-io"]
latest_commit: 3b045775c30fd191f92cf358988580b948d9f081
openGraphImageUrl:
  https://opengraph.githubassets.com/dcf4ad4e53b7b5928a85adfa16f92f1fbeb1f8271b8a613013ff951b6a043037/simonw/datasette-haversine


id: R_kgDOGeMHbg
nameWithOwner: simonw/datasette-hovercards
createdAt: 2021-12-02T17:11:59Z
topics: []
latest_commit: 9439ba46b7140fb03223faff0d21aeba5615a287
openGraphImageUrl:
  https://opengraph.githubassets.com/5a67979b4d81361f8f17ea27bc4e4c8fb6abc903ee262be898b7615bf2f70679/simonw/datasette-hovercards


id: MDEwOlJlcG9zaXRvcnkyOTkxOTgzNjk=
nameWithOwner: datasette/datasette-import-table
createdAt: 2020-09-28T05:30:07Z
topics: ["datasette", "datasette-io", "datasette-plugin"]
latest_commit: 19c740504a548f166809c5c270c3ab92310c6cc4
openGraphImageUrl:
  https://opengraph.githubassets.com/a24e49def0948ab006c165093e621507142b8459c604c877d0c37faf9099513a/datasette/datasette-import-table


id: MDEwOlJlcG9zaXRvcnkzMTI5MzQwMDE=
nameWithOwner: simonw/datasette-indieauth
createdAt: 2020-11-15T01:18:21Z
topics: ["datasette", "indieauth", "datasette-plugin", "datasette-io"]
latest_commit: 55b8eefa35aff9d7819fa040abf4f897aee28836
openGraphImageUrl:
  https://opengraph.githubassets.com/626d1afd700a9048c832eb7300d4d5cc08db99e3e2c21c9a7b42a96e13ba80da/simonw/datasette-indieauth


id: MDEwOlJlcG9zaXRvcnkyNzU2MjQzNDY=
nameWithOwner: simonw/datasette-init
createdAt: 2020-06-28T16:26:29Z
topics: []
latest_commit: f978d0308f82641fcc6c71bab45d9a542b7bb66d
openGraphImageUrl:
  https://opengraph.githubassets.com/ba7210de6970d4aedf76f57e036b13aa4b8bfb2f2f38c290ebfd7cbbf16b8943/simonw/datasette-init


id: MDEwOlJlcG9zaXRvcnkyODA1MDAwMjc=
nameWithOwner: simonw/datasette-insert
createdAt: 2020-07-17T18:40:34Z
topics: ["datasette", "datasette-plugin", "datasette-io"]
latest_commit: f3e36af6931fefff39471eaab3978c843aa16dc3
openGraphImageUrl:
  https://opengraph.githubassets.com/aa619c3e75c6d0468d2ed1e7364ccb4c28e3e8ba0b36d36cb45cdcebfdc5541a/simonw/datasette-insert


id: MDEwOlJlcG9zaXRvcnkxNzQ3MTUxNTM=
nameWithOwner: simonw/datasette-jellyfish
createdAt: 2019-03-09T16:02:01Z
topics: ["datasette", "datasette-plugin", "datasette-io"]
latest_commit: e9073cf8a5f6ca2f6fdca832f24f973476868bf8
openGraphImageUrl:
  https://opengraph.githubassets.com/d6d3b4d265e6110065e084fc3679d7dfeee044d5914caac40f495307905756f9/simonw/datasette-jellyfish


id: MDEwOlJlcG9zaXRvcnkxODkzMjE2NzE=
nameWithOwner: simonw/datasette-jq
createdAt: 2019-05-30T01:06:31Z
topics: ["datasette", "datasette-plugin", "jq", "datasette-io"]
latest_commit: 7c67aa7293cab1441d633d5b67c642bd0aaa42c6
openGraphImageUrl:
  https://opengraph.githubassets.com/519b9263bfb24f3f8742c5362091e01f5581dab0fc98d39841e32d50efba4565/simonw/datasette-jq


id: MDEwOlJlcG9zaXRvcnkzMzE3MjA4MjQ=
nameWithOwner: simonw/datasette-leaflet
createdAt: 2021-01-21T18:41:19Z
topics: ["datasette", "datasette-io", "datasette-plugin"]
latest_commit: 04ea172c684554cedd7e8efa40406b76ad4187d4
openGraphImageUrl:
  https://opengraph.githubassets.com/b873607e876dacb1c6411fc5dd038b4bd941c409bc542828741385fe42507b6d/simonw/datasette-leaflet


id: MDEwOlJlcG9zaXRvcnkxMzUwMDcyODc=
nameWithOwner: simonw/datasette-leaflet-geojson
createdAt: 2018-05-27T01:42:30Z
topics: ["datasette", "leaflet", "gis", "datasette-plugin", "datasette-io"]
latest_commit: 5da7e3c45d1290ddd996947fb4844b4872e4d163
openGraphImageUrl:
  https://opengraph.githubassets.com/8093a949817d2aa4a19d1a407fbad073473a1606f3364dd1af763dffd5daf442/simonw/datasette-leaflet-geojson


id: R_kgDOKRMeGQ
nameWithOwner: simonw/datasette-llm-embed
createdAt: 2023-09-08T21:09:40Z
topics: []
latest_commit: ebded67fa9ee19db2c4b1badb1895cef0d58ac4a
openGraphImageUrl:
  https://opengraph.githubassets.com/27d688ca5161b29d49fa94f7fa70f88acdf028e099440effa07d77d841add227/simonw/datasette-llm-embed


id: MDEwOlJlcG9zaXRvcnkyNDIyNjA1ODM=
nameWithOwner: simonw/datasette-mask-columns
createdAt: 2020-02-22T01:29:16Z
topics: ["datasette", "datasette-io", "datasette-plugin"]
latest_commit: 0f11442b5c2cfd9a7ad99e9025574409c5e2f715
openGraphImageUrl:
  https://opengraph.githubassets.com/2220fa86466d8247fc515e012b2bbee43d915ebce61c21648245d7dcfdc2897a/simonw/datasette-mask-columns


id: MDEwOlJlcG9zaXRvcnkyNjE2MzQ4MDc=
nameWithOwner: simonw/datasette-media
createdAt: 2020-05-06T02:42:57Z
topics: ["datasette", "datasette-plugin", "datasette-io"]
latest_commit: 99714794a0ef58e9808f68bf752a8b934591d832
openGraphImageUrl:
  https://opengraph.githubassets.com/0a2f9bc1ee18af876579949af4a41fdb6230d9a726e7fcccb1981feb77103708/simonw/datasette-media


id: R_kgDOH2kMZA
nameWithOwner: simonw/datasette-mp3-audio
createdAt: 2022-08-20T16:20:06Z
topics: []
latest_commit: 4402168792f452a46ab7b488e40ec49cd4b12185
openGraphImageUrl:
  https://opengraph.githubassets.com/62801a008e2061d79c82021040b2f085402755d2f34fdae6f047fb5d0951519f/simonw/datasette-mp3-audio


id: R_kgDOH4H41Q
nameWithOwner: simonw/datasette-multiline-links
createdAt: 2022-08-24T22:16:30Z
topics: []
latest_commit: c14032644c6c56bf5e4ea7bf78715db7e1d525b3
openGraphImageUrl:
  https://opengraph.githubassets.com/b2c7e4cf5c70e981021c89329d43045344b4c6ee065a26b563cb7e5fe8d65b15/simonw/datasette-multiline-links


id: R_kgDOItymQA
nameWithOwner: simonw/datasette-openai
createdAt: 2023-01-03T19:12:52Z
topics: ["datasette", "datasette-plugin", "gpt-3", "openai"]
latest_commit: 289dad171ac52b56f93140a3e1eec76c2accbe0e
openGraphImageUrl:
  https://opengraph.githubassets.com/4e938fb7102694e3a8544d36f8d22eee254194500af773d4b7099f1a56b4405f/simonw/datasette-openai


id: R_kgDOHENadA
nameWithOwner: simonw/datasette-packages
createdAt: 2022-03-25T22:04:23Z
topics: []
latest_commit: 953e0e636d5e6ceb3875fd92c8d23f7a8358c341
openGraphImageUrl:
  https://opengraph.githubassets.com/a47d208018ed468db295c7157e0d3892a45ea1d5f8fe29398f385a1e4d30aa33/simonw/datasette-packages


id: MDEwOlJlcG9zaXRvcnkyNzE0MDg4OTU=
nameWithOwner: simonw/datasette-permissions-sql
createdAt: 2020-06-10T23:48:13Z
topics: ["datasette", "datasette-io", "datasette-plugin"]
latest_commit: e0103ea1c13389391a3e40241485df45739aa638
openGraphImageUrl:
  https://opengraph.githubassets.com/600e916c2750a879682d98d2195e2d91b6b044fb5f7706fe10c6cc4b0a88e0b4/simonw/datasette-permissions-sql


id: MDEwOlJlcG9zaXRvcnkzNzU1NDY2NzU=
nameWithOwner: simonw/datasette-placekey
createdAt: 2021-06-10T02:31:27Z
topics: ["datasette", "datasette-io", "datasette-plugin", "placekey"]
latest_commit: 7bf3497bcec6c4cd7b351cfb87d76b6877ca910a
openGraphImageUrl:
  https://opengraph.githubassets.com/94e136f8c215ad1d213923e121e8ad784737725f80ae0474322a3472453c30ae/simonw/datasette-placekey


id: MDEwOlJlcG9zaXRvcnkxNjc3MzAwNzE=
nameWithOwner: simonw/datasette-pretty-json
createdAt: 2019-01-26T19:30:43Z
topics: ["datasette", "datasette-plugin", "json", "datasette-io"]
latest_commit: 9e5d80a5025aba9408b472b4ccc8b4fdc3586bac
openGraphImageUrl:
  https://opengraph.githubassets.com/3748c85c9910699076ad7a6f58edcf9c04e3c172d291135542595b5e19232ea4/simonw/datasette-pretty-json


id: MDEwOlJlcG9zaXRvcnkyNzIwOTg0ODY=
nameWithOwner: simonw/datasette-psutil
createdAt: 2020-06-13T22:57:07Z
topics: ["datasette", "datasette-io", "datasette-plugin", "psutil"]
latest_commit: ef767d6276f87257d5b72c93e3de57ad4aec1f34
openGraphImageUrl:
  https://opengraph.githubassets.com/ab02883a9d29a90972365f25bfa51eb8223240ff9803bce0c07b9358addd6a27/simonw/datasette-psutil


id: R_kgDOIIe2Hg
nameWithOwner: datasette/datasette-public
createdAt: 2022-10-05T00:03:28Z
topics: []
latest_commit: 0a5dd876f9887404a0e5458f5d8feb9fca31bee4
openGraphImageUrl:
  https://opengraph.githubassets.com/496c12245afa3ca847cba25110534b675a4914817bd740385af619ea57f080cc/datasette/datasette-public


id: MDEwOlJlcG9zaXRvcnkyNDgzODUyOTk=
nameWithOwner: simonw/datasette-publish-fly
createdAt: 2020-03-19T01:47:01Z
topics: ["datasette", "datasette-io", "datasette-plugin", "fly"]
latest_commit: 00706375914175493b4ba136d78ae505c29852c3
openGraphImageUrl:
  https://opengraph.githubassets.com/8a31e342fc4724d98d4df6c2f306d619150f0fdd07984367c0de2973e99f3f17/simonw/datasette-publish-fly


id: MDEwOlJlcG9zaXRvcnkyNTM2MzI5NDg=
nameWithOwner: simonw/datasette-publish-vercel
createdAt: 2020-04-06T22:47:13Z
topics: ["datasette", "datasette-plugin", "zeit-now", "vercel", "datasette-io"]
latest_commit: 1559d979b4e3b1f2f83c51c3c0c10192ff9a6d0c
openGraphImageUrl:
  https://opengraph.githubassets.com/de89462d55482fcd6ca53c329dc1536b651e46b11ce19217c0735c8d8b285031/simonw/datasette-publish-vercel


id: MDEwOlJlcG9zaXRvcnkzOTQxMDc2MTQ=
nameWithOwner: simonw/datasette-query-links
createdAt: 2021-08-09T01:16:59Z
topics: []
latest_commit: 130776d43660716193601ad07cb4059eacf0f393
openGraphImageUrl:
  https://opengraph.githubassets.com/f9b5c654a89737f4c3110c6d02c3370060ff1c080ba32051c07457bd22c1fb7b/simonw/datasette-query-links


id: R_kgDOG5dZxg
nameWithOwner: simonw/datasette-redirect-forbidden
createdAt: 2022-02-23T20:59:26Z
topics: []
latest_commit: 058e5362de44a550cb0675757f145d461211852f
openGraphImageUrl:
  https://opengraph.githubassets.com/c4159d43e4b9afb1b970a1c066ad3e141e3bb8ef9740fd9b23600d0f9e3a4932/simonw/datasette-redirect-forbidden


id: R_kgDOGaS1TA
nameWithOwner: simonw/datasette-redirect-to-https
createdAt: 2021-11-20T22:43:33Z
topics: ["datasette", "datasette-io", "datasette-plugin", "asgi"]
latest_commit: 2a5be0f8b339b0558499a36c4d1be2180c35d58f
openGraphImageUrl:
  https://opengraph.githubassets.com/ef576388384adec6ec97345939953ceef1b074be5fa6d6c30a6fac4cc56d8c7f/simonw/datasette-redirect-to-https


id: MDEwOlJlcG9zaXRvcnkzOTA1MzU1MDA=
nameWithOwner: simonw/datasette-remote-metadata
createdAt: 2021-07-28T23:17:19Z
topics: []
latest_commit: 25c3c3bda7de4efe7e3af0056d3e1bcb95584c9c
openGraphImageUrl:
  https://opengraph.githubassets.com/f785ba9de2950e551bdb1f1850691028844818e3a3ebee66133bbdb7f63ff8c7/simonw/datasette-remote-metadata


id: MDEwOlJlcG9zaXRvcnkxOTEwMjI5Mjg=
nameWithOwner: simonw/datasette-render-binary
createdAt: 2019-06-09T15:25:52Z
topics: ["datasette", "datasette-plugin", "datasette-io"]
latest_commit: b9efb24abe192e7652582731db57e97941753a1e
openGraphImageUrl:
  https://opengraph.githubassets.com/8295c40704d073d5acf25b103d87d63e217774b08842ae3a73d5a18491752926/simonw/datasette-render-binary


id: MDEwOlJlcG9zaXRvcnkxODQxNjg4NjQ=
nameWithOwner: simonw/datasette-render-html
createdAt: 2019-04-30T01:21:25Z
topics: ["datasette-plugin", "datasette-io", "datasette"]
latest_commit: a2a1114d3fbc6b7b2960a3b916aa3afc9aedb6bf
openGraphImageUrl:
  https://opengraph.githubassets.com/b232c6161284ecb9f4b6b6ed91f8929cbb0d7b7cd60ebcc85ef3ac8eadfaf89e/simonw/datasette-render-html


id: R_kgDOH7vZQw
nameWithOwner: simonw/datasette-render-image-tags
createdAt: 2022-09-04T00:36:44Z
topics: []
latest_commit: b22e954e83815b01d82b5f4004edb1a9f24d2e8e
openGraphImageUrl:
  https://opengraph.githubassets.com/de042340b5d4cbfc88d460dd497c9e3ea175d071381d6cd1f5014637aa48bfee/simonw/datasette-render-image-tags


id: MDEwOlJlcG9zaXRvcnkxNDU0ODMwNzc=
nameWithOwner: simonw/datasette-render-images
createdAt: 2018-08-21T00:05:47Z
topics: ["datasette", "plugin", "datasette-plugin", "datasette-io"]
latest_commit: c7a3c287ef53e597df984506ac5161c9e279fb7b
openGraphImageUrl:
  https://opengraph.githubassets.com/088e1feade6190891592817e5b8960e6112a281f08ed184e643ca793cf673af4/simonw/datasette-render-images


id: MDEwOlJlcG9zaXRvcnkyMjA3MTY4MjI=
nameWithOwner: simonw/datasette-render-markdown
createdAt: 2019-11-09T23:28:31Z
topics: ["datasette", "datasette-plugin", "markdown", "datasette-io"]
latest_commit: 65051d4db2681438e4b1bd2442b8ad5a98979a6d
openGraphImageUrl:
  https://opengraph.githubassets.com/d1f5a183edcf5df0346b5dbd4ca8d5eade8affe1a637d3d5cb81a849223cac01/simonw/datasette-render-markdown


id: MDEwOlJlcG9zaXRvcnkyMTQyOTkyNjc=
nameWithOwner: simonw/datasette-render-timestamps
createdAt: 2019-10-10T22:50:50Z
topics: ["datasette", "datasette-plugin", "datasette-io"]
latest_commit: 53b4a495cf2d8145536e5f9dd42bff27fdd2ef91
openGraphImageUrl:
  https://opengraph.githubassets.com/ae5c12788767e934abafdd976e7203d00f38c26f9ed82aa331372a16bcb2dd16/simonw/datasette-render-timestamps


id: MDEwOlJlcG9zaXRvcnkyMDc2MzAxNzQ=
nameWithOwner: simonw/datasette-rure
createdAt: 2019-09-10T18:09:33Z
topics: ["datasette", "datasette-plugin", "sqlite", "regular-expressions", "datasette-io"]
latest_commit: 043fa612c86e0a4a64970b8f28b98ad7507da9a1
openGraphImageUrl:
  https://opengraph.githubassets.com/729bc5aa30ec7e63bad9953fd1ef5a40d0d34f88aa33047873f2f1dca6e20a35/simonw/datasette-rure


id: R_kgDOH99rFw
nameWithOwner: simonw/datasette-sandstorm-support
createdAt: 2022-09-09T17:11:04Z
topics: []
latest_commit: 9d2eafe1345c4aa1ae6355d511286801be8be8ff
openGraphImageUrl:
  https://opengraph.githubassets.com/f6216b8113799d766e11ba448735c466a8f29df54cd65d916dd94b4dc6504426/simonw/datasette-sandstorm-support


id: MDEwOlJlcG9zaXRvcnkyNzM2MDk4Nzk=
nameWithOwner: simonw/datasette-saved-queries
createdAt: 2020-06-20T00:20:42Z
topics: ["datasette", "datasette-io", "datasette-plugin"]
latest_commit: 567ea9ccc65fb2367ae17eef3b9c4e43ef028c5b
openGraphImageUrl:
  https://opengraph.githubassets.com/1fe4f92dd3c9560709e20159384a5355521f580ca0acf1e62365e7e673a3a250/simonw/datasette-saved-queries


id: R_kgDOHilbpw
nameWithOwner: simonw/datasette-scale-to-zero
createdAt: 2022-06-21T22:49:04Z
topics: ["datasette", "datasette-plugin"]
latest_commit: f8ad9bbcf77c28df2e0b7d8e2f871fe08bfc9f5a
openGraphImageUrl:
  https://opengraph.githubassets.com/75a2f8094544385a287f9f7a6fc245fa6403453768d772516bd0153ad52fc458/simonw/datasette-scale-to-zero


id: MDEwOlJlcG9zaXRvcnkyODg2Mjk3NjY=
nameWithOwner: simonw/datasette-schema-versions
createdAt: 2020-08-19T04:04:39Z
topics: ["datasette", "datasette-io", "datasette-plugin"]
latest_commit: 138cfacdac5994404f437e82f7895e7ab7889a63
openGraphImageUrl:
  https://opengraph.githubassets.com/c9e09a10835d31ebccb937443bb86f583ec15d45b68677f371c1ec77cb8206b1/simonw/datasette-schema-versions


id: MDEwOlJlcG9zaXRvcnkyOTQ3MDYyNjc=
nameWithOwner: simonw/datasette-seaborn
createdAt: 2020-09-11T13:43:08Z
topics: ["datasette", "datasette-plugin", "datasette-io", "seaborn", "visualization"]
latest_commit: 61ad490dad04a964d512aebaf3b7af7d965e51ae
openGraphImageUrl:
  https://opengraph.githubassets.com/ae612731cbc7aa61566d1e7bd10f3edaaeea4f711ee49bc903456adba81dcf7c/simonw/datasette-seaborn


id: R_kgDOIl_N4w
nameWithOwner: simonw/datasette-secret-santa
createdAt: 2022-12-10T17:54:39Z
topics: []
latest_commit: 06737db364bab57f2ed80822d7468f22b3086872
openGraphImageUrl:
  https://opengraph.githubassets.com/b4212c4b3a80fae670d388ff16ed50ad881c35828fbe398ee112630852168113/simonw/datasette-secret-santa


id: MDEwOlJlcG9zaXRvcnkyMzY4NjcwMjc=
nameWithOwner: simonw/datasette-sentry
createdAt: 2020-01-28T23:41:27Z
topics: ["datasette", "datasette-plugin", "sentry", "datasette-io"]
latest_commit: 83022b92be7aab45eead571e0c54a6fe87082891
openGraphImageUrl:
  https://opengraph.githubassets.com/19e14802b4d3bf01a2e0b30a44ed401f4c3fd5df2fd43a060f1b8269b22d00a8/simonw/datasette-sentry


id: MDEwOlJlcG9zaXRvcnkyNDg5OTk5OTQ=
nameWithOwner: simonw/datasette-show-errors
createdAt: 2020-03-21T15:06:04Z
topics: ["datasette", "datasette-io", "datasette-plugin", "starlette", "asgi"]
latest_commit: 07c01e54405531fa780e457543de2af99bf53ab6
openGraphImageUrl:
  https://opengraph.githubassets.com/ded7026b31328d63bdd8757fae16f129a7e9eedb195ecf0001cc27fb81f1ba8e/simonw/datasette-show-errors


id: R_kgDOJDyulQ
nameWithOwner: simonw/datasette-simple-html
createdAt: 2023-03-01T02:29:26Z
topics: ["datasette-plugin"]
latest_commit: 7ae1e2d693ea69d88ae11acf26ed81bd0026121c
openGraphImageUrl:
  https://opengraph.githubassets.com/7b0be185149820dc1091311231b9b757589df38d66242ba91ce52abcc65d1414/simonw/datasette-simple-html


id: R_kgDOH50_7g
nameWithOwner: simonw/datasette-sitemap
createdAt: 2022-08-29T21:22:46Z
topics: []
latest_commit: a825654faa4b4802c3bf9ccff4c15bbffe04b224
openGraphImageUrl:
  https://opengraph.githubassets.com/38b356f49b2844ee328e2f7c29281d8a241368d268375ae88f04e77aae3ca0a7/simonw/datasette-sitemap


id: R_kgDOHeFHyg
nameWithOwner: simonw/datasette-socrata
createdAt: 2022-06-08T15:13:44Z
topics: []
latest_commit: 1409aa9b4d2fc3aff286b52e73af33b5786d56d0
openGraphImageUrl:
  https://opengraph.githubassets.com/663cf90ba3034823f9734b74a7a850ebf2d47e0b55b4bbed3fbe4a5c9d993a64/simonw/datasette-socrata


id: MDEwOlJlcG9zaXRvcnkxNjM3OTA4MjI=
nameWithOwner: simonw/datasette-sqlite-fts4
createdAt: 2019-01-02T03:40:41Z
topics: ["datasette", "plugin", "datasette-plugin", "datasette-io"]
latest_commit: 7e46394c63f75536a9dce3796ed03734a7d1e9ee
openGraphImageUrl:
  https://opengraph.githubassets.com/a8f943dfb3309b36192b860a9c99bb717db17535fce3c33f7b5481ca923f2008/simonw/datasette-sqlite-fts4


id: MDEwOlJlcG9zaXRvcnk0MDY5OTY0NzI=
nameWithOwner: simonw/datasette-statistics
createdAt: 2021-09-16T02:51:09Z
topics: []
latest_commit: d5d53539bd22033e92678460d2d92a5a21a6f66a
openGraphImageUrl:
  https://opengraph.githubassets.com/a31591fc4584af66fa122f957b1a18f2efe657ffab211c2b697bbbc18eb35eae/simonw/datasette-statistics


id: R_kgDOGGsxew
nameWithOwner: simonw/datasette-template-request
createdAt: 2021-09-23T17:07:00Z
topics: ["datasette", "datasette-plugin", "datasette-io"]
latest_commit: b379518cffbd0879010babc7d16fb5cbce550432
openGraphImageUrl:
  https://opengraph.githubassets.com/cca6bd7f74490c5d27fd2c160e3e1503e7f256bcd5cd84645a223daa24b0af7f/simonw/datasette-template-request


id: MDEwOlJlcG9zaXRvcnkyMjE4MDIyOTY=
nameWithOwner: simonw/datasette-template-sql
createdAt: 2019-11-14T23:05:34Z
topics: ["datasette", "datasette-plugin", "datasette-io"]
latest_commit: 70395fb245051df5499e10fba596c255b50c8b74
openGraphImageUrl:
  https://opengraph.githubassets.com/ff5e31b5d068acb2ae1ad7883955b1a04cbc11a20eb0cfbc0561169c329a7d63/simonw/datasette-template-sql


id: R_kgDOGkmBIg
nameWithOwner: simonw/datasette-tiddlywiki
createdAt: 2021-12-23T01:05:56Z
topics: []
latest_commit: d6e0b6b9a4dcbe92e27eb4cfa989ede9e6a8991d
openGraphImageUrl:
  https://opengraph.githubassets.com/10a0111dcf53b169c3bcca74c0f3723dd66f601826a5ea1b5459e3da4bfadd2a/simonw/datasette-tiddlywiki


id: R_kgDOHPc0Nw
nameWithOwner: simonw/datasette-total-page-time
createdAt: 2022-04-26T22:09:58Z
topics: []
latest_commit: 0ee52a9bb7b0b4c9de5afdd04190c61410a7ada1
openGraphImageUrl:
  https://opengraph.githubassets.com/c5ec52760212756491bdef31d04cc6799f8d0d08af2573890d0a919245a72d30/simonw/datasette-total-page-time


id: MDEwOlJlcG9zaXRvcnkyMzQ4MjU3OTA=
nameWithOwner: simonw/datasette-upload-csvs
createdAt: 2020-01-19T02:07:05Z
topics: ["datasette-plugin", "datasette", "csvs", "datasette-io"]
latest_commit: 0c167c8db7352851f7dd4cac3ff3a19d9c4f677a
openGraphImageUrl:
  https://opengraph.githubassets.com/b1391382cbd4898c327a83553adc9f265c71d33d4f57b8f5a1fe581d07a0af2f/simonw/datasette-upload-csvs


id: R_kgDOHSfuMg
nameWithOwner: simonw/datasette-upload-dbs
createdAt: 2022-05-05T23:36:51Z
topics: []
latest_commit: 3062f7a3293b6125d60422a72ba2dbf4bb914b3c
openGraphImageUrl:
  https://opengraph.githubassets.com/ba30ad7378dce961967116b869c5de7a3ba67b201f28ac432a87e519e2e82cde/simonw/datasette-upload-dbs


id: MDEwOlJlcG9zaXRvcnk0MDA2NzgzMTc=
nameWithOwner: simonw/datasette-verify
createdAt: 2021-08-28T01:59:12Z
topics: ["datasette-io", "datasette-plugin", "datasette"]
latest_commit: 77bbc92bd670d4098bd3738d9c7ebc3ed0a33014
openGraphImageUrl:
  https://opengraph.githubassets.com/fe6b09e7f27347d0d51bc45e9d93c093d3b1b2a35d9243d94e40f74e827f1858/simonw/datasette-verify


id: MDEwOlJlcG9zaXRvcnkyNzU3MTEyNTQ=
nameWithOwner: simonw/datasette-write
createdAt: 2020-06-29T02:27:31Z
topics: ["datasette-io", "datasette-plugin"]
latest_commit: 7ae4776affb5417892dc1a57e9bce0033d4e3b11
openGraphImageUrl:
  https://opengraph.githubassets.com/8a80add41d26f379748f532032e90412b06427fe57352c6464ca93a6008b6b06/simonw/datasette-write


id: MDEwOlJlcG9zaXRvcnkzOTUxMzc1MTM=
nameWithOwner: simonw/datasette-x-forwarded-host
createdAt: 2021-08-11T23:10:44Z
topics: ["datasette-io", "datasette-plugin"]
latest_commit: f1f6650cb6695c5e43a4893c06230b96a150b6ba
openGraphImageUrl:
  https://opengraph.githubassets.com/26377d7ed775716a5a0b40bf87765570f875fbcd9fdd7dc986d184141cfd1833/simonw/datasette-x-forwarded-host


id: MDEwOlJlcG9zaXRvcnkyOTEzNTkzNTg=
nameWithOwner: simonw/datasette-yaml
createdAt: 2020-08-29T22:32:15Z
topics: ["datasette", "datasette-plugin", "datasette-io", "yaml"]
latest_commit: 448cb722a8b8d6c1664af2e516ece8ca75694a43
openGraphImageUrl:
  https://opengraph.githubassets.com/e3190874a5fbbe78a3b06d44e13ccecaae1dc83578cb8ca042d2def817dd458c/simonw/datasette-yaml


id: R_kgDOI2nvlA
nameWithOwner: simonw/datasette-youtube-embed
createdAt: 2023-01-27T17:52:05Z
topics: []
latest_commit: 95287be382b6e3664bc9c63bcfd1baa8a926ed3d
openGraphImageUrl:
  https://opengraph.githubassets.com/41acc8b60cca77e7fe2ce0d3e51a18a90f0fbad4a544a8d900a0f6837424aa64/simonw/datasette-youtube-embed


id: MDEwOlJlcG9zaXRvcnkxNjYxNTkwNzI=
nameWithOwner: simonw/db-to-sqlite
createdAt: 2019-01-17T04:16:48Z
topics: ["sqlite", "datasette", "sqlalchemy", "datasette-io", "datasette-tool"]
latest_commit: b7c95fcfe853ca5fc2236bfbf7e4a80ca9239969
openGraphImageUrl:
  https://opengraph.githubassets.com/b4998cc299c1fa3c875bed476b2e3a25f98cb3c73df833f0802a98040578427b/simonw/db-to-sqlite


id: MDEwOlJlcG9zaXRvcnkxNjg0NzQ5NzA=
nameWithOwner: simonw/dbf-to-sqlite
createdAt: 2019-01-31T06:30:46Z
topics: ["sqlite", "dbf", "dbase", "foxpro", "datasette-io", "datasette-tool"]
latest_commit: 1d1e1e8d64feb74d8a96b99200fb558b06a6957b
openGraphImageUrl:
  https://opengraph.githubassets.com/90c8f32b35847616a4a9808b393e2d78127d7e436d9e7dee94cf9bd67f1242f9/simonw/dbf-to-sqlite


id: MDEwOlJlcG9zaXRvcnkzMzUzNzIwNTA=
nameWithOwner: simonw/download-tiles
createdAt: 2021-02-02T17:37:49Z
topics: ["mbtiles", "openstreetmap", "datasette-io", "datasette-tool"]
latest_commit: 7e4e0335cad7cbe39b4305651e35a85f14d5403b
openGraphImageUrl:
  https://opengraph.githubassets.com/3c6930d77d0791cc66a4b6831fc24b570b04e6e0effbf1e16b5cbbfd622b005f/simonw/download-tiles


id: MDEwOlJlcG9zaXRvcnkyNDU2NzA2NzA=
nameWithOwner: simonw/fec-to-sqlite
createdAt: 2020-03-07T16:52:49Z
topics: ["sqlite", "datasette", "datasette-io", "fec", "datasette-tool"]
latest_commit: 0d3e1a9b2d2415327762eb078b1e9d3008fac601
openGraphImageUrl:
  https://opengraph.githubassets.com/7a3f7b1e5542c2c2660e6e77e0178e5e93826a218c18aca41cf477f8899b17bc/simonw/fec-to-sqlite


id: MDEwOlJlcG9zaXRvcnkyMzczMjEyNjc=
nameWithOwner: simonw/geojson-to-sqlite
createdAt: 2020-01-30T22:51:05Z
topics: ["sqlite", "geojson", "gis", "datasette-io", "datasette-tool"]
latest_commit: 9c6c1c8ce2f27c2e1374c9a5f2f90996268ade1f
openGraphImageUrl:
  https://opengraph.githubassets.com/09d9790ff3ed575c47fd85f66d199c2aef97da26f71b2b3d3aff0f758f3e8279/simonw/geojson-to-sqlite


id: R_kgDOGXV4Ig
nameWithOwner: simonw/git-history
createdAt: 2021-11-11T20:07:06Z
topics: ["git", "sqlite"]
latest_commit: f6bf5e248c5fd56e37a0ab9c76bc8ce29a91e881
openGraphImageUrl:
  https://opengraph.githubassets.com/f1432f4e812331d23736e80cabf689e4840758eb767c70d386d03e8f18a85db3/simonw/git-history


id: R_kgDOG2hQNg
nameWithOwner: simonw/google-drive-to-sqlite
createdAt: 2022-02-16T02:16:29Z
topics: []
latest_commit: 60c96e5c909285afb32978e16ec8d6a4aeb851db
openGraphImageUrl:
  https://opengraph.githubassets.com/820e7063dcba582fc1d14a684600869fe58533437970e84da6e16e0de13be156/simonw/google-drive-to-sqlite


id: R_kgDOIwP5mQ
nameWithOwner: simonw/json-to-files
createdAt: 2023-01-10T20:09:34Z
topics: []
latest_commit: 3b653d8f06868a3fad2c34b5b8e3d69e5de41f1b
openGraphImageUrl:
  https://opengraph.githubassets.com/89af3d7031117ee9e0ceb9b1242a7a11827d208cfd2ddbd5813b788327384d4a/simonw/json-to-files


id: R_kgDOJRhX7A
nameWithOwner: simonw/llm
createdAt: 2023-04-01T21:16:57Z
topics: ["llms", "openai", "ai"]
latest_commit: 0d593ea2a45d81cb622ded87ebd5ed47eeb6a263
openGraphImageUrl:
  https://opengraph.githubassets.com/5a156011a590a081cfa3115a4196965d16939cea0d4d16e9fca89352da2486ab/simonw/llm


id: MDEwOlJlcG9zaXRvcnkxNjc3NTk4NDY=
nameWithOwner: simonw/markdown-to-sqlite
createdAt: 2019-01-27T02:04:54Z
topics: ["sqlite", "markdown", "yaml", "datasette-io", "datasette-tool"]
latest_commit: e6b852a8c1af51d6d42bfb44d8ee6f5f06debf03
openGraphImageUrl:
  https://opengraph.githubassets.com/4ea90699fb52709a0fce679b5dca56cab0aa0a6e75279d3e371799285c5e6219/simonw/markdown-to-sqlite


id: R_kgDOItf8Rg
nameWithOwner: simonw/openai-to-sqlite
createdAt: 2023-01-03T01:14:22Z
topics: []
latest_commit: 2cad6fcd55238a308d957fdd2f16767e9cb71ccd
openGraphImageUrl:
  https://opengraph.githubassets.com/9cc75185c767aedf2d3a4172c2d0b71c1683294ef3955200583c785aed4ed451/simonw/openai-to-sqlite


id: R_kgDOHI-jKw
nameWithOwner: simonw/pypi-to-sqlite
createdAt: 2022-04-07T23:09:02Z
topics: []
latest_commit: c47bc0e819be4cafd3af6e94c95ce0ed8e236281
openGraphImageUrl:
  https://opengraph.githubassets.com/989306a0ec45744d36e822ee8ce66cbcece96c46a81566f811922e3689f2e99c/simonw/pypi-to-sqlite


id: MDEwOlJlcG9zaXRvcnkyNDA4MTU5Mzg=
nameWithOwner: simonw/shapefile-to-sqlite
createdAt: 2020-02-16T01:55:29Z
topics: ["sqlite", "datasette", "spatialite", "shapefiles", "gis", "datasette-io", "datasette-tool"]
latest_commit: da5ce1fbd15de4440192c4403fb7d8de0d327a9d
openGraphImageUrl:
  https://opengraph.githubassets.com/39a135cecbe0303a0fd1b098c347505f8fe0d1b5e98287ae336b9489083e8d0a/simonw/shapefile-to-sqlite


id: MDEwOlJlcG9zaXRvcnkzMDUxOTk2NjE=
nameWithOwner: simonw/sphinx-to-sqlite
createdAt: 2020-10-18T21:26:55Z
topics: ["sqlite", "sphinx", "datasette-io", "datasette-tool"]
latest_commit: 950a34d7a0aac47a201f079c53d32f806a3bbfc1
openGraphImageUrl:
  https://opengraph.githubassets.com/ff0803f368f2da301c195089ea034217a40de812f8dabeb1469b4c2e77ba22ff/simonw/sphinx-to-sqlite


id: R_kgDOHoFAng
nameWithOwner: simonw/sqlite-comprehend
createdAt: 2022-07-08T06:26:15Z
topics: []
latest_commit: 2a1c9baa9664b3a4f9e2e3e59abd88db5ebb32d7
openGraphImageUrl:
  https://opengraph.githubassets.com/50640839e12a039afda358ee81fe98a64c86c304dfe69fd2cb7813d2c4fcf192/simonw/sqlite-comprehend


id: MDEwOlJlcG9zaXRvcnkxOTUxNDU2Nzg=
nameWithOwner: simonw/sqlite-diffable
createdAt: 2019-07-04T00:58:46Z
topics: ["sqlite", "datasette-tool", "datasette-io"]
latest_commit: dfe290c982bfc5a94bb1df199badde5a931606a1
openGraphImageUrl:
  https://opengraph.githubassets.com/5136967794d5664a615c5178013f34719ce487c0cdcb9594762980bab4fa4c0a/simonw/sqlite-diffable


id: MDEwOlJlcG9zaXRvcnkyNzQyNjQ0ODQ=
nameWithOwner: simonw/sqlite-generate
createdAt: 2020-06-22T23:36:44Z
topics: ["sqlite", "datasette-io", "datasette-tool"]
latest_commit: 84267539f3874d7aee7f868827268f479b480423
openGraphImageUrl:
  https://opengraph.githubassets.com/05b6dcb8caa27d79b609a3aa530af0a0bbc50bf70e41f9af058a2254169d880e/simonw/sqlite-generate


id: MDEwOlJlcG9zaXRvcnkyMTkzNzIxMzM=
nameWithOwner: simonw/sqlite-transform
createdAt: 2019-11-03T22:07:53Z
topics: ["sqlite", "datasette-io", "datasette-tool"]
latest_commit: 049b2fcb6671f86a0916392e0a84dedc160d8fa3
openGraphImageUrl:
  https://opengraph.githubassets.com/ab58080809a86f4a3db6de21d81048d9e9e5da5bdc9c5356ebd32b4d5e9e1679/simonw/sqlite-transform


id: MDEwOlJlcG9zaXRvcnkxNDA5MTI0MzI=
nameWithOwner: simonw/sqlite-utils
createdAt: 2018-07-14T03:21:46Z
openGraphImageUrl: https://repository-images.githubusercontent.com/140912432/871d0480-8b28-11ea-9d87-0068901541de
latest_commit: 8f0c06e1889513ed0f01cb57783ddf07c442d4be
topics:
  ["sqlite", "python", "datasette", "sqlite-database", "click", "cli", "datasette-io", "datasette-tool"]


id: R_kgDOJksOkw
nameWithOwner: simonw/strip-tags
createdAt: 2023-05-18T15:44:34Z
topics: []
latest_commit: 868072957a9502a741b1ca947a992a988f58dc2e
openGraphImageUrl:
  https://opengraph.githubassets.com/b44de664be6a3d50166cff20dec6f86815a7c1a5c61e3d0acf73815df903edec/simonw/strip-tags


id: MDEwOlJlcG9zaXRvcnkzNDY1OTc1NTc=
nameWithOwner: simonw/tableau-to-sqlite
createdAt: 2021-03-11T06:12:02Z
topics: ["datasette-io", "datasette-tool"]
latest_commit: 256aca60d425cea1ef3a2193655bd90f526c171d
openGraphImageUrl:
  https://opengraph.githubassets.com/9a80da85c9e6b4670606581c972c845e114751578ad7883ce433a3c57f2537c9/simonw/tableau-to-sqlite


id: R_kgDOJkvrZQ
nameWithOwner: simonw/ttok
createdAt: 2023-05-18T18:22:59Z
topics: []
latest_commit: 9ad9394c6b886d2ca26e53ea27205192325f3744
openGraphImageUrl:
  https://opengraph.githubassets.com/e302e444a467efa5eb65e2c234eb607105f1e895e329b8cbb43b9bbfc56f3a61/simonw/ttok


id: MDEwOlJlcG9zaXRvcnkxNzU1NTAxMjc=
nameWithOwner: simonw/yaml-to-sqlite
createdAt: 2019-03-14T04:49:08Z
topics: ["yaml", "sqlite", "datasette-io", "datasette-tool"]
latest_commit: 1405bc83c8f2d4d8cd772ccacc340754f22b153e
openGraphImageUrl:
  https://opengraph.githubassets.com/6c2d908dc8dbec52eacf8ff36dd19515254eea80d2aab8e798f2d47a32ee309c/simonw/yaml-to-sqlite


# repos

node_id: MDEwOlJlcG9zaXRvcnkxMDc5MTQ0OTM=
name: datasette
full_name: simonw/datasette
html_url: https://github.com/simonw/datasette
description: An open source multi-tool for exploring and publishing data
created_at: 2017-10-23T00:39:03Z
updated_at: 2026-06-15T03:30:07Z
pushed_at: 2026-06-15T03:30:03Z
homepage: https://datasette.io
language: Python
license: apache-2.0
visibility: public
default_branch: main
permissions: {"admin": false, "maintain": false, "push": false, "triage": false, "pull": false}
pull_request_creation_policy: all
topics:
  ["asgi", "automatic-api", "csv", "datasets", "datasette", "datasette-io", "docker", "json", "python", "sql", "sqlite"]


node_id: MDEwOlJlcG9zaXRvcnkxMTA1MDk4MTY=
name: csvs-to-sqlite
full_name: simonw/csvs-to-sqlite
html_url: https://github.com/simonw/csvs-to-sqlite
description: Convert CSV files into a SQLite database
created_at: 2017-11-13T06:38:21Z
updated_at: 2025-05-01T19:28:38Z
pushed_at: 2025-04-10T17:00:54Z
language: Python
license: apache-2.0
topics: ["click", "csv", "datasette", "datasette-io", "datasette-tool", "pandas", "python", "sqlite"]
visibility: public
default_branch: main
permissions: {"admin": false, "maintain": false, "push": false, "triage": false, "pull": true}
readme:
  # csvs-to-sqlite

  [![PyPI](https://img.shields.io/pypi/v/csvs-to-sqlite.svg)](https://pypi.org/project/csvs-to-sqlite/)
  [![Changelog](https://img.shields.io/github/v/release/simonw/csvs-to-sqlite?include_prereleases&label=changelog)](https://github.com/simonw/csvs-to-sqlite/releases)
  [![Tests](https://github.com/simonw/csvs-to-sqlite/workflows/Test/badge.svg)](https://github.com/simonw/csvs-to-sqlite/actions?query=workflow%3ATest)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/csvs-to-sqlite/blob/main/LICENSE)

  Convert CSV files into a SQLite database. Browse and publish that SQLite database with [Datasette](https://github.com/simonw/datasette).

  > [!NOTE]
  > This tool is **infrequently maintained**. I suggest [using sqlite-utils](https://sqlite-utils.datasette.io/en/stable/cli.html#inserting-csv-or-tsv-data) for importing CSV and TSV to SQLite instead for most cases.

  Basic usage:
  ```bash
  csvs-to-sqlite myfile.csv mydatabase.db
  ```
  This will create a new SQLite database called `mydatabase.db` containing a
  single table, `myfile`, containing the CSV content.

  You can provide multiple CSV files:
  ```
  csvs-to-sqlite one.csv two.csv bundle.db
  ```
  The `bundle.db` database will contain two tables, `one` and `two`.

  This means you can use wildcards:
  ```bash
  csvs-to-sqlite ~/Downloads/*.csv my-downloads.db
  ```
  If you pass a path to one or more directories, the script will recursively
  search those directories for CSV files and create tables for each one.
  ```bash
  csvs-to-sqlite ~/path/to/directory all-my-csvs.db
  ```
  ## Handling TSV (tab-separated values)

  You can use the `-s` option to specify a different delimiter. If you want
  to use a tab character you'll need to apply shell escaping like so:
  ```bash
  csvs-to-sqlite my-file.tsv my-file.db -s $'\t'
  ```
  ## Refactoring columns into separate lookup tables

  Let's say you have a CSV file that looks like this:
  ```csv
  county,precinct,office,district,party,candidate,votes
  Clark,1,President,,REP,John R. Kasich,5
  Clark,2,President,,REP,John R. Kasich,0
  Clark,3,President,,REP,John R. Kasich,7
  ```
  ([Real example taken from the Open Elections project](https://github.com/openelections/openelections-data-sd/blob/master/2016/20160607__sd__primary__clark__precinct.csv))

  You can now convert selected columns into separate lookup tables using the new
  `--extract-column` option (shortname: `-c`) - for example:
  ```bash
  csvs-to-sqlite openelections-data-*/*.csv \
      -c county:County:name \
      -c precinct:Precinct:name \
      -c office -c district -c party -c candidate \
      openelections.db
  ```
  The format is as follows:
  ```bash
  column_name:optional_table_name:optional_table_value_column_name
  ```
  If you just specify the column name e.g. `-c office`, the following table will
  be created:
  ```sql
  CREATE TABLE "office" (
      "id" INTEGER PRIMARY KEY,
      "value" TEXT
  );
  ```
  If you specify all three options, e.g. `-c precinct:Precinct:name` the table
  will look like this:
  ```sql
  CREATE TABLE "Precinct" (
      "id" INTEGER PRIMARY KEY,
      "name" TEXT
  );
  ```
  The original tables will be created like this:
  ```sql
  CREATE TABLE "ca__primary__san_francisco__precinct" (
      "county" INTEGER,
      "precinct" INTEGER,
      "office" INTEGER,
      "district" INTEGER,
      "party" INTEGER,
      "candidate" INTEGER,
      "votes" INTEGER,
      FOREIGN KEY (county) REFERENCES County(id),
      FOREIGN KEY (party) REFERENCES party(id),
      FOREIGN KEY (precinct) REFERENCES Precinct(id),
      FOREIGN KEY (office) REFERENCES office(id),
      FOREIGN KEY (candidate) REFERENCES candidate(id)
  );
  ```
  They will be populated with IDs that reference the new derived tables.

  ## Installation

  ```bash
  pip install csvs-to-sqlite
  ```

  `csvs-to-sqlite` now requires Python 3. If you are running Python 2 you can install the last version to support Python 2:
  ```bash
  pip install csvs-to-sqlite==0.9.2
  ```

  ## csvs-to-sqlite --help

  <!-- [[[cog
  import cog
  from csvs_to_sqlite import cli
  from click.testing import CliRunner
  runner = CliRunner()
  result = runner.invoke(cli.cli, ["--help"])
  help = result.output.replace("Usage: cli", "Usage: csvs-to-sqlite")
  cog.out(
      "```\n{}\n```".format(help)
  )
  ]]] -->
  ```
  Usage: csvs-to-sqlite [OPTIONS] PATHS... DBNAME

    PATHS: paths to individual .csv files or to directories containing .csvs

    DBNAME: name of the SQLite database file to create

  Options:
    -s, --separator TEXT            Field separator in input .csv
    -q, --quoting INTEGER           Control field quoting behavior per csv.QUOTE_*
                                    constants. Use one of QUOTE_MINIMAL (0),
                                    QUOTE_ALL (1), QUOTE_NONNUMERIC (2) or
                                    QUOTE_NONE (3).
    --skip-errors                   Skip lines with too many fields instead of
                                    stopping the import
    --replace-tables                Replace tables if they already exist
    -t, --table TEXT                Table to use (instead of using CSV filename)
    -c, --extract-column TEXT       One or more columns to 'extract' into a
                                    separate lookup table. If you pass a simple
                                    column name that column will be replaced with
                                    integer foreign key references to a new table
                                    of that name. You can customize the name of
                                    the table like so:     state:States:state_name
                                    
                                    This will pull unique values from the 'state'
                                    column and use them to populate a new 'States'
                                    table, with an id column primary key and a
                                    state_name column containing the strings from
                                    the original column.
    -d, --date TEXT                 One or more columns to parse into ISO
                                    formatted dates
    -dt, --datetime TEXT            One or more columns to parse into ISO
                                    formatted datetimes
    -df, --datetime-format TEXT     One or more custom date format strings to try
                                    when parsing dates/datetimes
    -pk, --primary-key TEXT         One or more columns to use as the primary key
    -f, --fts TEXT                  One or more columns to use to populate a full-
                                    text index
    -i, --index TEXT                Add index on this column (or a compound index
                                    with -i col1,col2)
    --shape TEXT                    Custom shape for the DB table - format is
                                    csvcol:dbcol(TYPE),...
    --filename-column TEXT          Add a column with this name and populate with
                                    CSV file name
    --fixed-column <TEXT TEXT>...   Populate column with a fixed string
    --fixed-column-int <TEXT INTEGER>...
                                    Populate column with a fixed integer
    --fixed-column-float <TEXT FLOAT>...
                                    Populate column with a fixed float
    --no-index-fks                  Skip adding index to foreign key columns
                                    created using --extract-column (default is to
                                    add them)
    --no-fulltext-fks               Skip adding full-text index on values
                                    extracted using --extract-column (default is
                                    to add them)
    --just-strings                  Import all columns as text strings by default
                                    (and, if specified, still obey --shape,
                                    --date/datetime, and --datetime-format)
    --version                       Show the version and exit.
    --help                          Show this message and exit.

  ```
  <!-- [[[end]]] -->

readme_html:
  <div id="readme" class="md" data-path="README.md"><article class="markdown-body entry-content container-lg" itemprop="text"><div class="markdown-heading" dir="auto"><h1 class="heading-element" dir="auto">csvs-to-sqlite</h1><a id="user-content-csvs-to-sqlite" class="anchor" aria-label="Permalink: csvs-to-sqlite" href="#csvs-to-sqlite"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto"><a href="https://pypi.org/project/csvs-to-sqlite/" rel="nofollow"><img src="https://img.shields.io/pypi/v/csvs-to-sqlite.svg" alt="PyPI" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/206cee505592c913716af0f31232c8b4278c80879cd5bcc9ff71f6c233f0d867/68747470733a2f2f696d672e736869656c64732e696f2f707970692f762f637376732d746f2d73716c6974652e737667"></a>
  <a href="https://github.com/simonw/csvs-to-sqlite/releases"><img src="https://img.shields.io/github/v/release/simonw/csvs-to-sqlite?include_prereleases&label=changelog" alt="Changelog" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/1cf897f943199796c3ac886537c0b2f118c1fb8d810f80f4cbe7ff882e363cd8/68747470733a2f2f696d672e736869656c64732e696f2f6769746875622f762f72656c656173652f73696d6f6e772f637376732d746f2d73716c6974653f696e636c7564655f70726572656c6561736573266c6162656c3d6368616e67656c6f67"></a>
  <a href="https://github.com/simonw/csvs-to-sqlite/actions?query=workflow%3ATest"><img src="https://github.com/simonw/csvs-to-sqlite/workflows/Test/badge.svg" alt="Tests" style="max-width: 100%;"></a>
  <a href="https://github.com/simonw/csvs-to-sqlite/blob/main/LICENSE"><img src="https://img.shields.io/badge/license-Apache%202.0-blue.svg" alt="License" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/c355f200ea90fddaa407b6eaab303663a669248ea3ca7b1fcf77dbe04ff5f48c/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6c6963656e73652d417061636865253230322e302d626c75652e737667"></a></p>
  <p dir="auto">Convert CSV files into a SQLite database. Browse and publish that SQLite database with <a href="https://github.com/simonw/datasette">Datasette</a>.</p>
  <div class="markdown-alert markdown-alert-note" dir="auto"><p class="markdown-alert-title" dir="auto"><svg class="octicon octicon-info mr-2" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="M0 8a8 8 0 1 1 16 0A8 8 0 0 1 0 8Zm8-6.5a6.5 6.5 0 1 0 0 13 6.5 6.5 0 0 0 0-13ZM6.5 7.75A.75.75 0 0 1 7.25 7h1a.75.75 0 0 1 .75.75v2.75h.25a.75.75 0 0 1 0 1.5h-2a.75.75 0 0 1 0-1.5h.25v-2h-.25a.75.75 0 0 1-.75-.75ZM8 6a1 1 0 1 1 0-2 1 1 0 0 1 0 2Z"></path></svg>Note</p><p dir="auto">This tool is <strong>infrequently maintained</strong>. I suggest <a href="https://sqlite-utils.datasette.io/en/stable/cli.html#inserting-csv-or-tsv-data" rel="nofollow">using sqlite-utils</a> for importing CSV and TSV to SQLite instead for most cases.</p>
  </div>
  <p dir="auto">Basic usage:</p>
  <div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="csvs-to-sqlite myfile.csv mydatabase.db"><pre>csvs-to-sqlite myfile.csv mydatabase.db</pre></div>
  <p dir="auto">This will create a new SQLite database called <code>mydatabase.db</code> containing a
  single table, <code>myfile</code>, containing the CSV content.</p>
  <p dir="auto">You can provide multiple CSV files:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="csvs-to-sqlite one.csv two.csv bundle.db"><pre class="notranslate"><code>csvs-to-sqlite one.csv two.csv bundle.db
  </code></pre></div>
  <p dir="auto">The <code>bundle.db</code> database will contain two tables, <code>one</code> and <code>two</code>.</p>
  <p dir="auto">This means you can use wildcards:</p>
  <div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="csvs-to-sqlite ~/Downloads/*.csv my-downloads.db"><pre>csvs-to-sqlite <span class="pl-k">~</span>/Downloads/<span class="pl-k">*</span>.csv my-downloads.db</pre></div>
  <p dir="auto">If you pass a path to one or more directories, the script will recursively
  search those directories for CSV files and create tables for each one.</p>
  <div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="csvs-to-sqlite ~/path/to/directory all-my-csvs.db"><pre>csvs-to-sqlite <span class="pl-k">~</span>/path/to/directory all-my-csvs.db</pre></div>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Handling TSV (tab-separated values)</h2><a id="user-content-handling-tsv-tab-separated-values" class="anchor" aria-label="Permalink: Handling TSV (tab-separated values)" href="#handling-tsv-tab-separated-values"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">You can use the <code>-s</code> option to specify a different delimiter. If you want
  to use a tab character you'll need to apply shell escaping like so:</p>
  <div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="csvs-to-sqlite my-file.tsv my-file.db -s $'\t'"><pre>csvs-to-sqlite my-file.tsv my-file.db -s <span class="pl-s"><span class="pl-pds">$'</span><span class="pl-cce">\t</span><span class="pl-pds">'</span></span></pre></div>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Refactoring columns into separate lookup tables</h2><a id="user-content-refactoring-columns-into-separate-lookup-tables" class="anchor" aria-label="Permalink: Refactoring columns into separate lookup tables" href="#refactoring-columns-into-separate-lookup-tables"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">Let's say you have a CSV file that looks like this:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="county,precinct,office,district,party,candidate,votes
  Clark,1,President,,REP,John R. Kasich,5
  Clark,2,President,,REP,John R. Kasich,0
  Clark,3,President,,REP,John R. Kasich,7"><pre lang="csv" class="notranslate"><code>county,precinct,office,district,party,candidate,votes
  Clark,1,President,,REP,John R. Kasich,5
  Clark,2,President,,REP,John R. Kasich,0
  Clark,3,President,,REP,John R. Kasich,7
  </code></pre></div>
  <p dir="auto">(<a href="https://github.com/openelections/openelections-data-sd/blob/master/2016/20160607__sd__primary__clark__precinct.csv">Real example taken from the Open Elections project</a>)</p>
  <p dir="auto">You can now convert selected columns into separate lookup tables using the new
  <code>--extract-column</code> option (shortname: <code>-c</code>) - for example:</p>
  <div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="csvs-to-sqlite openelections-data-*/*.csv \
      -c county:County:name \
      -c precinct:Precinct:name \
      -c office -c district -c party -c candidate \
      openelections.db"><pre>csvs-to-sqlite openelections-data-<span class="pl-k">*</span>/<span class="pl-k">*</span>.csv \
      -c county:County:name \
      -c precinct:Precinct:name \
      -c office -c district -c party -c candidate \
      openelections.db</pre></div>
  <p dir="auto">The format is as follows:</p>
  <div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="column_name:optional_table_name:optional_table_value_column_name"><pre>column_name:optional_table_name:optional_table_value_column_name</pre></div>
  <p dir="auto">If you just specify the column name e.g. <code>-c office</code>, the following table will
  be created:</p>
  <div class="highlight highlight-source-sql notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="CREATE TABLE "office" (
      "id" INTEGER PRIMARY KEY,
      "value" TEXT
  );"><pre><span class="pl-k">CREATE</span> <span class="pl-k">TABLE</span> "<span class="pl-en">office</span>" (
      <span class="pl-s"><span class="pl-pds">"</span>id<span class="pl-pds">"</span></span> <span class="pl-k">INTEGER</span> <span class="pl-k">PRIMARY KEY</span>,
      <span class="pl-s"><span class="pl-pds">"</span>value<span class="pl-pds">"</span></span> <span class="pl-k">TEXT</span>
  );</pre></div>
  <p dir="auto">If you specify all three options, e.g. <code>-c precinct:Precinct:name</code> the table
  will look like this:</p>
  <div class="highlight highlight-source-sql notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="CREATE TABLE "Precinct" (
      "id" INTEGER PRIMARY KEY,
      "name" TEXT
  );"><pre><span class="pl-k">CREATE</span> <span class="pl-k">TABLE</span> "<span class="pl-en">Precinct</span>" (
      <span class="pl-s"><span class="pl-pds">"</span>id<span class="pl-pds">"</span></span> <span class="pl-k">INTEGER</span> <span class="pl-k">PRIMARY KEY</span>,
      <span class="pl-s"><span class="pl-pds">"</span>name<span class="pl-pds">"</span></span> <span class="pl-k">TEXT</span>
  );</pre></div>
  <p dir="auto">The original tables will be created like this:</p>
  <div class="highlight highlight-source-sql notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="CREATE TABLE "ca__primary__san_francisco__precinct" (
      "county" INTEGER,
      "precinct" INTEGER,
      "office" INTEGER,
      "district" INTEGER,
      "party" INTEGER,
      "candidate" INTEGER,
      "votes" INTEGER,
      FOREIGN KEY (county) REFERENCES County(id),
      FOREIGN KEY (party) REFERENCES party(id),
      FOREIGN KEY (precinct) REFERENCES Precinct(id),
      FOREIGN KEY (office) REFERENCES office(id),
      FOREIGN KEY (candidate) REFERENCES candidate(id)
  );"><pre><span class="pl-k">CREATE</span> <span class="pl-k">TABLE</span> "<span class="pl-en">ca__primary__san_francisco__precinct</span>" (
      <span class="pl-s"><span class="pl-pds">"</span>county<span class="pl-pds">"</span></span> <span class="pl-k">INTEGER</span>,
      <span class="pl-s"><span class="pl-pds">"</span>precinct<span class="pl-pds">"</span></span> <span class="pl-k">INTEGER</span>,
      <span class="pl-s"><span class="pl-pds">"</span>office<span class="pl-pds">"</span></span> <span class="pl-k">INTEGER</span>,
      <span class="pl-s"><span class="pl-pds">"</span>district<span class="pl-pds">"</span></span> <span class="pl-k">INTEGER</span>,
      <span class="pl-s"><span class="pl-pds">"</span>party<span class="pl-pds">"</span></span> <span class="pl-k">INTEGER</span>,
      <span class="pl-s"><span class="pl-pds">"</span>candidate<span class="pl-pds">"</span></span> <span class="pl-k">INTEGER</span>,
      <span class="pl-s"><span class="pl-pds">"</span>votes<span class="pl-pds">"</span></span> <span class="pl-k">INTEGER</span>,
      <span class="pl-k">FOREIGN KEY</span> (county) <span class="pl-k">REFERENCES</span> County(id),
      <span class="pl-k">FOREIGN KEY</span> (party) <span class="pl-k">REFERENCES</span> party(id),
      <span class="pl-k">FOREIGN KEY</span> (precinct) <span class="pl-k">REFERENCES</span> Precinct(id),
      <span class="pl-k">FOREIGN KEY</span> (office) <span class="pl-k">REFERENCES</span> office(id),
      <span class="pl-k">FOREIGN KEY</span> (candidate) <span class="pl-k">REFERENCES</span> candidate(id)
  );</pre></div>
  <p dir="auto">They will be populated with IDs that reference the new derived tables.</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Installation</h2><a id="user-content-installation" class="anchor" aria-label="Permalink: Installation" href="#installation"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="pip install csvs-to-sqlite"><pre>pip install csvs-to-sqlite</pre></div>
  <p dir="auto"><code>csvs-to-sqlite</code> now requires Python 3. If you are running Python 2 you can install the last version to support Python 2:</p>
  <div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="pip install csvs-to-sqlite==0.9.2"><pre>pip install csvs-to-sqlite==0.9.2</pre></div>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">csvs-to-sqlite --help</h2><a id="user-content-csvs-to-sqlite---help" class="anchor" aria-label="Permalink: csvs-to-sqlite --help" href="#csvs-to-sqlite---help"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>

  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="Usage: csvs-to-sqlite [OPTIONS] PATHS... DBNAME

    PATHS: paths to individual .csv files or to directories containing .csvs

    DBNAME: name of the SQLite database file to create

  Options:
    -s, --separator TEXT            Field separator in input .csv
    -q, --quoting INTEGER           Control field quoting behavior per csv.QUOTE_*
                                    constants. Use one of QUOTE_MINIMAL (0),
                                    QUOTE_ALL (1), QUOTE_NONNUMERIC (2) or
                                    QUOTE_NONE (3).
    --skip-errors                   Skip lines with too many fields instead of
                                    stopping the import
    --replace-tables                Replace tables if they already exist
    -t, --table TEXT                Table to use (instead of using CSV filename)
    -c, --extract-column TEXT       One or more columns to 'extract' into a
                                    separate lookup table. If you pass a simple
                                    column name that column will be replaced with
                                    integer foreign key references to a new table
                                    of that name. You can customize the name of
                                    the table like so:     state:States:state_name
                                    
                                    This will pull unique values from the 'state'
                                    column and use them to populate a new 'States'
                                    table, with an id column primary key and a
                                    state_name column containing the strings from
                                    the original column.
    -d, --date TEXT                 One or more columns to parse into ISO
                                    formatted dates
    -dt, --datetime TEXT            One or more columns to parse into ISO
                                    formatted datetimes
    -df, --datetime-format TEXT     One or more custom date format strings to try
                                    when parsing dates/datetimes
    -pk, --primary-key TEXT         One or more columns to use as the primary key
    -f, --fts TEXT                  One or more columns to use to populate a full-
                                    text index
    -i, --index TEXT                Add index on this column (or a compound index
                                    with -i col1,col2)
    --shape TEXT                    Custom shape for the DB table - format is
                                    csvcol:dbcol(TYPE),...
    --filename-column TEXT          Add a column with this name and populate with
                                    CSV file name
    --fixed-column <TEXT TEXT>...   Populate column with a fixed string
    --fixed-column-int <TEXT INTEGER>...
                                    Populate column with a fixed integer
    --fixed-column-float <TEXT FLOAT>...
                                    Populate column with a fixed float
    --no-index-fks                  Skip adding index to foreign key columns
                                    created using --extract-column (default is to
                                    add them)
    --no-fulltext-fks               Skip adding full-text index on values
                                    extracted using --extract-column (default is
                                    to add them)
    --just-strings                  Import all columns as text strings by default
                                    (and, if specified, still obey --shape,
                                    --date/datetime, and --datetime-format)
    --version                       Show the version and exit.
    --help                          Show this message and exit.
  "><pre class="notranslate"><code>Usage: csvs-to-sqlite [OPTIONS] PATHS... DBNAME

    PATHS: paths to individual .csv files or to directories containing .csvs

    DBNAME: name of the SQLite database file to create

  Options:
    -s, --separator TEXT            Field separator in input .csv
    -q, --quoting INTEGER           Control field quoting behavior per csv.QUOTE_*
                                    constants. Use one of QUOTE_MINIMAL (0),
                                    QUOTE_ALL (1), QUOTE_NONNUMERIC (2) or
                                    QUOTE_NONE (3).
    --skip-errors                   Skip lines with too many fields instead of
                                    stopping the import
    --replace-tables                Replace tables if they already exist
    -t, --table TEXT                Table to use (instead of using CSV filename)
    -c, --extract-column TEXT       One or more columns to 'extract' into a
                                    separate lookup table. If you pass a simple
                                    column name that column will be replaced with
                                    integer foreign key references to a new table
                                    of that name. You can customize the name of
                                    the table like so:     state:States:state_name
                                    
                                    This will pull unique values from the 'state'
                                    column and use them to populate a new 'States'
                                    table, with an id column primary key and a
                                    state_name column containing the strings from
                                    the original column.
    -d, --date TEXT                 One or more columns to parse into ISO
                                    formatted dates
    -dt, --datetime TEXT            One or more columns to parse into ISO
                                    formatted datetimes
    -df, --datetime-format TEXT     One or more custom date format strings to try
                                    when parsing dates/datetimes
    -pk, --primary-key TEXT         One or more columns to use as the primary key
    -f, --fts TEXT                  One or more columns to use to populate a full-
                                    text index
    -i, --index TEXT                Add index on this column (or a compound index
                                    with -i col1,col2)
    --shape TEXT                    Custom shape for the DB table - format is
                                    csvcol:dbcol(TYPE),...
    --filename-column TEXT          Add a column with this name and populate with
                                    CSV file name
    --fixed-column <TEXT TEXT>...   Populate column with a fixed string
    --fixed-column-int <TEXT INTEGER>...
                                    Populate column with a fixed integer
    --fixed-column-float <TEXT FLOAT>...
                                    Populate column with a fixed float
    --no-index-fks                  Skip adding index to foreign key columns
                                    created using --extract-column (default is to
                                    add them)
    --no-fulltext-fks               Skip adding full-text index on values
                                    extracted using --extract-column (default is
                                    to add them)
    --just-strings                  Import all columns as text strings by default
                                    (and, if specified, still obey --shape,
                                    --date/datetime, and --datetime-format)
    --version                       Show the version and exit.
    --help                          Show this message and exit.

  </code></pre></div>

  </article></div>


node_id: MDEwOlJlcG9zaXRvcnkxMzAyMzY3NjI=
name: datasette-cluster-map
full_name: simonw/datasette-cluster-map
html_url: https://github.com/simonw/datasette-cluster-map
description: Datasette plugin that shows a map for any data with latitude/longitude columns
created_at: 2018-04-19T15:31:55Z
updated_at: 2025-11-02T15:47:03Z
pushed_at: 2025-11-02T15:46:59Z
language: JavaScript
license: apache-2.0
topics: ["datasette", "datasette-io", "datasette-plugin", "leafletjs"]
visibility: public
default_branch: main
permissions: {"admin": false, "maintain": false, "push": false, "triage": false, "pull": false}
readme:
  # datasette-cluster-map

  [![PyPI](https://img.shields.io/pypi/v/datasette-cluster-map.svg)](https://pypi.org/project/datasette-cluster-map/)
  [![Changelog](https://img.shields.io/github/v/release/simonw/datasette-cluster-map?include_prereleases&label=changelog)](https://github.com/simonw/datasette-cluster-map/releases)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/datasette-cluster-map/blob/main/LICENSE)

  A [Datasette plugin](https://docs.datasette.io/en/stable/plugins.html) that detects tables with `latitude` and `longitude` columns and then plots them on a map using [Leaflet.markercluster](https://github.com/Leaflet/Leaflet.markercluster).

  More about this project: [Datasette plugins, and building a clustered map visualization](https://simonwillison.net/2018/Apr/20/datasette-plugins/).

  ## Demo

  [global-power-plants.datasettes.com](https://global-power-plants.datasettes.com/global-power-plants/global-power-plants) hosts a demo of this plugin running against a database of 33,000 power plants around the world.

  ![Cluster map demo](https://static.simonwillison.net/static/2020/global-power-plants.png)

  ## Installation

  Run `datasette install datasette-cluster-map` to add this plugin to your Datasette virtual environment. Datasette will automatically load the plugin if it is installed in this way.

  If you are deploying using the `datasette publish` command you can use the `--install` option:

      datasette publish cloudrun mydb.db --install=datasette-cluster-map

  If any of your tables have one of the following pairs of columns a map will be automatically displayed:

  - `latitude` and `longitude`
  - `lat` and `lng`
  - `lat` and `lon`
  - `lat` and `long`
  - `*_latitude` and `*_longitude`
  - `*_lat` and `*_lng` for any of the three variants of `lng`

  ## Configuration

  If your columns are called something else you can configure the column names using [plugin configuration](https://docs.datasette.io/en/stable/plugins.html#plugin-configuration) in a `metadata.json` file. For example, if all of your columns are called `xlat` and `xlng` you can create a `metadata.json` file like this:

  ```json
  {
      "title": "Regular metadata keys can go here too",
      "plugins": {
          "datasette-cluster-map": {
              "latitude_column": "xlat",
              "longitude_column": "xlng"
          }
      }
  }
  ```

  Then run Datasette like this:

      datasette mydata.db -m metadata.json

  This will configure the required column names for every database loaded by that Datasette instance.

  If you want to customize the column names for just one table in one database, you can do something like this:

  ```json
  {
      "databases": {
          "polar-bears": {
              "tables": {
                  "USGS_WC_eartag_deployments_2009-2011": {
                      "plugins": {
                          "datasette-cluster-map": {
                              "latitude_column": "Capture Latitude",
                              "longitude_column": "Capture Longitude"
                          }
                      }
                  }
              }
          }
      }
  }
  ```

  You can also use a custom SQL query to rename those columns to `latitude` and `longitude`, [for example](https://polar-bears.now.sh/polar-bears?sql=select+*%2C%0D%0A++++%22Capture+Latitude%22+as+latitude%2C%0D%0A++++%22Capture+Longitude%22+as+longitude%0D%0Afrom+%5BUSGS_WC_eartag_deployments_2009-2011%5D):

  ```sql
  select *,
      "Capture Latitude" as latitude,
      "Capture Longitude" as longitude
  from [USGS_WC_eartag_deployments_2009-2011]
  ```

  The map defaults to being displayed above the main results table on the page. You can use the `"container"` plugin setting to provide a CSS selector indicating an element that the map should be appended to instead.

  ## Custom tile layers

  You can customize the tile layer used  by the maps using the `tile_layer` and `tile_layer_options` configuration settings. For example, to use the [OpenTopoMap](https://opentopomap.org/) you can use these settings:

  ```json
  {
      "plugins": {
          "datasette-cluster-map": {
              "tile_layer": "https://{s}.tile.opentopomap.org/{z}/{x}/{y}.png",
              "tile_layer_options": {
                  "attribution": "Map data: &copy; <a href='https://www.openstreetmap.org/copyright'>OpenStreetMap</a> contributors, <a href='http://viewfinderpanoramas.org'>SRTM</a> | Map style: &copy; <a href='https://opentopomap.org'>OpenTopoMap</a> (<a href='https://creativecommons.org/licenses/by-sa/3.0/'>CC-BY-SA</a>)",
                  "maxZoom": 17
              }
          }
      }
  }
  ```
  If your tile server provides retina tiles, add `"detectRetina": true` to `tile_layer_options` to enable them.

  The [Leaflet Providers preview list](https://leaflet-extras.github.io/leaflet-providers/preview/index.html) has details of many other tile layers you can use.

  ## Custom marker popups

  The marker popup defaults to displaying the data for the underlying database row.

  You can customize this by including a `popup` column in your results containing JSON that defines a more useful popup.

  The JSON in the popup column should look something like this:

  ```json
  {
      "image": "https://niche-museums.imgix.net/dodgems.heic?w=800&h=400&fit=crop",
      "alt": "Dingles Fairground Heritage Centre",
      "title": "Dingles Fairground Heritage Centre",
      "description": "Home of the National Fairground Collection, Dingles has over 45,000 indoor square feet of vintage fairground rides... and you can go on them! Highlights include the last complete surviving and opera",
      "link": "/browse/museums/26"
  }
  ```

  Each of these columns is optional.

  - `title` is the title to show at the top of the popup
  - `image` is the URL to an image to display in the popup
  - `alt` is the alt attribute to use for that image
  - `description` is a longer string of text to use as a description
  - `link` is a URL that the marker content should link to

  You can use the SQLite `json_object()` function to construct this data dynamically as part of your SQL query. Here's an example:

  ```sql
  select json_object(
    'image', photo_url || '?w=800&h=400&fit=crop',
    'title', name,
    'description', substr(description, 0, 200),
    'link', '/browse/museums/' || id
    ) as popup,
    latitude, longitude from museums
  where id in (26, 27) order by id
  ```

  [Try that example here](https://www.niche-museums.com/browse?sql=select+json_object%28%0D%0A++%27image%27%2C+photo_url+%7C%7C+%27%3Fw%3D800%26h%3D400%26fit%3Dcrop%27%2C%0D%0A++%27title%27%2C+name%2C%0D%0A++%27description%27%2C+substr%28description%2C+0%2C+200%29%2C%0D%0A++%27link%27%2C+%27%2Fbrowse%2Fmuseums%2F%27+%7C%7C+id%0D%0A++%29+as+popup%2C%0D%0A++latitude%2C+longitude+from+museums) or take a look at [this demo built using a SQL view](https://dogsheep-photos.dogsheep.net/public/photos_on_a_map).

  ## How I deployed the demo

      datasette publish cloudrun global-power-plants.db \
          --service global-power-plants \
          --metadata metadata.json \
          --install=datasette-cluster-map \
          --extra-options="--config facet_time_limit_ms:1000"

  ## Development

  To set up this plugin locally, first checkout the code. Then create a new virtual environment:

      cd datasette-cluster-map
      python3 -mvenv venv
      source venv/bin/activate

  Or if you are using `pipenv`:

      pipenv shell

  Now install the dependencies and tests:

      pip install -e '.[test]'

  To run the tests:

      pytest

readme_html:
  <div id="readme" class="md" data-path="README.md"><article class="markdown-body entry-content container-lg" itemprop="text"><div class="markdown-heading" dir="auto"><h1 class="heading-element" dir="auto">datasette-cluster-map</h1><a id="user-content-datasette-cluster-map" class="anchor" aria-label="Permalink: datasette-cluster-map" href="#datasette-cluster-map"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto"><a href="https://pypi.org/project/datasette-cluster-map/" rel="nofollow"><img src="https://img.shields.io/pypi/v/datasette-cluster-map.svg" alt="PyPI" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/660c153cf92a4cd1f0c7f75ca564eddacdd234d35a86f4a93741f8c21e71ca6b/68747470733a2f2f696d672e736869656c64732e696f2f707970692f762f6461746173657474652d636c75737465722d6d61702e737667"></a>
  <a href="https://github.com/simonw/datasette-cluster-map/releases"><img src="https://img.shields.io/github/v/release/simonw/datasette-cluster-map?include_prereleases&label=changelog" alt="Changelog" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/eea756791168d49c4fb62707ac50e78daf47bbd0dd468e6415b4dd419adef305/68747470733a2f2f696d672e736869656c64732e696f2f6769746875622f762f72656c656173652f73696d6f6e772f6461746173657474652d636c75737465722d6d61703f696e636c7564655f70726572656c6561736573266c6162656c3d6368616e67656c6f67"></a>
  <a href="https://github.com/simonw/datasette-cluster-map/blob/main/LICENSE"><img src="https://img.shields.io/badge/license-Apache%202.0-blue.svg" alt="License" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/c355f200ea90fddaa407b6eaab303663a669248ea3ca7b1fcf77dbe04ff5f48c/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6c6963656e73652d417061636865253230322e302d626c75652e737667"></a></p>
  <p dir="auto">A <a href="https://docs.datasette.io/en/stable/plugins.html" rel="nofollow">Datasette plugin</a> that detects tables with <code>latitude</code> and <code>longitude</code> columns and then plots them on a map using <a href="https://github.com/Leaflet/Leaflet.markercluster">Leaflet.markercluster</a>.</p>
  <p dir="auto">More about this project: <a href="https://simonwillison.net/2018/Apr/20/datasette-plugins/" rel="nofollow">Datasette plugins, and building a clustered map visualization</a>.</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Demo</h2><a id="user-content-demo" class="anchor" aria-label="Permalink: Demo" href="#demo"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto"><a href="https://global-power-plants.datasettes.com/global-power-plants/global-power-plants" rel="nofollow">global-power-plants.datasettes.com</a> hosts a demo of this plugin running against a database of 33,000 power plants around the world.</p>
  <p dir="auto"><a target="_blank" rel="noopener noreferrer nofollow" href="https://camo.githubusercontent.com/25ada088b85ce62c018a3b5e2597f669b4bbd61e46e43d86ff5635f84cc4f0f5/68747470733a2f2f7374617469632e73696d6f6e77696c6c69736f6e2e6e65742f7374617469632f323032302f676c6f62616c2d706f7765722d706c616e74732e706e67"><img src="https://static.simonwillison.net/static/2020/global-power-plants.png" alt="Cluster map demo" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/25ada088b85ce62c018a3b5e2597f669b4bbd61e46e43d86ff5635f84cc4f0f5/68747470733a2f2f7374617469632e73696d6f6e77696c6c69736f6e2e6e65742f7374617469632f323032302f676c6f62616c2d706f7765722d706c616e74732e706e67"></a></p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Installation</h2><a id="user-content-installation" class="anchor" aria-label="Permalink: Installation" href="#installation"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">Run <code>datasette install datasette-cluster-map</code> to add this plugin to your Datasette virtual environment. Datasette will automatically load the plugin if it is installed in this way.</p>
  <p dir="auto">If you are deploying using the <code>datasette publish</code> command you can use the <code>--install</code> option:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="datasette publish cloudrun mydb.db --install=datasette-cluster-map"><pre class="notranslate"><code>datasette publish cloudrun mydb.db --install=datasette-cluster-map
  </code></pre></div>
  <p dir="auto">If any of your tables have one of the following pairs of columns a map will be automatically displayed:</p>
  <ul dir="auto">
  <li><code>latitude</code> and <code>longitude</code></li>
  <li><code>lat</code> and <code>lng</code></li>
  <li><code>lat</code> and <code>lon</code></li>
  <li><code>lat</code> and <code>long</code></li>
  <li><code>*_latitude</code> and <code>*_longitude</code></li>
  <li><code>*_lat</code> and <code>*_lng</code> for any of the three variants of <code>lng</code></li>
  </ul>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Configuration</h2><a id="user-content-configuration" class="anchor" aria-label="Permalink: Configuration" href="#configuration"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">If your columns are called something else you can configure the column names using <a href="https://docs.datasette.io/en/stable/plugins.html#plugin-configuration" rel="nofollow">plugin configuration</a> in a <code>metadata.json</code> file. For example, if all of your columns are called <code>xlat</code> and <code>xlng</code> you can create a <code>metadata.json</code> file like this:</p>
  <div class="highlight highlight-source-json notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="{
      "title": "Regular metadata keys can go here too",
      "plugins": {
          "datasette-cluster-map": {
              "latitude_column": "xlat",
              "longitude_column": "xlng"
          }
      }
  }"><pre>{
      <span class="pl-ent">"title"</span>: <span class="pl-s"><span class="pl-pds">"</span>Regular metadata keys can go here too<span class="pl-pds">"</span></span>,
      <span class="pl-ent">"plugins"</span>: {
          <span class="pl-ent">"datasette-cluster-map"</span>: {
              <span class="pl-ent">"latitude_column"</span>: <span class="pl-s"><span class="pl-pds">"</span>xlat<span class="pl-pds">"</span></span>,
              <span class="pl-ent">"longitude_column"</span>: <span class="pl-s"><span class="pl-pds">"</span>xlng<span class="pl-pds">"</span></span>
          }
      }
  }</pre></div>
  <p dir="auto">Then run Datasette like this:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="datasette mydata.db -m metadata.json"><pre class="notranslate"><code>datasette mydata.db -m metadata.json
  </code></pre></div>
  <p dir="auto">This will configure the required column names for every database loaded by that Datasette instance.</p>
  <p dir="auto">If you want to customize the column names for just one table in one database, you can do something like this:</p>
  <div class="highlight highlight-source-json notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="{
      "databases": {
          "polar-bears": {
              "tables": {
                  "USGS_WC_eartag_deployments_2009-2011": {
                      "plugins": {
                          "datasette-cluster-map": {
                              "latitude_column": "Capture Latitude",
                              "longitude_column": "Capture Longitude"
                          }
                      }
                  }
              }
          }
      }
  }"><pre>{
      <span class="pl-ent">"databases"</span>: {
          <span class="pl-ent">"polar-bears"</span>: {
              <span class="pl-ent">"tables"</span>: {
                  <span class="pl-ent">"USGS_WC_eartag_deployments_2009-2011"</span>: {
                      <span class="pl-ent">"plugins"</span>: {
                          <span class="pl-ent">"datasette-cluster-map"</span>: {
                              <span class="pl-ent">"latitude_column"</span>: <span class="pl-s"><span class="pl-pds">"</span>Capture Latitude<span class="pl-pds">"</span></span>,
                              <span class="pl-ent">"longitude_column"</span>: <span class="pl-s"><span class="pl-pds">"</span>Capture Longitude<span class="pl-pds">"</span></span>
                          }
                      }
                  }
              }
          }
      }
  }</pre></div>
  <p dir="auto">You can also use a custom SQL query to rename those columns to <code>latitude</code> and <code>longitude</code>, <a href="https://polar-bears.now.sh/polar-bears?sql=select+*%2C%0D%0A++++%22Capture+Latitude%22+as+latitude%2C%0D%0A++++%22Capture+Longitude%22+as+longitude%0D%0Afrom+%5BUSGS_WC_eartag_deployments_2009-2011%5D" rel="nofollow">for example</a>:</p>
  <div class="highlight highlight-source-sql notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="select *,
      "Capture Latitude" as latitude,
      "Capture Longitude" as longitude
  from [USGS_WC_eartag_deployments_2009-2011]"><pre><span class="pl-k">select</span> <span class="pl-k">*</span>,
      <span class="pl-s"><span class="pl-pds">"</span>Capture Latitude<span class="pl-pds">"</span></span> <span class="pl-k">as</span> latitude,
      <span class="pl-s"><span class="pl-pds">"</span>Capture Longitude<span class="pl-pds">"</span></span> <span class="pl-k">as</span> longitude
  <span class="pl-k">from</span> [USGS_WC_eartag_deployments_2009<span class="pl-k">-</span><span class="pl-c1">2011</span>]</pre></div>
  <p dir="auto">The map defaults to being displayed above the main results table on the page. You can use the <code>"container"</code> plugin setting to provide a CSS selector indicating an element that the map should be appended to instead.</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Custom tile layers</h2><a id="user-content-custom-tile-layers" class="anchor" aria-label="Permalink: Custom tile layers" href="#custom-tile-layers"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">You can customize the tile layer used  by the maps using the <code>tile_layer</code> and <code>tile_layer_options</code> configuration settings. For example, to use the <a href="https://opentopomap.org/" rel="nofollow">OpenTopoMap</a> you can use these settings:</p>
  <div class="highlight highlight-source-json notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="{
      "plugins": {
          "datasette-cluster-map": {
              "tile_layer": "https://{s}.tile.opentopomap.org/{z}/{x}/{y}.png",
              "tile_layer_options": {
                  "attribution": "Map data: &copy; <a href='https://www.openstreetmap.org/copyright'>OpenStreetMap</a> contributors, <a href='http://viewfinderpanoramas.org'>SRTM</a> | Map style: &copy; <a href='https://opentopomap.org'>OpenTopoMap</a> (<a href='https://creativecommons.org/licenses/by-sa/3.0/'>CC-BY-SA</a>)",
                  "maxZoom": 17
              }
          }
      }
  }"><pre>{
      <span class="pl-ent">"plugins"</span>: {
          <span class="pl-ent">"datasette-cluster-map"</span>: {
              <span class="pl-ent">"tile_layer"</span>: <span class="pl-s"><span class="pl-pds">"</span>https://{s}.tile.opentopomap.org/{z}/{x}/{y}.png<span class="pl-pds">"</span></span>,
              <span class="pl-ent">"tile_layer_options"</span>: {
                  <span class="pl-ent">"attribution"</span>: <span class="pl-s"><span class="pl-pds">"</span>Map data: &copy; <a href='https://www.openstreetmap.org/copyright'>OpenStreetMap</a> contributors, <a href='http://viewfinderpanoramas.org'>SRTM</a> | Map style: &copy; <a href='https://opentopomap.org'>OpenTopoMap</a> (<a href='https://creativecommons.org/licenses/by-sa/3.0/'>CC-BY-SA</a>)<span class="pl-pds">"</span></span>,
                  <span class="pl-ent">"maxZoom"</span>: <span class="pl-c1">17</span>
              }
          }
      }
  }</pre></div>
  <p dir="auto">If your tile server provides retina tiles, add <code>"detectRetina": true</code> to <code>tile_layer_options</code> to enable them.</p>
  <p dir="auto">The <a href="https://leaflet-extras.github.io/leaflet-providers/preview/index.html" rel="nofollow">Leaflet Providers preview list</a> has details of many other tile layers you can use.</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Custom marker popups</h2><a id="user-content-custom-marker-popups" class="anchor" aria-label="Permalink: Custom marker popups" href="#custom-marker-popups"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">The marker popup defaults to displaying the data for the underlying database row.</p>
  <p dir="auto">You can customize this by including a <code>popup</code> column in your results containing JSON that defines a more useful popup.</p>
  <p dir="auto">The JSON in the popup column should look something like this:</p>
  <div class="highlight highlight-source-json notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="{
      "image": "https://niche-museums.imgix.net/dodgems.heic?w=800&h=400&fit=crop",
      "alt": "Dingles Fairground Heritage Centre",
      "title": "Dingles Fairground Heritage Centre",
      "description": "Home of the National Fairground Collection, Dingles has over 45,000 indoor square feet of vintage fairground rides... and you can go on them! Highlights include the last complete surviving and opera",
      "link": "/browse/museums/26"
  }"><pre>{
      <span class="pl-ent">"image"</span>: <span class="pl-s"><span class="pl-pds">"</span>https://niche-museums.imgix.net/dodgems.heic?w=800&h=400&fit=crop<span class="pl-pds">"</span></span>,
      <span class="pl-ent">"alt"</span>: <span class="pl-s"><span class="pl-pds">"</span>Dingles Fairground Heritage Centre<span class="pl-pds">"</span></span>,
      <span class="pl-ent">"title"</span>: <span class="pl-s"><span class="pl-pds">"</span>Dingles Fairground Heritage Centre<span class="pl-pds">"</span></span>,
      <span class="pl-ent">"description"</span>: <span class="pl-s"><span class="pl-pds">"</span>Home of the National Fairground Collection, Dingles has over 45,000 indoor square feet of vintage fairground rides... and you can go on them! Highlights include the last complete surviving and opera<span class="pl-pds">"</span></span>,
      <span class="pl-ent">"link"</span>: <span class="pl-s"><span class="pl-pds">"</span>/browse/museums/26<span class="pl-pds">"</span></span>
  }</pre></div>
  <p dir="auto">Each of these columns is optional.</p>
  <ul dir="auto">
  <li><code>title</code> is the title to show at the top of the popup</li>
  <li><code>image</code> is the URL to an image to display in the popup</li>
  <li><code>alt</code> is the alt attribute to use for that image</li>
  <li><code>description</code> is a longer string of text to use as a description</li>
  <li><code>link</code> is a URL that the marker content should link to</li>
  </ul>
  <p dir="auto">You can use the SQLite <code>json_object()</code> function to construct this data dynamically as part of your SQL query. Here's an example:</p>
  <div class="highlight highlight-source-sql notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="select json_object(
    'image', photo_url || '?w=800&h=400&fit=crop',
    'title', name,
    'description', substr(description, 0, 200),
    'link', '/browse/museums/' || id
    ) as popup,
    latitude, longitude from museums
  where id in (26, 27) order by id"><pre><span class="pl-k">select</span> json_object(
    <span class="pl-s"><span class="pl-pds">'</span>image<span class="pl-pds">'</span></span>, photo_url <span class="pl-k">||</span> <span class="pl-s"><span class="pl-pds">'</span>?w=800&h=400&fit=crop<span class="pl-pds">'</span></span>,
    <span class="pl-s"><span class="pl-pds">'</span>title<span class="pl-pds">'</span></span>, name,
    <span class="pl-s"><span class="pl-pds">'</span>description<span class="pl-pds">'</span></span>, substr(description, <span class="pl-c1">0</span>, <span class="pl-c1">200</span>),
    <span class="pl-s"><span class="pl-pds">'</span>link<span class="pl-pds">'</span></span>, <span class="pl-s"><span class="pl-pds">'</span>/browse/museums/<span class="pl-pds">'</span></span> <span class="pl-k">||</span> id
    ) <span class="pl-k">as</span> popup,
    latitude, longitude <span class="pl-k">from</span> museums
  <span class="pl-k">where</span> id <span class="pl-k">in</span> (<span class="pl-c1">26</span>, <span class="pl-c1">27</span>) <span class="pl-k">order by</span> id</pre></div>
  <p dir="auto"><a href="https://www.niche-museums.com/browse?sql=select+json_object%28%0D%0A++%27image%27%2C+photo_url+%7C%7C+%27%3Fw%3D800%26h%3D400%26fit%3Dcrop%27%2C%0D%0A++%27title%27%2C+name%2C%0D%0A++%27description%27%2C+substr%28description%2C+0%2C+200%29%2C%0D%0A++%27link%27%2C+%27%2Fbrowse%2Fmuseums%2F%27+%7C%7C+id%0D%0A++%29+as+popup%2C%0D%0A++latitude%2C+longitude+from+museums" rel="nofollow">Try that example here</a> or take a look at <a href="https://dogsheep-photos.dogsheep.net/public/photos_on_a_map" rel="nofollow">this demo built using a SQL view</a>.</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">How I deployed the demo</h2><a id="user-content-how-i-deployed-the-demo" class="anchor" aria-label="Permalink: How I deployed the demo" href="#how-i-deployed-the-demo"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="datasette publish cloudrun global-power-plants.db \
      --service global-power-plants \
      --metadata metadata.json \
      --install=datasette-cluster-map \
      --extra-options="--config facet_time_limit_ms:1000""><pre class="notranslate"><code>datasette publish cloudrun global-power-plants.db \
      --service global-power-plants \
      --metadata metadata.json \
      --install=datasette-cluster-map \
      --extra-options="--config facet_time_limit_ms:1000"
  </code></pre></div>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Development</h2><a id="user-content-development" class="anchor" aria-label="Permalink: Development" href="#development"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">To set up this plugin locally, first checkout the code. Then create a new virtual environment:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="cd datasette-cluster-map
  python3 -mvenv venv
  source venv/bin/activate"><pre class="notranslate"><code>cd datasette-cluster-map
  python3 -mvenv venv
  source venv/bin/activate
  </code></pre></div>
  <p dir="auto">Or if you are using <code>pipenv</code>:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="pipenv shell"><pre class="notranslate"><code>pipenv shell
  </code></pre></div>
  <p dir="auto">Now install the dependencies and tests:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="pip install -e '.[test]'"><pre class="notranslate"><code>pip install -e '.[test]'
  </code></pre></div>
  <p dir="auto">To run the tests:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="pytest"><pre class="notranslate"><code>pytest
  </code></pre></div>
  </article></div>


node_id: MDEwOlJlcG9zaXRvcnkxMzUwMDcyODc=
name: datasette-leaflet-geojson
full_name: simonw/datasette-leaflet-geojson
html_url: https://github.com/simonw/datasette-leaflet-geojson
description: Datasette plugin that replaces any GeoJSON column values with a Leaflet map.
created_at: 2018-05-27T01:42:30Z
updated_at: 2025-03-14T03:59:21Z
pushed_at: 2024-04-25T05:04:35Z
language: Python
topics: ["datasette", "datasette-io", "datasette-plugin", "gis", "leaflet"]
visibility: public
default_branch: main
permissions: {"admin": false, "maintain": false, "push": false, "triage": false, "pull": true}
readme:
  # datasette-leaflet-geojson

  [![PyPI](https://img.shields.io/pypi/v/datasette-leaflet-geojson.svg)](https://pypi.org/project/datasette-leaflet-geojson/)
  [![Changelog](https://img.shields.io/github/v/release/simonw/datasette-leaflet-geojson?include_prereleases&label=changelog)](https://github.com/simonw/datasette-leaflet-geojson/releases)
  [![Tests](https://github.com/simonw/datasette-leaflet-geojson/workflows/Test/badge.svg)](https://github.com/simonw/datasette-leaflet-geojson/actions?query=workflow%3ATest)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/datasette-leaflet-geojson/blob/main/LICENSE)

  Datasette plugin that replaces any GeoJSON column values with a Leaflet map

  ## Installation

  Install this plugin in the same environment as Datasette.

      $ datasette install datasette-leaflet-geojson

  ## Usage

  Any columns containing valid GeoJSON strings will have their contents replaced with a Leaflet map when they are displayed in the Datasette interface.

  ## Demo

  You can try this plugin out at https://calands.datasettes.com/calands/superunits_with_maps

  ![datasette-leaflet-geojson in action](https://raw.github.com/simonw/datasette-leaflet-geojson/main/datasette-leaflet-geojson.png)

  ## Configuration

  By default this plugin displays maps for the first ten rows, and shows a "Click to load map" prompt for rows past the first ten.

  You can change this limit using the `default_maps_to_load` plugin configuration setting. Add this to your `metadata.json`:

  ```json
  {
      "plugins": {
          "datasette-leaflet-geojson": {
              "default_maps_to_load": 20
          }
      }
  }
  ```
  Then run Datasette with `datasette mydb.db -m metadata.json`.

readme_html:
  <div id="readme" class="md" data-path="README.md"><article class="markdown-body entry-content container-lg" itemprop="text"><div class="markdown-heading" dir="auto"><h1 class="heading-element" dir="auto">datasette-leaflet-geojson</h1><a id="user-content-datasette-leaflet-geojson" class="anchor" aria-label="Permalink: datasette-leaflet-geojson" href="#datasette-leaflet-geojson"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto"><a href="https://pypi.org/project/datasette-leaflet-geojson/" rel="nofollow"><img src="https://img.shields.io/pypi/v/datasette-leaflet-geojson.svg" alt="PyPI" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/5db364a483b3a360e705cc327f1857f42f534e0d7a0a8c22260813134d6f7bdd/68747470733a2f2f696d672e736869656c64732e696f2f707970692f762f6461746173657474652d6c6561666c65742d67656f6a736f6e2e737667"></a>
  <a href="https://github.com/simonw/datasette-leaflet-geojson/releases"><img src="https://img.shields.io/github/v/release/simonw/datasette-leaflet-geojson?include_prereleases&label=changelog" alt="Changelog" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/424a7bf6cf2133039095c78623782c43777e87d1e7d673ac6128607150008ee0/68747470733a2f2f696d672e736869656c64732e696f2f6769746875622f762f72656c656173652f73696d6f6e772f6461746173657474652d6c6561666c65742d67656f6a736f6e3f696e636c7564655f70726572656c6561736573266c6162656c3d6368616e67656c6f67"></a>
  <a href="https://github.com/simonw/datasette-leaflet-geojson/actions?query=workflow%3ATest"><img src="https://github.com/simonw/datasette-leaflet-geojson/workflows/Test/badge.svg" alt="Tests" style="max-width: 100%;"></a>
  <a href="https://github.com/simonw/datasette-leaflet-geojson/blob/main/LICENSE"><img src="https://img.shields.io/badge/license-Apache%202.0-blue.svg" alt="License" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/c355f200ea90fddaa407b6eaab303663a669248ea3ca7b1fcf77dbe04ff5f48c/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6c6963656e73652d417061636865253230322e302d626c75652e737667"></a></p>
  <p dir="auto">Datasette plugin that replaces any GeoJSON column values with a Leaflet map</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Installation</h2><a id="user-content-installation" class="anchor" aria-label="Permalink: Installation" href="#installation"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">Install this plugin in the same environment as Datasette.</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ datasette install datasette-leaflet-geojson"><pre class="notranslate"><code>$ datasette install datasette-leaflet-geojson
  </code></pre></div>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Usage</h2><a id="user-content-usage" class="anchor" aria-label="Permalink: Usage" href="#usage"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">Any columns containing valid GeoJSON strings will have their contents replaced with a Leaflet map when they are displayed in the Datasette interface.</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Demo</h2><a id="user-content-demo" class="anchor" aria-label="Permalink: Demo" href="#demo"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">You can try this plugin out at <a href="https://calands.datasettes.com/calands/superunits_with_maps" rel="nofollow">https://calands.datasettes.com/calands/superunits_with_maps</a></p>
  <p dir="auto"><a target="_blank" rel="noopener noreferrer nofollow" href="https://camo.githubusercontent.com/4ba328083094f06ce1b30e7620dace7df4ecc8bbbf3f84535a495c3ad29f5adc/68747470733a2f2f7261772e6769746875622e636f6d2f73696d6f6e772f6461746173657474652d6c6561666c65742d67656f6a736f6e2f6d61696e2f6461746173657474652d6c6561666c65742d67656f6a736f6e2e706e67"><img src="https://raw.github.com/simonw/datasette-leaflet-geojson/main/datasette-leaflet-geojson.png" alt="datasette-leaflet-geojson in action" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/4ba328083094f06ce1b30e7620dace7df4ecc8bbbf3f84535a495c3ad29f5adc/68747470733a2f2f7261772e6769746875622e636f6d2f73696d6f6e772f6461746173657474652d6c6561666c65742d67656f6a736f6e2f6d61696e2f6461746173657474652d6c6561666c65742d67656f6a736f6e2e706e67"></a></p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Configuration</h2><a id="user-content-configuration" class="anchor" aria-label="Permalink: Configuration" href="#configuration"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">By default this plugin displays maps for the first ten rows, and shows a "Click to load map" prompt for rows past the first ten.</p>
  <p dir="auto">You can change this limit using the <code>default_maps_to_load</code> plugin configuration setting. Add this to your <code>metadata.json</code>:</p>
  <div class="highlight highlight-source-json notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="{
      "plugins": {
          "datasette-leaflet-geojson": {
              "default_maps_to_load": 20
          }
      }
  }"><pre>{
      <span class="pl-ent">"plugins"</span>: {
          <span class="pl-ent">"datasette-leaflet-geojson"</span>: {
              <span class="pl-ent">"default_maps_to_load"</span>: <span class="pl-c1">20</span>
          }
      }
  }</pre></div>
  <p dir="auto">Then run Datasette with <code>datasette mydb.db -m metadata.json</code>.</p>
  </article></div>


node_id: MDEwOlJlcG9zaXRvcnkxMzg2Njk2NzM=
name: datasette-vega
full_name: simonw/datasette-vega
html_url: https://github.com/simonw/datasette-vega
description: Datasette plugin for visualizing data using Vega
created_at: 2018-06-26T01:40:54Z
updated_at: 2025-11-02T18:27:04Z
pushed_at: 2025-11-02T18:27:01Z
language: JavaScript
license: apache-2.0
topics: ["datasette", "datasette-io", "datasette-plugin", "plugin", "react", "vega"]
visibility: public
default_branch: master
permissions: {"admin": false, "maintain": false, "push": false, "triage": false, "pull": false}
readme:
  # datasette-vega

  [![PyPI](https://img.shields.io/pypi/v/datasette-vega.svg)](https://pypi.org/project/datasette-vega/)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/datasette-vega/blob/master/LICENSE)

  A [Datasette](https://github.com/simonw/datasette) plugin that provides tools
  for generating charts using [Vega](https://vega.github.io/).

  ![Datasette Vega interface](https://raw.githubusercontent.com/simonw/datasette-vega/master/datasette-vega.png)

  Try out the latest master build as a live demo at https://datasette-vega-latest.datasette.io/ or try the latest release installed as a plugin at https://fivethirtyeight.datasettes.com/

  To add this to your Datasette installation, install the plugin like so:

      pip install datasette-vega

  The plugin will then add itself to every Datasette table view.

  If you are publishing data using the `datasette publish` command, you can
  include this plugin like so:

      datasette publish now mydatabase.db --install=datasette-vega

readme_html:
  <div id="readme" class="md" data-path="README.md"><article class="markdown-body entry-content container-lg" itemprop="text"><div class="markdown-heading" dir="auto"><h1 class="heading-element" dir="auto">datasette-vega</h1><a id="user-content-datasette-vega" class="anchor" aria-label="Permalink: datasette-vega" href="#datasette-vega"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto"><a href="https://pypi.org/project/datasette-vega/" rel="nofollow"><img src="https://img.shields.io/pypi/v/datasette-vega.svg" alt="PyPI" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/a89d96c7a21dc24e6662344cac1d8a6fc3b71e8f50f2733650ffa6784eba6447/68747470733a2f2f696d672e736869656c64732e696f2f707970692f762f6461746173657474652d766567612e737667"></a>
  <a href="https://github.com/simonw/datasette-vega/blob/master/LICENSE"><img src="https://img.shields.io/badge/license-Apache%202.0-blue.svg" alt="License" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/c355f200ea90fddaa407b6eaab303663a669248ea3ca7b1fcf77dbe04ff5f48c/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6c6963656e73652d417061636865253230322e302d626c75652e737667"></a></p>
  <p dir="auto">A <a href="https://github.com/simonw/datasette">Datasette</a> plugin that provides tools
  for generating charts using <a href="https://vega.github.io/" rel="nofollow">Vega</a>.</p>
  <p dir="auto"><a target="_blank" rel="noopener noreferrer nofollow" href="https://raw.githubusercontent.com/simonw/datasette-vega/master/datasette-vega.png"><img src="https://raw.githubusercontent.com/simonw/datasette-vega/master/datasette-vega.png" alt="Datasette Vega interface" style="max-width: 100%;"></a></p>
  <p dir="auto">Try out the latest master build as a live demo at <a href="https://datasette-vega-latest.datasette.io/" rel="nofollow">https://datasette-vega-latest.datasette.io/</a> or try the latest release installed as a plugin at <a href="https://fivethirtyeight.datasettes.com/" rel="nofollow">https://fivethirtyeight.datasettes.com/</a></p>
  <p dir="auto">To add this to your Datasette installation, install the plugin like so:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="pip install datasette-vega"><pre class="notranslate"><code>pip install datasette-vega
  </code></pre></div>
  <p dir="auto">The plugin will then add itself to every Datasette table view.</p>
  <p dir="auto">If you are publishing data using the <code>datasette publish</code> command, you can
  include this plugin like so:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="datasette publish now mydatabase.db --install=datasette-vega"><pre class="notranslate"><code>datasette publish now mydatabase.db --install=datasette-vega
  </code></pre></div>
  </article></div>


node_id: MDEwOlJlcG9zaXRvcnkxNDA5MTI0MzI=
name: sqlite-utils
full_name: simonw/sqlite-utils
html_url: https://github.com/simonw/sqlite-utils
description: Python CLI utility and library for manipulating SQLite databases
created_at: 2018-07-14T03:21:46Z
updated_at: 2026-05-20T16:30:41Z
pushed_at: 2026-05-17T23:52:48Z
homepage: https://sqlite-utils.datasette.io
language: Python
license: apache-2.0
visibility: public
default_branch: main
permissions: {"admin": false, "maintain": false, "push": false, "triage": false, "pull": false}
pull_request_creation_policy: all
topics:
  ["cli", "click", "datasette", "datasette-io", "datasette-tool", "python", "sqlite", "sqlite-database"]

readme:
  # sqlite-utils

  [![PyPI](https://img.shields.io/pypi/v/sqlite-utils.svg)](https://pypi.org/project/sqlite-utils/)
  [![Changelog](https://img.shields.io/github/v/release/simonw/sqlite-utils?include_prereleases&label=changelog)](https://sqlite-utils.datasette.io/en/stable/changelog.html)
  [![Python 3.x](https://img.shields.io/pypi/pyversions/sqlite-utils.svg?logo=python&logoColor=white)](https://pypi.org/project/sqlite-utils/)
  [![Tests](https://github.com/simonw/sqlite-utils/workflows/Test/badge.svg)](https://github.com/simonw/sqlite-utils/actions?query=workflow%3ATest)
  [![Documentation Status](https://readthedocs.org/projects/sqlite-utils/badge/?version=stable)](http://sqlite-utils.datasette.io/en/stable/?badge=stable)
  [![codecov](https://codecov.io/gh/simonw/sqlite-utils/branch/main/graph/badge.svg)](https://codecov.io/gh/simonw/sqlite-utils)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/sqlite-utils/blob/main/LICENSE)
  [![discord](https://img.shields.io/discord/823971286308356157?label=discord)](https://discord.gg/Ass7bCAMDw)

  Python CLI utility and library for manipulating SQLite databases.

  ## Some feature highlights

  - [Pipe JSON](https://sqlite-utils.datasette.io/en/stable/cli.html#inserting-json-data) (or [CSV or TSV](https://sqlite-utils.datasette.io/en/stable/cli.html#inserting-csv-or-tsv-data)) directly into a new SQLite database file, automatically creating a table with the appropriate schema
  - [Run in-memory SQL queries](https://sqlite-utils.datasette.io/en/stable/cli.html#querying-data-directly-using-an-in-memory-database), including joins, directly against data in CSV, TSV or JSON files and view the results
  - [Configure SQLite full-text search](https://sqlite-utils.datasette.io/en/stable/cli.html#configuring-full-text-search) against your database tables and run search queries against them, ordered by relevance
  - Run [transformations against your tables](https://sqlite-utils.datasette.io/en/stable/cli.html#transforming-tables) to make schema changes that SQLite `ALTER TABLE` does not directly support, such as changing the type of a column
  - [Extract columns](https://sqlite-utils.datasette.io/en/stable/cli.html#extracting-columns-into-a-separate-table) into separate tables to better normalize your existing data
  - [Install plugins](https://sqlite-utils.datasette.io/en/stable/plugins.html) to add custom SQL functions and additional features

  Read more on my blog, in this series of posts on [New features in sqlite-utils](https://simonwillison.net/series/sqlite-utils-features/) and other [entries tagged sqliteutils](https://simonwillison.net/tags/sqliteutils/).

  ## Installation

      pip install sqlite-utils

  Or if you use [Homebrew](https://brew.sh/) for macOS:

      brew install sqlite-utils

  ## Using as a CLI tool

  Now you can do things with the CLI utility like this:

      $ sqlite-utils memory dogs.csv "select * from t"
      [{"id": 1, "age": 4, "name": "Cleo"},
       {"id": 2, "age": 2, "name": "Pancakes"}]

      $ sqlite-utils insert dogs.db dogs dogs.csv --csv
      [####################################]  100%

      $ sqlite-utils tables dogs.db --counts
      [{"table": "dogs", "count": 2}]

      $ sqlite-utils dogs.db "select id, name from dogs"
      [{"id": 1, "name": "Cleo"},
       {"id": 2, "name": "Pancakes"}]

      $ sqlite-utils dogs.db "select * from dogs" --csv
      id,age,name
      1,4,Cleo
      2,2,Pancakes

      $ sqlite-utils dogs.db "select * from dogs" --table
        id    age  name
      ----  -----  --------
         1      4  Cleo
         2      2  Pancakes

  You can import JSON data into a new database table like this:

      $ curl https://api.github.com/repos/simonw/sqlite-utils/releases \
          | sqlite-utils insert releases.db releases - --pk id

  Or for data in a CSV file:

      $ sqlite-utils insert dogs.db dogs dogs.csv --csv

  `sqlite-utils memory` lets you import CSV or JSON data into an in-memory database and run SQL queries against it in a single command:

      $ cat dogs.csv | sqlite-utils memory - "select name, age from stdin"

  See the [full CLI documentation](https://sqlite-utils.datasette.io/en/stable/cli.html) for comprehensive coverage of many more commands.

  ## Using as a library

  You can also `import sqlite_utils` and use it as a Python library like this:

  ```python
  import sqlite_utils
  db = sqlite_utils.Database("demo_database.db")
  # This line creates a "dogs" table if one does not already exist:
  db["dogs"].insert_all([
      {"id": 1, "age": 4, "name": "Cleo"},
      {"id": 2, "age": 2, "name": "Pancakes"}
  ], pk="id")
  ```

  Check out the [full library documentation](https://sqlite-utils.datasette.io/en/stable/python-api.html) for everything else you can do with the Python library.

  ## Related projects

  * [Datasette](https://datasette.io/): A tool for exploring and publishing data
  * [csvs-to-sqlite](https://github.com/simonw/csvs-to-sqlite): Convert CSV files into a SQLite database
  * [db-to-sqlite](https://github.com/simonw/db-to-sqlite): CLI tool for exporting a MySQL or PostgreSQL database as a SQLite file
  * [dogsheep](https://dogsheep.github.io/): A family of tools for personal analytics, built on top of `sqlite-utils`

readme_html:
  <div id="readme" class="md" data-path="README.md"><article class="markdown-body entry-content container-lg" itemprop="text"><div class="markdown-heading" dir="auto"><h1 class="heading-element" dir="auto">sqlite-utils</h1><a id="user-content-sqlite-utils" class="anchor" aria-label="Permalink: sqlite-utils" href="#sqlite-utils"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto"><a href="https://pypi.org/project/sqlite-utils/" rel="nofollow"><img src="https://img.shields.io/pypi/v/sqlite-utils.svg" alt="PyPI" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/f0fdc22916d7814f085f303a7e71af95a03bb3443467f9ea1a33d6df693a3bb6/68747470733a2f2f696d672e736869656c64732e696f2f707970692f762f73716c6974652d7574696c732e737667"></a>
  <a href="https://sqlite-utils.datasette.io/en/stable/changelog.html" rel="nofollow"><img src="https://img.shields.io/github/v/release/simonw/sqlite-utils?include_prereleases&label=changelog" alt="Changelog" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/168846796f4f720b7e88978c1e677781a14f74c7940886a6f67aed4419977abe/68747470733a2f2f696d672e736869656c64732e696f2f6769746875622f762f72656c656173652f73696d6f6e772f73716c6974652d7574696c733f696e636c7564655f70726572656c6561736573266c6162656c3d6368616e67656c6f67"></a>
  <a href="https://pypi.org/project/sqlite-utils/" rel="nofollow"><img src="https://img.shields.io/pypi/pyversions/sqlite-utils.svg?logo=python&logoColor=white" alt="Python 3.x" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/62c68f7db551b609c8b020fc1072c2eddbb2e59ac33f30b6eb4a931e9f7d7dcf/68747470733a2f2f696d672e736869656c64732e696f2f707970692f707976657273696f6e732f73716c6974652d7574696c732e7376673f6c6f676f3d707974686f6e266c6f676f436f6c6f723d7768697465"></a>
  <a href="https://github.com/simonw/sqlite-utils/actions?query=workflow%3ATest"><img src="https://github.com/simonw/sqlite-utils/workflows/Test/badge.svg" alt="Tests" style="max-width: 100%;"></a>
  <a href="http://sqlite-utils.datasette.io/en/stable/?badge=stable" rel="nofollow"><img src="https://readthedocs.org/projects/sqlite-utils/badge/?version=stable" alt="Documentation Status" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/ed6f39d15e1e159f94594cc8e0a89d064b28a598bdb1134dd920c254da75b541/68747470733a2f2f72656164746865646f63732e6f72672f70726f6a656374732f73716c6974652d7574696c732f62616467652f3f76657273696f6e3d737461626c65"></a>
  <a href="https://codecov.io/gh/simonw/sqlite-utils" rel="nofollow"><img src="https://codecov.io/gh/simonw/sqlite-utils/branch/main/graph/badge.svg" alt="codecov" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/c748046f78d5cb475ef7fa682b9d2638427d5fb5ec085109e5890c5187e76b70/68747470733a2f2f636f6465636f762e696f2f67682f73696d6f6e772f73716c6974652d7574696c732f6272616e63682f6d61696e2f67726170682f62616467652e737667"></a>
  <a href="https://github.com/simonw/sqlite-utils/blob/main/LICENSE"><img src="https://img.shields.io/badge/license-Apache%202.0-blue.svg" alt="License" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/c355f200ea90fddaa407b6eaab303663a669248ea3ca7b1fcf77dbe04ff5f48c/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6c6963656e73652d417061636865253230322e302d626c75652e737667"></a>
  <a href="https://discord.gg/Ass7bCAMDw" rel="nofollow"><img src="https://img.shields.io/discord/823971286308356157?label=discord" alt="discord" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/94e396359356f1b1dd8fed242a2f694cb88a3b94a9eef104404225094357e912/68747470733a2f2f696d672e736869656c64732e696f2f646973636f72642f3832333937313238363330383335363135373f6c6162656c3d646973636f7264"></a></p>
  <p dir="auto">Python CLI utility and library for manipulating SQLite databases.</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Some feature highlights</h2><a id="user-content-some-feature-highlights" class="anchor" aria-label="Permalink: Some feature highlights" href="#some-feature-highlights"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <ul dir="auto">
  <li><a href="https://sqlite-utils.datasette.io/en/stable/cli.html#inserting-json-data" rel="nofollow">Pipe JSON</a> (or <a href="https://sqlite-utils.datasette.io/en/stable/cli.html#inserting-csv-or-tsv-data" rel="nofollow">CSV or TSV</a>) directly into a new SQLite database file, automatically creating a table with the appropriate schema</li>
  <li><a href="https://sqlite-utils.datasette.io/en/stable/cli.html#querying-data-directly-using-an-in-memory-database" rel="nofollow">Run in-memory SQL queries</a>, including joins, directly against data in CSV, TSV or JSON files and view the results</li>
  <li><a href="https://sqlite-utils.datasette.io/en/stable/cli.html#configuring-full-text-search" rel="nofollow">Configure SQLite full-text search</a> against your database tables and run search queries against them, ordered by relevance</li>
  <li>Run <a href="https://sqlite-utils.datasette.io/en/stable/cli.html#transforming-tables" rel="nofollow">transformations against your tables</a> to make schema changes that SQLite <code>ALTER TABLE</code> does not directly support, such as changing the type of a column</li>
  <li><a href="https://sqlite-utils.datasette.io/en/stable/cli.html#extracting-columns-into-a-separate-table" rel="nofollow">Extract columns</a> into separate tables to better normalize your existing data</li>
  <li><a href="https://sqlite-utils.datasette.io/en/stable/plugins.html" rel="nofollow">Install plugins</a> to add custom SQL functions and additional features</li>
  </ul>
  <p dir="auto">Read more on my blog, in this series of posts on <a href="https://simonwillison.net/series/sqlite-utils-features/" rel="nofollow">New features in sqlite-utils</a> and other <a href="https://simonwillison.net/tags/sqliteutils/" rel="nofollow">entries tagged sqliteutils</a>.</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Installation</h2><a id="user-content-installation" class="anchor" aria-label="Permalink: Installation" href="#installation"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="pip install sqlite-utils"><pre class="notranslate"><code>pip install sqlite-utils
  </code></pre></div>
  <p dir="auto">Or if you use <a href="https://brew.sh/" rel="nofollow">Homebrew</a> for macOS:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="brew install sqlite-utils"><pre class="notranslate"><code>brew install sqlite-utils
  </code></pre></div>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Using as a CLI tool</h2><a id="user-content-using-as-a-cli-tool" class="anchor" aria-label="Permalink: Using as a CLI tool" href="#using-as-a-cli-tool"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">Now you can do things with the CLI utility like this:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ sqlite-utils memory dogs.csv "select * from t"
  [{"id": 1, "age": 4, "name": "Cleo"},
   {"id": 2, "age": 2, "name": "Pancakes"}]

  $ sqlite-utils insert dogs.db dogs dogs.csv --csv
  [####################################]  100%

  $ sqlite-utils tables dogs.db --counts
  [{"table": "dogs", "count": 2}]

  $ sqlite-utils dogs.db "select id, name from dogs"
  [{"id": 1, "name": "Cleo"},
   {"id": 2, "name": "Pancakes"}]

  $ sqlite-utils dogs.db "select * from dogs" --csv
  id,age,name
  1,4,Cleo
  2,2,Pancakes

  $ sqlite-utils dogs.db "select * from dogs" --table
    id    age  name
  ----  -----  --------
     1      4  Cleo
     2      2  Pancakes"><pre class="notranslate"><code>$ sqlite-utils memory dogs.csv "select * from t"
  [{"id": 1, "age": 4, "name": "Cleo"},
   {"id": 2, "age": 2, "name": "Pancakes"}]

  $ sqlite-utils insert dogs.db dogs dogs.csv --csv
  [####################################]  100%

  $ sqlite-utils tables dogs.db --counts
  [{"table": "dogs", "count": 2}]

  $ sqlite-utils dogs.db "select id, name from dogs"
  [{"id": 1, "name": "Cleo"},
   {"id": 2, "name": "Pancakes"}]

  $ sqlite-utils dogs.db "select * from dogs" --csv
  id,age,name
  1,4,Cleo
  2,2,Pancakes

  $ sqlite-utils dogs.db "select * from dogs" --table
    id    age  name
  ----  -----  --------
     1      4  Cleo
     2      2  Pancakes
  </code></pre></div>
  <p dir="auto">You can import JSON data into a new database table like this:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ curl https://api.github.com/repos/simonw/sqlite-utils/releases \
      | sqlite-utils insert releases.db releases - --pk id"><pre class="notranslate"><code>$ curl https://api.github.com/repos/simonw/sqlite-utils/releases \
      | sqlite-utils insert releases.db releases - --pk id
  </code></pre></div>
  <p dir="auto">Or for data in a CSV file:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ sqlite-utils insert dogs.db dogs dogs.csv --csv"><pre class="notranslate"><code>$ sqlite-utils insert dogs.db dogs dogs.csv --csv
  </code></pre></div>
  <p dir="auto"><code>sqlite-utils memory</code> lets you import CSV or JSON data into an in-memory database and run SQL queries against it in a single command:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ cat dogs.csv | sqlite-utils memory - "select name, age from stdin""><pre class="notranslate"><code>$ cat dogs.csv | sqlite-utils memory - "select name, age from stdin"
  </code></pre></div>
  <p dir="auto">See the <a href="https://sqlite-utils.datasette.io/en/stable/cli.html" rel="nofollow">full CLI documentation</a> for comprehensive coverage of many more commands.</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Using as a library</h2><a id="user-content-using-as-a-library" class="anchor" aria-label="Permalink: Using as a library" href="#using-as-a-library"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">You can also <code>import sqlite_utils</code> and use it as a Python library like this:</p>
  <div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="import sqlite_utils
  db = sqlite_utils.Database("demo_database.db")
  # This line creates a "dogs" table if one does not already exist:
  db["dogs"].insert_all([
      {"id": 1, "age": 4, "name": "Cleo"},
      {"id": 2, "age": 2, "name": "Pancakes"}
  ], pk="id")"><pre><span class="pl-k">import</span> <span class="pl-s1">sqlite_utils</span>
  <span class="pl-s1">db</span> <span class="pl-c1">=</span> <span class="pl-s1">sqlite_utils</span>.<span class="pl-c1">Database</span>(<span class="pl-s">"demo_database.db"</span>)
  <span class="pl-c"># This line creates a "dogs" table if one does not already exist:</span>
  <span class="pl-s1">db</span>[<span class="pl-s">"dogs"</span>].<span class="pl-c1">insert_all</span>([
      {<span class="pl-s">"id"</span>: <span class="pl-c1">1</span>, <span class="pl-s">"age"</span>: <span class="pl-c1">4</span>, <span class="pl-s">"name"</span>: <span class="pl-s">"Cleo"</span>},
      {<span class="pl-s">"id"</span>: <span class="pl-c1">2</span>, <span class="pl-s">"age"</span>: <span class="pl-c1">2</span>, <span class="pl-s">"name"</span>: <span class="pl-s">"Pancakes"</span>}
  ], <span class="pl-s1">pk</span><span class="pl-c1">=</span><span class="pl-s">"id"</span>)</pre></div>
  <p dir="auto">Check out the <a href="https://sqlite-utils.datasette.io/en/stable/python-api.html" rel="nofollow">full library documentation</a> for everything else you can do with the Python library.</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Related projects</h2><a id="user-content-related-projects" class="anchor" aria-label="Permalink: Related projects" href="#related-projects"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <ul dir="auto">
  <li><a href="https://datasette.io/" rel="nofollow">Datasette</a>: A tool for exploring and publishing data</li>
  <li><a href="https://github.com/simonw/csvs-to-sqlite">csvs-to-sqlite</a>: Convert CSV files into a SQLite database</li>
  <li><a href="https://github.com/simonw/db-to-sqlite">db-to-sqlite</a>: CLI tool for exporting a MySQL or PostgreSQL database as a SQLite file</li>
  <li><a href="https://dogsheep.github.io/" rel="nofollow">dogsheep</a>: A family of tools for personal analytics, built on top of <code>sqlite-utils</code></li>
  </ul>
  </article></div>


node_id: MDEwOlJlcG9zaXRvcnkxNDI5NjczNDc=
name: datasette-json-html
full_name: simonw/datasette-json-html
html_url: https://github.com/simonw/datasette-json-html
description: Datasette plugin for rendering HTML based on JSON values
created_at: 2018-07-31T05:41:39Z
updated_at: 2024-09-04T07:35:36Z
pushed_at: 2022-03-22T01:43:59Z
language: Python
license: apache-2.0
topics: ["datasette", "datasette-io", "datasette-plugin", "plugin"]
visibility: public
default_branch: main
permissions: {"admin": false, "maintain": false, "push": false, "triage": false, "pull": true}
readme:
  # datasette-json-html

  [![PyPI](https://img.shields.io/pypi/v/datasette-json-html.svg)](https://pypi.org/project/datasette-json-html/)
  [![Changelog](https://img.shields.io/github/v/release/simonw/datasette-json-html?include_prereleases&label=changelog)](https://github.com/simonw/datasette-json-html/releases)
  [![Tests](https://github.com/simonw/datasette-json-html/workflows/Test/badge.svg)](https://github.com/simonw/datasette-remote-metadata/actions?query=workflow%3ATest)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/datasette-json-html/blob/main/LICENSE)

  Datasette plugin for rendering HTML based on JSON values, using the [render_cell plugin hook](https://docs.datasette.io/en/stable/plugin_hooks.html#render-cell-value-column-table-database-datasette).

  This plugin looks for cell values that match a very specific JSON format and converts them into HTML when they are rendered by the Datasette interface.

  ## Links

      {
          "href": "https://simonwillison.net/",
          "label": "Simon Willison"
      }

  Will be rendered as an `<a href="">` link:

      <a href="https://simonwillison.net/">Simon Willison</a>

  You can set a tooltip on the link using a `"title"` key:


      {
          "href": "https://simonwillison.net/",
          "label": "Simon Willison",
          "title": "My blog"
      }

  Produces:

      <a href="https://simonwillison.net/" title="My blog">Simon Willison</a>

  You can also include a description, which will be displayed below the link. If descriptions include newlines they will be converted to `<br>` elements:

      select json_object(
          "href", "https://simonwillison.net/",
          "label", "Simon Willison",
          "description", "This can contain" || x'0a' || "newlines"
      )

  Produces:

      <strong><a href="https://simonwillison.net/">Simon Willison</a></strong><br>This can contain<br>newlines

  * [Literal JSON link demo](https://datasette-json-html.datasette.io/demo?sql=select+%27%7B%0D%0A++++%22href%22%3A+%22https%3A%2F%2Fsimonwillison.net%2F%22%2C%0D%0A++++%22label%22%3A+%22Simon+Willison%22%2C%0D%0A++++%22title%22%3A+%22My+blog%22%0D%0A%7D%27)

  ## List of links

      [
          {
              "href": "https://simonwillison.net/",
              "label": "Simon Willison"
          },
          {
              "href": "https://github.com/simonw/datasette",
              "label": "Datasette"
          }
      ]

  Will be rendered as a comma-separated list of `<a href="">` links:

      <a href="https://simonwillison.net/">Simon Willison</a>,
      <a href="https://github.com/simonw/datasette">Datasette</a>

  The `href` property must begin with `https://` or `http://` or `/`, to avoid potential XSS injection attacks (for example URLs that begin with `javascript:`).

  Lists of links cannot include `"description"` keys.

  * [Literal list of links demo](https://datasette-json-html.datasette.io/demo?sql=select+%27%5B%0D%0A++++%7B%0D%0A++++++++%22href%22%3A+%22https%3A%2F%2Fsimonwillison.net%2F%22%2C%0D%0A++++++++%22label%22%3A+%22Simon+Willison%22%0D%0A++++%7D%2C%0D%0A++++%7B%0D%0A++++++++%22href%22%3A+%22https%3A%2F%2Fgithub.com%2Fsimonw%2Fdatasette%22%2C%0D%0A++++++++%22label%22%3A+%22Datasette%22%0D%0A++++%7D%0D%0A%5D%27)

  ## Images

  The image tag is more complex. The most basic version looks like this:

      {
          "img_src": "https://placekitten.com/200/300"
      }

  This will render as:

      <img src="https://placekitten.com/200/300">

  But you can also include one or more of `alt`, `caption`, `width` and `href`.

  If you include width or alt, they will be added as attributes:

      {
          "img_src": "https://placekitten.com/200/300",
          "alt": "Kitten",
          "width": 200
      }

  Produces:

      <img src="https://placekitten.com/200/300"
          alt="Kitten" width="200">

  * [Literal image demo](https://datasette-json-html.datasette.io/demo?sql=select+%27%7B%0D%0A++++%22img_src%22%3A+%22https%3A%2F%2Fplacekitten.com%2F200%2F300%22%2C%0D%0A++++%22alt%22%3A+%22Kitten%22%2C%0D%0A++++%22width%22%3A+200%0D%0A%7D%27)

  The `href` key will cause the image to be wrapped in a link:

      {
          "img_src": "https://placekitten.com/200/300",
          "href": "http://www.example.com"
      }

  Produces:

      <a href="http://www.example.com">
          <img src="https://placekitten.com/200/300">
      </a>

  The `caption` key wraps everything in a fancy figure/figcaption block:

      {
          "img_src": "https://placekitten.com/200/300",
          "caption": "Kitten caption"
      }

  Produces:

      <figure>
          <img src="https://placekitten.com/200/300"></a>
          <figcaption>Kitten caption</figcaption>
      </figure>

  ## Preformatted text

  You can use `{"pre": "text"}` to render text in a `<pre>` HTML tag:

      {
          "pre": "This\nhas\nnewlines"
      }

  Produces:

      <pre>This
      has
      newlines</pre>

  If the value attached to the `"pre"` key is itself a JSON object, that JSON will be pretty-printed:

      {
          "pre": {
              "this": {
                  "object": ["is", "nested"]
              }
          }
      }

  Produces:

      <pre>{
        &#34;this&#34;: {
          &#34;object&#34;: [
            &#34;is&#34;,
            &#34;nested&#34;
          ]
        }
      }</pre>

  * [Preformatted text with JSON demo](https://datasette-json-html.datasette.io/demo?sql=select+%27%7B%0D%0A++++%22pre%22%3A+%7B%0D%0A++++++++%22this%22%3A+%7B%0D%0A++++++++++++%22object%22%3A+%5B%22is%22%2C+%22nested%22%5D%0D%0A++++++++%7D%0D%0A++++%7D%0D%0A%7D%27)
  * [Preformatted text demo showing the Mandelbrot Set](https://datasette-json-html.datasette.io/demo?sql=WITH+RECURSIVE%0D%0A++xaxis%28x%29+AS+%28VALUES%28-2.0%29+UNION+ALL+SELECT+x%2B0.05+FROM+xaxis+WHERE+x%3C1.2%29%2C%0D%0A++yaxis%28y%29+AS+%28VALUES%28-1.0%29+UNION+ALL+SELECT+y%2B0.1+FROM+yaxis+WHERE+y%3C1.0%29%2C%0D%0A++m%28iter%2C+cx%2C+cy%2C+x%2C+y%29+AS+%28%0D%0A++++SELECT+0%2C+x%2C+y%2C+0.0%2C+0.0+FROM+xaxis%2C+yaxis%0D%0A++++UNION+ALL%0D%0A++++SELECT+iter%2B1%2C+cx%2C+cy%2C+x*x-y*y+%2B+cx%2C+2.0*x*y+%2B+cy+FROM+m+%0D%0A+++++WHERE+%28x*x+%2B+y*y%29+%3C+4.0+AND+iter%3C28%0D%0A++%29%2C%0D%0A++m2%28iter%2C+cx%2C+cy%29+AS+%28%0D%0A++++SELECT+max%28iter%29%2C+cx%2C+cy+FROM+m+GROUP+BY+cx%2C+cy%0D%0A++%29%2C%0D%0A++a%28t%29+AS+%28%0D%0A++++SELECT+group_concat%28+substr%28%27+.%2B*%23%27%2C+1%2Bmin%28iter%2F7%2C4%29%2C+1%29%2C+%27%27%29+%0D%0A++++FROM+m2+GROUP+BY+cy%0D%0A++%29%0D%0ASELECT+json_object%28%27pre%27%2C+group_concat%28rtrim%28t%29%2Cx%270a%27%29%29+FROM+a%3B) using [this example](https://www.sqlite.org/lang_with.html#outlandish_recursive_query_examples) from the SQLite documentation

  ## Using these with SQLite JSON functions

  The most powerful way to make use of this plugin is in conjunction with SQLite's [JSON functions](https://www.sqlite.org/json1.html). For example:

      select json_object(
          "href", "https://simonwillison.net/",
          "label", "Simon Willison"
      );

  * [json_object() link demo](https://datasette-json-html.datasette.io/demo?sql=select+json_object%28%0D%0A++++%22href%22%2C+%22https%3A%2F%2Fsimonwillison.net%2F%22%2C%0D%0A++++%22label%22%2C+%22Simon+Willison%22%0D%0A%29%3B)

  You can use these functions to construct JSON objects that work with the plugin from data in a table:

      select id, json_object(
          "href", url, "label", text
      ) from mytable;

  * [Demo that builds links against a table](https://datasette-json-html.datasette.io/demo?sql=select+json_object%28%22href%22%2C+url%2C+%22label%22%2C+package%2C+%22title%22%2C+package+%7C%7C+%22+%22+%7C%7C+url%29+as+package+from+packages)

  The `json_group_array()` function is an aggregate function similar to `group_concat()` - it allows you to construct lists of JSON objects in conjunction with a `GROUP BY` clause.

  This means you can use it to construct dynamic lists of links, for example:

      select
          substr(package, 0, 12) as prefix,
          json_group_array(
              json_object(
                  "href", url,
                  "label", package
              )
          ) as package_links
      from packages
      group by prefix

  * [Demo of json_group_array()](https://datasette-json-html.datasette.io/demo?sql=select%0D%0A++++substr%28package%2C+0%2C+12%29+as+prefix%2C%0D%0A++++json_group_array%28%0D%0A++++++++json_object%28%0D%0A++++++++++++%22href%22%2C+url%2C%0D%0A++++++++++++%22label%22%2C+package%0D%0A++++++++%29%0D%0A++++%29+as+package_links%0D%0Afrom+packages%0D%0Agroup+by+prefix)

  ## The `urllib_quote_plus()` SQL function

  Since this plugin is designed to be used with SQL that constructs the underlying JSON structure, it is likely you will need to construct dynamic URLs from results returned by a SQL query.

  This plugin registers a custom SQLite function called `urllib_quote_plus()` to help you do that. It lets you use Python's [urllib.parse.quote\_plus() function](https://docs.python.org/3/library/urllib.parse.html#urllib.parse.quote_plus) from within a SQL query.

  Here's an example of how you might use it:

      select id, json_object(
          "href",
          "/mydatabase/other_table?_search=" || urllib_quote_plus(text),
          "label", text
      ) from mytable;

readme_html:
  <div id="readme" class="md" data-path="README.md"><article class="markdown-body entry-content container-lg" itemprop="text"><div class="markdown-heading" dir="auto"><h1 class="heading-element" dir="auto">datasette-json-html</h1><a id="user-content-datasette-json-html" class="anchor" aria-label="Permalink: datasette-json-html" href="#datasette-json-html"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto"><a href="https://pypi.org/project/datasette-json-html/" rel="nofollow"><img src="https://img.shields.io/pypi/v/datasette-json-html.svg" alt="PyPI" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/0ba84cbf350e26d32701ac23d28024488afe2073e4c0557dfccb7c4623792221/68747470733a2f2f696d672e736869656c64732e696f2f707970692f762f6461746173657474652d6a736f6e2d68746d6c2e737667"></a>
  <a href="https://github.com/simonw/datasette-json-html/releases"><img src="https://img.shields.io/github/v/release/simonw/datasette-json-html?include_prereleases&label=changelog" alt="Changelog" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/a6be05629ebfb4b6c04d44064ed5769c00f9c442bf5e07a5dd1d0f4e3daf19ab/68747470733a2f2f696d672e736869656c64732e696f2f6769746875622f762f72656c656173652f73696d6f6e772f6461746173657474652d6a736f6e2d68746d6c3f696e636c7564655f70726572656c6561736573266c6162656c3d6368616e67656c6f67"></a>
  <a href="https://github.com/simonw/datasette-remote-metadata/actions?query=workflow%3ATest"><img src="https://github.com/simonw/datasette-json-html/workflows/Test/badge.svg" alt="Tests" style="max-width: 100%;"></a>
  <a href="https://github.com/simonw/datasette-json-html/blob/main/LICENSE"><img src="https://img.shields.io/badge/license-Apache%202.0-blue.svg" alt="License" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/c355f200ea90fddaa407b6eaab303663a669248ea3ca7b1fcf77dbe04ff5f48c/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6c6963656e73652d417061636865253230322e302d626c75652e737667"></a></p>
  <p dir="auto">Datasette plugin for rendering HTML based on JSON values, using the <a href="https://docs.datasette.io/en/stable/plugin_hooks.html#render-cell-value-column-table-database-datasette" rel="nofollow">render_cell plugin hook</a>.</p>
  <p dir="auto">This plugin looks for cell values that match a very specific JSON format and converts them into HTML when they are rendered by the Datasette interface.</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Links</h2><a id="user-content-links" class="anchor" aria-label="Permalink: Links" href="#links"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="{
      "href": "https://simonwillison.net/",
      "label": "Simon Willison"
  }"><pre class="notranslate"><code>{
      "href": "https://simonwillison.net/",
      "label": "Simon Willison"
  }
  </code></pre></div>
  <p dir="auto">Will be rendered as an <code><a href=""></code> link:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="<a href="https://simonwillison.net/">Simon Willison</a>"><pre class="notranslate"><code><a href="https://simonwillison.net/">Simon Willison</a>
  </code></pre></div>
  <p dir="auto">You can set a tooltip on the link using a <code>"title"</code> key:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="{
      "href": "https://simonwillison.net/",
      "label": "Simon Willison",
      "title": "My blog"
  }"><pre class="notranslate"><code>{
      "href": "https://simonwillison.net/",
      "label": "Simon Willison",
      "title": "My blog"
  }
  </code></pre></div>
  <p dir="auto">Produces:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="<a href="https://simonwillison.net/" title="My blog">Simon Willison</a>"><pre class="notranslate"><code><a href="https://simonwillison.net/" title="My blog">Simon Willison</a>
  </code></pre></div>
  <p dir="auto">You can also include a description, which will be displayed below the link. If descriptions include newlines they will be converted to <code><br></code> elements:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="select json_object(
      "href", "https://simonwillison.net/",
      "label", "Simon Willison",
      "description", "This can contain" || x'0a' || "newlines"
  )"><pre class="notranslate"><code>select json_object(
      "href", "https://simonwillison.net/",
      "label", "Simon Willison",
      "description", "This can contain" || x'0a' || "newlines"
  )
  </code></pre></div>
  <p dir="auto">Produces:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="<strong><a href="https://simonwillison.net/">Simon Willison</a></strong><br>This can contain<br>newlines"><pre class="notranslate"><code><strong><a href="https://simonwillison.net/">Simon Willison</a></strong><br>This can contain<br>newlines
  </code></pre></div>
  <ul dir="auto">
  <li><a href="https://datasette-json-html.datasette.io/demo?sql=select+%27%7B%0D%0A++++%22href%22%3A+%22https%3A%2F%2Fsimonwillison.net%2F%22%2C%0D%0A++++%22label%22%3A+%22Simon+Willison%22%2C%0D%0A++++%22title%22%3A+%22My+blog%22%0D%0A%7D%27" rel="nofollow">Literal JSON link demo</a></li>
  </ul>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">List of links</h2><a id="user-content-list-of-links" class="anchor" aria-label="Permalink: List of links" href="#list-of-links"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="[
      {
          "href": "https://simonwillison.net/",
          "label": "Simon Willison"
      },
      {
          "href": "https://github.com/simonw/datasette",
          "label": "Datasette"
      }
  ]"><pre class="notranslate"><code>[
      {
          "href": "https://simonwillison.net/",
          "label": "Simon Willison"
      },
      {
          "href": "https://github.com/simonw/datasette",
          "label": "Datasette"
      }
  ]
  </code></pre></div>
  <p dir="auto">Will be rendered as a comma-separated list of <code><a href=""></code> links:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="<a href="https://simonwillison.net/">Simon Willison</a>,
  <a href="https://github.com/simonw/datasette">Datasette</a>"><pre class="notranslate"><code><a href="https://simonwillison.net/">Simon Willison</a>,
  <a href="https://github.com/simonw/datasette">Datasette</a>
  </code></pre></div>
  <p dir="auto">The <code>href</code> property must begin with <code>https://</code> or <code>http://</code> or <code>/</code>, to avoid potential XSS injection attacks (for example URLs that begin with <code>javascript:</code>).</p>
  <p dir="auto">Lists of links cannot include <code>"description"</code> keys.</p>
  <ul dir="auto">
  <li><a href="https://datasette-json-html.datasette.io/demo?sql=select+%27%5B%0D%0A++++%7B%0D%0A++++++++%22href%22%3A+%22https%3A%2F%2Fsimonwillison.net%2F%22%2C%0D%0A++++++++%22label%22%3A+%22Simon+Willison%22%0D%0A++++%7D%2C%0D%0A++++%7B%0D%0A++++++++%22href%22%3A+%22https%3A%2F%2Fgithub.com%2Fsimonw%2Fdatasette%22%2C%0D%0A++++++++%22label%22%3A+%22Datasette%22%0D%0A++++%7D%0D%0A%5D%27" rel="nofollow">Literal list of links demo</a></li>
  </ul>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Images</h2><a id="user-content-images" class="anchor" aria-label="Permalink: Images" href="#images"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">The image tag is more complex. The most basic version looks like this:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="{
      "img_src": "https://placekitten.com/200/300"
  }"><pre class="notranslate"><code>{
      "img_src": "https://placekitten.com/200/300"
  }
  </code></pre></div>
  <p dir="auto">This will render as:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="<img src="https://placekitten.com/200/300">"><pre class="notranslate"><code><img src="https://placekitten.com/200/300">
  </code></pre></div>
  <p dir="auto">But you can also include one or more of <code>alt</code>, <code>caption</code>, <code>width</code> and <code>href</code>.</p>
  <p dir="auto">If you include width or alt, they will be added as attributes:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="{
      "img_src": "https://placekitten.com/200/300",
      "alt": "Kitten",
      "width": 200
  }"><pre class="notranslate"><code>{
      "img_src": "https://placekitten.com/200/300",
      "alt": "Kitten",
      "width": 200
  }
  </code></pre></div>
  <p dir="auto">Produces:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="<img src="https://placekitten.com/200/300"
      alt="Kitten" width="200">"><pre class="notranslate"><code><img src="https://placekitten.com/200/300"
      alt="Kitten" width="200">
  </code></pre></div>
  <ul dir="auto">
  <li><a href="https://datasette-json-html.datasette.io/demo?sql=select+%27%7B%0D%0A++++%22img_src%22%3A+%22https%3A%2F%2Fplacekitten.com%2F200%2F300%22%2C%0D%0A++++%22alt%22%3A+%22Kitten%22%2C%0D%0A++++%22width%22%3A+200%0D%0A%7D%27" rel="nofollow">Literal image demo</a></li>
  </ul>
  <p dir="auto">The <code>href</code> key will cause the image to be wrapped in a link:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="{
      "img_src": "https://placekitten.com/200/300",
      "href": "http://www.example.com"
  }"><pre class="notranslate"><code>{
      "img_src": "https://placekitten.com/200/300",
      "href": "http://www.example.com"
  }
  </code></pre></div>
  <p dir="auto">Produces:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="<a href="http://www.example.com">
      <img src="https://placekitten.com/200/300">
  </a>"><pre class="notranslate"><code><a href="http://www.example.com">
      <img src="https://placekitten.com/200/300">
  </a>
  </code></pre></div>
  <p dir="auto">The <code>caption</code> key wraps everything in a fancy figure/figcaption block:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="{
      "img_src": "https://placekitten.com/200/300",
      "caption": "Kitten caption"
  }"><pre class="notranslate"><code>{
      "img_src": "https://placekitten.com/200/300",
      "caption": "Kitten caption"
  }
  </code></pre></div>
  <p dir="auto">Produces:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="<figure>
      <img src="https://placekitten.com/200/300"></a>
      <figcaption>Kitten caption</figcaption>
  </figure>"><pre class="notranslate"><code><figure>
      <img src="https://placekitten.com/200/300"></a>
      <figcaption>Kitten caption</figcaption>
  </figure>
  </code></pre></div>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Preformatted text</h2><a id="user-content-preformatted-text" class="anchor" aria-label="Permalink: Preformatted text" href="#preformatted-text"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">You can use <code>{"pre": "text"}</code> to render text in a <code><pre></code> HTML tag:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="{
      "pre": "This\nhas\nnewlines"
  }"><pre class="notranslate"><code>{
      "pre": "This\nhas\nnewlines"
  }
  </code></pre></div>
  <p dir="auto">Produces:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="<pre>This
  has
  newlines</pre>"><pre class="notranslate"><code><pre>This
  has
  newlines</pre>
  </code></pre></div>
  <p dir="auto">If the value attached to the <code>"pre"</code> key is itself a JSON object, that JSON will be pretty-printed:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="{
      "pre": {
          "this": {
              "object": ["is", "nested"]
          }
      }
  }"><pre class="notranslate"><code>{
      "pre": {
          "this": {
              "object": ["is", "nested"]
          }
      }
  }
  </code></pre></div>
  <p dir="auto">Produces:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="<pre>{
    &#34;this&#34;: {
      &#34;object&#34;: [
        &#34;is&#34;,
        &#34;nested&#34;
      ]
    }
  }</pre>"><pre class="notranslate"><code><pre>{
    &#34;this&#34;: {
      &#34;object&#34;: [
        &#34;is&#34;,
        &#34;nested&#34;
      ]
    }
  }</pre>
  </code></pre></div>
  <ul dir="auto">
  <li><a href="https://datasette-json-html.datasette.io/demo?sql=select+%27%7B%0D%0A++++%22pre%22%3A+%7B%0D%0A++++++++%22this%22%3A+%7B%0D%0A++++++++++++%22object%22%3A+%5B%22is%22%2C+%22nested%22%5D%0D%0A++++++++%7D%0D%0A++++%7D%0D%0A%7D%27" rel="nofollow">Preformatted text with JSON demo</a></li>
  <li><a href="https://datasette-json-html.datasette.io/demo?sql=WITH+RECURSIVE%0D%0A++xaxis%28x%29+AS+%28VALUES%28-2.0%29+UNION+ALL+SELECT+x%2B0.05+FROM+xaxis+WHERE+x%3C1.2%29%2C%0D%0A++yaxis%28y%29+AS+%28VALUES%28-1.0%29+UNION+ALL+SELECT+y%2B0.1+FROM+yaxis+WHERE+y%3C1.0%29%2C%0D%0A++m%28iter%2C+cx%2C+cy%2C+x%2C+y%29+AS+%28%0D%0A++++SELECT+0%2C+x%2C+y%2C+0.0%2C+0.0+FROM+xaxis%2C+yaxis%0D%0A++++UNION+ALL%0D%0A++++SELECT+iter%2B1%2C+cx%2C+cy%2C+x*x-y*y+%2B+cx%2C+2.0*x*y+%2B+cy+FROM+m+%0D%0A+++++WHERE+%28x*x+%2B+y*y%29+%3C+4.0+AND+iter%3C28%0D%0A++%29%2C%0D%0A++m2%28iter%2C+cx%2C+cy%29+AS+%28%0D%0A++++SELECT+max%28iter%29%2C+cx%2C+cy+FROM+m+GROUP+BY+cx%2C+cy%0D%0A++%29%2C%0D%0A++a%28t%29+AS+%28%0D%0A++++SELECT+group_concat%28+substr%28%27+.%2B*%23%27%2C+1%2Bmin%28iter%2F7%2C4%29%2C+1%29%2C+%27%27%29+%0D%0A++++FROM+m2+GROUP+BY+cy%0D%0A++%29%0D%0ASELECT+json_object%28%27pre%27%2C+group_concat%28rtrim%28t%29%2Cx%270a%27%29%29+FROM+a%3B" rel="nofollow">Preformatted text demo showing the Mandelbrot Set</a> using <a href="https://www.sqlite.org/lang_with.html#outlandish_recursive_query_examples" rel="nofollow">this example</a> from the SQLite documentation</li>
  </ul>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Using these with SQLite JSON functions</h2><a id="user-content-using-these-with-sqlite-json-functions" class="anchor" aria-label="Permalink: Using these with SQLite JSON functions" href="#using-these-with-sqlite-json-functions"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">The most powerful way to make use of this plugin is in conjunction with SQLite's <a href="https://www.sqlite.org/json1.html" rel="nofollow">JSON functions</a>. For example:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="select json_object(
      "href", "https://simonwillison.net/",
      "label", "Simon Willison"
  );"><pre class="notranslate"><code>select json_object(
      "href", "https://simonwillison.net/",
      "label", "Simon Willison"
  );
  </code></pre></div>
  <ul dir="auto">
  <li><a href="https://datasette-json-html.datasette.io/demo?sql=select+json_object%28%0D%0A++++%22href%22%2C+%22https%3A%2F%2Fsimonwillison.net%2F%22%2C%0D%0A++++%22label%22%2C+%22Simon+Willison%22%0D%0A%29%3B" rel="nofollow">json_object() link demo</a></li>
  </ul>
  <p dir="auto">You can use these functions to construct JSON objects that work with the plugin from data in a table:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="select id, json_object(
      "href", url, "label", text
  ) from mytable;"><pre class="notranslate"><code>select id, json_object(
      "href", url, "label", text
  ) from mytable;
  </code></pre></div>
  <ul dir="auto">
  <li><a href="https://datasette-json-html.datasette.io/demo?sql=select+json_object%28%22href%22%2C+url%2C+%22label%22%2C+package%2C+%22title%22%2C+package+%7C%7C+%22+%22+%7C%7C+url%29+as+package+from+packages" rel="nofollow">Demo that builds links against a table</a></li>
  </ul>
  <p dir="auto">The <code>json_group_array()</code> function is an aggregate function similar to <code>group_concat()</code> - it allows you to construct lists of JSON objects in conjunction with a <code>GROUP BY</code> clause.</p>
  <p dir="auto">This means you can use it to construct dynamic lists of links, for example:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="select
      substr(package, 0, 12) as prefix,
      json_group_array(
          json_object(
              "href", url,
              "label", package
          )
      ) as package_links
  from packages
  group by prefix"><pre class="notranslate"><code>select
      substr(package, 0, 12) as prefix,
      json_group_array(
          json_object(
              "href", url,
              "label", package
          )
      ) as package_links
  from packages
  group by prefix
  </code></pre></div>
  <ul dir="auto">
  <li><a href="https://datasette-json-html.datasette.io/demo?sql=select%0D%0A++++substr%28package%2C+0%2C+12%29+as+prefix%2C%0D%0A++++json_group_array%28%0D%0A++++++++json_object%28%0D%0A++++++++++++%22href%22%2C+url%2C%0D%0A++++++++++++%22label%22%2C+package%0D%0A++++++++%29%0D%0A++++%29+as+package_links%0D%0Afrom+packages%0D%0Agroup+by+prefix" rel="nofollow">Demo of json_group_array()</a></li>
  </ul>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">The <code>urllib_quote_plus()</code> SQL function</h2><a id="user-content-the-urllib_quote_plus-sql-function" class="anchor" aria-label="Permalink: The urllib_quote_plus() SQL function" href="#the-urllib_quote_plus-sql-function"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">Since this plugin is designed to be used with SQL that constructs the underlying JSON structure, it is likely you will need to construct dynamic URLs from results returned by a SQL query.</p>
  <p dir="auto">This plugin registers a custom SQLite function called <code>urllib_quote_plus()</code> to help you do that. It lets you use Python's <a href="https://docs.python.org/3/library/urllib.parse.html#urllib.parse.quote_plus" rel="nofollow">urllib.parse.quote_plus() function</a> from within a SQL query.</p>
  <p dir="auto">Here's an example of how you might use it:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="select id, json_object(
      "href",
      "/mydatabase/other_table?_search=" || urllib_quote_plus(text),
      "label", text
  ) from mytable;"><pre class="notranslate"><code>select id, json_object(
      "href",
      "/mydatabase/other_table?_search=" || urllib_quote_plus(text),
      "label", text
  ) from mytable;
  </code></pre></div>
  </article></div>


node_id: MDEwOlJlcG9zaXRvcnkxNDU0ODMwNzc=
name: datasette-render-images
full_name: simonw/datasette-render-images
html_url: https://github.com/simonw/datasette-render-images
description: Datasette plugin that renders binary blob images using data-uris
created_at: 2018-08-21T00:05:47Z
updated_at: 2024-09-04T07:37:12Z
pushed_at: 2023-08-19T17:54:36Z
homepage: https://datasette-render-images-demo.datasette.io/favicons/favicons
language: Python
topics: ["datasette", "datasette-io", "datasette-plugin", "plugin"]
visibility: public
default_branch: main
permissions: {"admin": false, "maintain": false, "push": false, "triage": false, "pull": true}
readme:
  # datasette-render-images

  [![PyPI](https://img.shields.io/pypi/v/datasette-render-images.svg)](https://pypi.org/project/datasette-render-images/)
  [![Changelog](https://img.shields.io/github/v/release/simonw/datasette-render-images?include_prereleases&label=changelog)](https://github.com/simonw/datasette-render-images/releases)
  [![Tests](https://github.com/simonw/datasette-render-images/workflows/Test/badge.svg)](https://github.com/simonw/datasette-render-images/actions?query=workflow%3ATest)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/datasette-render-images/blob/main/LICENSE)

  A Datasette plugin that renders binary blob images with data-uris, using the [render_cell() plugin hook](https://docs.datasette.io/en/stable/plugins.html#render-cell-value-column-table-database-datasette).

  ## Installation

  Install this plugin in the same environment as Datasette.

      $ pip install datasette-render-images

  ## Usage

  If a database row contains binary image data (PNG, GIF or JPEG), this plugin will detect that it is an image (using the [imghdr module](https://docs.python.org/3/library/imghdr.html) and render that cell using an `<img src="data:image/png;base64,...">` element.

  Here's a [demo of the plugin in action](https://datasette-render-images-demo.datasette.io/favicons/favicons).

  ## Creating a compatible database table

  You can use the [sqlite-utils insert-files](https://sqlite-utils.datasette.io/en/stable/cli.html#inserting-data-from-files) command to insert image files into a database table:

      $ pip install sqlite-utils
      $ sqlite-utils insert-files gifs.db images *.gif

  See [Fun with binary data and SQLite](https://simonwillison.net/2020/Jul/30/fun-binary-data-and-sqlite/) for more on this tool.

  ## Configuration

  By default the plugin will only render images that are smaller than 100KB. You can adjust this limit using the `size_limit` plugin configuration option - for example, to increase the limit to 1MB (1000000 bytes) use the following in `metadata.json`:

  ```json
  {
      "plugins": {
          "datasette-render-images": {
              "size_limit": 1000000
          }
      }
  }
  ```

readme_html:
  <div id="readme" class="md" data-path="README.md"><article class="markdown-body entry-content container-lg" itemprop="text"><div class="markdown-heading" dir="auto"><h1 class="heading-element" dir="auto">datasette-render-images</h1><a id="user-content-datasette-render-images" class="anchor" aria-label="Permalink: datasette-render-images" href="#datasette-render-images"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto"><a href="https://pypi.org/project/datasette-render-images/" rel="nofollow"><img src="https://img.shields.io/pypi/v/datasette-render-images.svg" alt="PyPI" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/b91c750e885918e7dbf1b60eb7d8fa50210a1ee6a1d7f3c8fc02b49cdea5ee3d/68747470733a2f2f696d672e736869656c64732e696f2f707970692f762f6461746173657474652d72656e6465722d696d616765732e737667"></a>
  <a href="https://github.com/simonw/datasette-render-images/releases"><img src="https://img.shields.io/github/v/release/simonw/datasette-render-images?include_prereleases&label=changelog" alt="Changelog" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/3115101287c1950b97c6ed138df64e3e957e7888f044aad0e1b16ec7fa4ddc9b/68747470733a2f2f696d672e736869656c64732e696f2f6769746875622f762f72656c656173652f73696d6f6e772f6461746173657474652d72656e6465722d696d616765733f696e636c7564655f70726572656c6561736573266c6162656c3d6368616e67656c6f67"></a>
  <a href="https://github.com/simonw/datasette-render-images/actions?query=workflow%3ATest"><img src="https://github.com/simonw/datasette-render-images/workflows/Test/badge.svg" alt="Tests" style="max-width: 100%;"></a>
  <a href="https://github.com/simonw/datasette-render-images/blob/main/LICENSE"><img src="https://img.shields.io/badge/license-Apache%202.0-blue.svg" alt="License" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/c355f200ea90fddaa407b6eaab303663a669248ea3ca7b1fcf77dbe04ff5f48c/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6c6963656e73652d417061636865253230322e302d626c75652e737667"></a></p>
  <p dir="auto">A Datasette plugin that renders binary blob images with data-uris, using the <a href="https://docs.datasette.io/en/stable/plugins.html#render-cell-value-column-table-database-datasette" rel="nofollow">render_cell() plugin hook</a>.</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Installation</h2><a id="user-content-installation" class="anchor" aria-label="Permalink: Installation" href="#installation"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">Install this plugin in the same environment as Datasette.</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ pip install datasette-render-images"><pre class="notranslate"><code>$ pip install datasette-render-images
  </code></pre></div>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Usage</h2><a id="user-content-usage" class="anchor" aria-label="Permalink: Usage" href="#usage"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">If a database row contains binary image data (PNG, GIF or JPEG), this plugin will detect that it is an image (using the <a href="https://docs.python.org/3/library/imghdr.html" rel="nofollow">imghdr module</a> and render that cell using an <code><img src="data:image/png;base64,..."></code> element.</p>
  <p dir="auto">Here's a <a href="https://datasette-render-images-demo.datasette.io/favicons/favicons" rel="nofollow">demo of the plugin in action</a>.</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Creating a compatible database table</h2><a id="user-content-creating-a-compatible-database-table" class="anchor" aria-label="Permalink: Creating a compatible database table" href="#creating-a-compatible-database-table"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">You can use the <a href="https://sqlite-utils.datasette.io/en/stable/cli.html#inserting-data-from-files" rel="nofollow">sqlite-utils insert-files</a> command to insert image files into a database table:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ pip install sqlite-utils
  $ sqlite-utils insert-files gifs.db images *.gif"><pre class="notranslate"><code>$ pip install sqlite-utils
  $ sqlite-utils insert-files gifs.db images *.gif
  </code></pre></div>
  <p dir="auto">See <a href="https://simonwillison.net/2020/Jul/30/fun-binary-data-and-sqlite/" rel="nofollow">Fun with binary data and SQLite</a> for more on this tool.</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Configuration</h2><a id="user-content-configuration" class="anchor" aria-label="Permalink: Configuration" href="#configuration"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">By default the plugin will only render images that are smaller than 100KB. You can adjust this limit using the <code>size_limit</code> plugin configuration option - for example, to increase the limit to 1MB (1000000 bytes) use the following in <code>metadata.json</code>:</p>
  <div class="highlight highlight-source-json notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="{
      "plugins": {
          "datasette-render-images": {
              "size_limit": 1000000
          }
      }
  }"><pre>{
      <span class="pl-ent">"plugins"</span>: {
          <span class="pl-ent">"datasette-render-images"</span>: {
              <span class="pl-ent">"size_limit"</span>: <span class="pl-c1">1000000</span>
          }
      }
  }</pre></div>
  </article></div>


node_id: MDEwOlJlcG9zaXRvcnkxNjM3OTA4MjI=
name: datasette-sqlite-fts4
full_name: simonw/datasette-sqlite-fts4
html_url: https://github.com/simonw/datasette-sqlite-fts4
description: Datasette plugin that adds custom SQL functions for working with SQLite FTS4
created_at: 2019-01-02T03:40:41Z
updated_at: 2022-07-31T16:33:25Z
pushed_at: 2022-07-31T14:46:26Z
homepage: https://datasette.io/plugins/datasette-sqlite-fts4
language: Python
license: apache-2.0
topics: ["datasette", "datasette-io", "datasette-plugin", "plugin"]
visibility: public
default_branch: main
permissions: {"admin": false, "maintain": false, "push": false, "triage": false, "pull": true}
readme:
  # datasette-sqlite-fts4

  [![PyPI](https://img.shields.io/pypi/v/datasette-sqlite-fts4.svg)](https://pypi.org/project/datasette-sqlite-fts4/)
  [![Changelog](https://img.shields.io/github/v/release/simonw/datasette-sqlite-fts4?include_prereleases&label=changelog)](https://github.com/simonw/datasette-sqlite-fts4/releases)
  [![Tests](https://github.com/simonw/datasette-sqlite-fts4/workflows/Test/badge.svg)](https://github.com/simonw/datasette-sqlite-fts4/actions?query=workflow%3ATest)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/datasette-sqlite-fts4/blob/main/LICENSE)

  Datasette plugin that exposes the custom SQL functions from [sqlite-fts4](https://github.com/simonw/sqlite-fts4).

  [Interactive demo](https://datasette-sqlite-fts4.datasette.io/24ways-fts4?sql=select%0D%0A++++json_object%28%0D%0A++++++++"label"%2C+articles.title%2C+"href"%2C+articles.url%0D%0A++++%29+as+article%2C%0D%0A++++articles.author%2C%0D%0A++++rank_score%28matchinfo%28articles_fts%2C+"pcx"%29%29+as+score%2C%0D%0A++++rank_bm25%28matchinfo%28articles_fts%2C+"pcnalx"%29%29+as+bm25%2C%0D%0A++++json_object%28%0D%0A++++++++"pre"%2C+annotate_matchinfo%28matchinfo%28articles_fts%2C+"pcxnalyb"%29%2C+"pcxnalyb"%29%0D%0A++++%29+as+annotated_matchinfo%2C%0D%0A++++matchinfo%28articles_fts%2C+"pcxnalyb"%29+as+matchinfo%2C%0D%0A++++decode_matchinfo%28matchinfo%28articles_fts%2C+"pcxnalyb"%29%29+as+decoded_matchinfo%0D%0Afrom%0D%0A++++articles_fts+join+articles+on+articles.rowid+%3D+articles_fts.rowid%0D%0Awhere%0D%0A++++articles_fts+match+%3Asearch%0D%0Aorder+by+bm25&search=jquery+maps). Read [Exploring search relevance algorithms with SQLite](https://simonwillison.net/2019/Jan/7/exploring-search-relevance-algorithms-sqlite/) for further details on this project.

  ## Installation

      pip install datasette-sqlite-fts4

  If you are deploying a database using `datasette publish` you can include this plugin using the `--install` option:

      datasette publish now mydb.db --install=datasette-sqlite-fts4

readme_html:
  <div id="readme" class="md" data-path="README.md"><article class="markdown-body entry-content container-lg" itemprop="text"><div class="markdown-heading" dir="auto"><h1 class="heading-element" dir="auto">datasette-sqlite-fts4</h1><a id="user-content-datasette-sqlite-fts4" class="anchor" aria-label="Permalink: datasette-sqlite-fts4" href="#datasette-sqlite-fts4"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto"><a href="https://pypi.org/project/datasette-sqlite-fts4/" rel="nofollow"><img src="https://img.shields.io/pypi/v/datasette-sqlite-fts4.svg" alt="PyPI" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/0a14b3c86b81c7a83b6516384bcc778ca1224a980d79a94fb634e322ecd630f9/68747470733a2f2f696d672e736869656c64732e696f2f707970692f762f6461746173657474652d73716c6974652d667473342e737667"></a>
  <a href="https://github.com/simonw/datasette-sqlite-fts4/releases"><img src="https://img.shields.io/github/v/release/simonw/datasette-sqlite-fts4?include_prereleases&label=changelog" alt="Changelog" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/83bc9498026874c9bae59ff0700a7491c9ebb954e524ddcc01015e8b754378bd/68747470733a2f2f696d672e736869656c64732e696f2f6769746875622f762f72656c656173652f73696d6f6e772f6461746173657474652d73716c6974652d667473343f696e636c7564655f70726572656c6561736573266c6162656c3d6368616e67656c6f67"></a>
  <a href="https://github.com/simonw/datasette-sqlite-fts4/actions?query=workflow%3ATest"><img src="https://github.com/simonw/datasette-sqlite-fts4/workflows/Test/badge.svg" alt="Tests" style="max-width: 100%;"></a>
  <a href="https://github.com/simonw/datasette-sqlite-fts4/blob/main/LICENSE"><img src="https://img.shields.io/badge/license-Apache%202.0-blue.svg" alt="License" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/c355f200ea90fddaa407b6eaab303663a669248ea3ca7b1fcf77dbe04ff5f48c/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6c6963656e73652d417061636865253230322e302d626c75652e737667"></a></p>
  <p dir="auto">Datasette plugin that exposes the custom SQL functions from <a href="https://github.com/simonw/sqlite-fts4">sqlite-fts4</a>.</p>
  <p dir="auto"><a href="https://datasette-sqlite-fts4.datasette.io/24ways-fts4?sql=select%0D%0A++++json_object%28%0D%0A++++++++%22label%22%2C+articles.title%2C+%22href%22%2C+articles.url%0D%0A++++%29+as+article%2C%0D%0A++++articles.author%2C%0D%0A++++rank_score%28matchinfo%28articles_fts%2C+%22pcx%22%29%29+as+score%2C%0D%0A++++rank_bm25%28matchinfo%28articles_fts%2C+%22pcnalx%22%29%29+as+bm25%2C%0D%0A++++json_object%28%0D%0A++++++++%22pre%22%2C+annotate_matchinfo%28matchinfo%28articles_fts%2C+%22pcxnalyb%22%29%2C+%22pcxnalyb%22%29%0D%0A++++%29+as+annotated_matchinfo%2C%0D%0A++++matchinfo%28articles_fts%2C+%22pcxnalyb%22%29+as+matchinfo%2C%0D%0A++++decode_matchinfo%28matchinfo%28articles_fts%2C+%22pcxnalyb%22%29%29+as+decoded_matchinfo%0D%0Afrom%0D%0A++++articles_fts+join+articles+on+articles.rowid+%3D+articles_fts.rowid%0D%0Awhere%0D%0A++++articles_fts+match+%3Asearch%0D%0Aorder+by+bm25&search=jquery+maps" rel="nofollow">Interactive demo</a>. Read <a href="https://simonwillison.net/2019/Jan/7/exploring-search-relevance-algorithms-sqlite/" rel="nofollow">Exploring search relevance algorithms with SQLite</a> for further details on this project.</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Installation</h2><a id="user-content-installation" class="anchor" aria-label="Permalink: Installation" href="#installation"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="pip install datasette-sqlite-fts4"><pre class="notranslate"><code>pip install datasette-sqlite-fts4
  </code></pre></div>
  <p dir="auto">If you are deploying a database using <code>datasette publish</code> you can include this plugin using the <code>--install</code> option:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="datasette publish now mydb.db --install=datasette-sqlite-fts4"><pre class="notranslate"><code>datasette publish now mydb.db --install=datasette-sqlite-fts4
  </code></pre></div>
  </article></div>


node_id: MDEwOlJlcG9zaXRvcnkxNjYxNTkwNzI=
name: db-to-sqlite
full_name: simonw/db-to-sqlite
html_url: https://github.com/simonw/db-to-sqlite
description: CLI tool for exporting tables or queries from any SQL database to a SQLite file
created_at: 2019-01-17T04:16:48Z
updated_at: 2025-05-13T17:24:43Z
pushed_at: 2023-02-28T07:16:21Z
language: Python
license: apache-2.0
topics: ["datasette", "datasette-io", "datasette-tool", "sqlalchemy", "sqlite"]
visibility: public
default_branch: main
permissions: {"admin": false, "maintain": false, "push": false, "triage": false, "pull": true}
readme:
  # db-to-sqlite

  [![PyPI](https://img.shields.io/pypi/v/db-to-sqlite.svg)](https://pypi.python.org/pypi/db-to-sqlite)
  [![Changelog](https://img.shields.io/github/v/release/simonw/db-to-sqlite?include_prereleases&label=changelog)](https://github.com/simonw/db-to-sqlite/releases)
  [![Tests](https://github.com/simonw/db-to-sqlite/workflows/Test/badge.svg)](https://github.com/simonw/db-to-sqlite/actions?query=workflow%3ATest)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/db-to-sqlite/blob/main/LICENSE)

  CLI tool for exporting tables or queries from any SQL database to a SQLite file.

  ## Installation

  Install from PyPI like so:

      pip install db-to-sqlite

  If you want to use it with MySQL, you can install the extra dependency like this:

      pip install 'db-to-sqlite[mysql]'

  Installing the `mysqlclient` library on OS X can be tricky - I've found [this recipe](https://gist.github.com/simonw/90ac0afd204cd0d6d9c3135c3888d116) to work (run that before installing `db-to-sqlite`).

  For PostgreSQL, use this:

      pip install 'db-to-sqlite[postgresql]'

  ## Usage
  ```
  Usage: db-to-sqlite [OPTIONS] CONNECTION PATH

    Load data from any database into SQLite.

    PATH is a path to the SQLite file to create, e.c. /tmp/my_database.db

    CONNECTION is a SQLAlchemy connection string, for example:

        postgresql://localhost/my_database
        postgresql://username:passwd@localhost/my_database

        mysql://root@localhost/my_database
        mysql://username:passwd@localhost/my_database

    More: https://docs.sqlalchemy.org/en/13/core/engines.html#database-urls

  Options:
    --version                     Show the version and exit.
    --all                         Detect and copy all tables
    --table TEXT                  Specific tables to copy
    --skip TEXT                   When using --all skip these tables
    --redact TEXT...              (table, column) pairs to redact with ***
    --sql TEXT                    Optional SQL query to run
    --output TEXT                 Table in which to save --sql query results
    --pk TEXT                     Optional column to use as a primary key
    --index-fks / --no-index-fks  Should foreign keys have indexes? Default on
    -p, --progress                Show progress bar
    --postgres-schema TEXT        PostgreSQL schema to use
    --help                        Show this message and exit.
  ```

  For example, to save the content of the `blog_entry` table from a PostgreSQL database to a local file called `blog.db` you could do this:

      db-to-sqlite "postgresql://localhost/myblog" blog.db \
          --table=blog_entry

  You can specify `--table` more than once.

  You can also save the data from all of your tables, effectively creating a SQLite copy of your entire database. Any foreign key relationships will be detected and added to the SQLite database. For example:

      db-to-sqlite "postgresql://localhost/myblog" blog.db \
          --all

  When running `--all` you can specify tables to skip using `--skip`:

      db-to-sqlite "postgresql://localhost/myblog" blog.db \
          --all \
          --skip=django_migrations

  If you want to save the results of a custom SQL query, do this:

      db-to-sqlite "postgresql://localhost/myblog" output.db \
          --output=query_results \
          --sql="select id, title, created from blog_entry" \
          --pk=id

  The `--output` option specifies the table that should contain the results of the query.

  ## Using db-to-sqlite with PostgreSQL schemas

  If the tables you want to copy from your PostgreSQL database aren't in the default schema, you can specify an alternate one with the `--postgres-schema` option:

      db-to-sqlite "postgresql://localhost/myblog" blog.db \
          --all \
          --postgres-schema my_schema

  ## Using db-to-sqlite with MS SQL

  The best way to get the connection string needed for the MS SQL connections below is to use urllib from the Standard Library as below

      params = urllib.parse.quote_plus(
          "DRIVER={SQL Server Native Client 11.0};"
          "SERVER=localhost;"
          "DATABASE=my_database;"
          "Trusted_Connection=yes;"
      )

  The above will resolve to

      DRIVER%3D%7BSQL+Server+Native+Client+11.0%7D%3B+SERVER%3Dlocalhost%3B+DATABASE%3Dmy_database%3B+Trusted_Connection%3Dyes

  You can then use the string above in the odbc_connect below

      mssql+pyodbc:///?odbc_connect=DRIVER%3D%7BSQL+Server+Native+Client+11.0%7D%3B+SERVER%3Dlocalhost%3B+DATABASE%3Dmy_database%3B+Trusted_Connection%3Dyes
      mssql+pyodbc:///?odbc_connect=DRIVER%3D%7BSQL+Server+Native+Client+11.0%7D%3B+SERVER%3Dlocalhost%3B+DATABASE%3Dmy_database%3B+UID%3Dusername%3B+PWD%3Dpasswd

  ## Using db-to-sqlite with Heroku Postgres

  If you run an application on [Heroku](https://www.heroku.com/) using their [Postgres database product](https://www.heroku.com/postgres), you can use the `heroku config` command to access a compatible connection string:

      $ heroku config --app myappname | grep HEROKU_POSTG
      HEROKU_POSTGRESQL_OLIVE_URL: postgres://username:password@ec2-xxx-xxx-xxx-x.compute-1.amazonaws.com:5432/dbname

  You can pass this to `db-to-sqlite` to create a local SQLite database with the data from your Heroku instance.

  You can even do this using a bash one-liner:

      $ db-to-sqlite $(heroku config --app myappname | grep HEROKU_POSTG | cut -d: -f 2-) \
          /tmp/heroku.db --all -p
      1/23: django_migrations
      ...
      17/23: blog_blogmark
      [####################################]  100%
      ...

  ## Related projects

  * [Datasette](https://github.com/simonw/datasette): A tool for exploring and publishing data. Works great with SQLite files generated using `db-to-sqlite`.
  * [sqlite-utils](https://github.com/simonw/sqlite-utils): Python CLI utility and library for manipulating SQLite databases.
  * [csvs-to-sqlite](https://github.com/simonw/csvs-to-sqlite): Convert CSV files into a SQLite database.

  ## Development

  To set up this tool locally, first checkout the code. Then create a new virtual environment:

      cd db-to-sqlite
      python3 -m venv venv
      source venv/bin/activate

  Or if you are using `pipenv`:

      pipenv shell

  Now install the dependencies and test dependencies:

      pip install -e '.[test]'

  To run the tests:

      pytest

  This will skip tests against MySQL or PostgreSQL if you do not have their additional dependencies installed.

  You can install those extra dependencies like so:

      pip install -e '.[test_mysql,test_postgresql]'

  You can alternative use `pip install psycopg2-binary` if you cannot install the `psycopg2` dependency used by the `test_postgresql` extra.

  See [Running a MySQL server using Homebrew](https://til.simonwillison.net/homebrew/mysql-homebrew) for tips on running the tests against MySQL on macOS, including how to install the `mysqlclient` dependency.

  The PostgreSQL and MySQL tests default to expecting to run against servers on localhost. You can use environment variables to point them at different test database servers:

  - `MYSQL_TEST_DB_CONNECTION` - defaults to `mysql://root@localhost/test_db_to_sqlite`
  - `POSTGRESQL_TEST_DB_CONNECTION` - defaults to `postgresql://localhost/test_db_to_sqlite`

  The database you indicate in the environment variable - `test_db_to_sqlite` by default - will be deleted and recreated on every test run.

readme_html:
  <div id="readme" class="md" data-path="README.md"><article class="markdown-body entry-content container-lg" itemprop="text"><div class="markdown-heading" dir="auto"><h1 class="heading-element" dir="auto">db-to-sqlite</h1><a id="user-content-db-to-sqlite" class="anchor" aria-label="Permalink: db-to-sqlite" href="#db-to-sqlite"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto"><a href="https://pypi.python.org/pypi/db-to-sqlite" rel="nofollow"><img src="https://img.shields.io/pypi/v/db-to-sqlite.svg" alt="PyPI" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/4745ee4e3efcb2d58ec635fa8b1845f6cae7fa75a617d903456e1b39cf45a01b/68747470733a2f2f696d672e736869656c64732e696f2f707970692f762f64622d746f2d73716c6974652e737667"></a>
  <a href="https://github.com/simonw/db-to-sqlite/releases"><img src="https://img.shields.io/github/v/release/simonw/db-to-sqlite?include_prereleases&label=changelog" alt="Changelog" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/35a8d7d7ef60c27a3049b807c7c33b92bc96e0ee59481716ae793250c4131ee5/68747470733a2f2f696d672e736869656c64732e696f2f6769746875622f762f72656c656173652f73696d6f6e772f64622d746f2d73716c6974653f696e636c7564655f70726572656c6561736573266c6162656c3d6368616e67656c6f67"></a>
  <a href="https://github.com/simonw/db-to-sqlite/actions?query=workflow%3ATest"><img src="https://github.com/simonw/db-to-sqlite/workflows/Test/badge.svg" alt="Tests" style="max-width: 100%;"></a>
  <a href="https://github.com/simonw/db-to-sqlite/blob/main/LICENSE"><img src="https://img.shields.io/badge/license-Apache%202.0-blue.svg" alt="License" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/c355f200ea90fddaa407b6eaab303663a669248ea3ca7b1fcf77dbe04ff5f48c/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6c6963656e73652d417061636865253230322e302d626c75652e737667"></a></p>
  <p dir="auto">CLI tool for exporting tables or queries from any SQL database to a SQLite file.</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Installation</h2><a id="user-content-installation" class="anchor" aria-label="Permalink: Installation" href="#installation"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">Install from PyPI like so:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="pip install db-to-sqlite"><pre class="notranslate"><code>pip install db-to-sqlite
  </code></pre></div>
  <p dir="auto">If you want to use it with MySQL, you can install the extra dependency like this:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="pip install 'db-to-sqlite[mysql]'"><pre class="notranslate"><code>pip install 'db-to-sqlite[mysql]'
  </code></pre></div>
  <p dir="auto">Installing the <code>mysqlclient</code> library on OS X can be tricky - I've found <a href="https://gist.github.com/simonw/90ac0afd204cd0d6d9c3135c3888d116">this recipe</a> to work (run that before installing <code>db-to-sqlite</code>).</p>
  <p dir="auto">For PostgreSQL, use this:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="pip install 'db-to-sqlite[postgresql]'"><pre class="notranslate"><code>pip install 'db-to-sqlite[postgresql]'
  </code></pre></div>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Usage</h2><a id="user-content-usage" class="anchor" aria-label="Permalink: Usage" href="#usage"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="Usage: db-to-sqlite [OPTIONS] CONNECTION PATH

    Load data from any database into SQLite.

    PATH is a path to the SQLite file to create, e.c. /tmp/my_database.db

    CONNECTION is a SQLAlchemy connection string, for example:

        postgresql://localhost/my_database
        postgresql://username:passwd@localhost/my_database

        mysql://root@localhost/my_database
        mysql://username:passwd@localhost/my_database

    More: https://docs.sqlalchemy.org/en/13/core/engines.html#database-urls

  Options:
    --version                     Show the version and exit.
    --all                         Detect and copy all tables
    --table TEXT                  Specific tables to copy
    --skip TEXT                   When using --all skip these tables
    --redact TEXT...              (table, column) pairs to redact with ***
    --sql TEXT                    Optional SQL query to run
    --output TEXT                 Table in which to save --sql query results
    --pk TEXT                     Optional column to use as a primary key
    --index-fks / --no-index-fks  Should foreign keys have indexes? Default on
    -p, --progress                Show progress bar
    --postgres-schema TEXT        PostgreSQL schema to use
    --help                        Show this message and exit."><pre class="notranslate"><code>Usage: db-to-sqlite [OPTIONS] CONNECTION PATH

    Load data from any database into SQLite.

    PATH is a path to the SQLite file to create, e.c. /tmp/my_database.db

    CONNECTION is a SQLAlchemy connection string, for example:

        postgresql://localhost/my_database
        postgresql://username:passwd@localhost/my_database

        mysql://root@localhost/my_database
        mysql://username:passwd@localhost/my_database

    More: https://docs.sqlalchemy.org/en/13/core/engines.html#database-urls

  Options:
    --version                     Show the version and exit.
    --all                         Detect and copy all tables
    --table TEXT                  Specific tables to copy
    --skip TEXT                   When using --all skip these tables
    --redact TEXT...              (table, column) pairs to redact with ***
    --sql TEXT                    Optional SQL query to run
    --output TEXT                 Table in which to save --sql query results
    --pk TEXT                     Optional column to use as a primary key
    --index-fks / --no-index-fks  Should foreign keys have indexes? Default on
    -p, --progress                Show progress bar
    --postgres-schema TEXT        PostgreSQL schema to use
    --help                        Show this message and exit.
  </code></pre></div>
  <p dir="auto">For example, to save the content of the <code>blog_entry</code> table from a PostgreSQL database to a local file called <code>blog.db</code> you could do this:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="db-to-sqlite "postgresql://localhost/myblog" blog.db \
      --table=blog_entry"><pre class="notranslate"><code>db-to-sqlite "postgresql://localhost/myblog" blog.db \
      --table=blog_entry
  </code></pre></div>
  <p dir="auto">You can specify <code>--table</code> more than once.</p>
  <p dir="auto">You can also save the data from all of your tables, effectively creating a SQLite copy of your entire database. Any foreign key relationships will be detected and added to the SQLite database. For example:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="db-to-sqlite "postgresql://localhost/myblog" blog.db \
      --all"><pre class="notranslate"><code>db-to-sqlite "postgresql://localhost/myblog" blog.db \
      --all
  </code></pre></div>
  <p dir="auto">When running <code>--all</code> you can specify tables to skip using <code>--skip</code>:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="db-to-sqlite "postgresql://localhost/myblog" blog.db \
      --all \
      --skip=django_migrations"><pre class="notranslate"><code>db-to-sqlite "postgresql://localhost/myblog" blog.db \
      --all \
      --skip=django_migrations
  </code></pre></div>
  <p dir="auto">If you want to save the results of a custom SQL query, do this:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="db-to-sqlite "postgresql://localhost/myblog" output.db \
      --output=query_results \
      --sql="select id, title, created from blog_entry" \
      --pk=id"><pre class="notranslate"><code>db-to-sqlite "postgresql://localhost/myblog" output.db \
      --output=query_results \
      --sql="select id, title, created from blog_entry" \
      --pk=id
  </code></pre></div>
  <p dir="auto">The <code>--output</code> option specifies the table that should contain the results of the query.</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Using db-to-sqlite with PostgreSQL schemas</h2><a id="user-content-using-db-to-sqlite-with-postgresql-schemas" class="anchor" aria-label="Permalink: Using db-to-sqlite with PostgreSQL schemas" href="#using-db-to-sqlite-with-postgresql-schemas"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">If the tables you want to copy from your PostgreSQL database aren't in the default schema, you can specify an alternate one with the <code>--postgres-schema</code> option:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="db-to-sqlite "postgresql://localhost/myblog" blog.db \
      --all \
      --postgres-schema my_schema"><pre class="notranslate"><code>db-to-sqlite "postgresql://localhost/myblog" blog.db \
      --all \
      --postgres-schema my_schema
  </code></pre></div>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Using db-to-sqlite with MS SQL</h2><a id="user-content-using-db-to-sqlite-with-ms-sql" class="anchor" aria-label="Permalink: Using db-to-sqlite with MS SQL" href="#using-db-to-sqlite-with-ms-sql"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">The best way to get the connection string needed for the MS SQL connections below is to use urllib from the Standard Library as below</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="params = urllib.parse.quote_plus(
      "DRIVER={SQL Server Native Client 11.0};"
      "SERVER=localhost;"
      "DATABASE=my_database;"
      "Trusted_Connection=yes;"
  )"><pre class="notranslate"><code>params = urllib.parse.quote_plus(
      "DRIVER={SQL Server Native Client 11.0};"
      "SERVER=localhost;"
      "DATABASE=my_database;"
      "Trusted_Connection=yes;"
  )
  </code></pre></div>
  <p dir="auto">The above will resolve to</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="DRIVER%3D%7BSQL+Server+Native+Client+11.0%7D%3B+SERVER%3Dlocalhost%3B+DATABASE%3Dmy_database%3B+Trusted_Connection%3Dyes"><pre class="notranslate"><code>DRIVER%3D%7BSQL+Server+Native+Client+11.0%7D%3B+SERVER%3Dlocalhost%3B+DATABASE%3Dmy_database%3B+Trusted_Connection%3Dyes
  </code></pre></div>
  <p dir="auto">You can then use the string above in the odbc_connect below</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="mssql+pyodbc:///?odbc_connect=DRIVER%3D%7BSQL+Server+Native+Client+11.0%7D%3B+SERVER%3Dlocalhost%3B+DATABASE%3Dmy_database%3B+Trusted_Connection%3Dyes
  mssql+pyodbc:///?odbc_connect=DRIVER%3D%7BSQL+Server+Native+Client+11.0%7D%3B+SERVER%3Dlocalhost%3B+DATABASE%3Dmy_database%3B+UID%3Dusername%3B+PWD%3Dpasswd"><pre class="notranslate"><code>mssql+pyodbc:///?odbc_connect=DRIVER%3D%7BSQL+Server+Native+Client+11.0%7D%3B+SERVER%3Dlocalhost%3B+DATABASE%3Dmy_database%3B+Trusted_Connection%3Dyes
  mssql+pyodbc:///?odbc_connect=DRIVER%3D%7BSQL+Server+Native+Client+11.0%7D%3B+SERVER%3Dlocalhost%3B+DATABASE%3Dmy_database%3B+UID%3Dusername%3B+PWD%3Dpasswd
  </code></pre></div>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Using db-to-sqlite with Heroku Postgres</h2><a id="user-content-using-db-to-sqlite-with-heroku-postgres" class="anchor" aria-label="Permalink: Using db-to-sqlite with Heroku Postgres" href="#using-db-to-sqlite-with-heroku-postgres"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">If you run an application on <a href="https://www.heroku.com/" rel="nofollow">Heroku</a> using their <a href="https://www.heroku.com/postgres" rel="nofollow">Postgres database product</a>, you can use the <code>heroku config</code> command to access a compatible connection string:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ heroku config --app myappname | grep HEROKU_POSTG
  HEROKU_POSTGRESQL_OLIVE_URL: postgres://username:password@ec2-xxx-xxx-xxx-x.compute-1.amazonaws.com:5432/dbname"><pre class="notranslate"><code>$ heroku config --app myappname | grep HEROKU_POSTG
  HEROKU_POSTGRESQL_OLIVE_URL: postgres://username:password@ec2-xxx-xxx-xxx-x.compute-1.amazonaws.com:5432/dbname
  </code></pre></div>
  <p dir="auto">You can pass this to <code>db-to-sqlite</code> to create a local SQLite database with the data from your Heroku instance.</p>
  <p dir="auto">You can even do this using a bash one-liner:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ db-to-sqlite $(heroku config --app myappname | grep HEROKU_POSTG | cut -d: -f 2-) \
      /tmp/heroku.db --all -p
  1/23: django_migrations
  ...
  17/23: blog_blogmark
  [####################################]  100%
  ..."><pre class="notranslate"><code>$ db-to-sqlite $(heroku config --app myappname | grep HEROKU_POSTG | cut -d: -f 2-) \
      /tmp/heroku.db --all -p
  1/23: django_migrations
  ...
  17/23: blog_blogmark
  [####################################]  100%
  ...
  </code></pre></div>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Related projects</h2><a id="user-content-related-projects" class="anchor" aria-label="Permalink: Related projects" href="#related-projects"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <ul dir="auto">
  <li><a href="https://github.com/simonw/datasette">Datasette</a>: A tool for exploring and publishing data. Works great with SQLite files generated using <code>db-to-sqlite</code>.</li>
  <li><a href="https://github.com/simonw/sqlite-utils">sqlite-utils</a>: Python CLI utility and library for manipulating SQLite databases.</li>
  <li><a href="https://github.com/simonw/csvs-to-sqlite">csvs-to-sqlite</a>: Convert CSV files into a SQLite database.</li>
  </ul>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Development</h2><a id="user-content-development" class="anchor" aria-label="Permalink: Development" href="#development"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">To set up this tool locally, first checkout the code. Then create a new virtual environment:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="cd db-to-sqlite
  python3 -m venv venv
  source venv/bin/activate"><pre class="notranslate"><code>cd db-to-sqlite
  python3 -m venv venv
  source venv/bin/activate
  </code></pre></div>
  <p dir="auto">Or if you are using <code>pipenv</code>:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="pipenv shell"><pre class="notranslate"><code>pipenv shell
  </code></pre></div>
  <p dir="auto">Now install the dependencies and test dependencies:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="pip install -e '.[test]'"><pre class="notranslate"><code>pip install -e '.[test]'
  </code></pre></div>
  <p dir="auto">To run the tests:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="pytest"><pre class="notranslate"><code>pytest
  </code></pre></div>
  <p dir="auto">This will skip tests against MySQL or PostgreSQL if you do not have their additional dependencies installed.</p>
  <p dir="auto">You can install those extra dependencies like so:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="pip install -e '.[test_mysql,test_postgresql]'"><pre class="notranslate"><code>pip install -e '.[test_mysql,test_postgresql]'
  </code></pre></div>
  <p dir="auto">You can alternative use <code>pip install psycopg2-binary</code> if you cannot install the <code>psycopg2</code> dependency used by the <code>test_postgresql</code> extra.</p>
  <p dir="auto">See <a href="https://til.simonwillison.net/homebrew/mysql-homebrew" rel="nofollow">Running a MySQL server using Homebrew</a> for tips on running the tests against MySQL on macOS, including how to install the <code>mysqlclient</code> dependency.</p>
  <p dir="auto">The PostgreSQL and MySQL tests default to expecting to run against servers on localhost. You can use environment variables to point them at different test database servers:</p>
  <ul dir="auto">
  <li><code>MYSQL_TEST_DB_CONNECTION</code> - defaults to <code>mysql://root@localhost/test_db_to_sqlite</code></li>
  <li><code>POSTGRESQL_TEST_DB_CONNECTION</code> - defaults to <code>postgresql://localhost/test_db_to_sqlite</code></li>
  </ul>
  <p dir="auto">The database you indicate in the environment variable - <code>test_db_to_sqlite</code> by default - will be deleted and recreated on every test run.</p>
  </article></div>


node_id: MDEwOlJlcG9zaXRvcnkxNjc3MzAwNzE=
name: datasette-pretty-json
full_name: simonw/datasette-pretty-json
html_url: https://github.com/simonw/datasette-pretty-json
description: Datasette plugin that pretty-prints any column values that are valid JSON objects or arrays
created_at: 2019-01-26T19:30:43Z
updated_at: 2025-02-28T21:21:59Z
pushed_at: 2024-03-05T02:33:02Z
language: Python
license: apache-2.0
topics: ["datasette", "datasette-io", "datasette-plugin", "json"]
visibility: public
default_branch: master
permissions: {"admin": false, "maintain": false, "push": false, "triage": false, "pull": true}
readme:
  # datasette-pretty-json

  [![PyPI](https://img.shields.io/pypi/v/datasette-pretty-json.svg)](https://pypi.org/project/datasette-pretty-json/)
  [![Changelog](https://img.shields.io/github/v/release/simonw/datasette-pretty-json?include_prereleases&label=changelog)](https://github.com/simonw/datasette-pretty-json/releases)
  [![Tests](https://github.com/simonw/datasette-pretty-json/workflows/Test/badge.svg)](https://github.com/simonw/datasette-pretty-json/actions?query=workflow%3ATest)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/datasette-pretty-json/blob/main/LICENSE)


  [Datasette](https://github.com/simonw/datasette) plugin that pretty-prints any column values that are valid JSON objects or arrays.

  You may also be interested in [datasette-json-html](https://github.com/simonw/datasette-json-html).

readme_html:
  <div id="readme" class="md" data-path="README.md"><article class="markdown-body entry-content container-lg" itemprop="text"><div class="markdown-heading" dir="auto"><h1 class="heading-element" dir="auto">datasette-pretty-json</h1><a id="user-content-datasette-pretty-json" class="anchor" aria-label="Permalink: datasette-pretty-json" href="#datasette-pretty-json"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto"><a href="https://pypi.org/project/datasette-pretty-json/" rel="nofollow"><img src="https://img.shields.io/pypi/v/datasette-pretty-json.svg" alt="PyPI" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/a9801bbcb2d60451a69f042506284e8b12f3e22c92bade3bc7817d8d6e689def/68747470733a2f2f696d672e736869656c64732e696f2f707970692f762f6461746173657474652d7072657474792d6a736f6e2e737667"></a>
  <a href="https://github.com/simonw/datasette-pretty-json/releases"><img src="https://img.shields.io/github/v/release/simonw/datasette-pretty-json?include_prereleases&label=changelog" alt="Changelog" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/171dc8c0e2e384e3f91359629e13a1d01ceabd0f6cf307cac1499811c59bf254/68747470733a2f2f696d672e736869656c64732e696f2f6769746875622f762f72656c656173652f73696d6f6e772f6461746173657474652d7072657474792d6a736f6e3f696e636c7564655f70726572656c6561736573266c6162656c3d6368616e67656c6f67"></a>
  <a href="https://github.com/simonw/datasette-pretty-json/actions?query=workflow%3ATest"><img src="https://github.com/simonw/datasette-pretty-json/workflows/Test/badge.svg" alt="Tests" style="max-width: 100%;"></a>
  <a href="https://github.com/simonw/datasette-pretty-json/blob/main/LICENSE"><img src="https://img.shields.io/badge/license-Apache%202.0-blue.svg" alt="License" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/c355f200ea90fddaa407b6eaab303663a669248ea3ca7b1fcf77dbe04ff5f48c/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6c6963656e73652d417061636865253230322e302d626c75652e737667"></a></p>
  <p dir="auto"><a href="https://github.com/simonw/datasette">Datasette</a> plugin that pretty-prints any column values that are valid JSON objects or arrays.</p>
  <p dir="auto">You may also be interested in <a href="https://github.com/simonw/datasette-json-html">datasette-json-html</a>.</p>
  </article></div>


node_id: MDEwOlJlcG9zaXRvcnkxNjc3NTk4NDY=
name: markdown-to-sqlite
full_name: simonw/markdown-to-sqlite
html_url: https://github.com/simonw/markdown-to-sqlite
description: CLI tool for loading markdown files into a SQLite database
created_at: 2019-01-27T02:04:54Z
updated_at: 2025-05-04T12:45:30Z
pushed_at: 2022-05-13T18:09:22Z
language: Python
license: apache-2.0
topics: ["datasette-io", "datasette-tool", "markdown", "sqlite", "yaml"]
visibility: public
default_branch: main
permissions: {"admin": false, "maintain": false, "push": false, "triage": false, "pull": true}
readme:
  # markdown-to-sqlite

  [![PyPI](https://img.shields.io/pypi/v/markdown-to-sqlite.svg)](https://pypi.python.org/pypi/markdown-to-sqlite)
  [![Changelog](https://img.shields.io/github/v/release/simonw/markdown-to-sqlite?include_prereleases&label=changelog)](https://github.com/simonw/markdown-to-sqlite/releases)
  [![Tests](https://github.com/simonw/markdown-to-sqlite/workflows/Test/badge.svg)](https://github.com/simonw/markdown-to-sqlite/actions?query=workflow%3ATest)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/markdown-to-sqlite/blob/main/LICENSE)

  CLI tool for loading markdown files into a SQLite database.

  YAML embedded in the markdown files will be used to populate additional columns.

      Usage: markdown-to-sqlite [OPTIONS] DBNAME TABLE PATHS...

  For example:

      $ markdown-to-sqlite docs.db documents file1.md file2.md

  ## Breaking change

  Prior to version 1.0 this argument order was different - markdown files were listed before the database and table.

readme_html:
  <div id="readme" class="md" data-path="README.md"><article class="markdown-body entry-content container-lg" itemprop="text"><div class="markdown-heading" dir="auto"><h1 class="heading-element" dir="auto">markdown-to-sqlite</h1><a id="user-content-markdown-to-sqlite" class="anchor" aria-label="Permalink: markdown-to-sqlite" href="#markdown-to-sqlite"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto"><a href="https://pypi.python.org/pypi/markdown-to-sqlite" rel="nofollow"><img src="https://img.shields.io/pypi/v/markdown-to-sqlite.svg" alt="PyPI" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/a09e2f678411368a2f507325c825306f27bd16bef5b42bbc358b5f324f6c41ce/68747470733a2f2f696d672e736869656c64732e696f2f707970692f762f6d61726b646f776e2d746f2d73716c6974652e737667"></a>
  <a href="https://github.com/simonw/markdown-to-sqlite/releases"><img src="https://img.shields.io/github/v/release/simonw/markdown-to-sqlite?include_prereleases&label=changelog" alt="Changelog" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/af0710f8c013d2323cc1afe6c1905ac5ab7ddd56ddf0113ca35d6d73882029b9/68747470733a2f2f696d672e736869656c64732e696f2f6769746875622f762f72656c656173652f73696d6f6e772f6d61726b646f776e2d746f2d73716c6974653f696e636c7564655f70726572656c6561736573266c6162656c3d6368616e67656c6f67"></a>
  <a href="https://github.com/simonw/markdown-to-sqlite/actions?query=workflow%3ATest"><img src="https://github.com/simonw/markdown-to-sqlite/workflows/Test/badge.svg" alt="Tests" style="max-width: 100%;"></a>
  <a href="https://github.com/simonw/markdown-to-sqlite/blob/main/LICENSE"><img src="https://img.shields.io/badge/license-Apache%202.0-blue.svg" alt="License" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/c355f200ea90fddaa407b6eaab303663a669248ea3ca7b1fcf77dbe04ff5f48c/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6c6963656e73652d417061636865253230322e302d626c75652e737667"></a></p>
  <p dir="auto">CLI tool for loading markdown files into a SQLite database.</p>
  <p dir="auto">YAML embedded in the markdown files will be used to populate additional columns.</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="Usage: markdown-to-sqlite [OPTIONS] DBNAME TABLE PATHS..."><pre class="notranslate"><code>Usage: markdown-to-sqlite [OPTIONS] DBNAME TABLE PATHS...
  </code></pre></div>
  <p dir="auto">For example:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ markdown-to-sqlite docs.db documents file1.md file2.md"><pre class="notranslate"><code>$ markdown-to-sqlite docs.db documents file1.md file2.md
  </code></pre></div>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Breaking change</h2><a id="user-content-breaking-change" class="anchor" aria-label="Permalink: Breaking change" href="#breaking-change"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">Prior to version 1.0 this argument order was different - markdown files were listed before the database and table.</p>
  </article></div>


node_id: MDEwOlJlcG9zaXRvcnkxNzQ3MTUxNTM=
name: datasette-jellyfish
full_name: simonw/datasette-jellyfish
html_url: https://github.com/simonw/datasette-jellyfish
description: Datasette plugin adding SQL functions for fuzzy text matching powered by Jellyfish
created_at: 2019-03-09T16:02:01Z
updated_at: 2023-08-27T07:23:28Z
pushed_at: 2023-08-24T21:45:20Z
homepage: https://datasette.io/plugins/datasette-jellyfish
language: Python
license: apache-2.0
topics: ["datasette", "datasette-io", "datasette-plugin"]
visibility: public
default_branch: main
permissions: {"admin": false, "maintain": false, "push": false, "triage": false, "pull": true}
readme:
  # datasette-jellyfish

  [![PyPI](https://img.shields.io/pypi/v/datasette-jellyfish.svg)](https://pypi.org/project/datasette-jellyfish/)
  [![Changelog](https://img.shields.io/github/v/release/simonw/datasette-jellyfish?include_prereleases&label=changelog)](https://github.com/simonw/datasette-jellyfish/releases)
  [![Tests](https://github.com/simonw/datasette-jellyfish/workflows/Test/badge.svg)](https://github.com/simonw/datasette-jellyfish/actions?query=workflow%3ATest)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/datasette-jellyfish/blob/main/LICENSE)

  Datasette plugin that adds custom SQL functions for fuzzy string matching, built on top of the [Jellyfish](https://github.com/jamesturk/jellyfish) Python library by James Turk and Michael Stephens.

  Interactive demos:

  * [soundex, metaphone, nysiis, match_rating_codex comparison](https://latest-with-plugins.datasette.io/fixtures?sql=SELECT%0D%0A++++soundex%28%3As%29%2C+%0D%0A++++metaphone%28%3As%29%2C+%0D%0A++++nysiis%28%3As%29%2C+%0D%0A++++match_rating_codex%28%3As%29&s=demo).
  * [distance functions comparison](https://latest-with-plugins.datasette.io/fixtures?sql=SELECT%0D%0A++++levenshtein_distance%28%3As1%2C+%3As2%29%2C%0D%0A++++damerau_levenshtein_distance%28%3As1%2C+%3As2%29%2C%0D%0A++++hamming_distance%28%3As1%2C+%3As2%29%2C%0D%0A++++jaro_similarity%28%3As1%2C+%3As2%29%2C%0D%0A++++jaro_winkler_similarity%28%3As1%2C+%3As2%29%2C%0D%0A++++match_rating_comparison%28%3As1%2C+%3As2%29%3B&s1=barrack+obama&s2=barrack+h+obama)

  Examples:

      SELECT soundex("hello");
          -- Outputs H400
      SELECT metaphone("hello");
          -- Outputs HL
      SELECT nysiis("hello");
          -- Outputs HAL
      SELECT match_rating_codex("hello");
          -- Outputs HLL
      SELECT levenshtein_distance("hello", "hello world");
          -- Outputs 6
      SELECT damerau_levenshtein_distance("hello", "hello world");
          -- Outputs 6
      SELECT hamming_distance("hello", "hello world");
          -- Outputs 6
      SELECT jaro_similarity("hello", "hello world");
          -- Outputs 0.8181818181818182
      SELECT jaro_winkler_similarity("hello", "hello world");
          -- Outputs 0.890909090909091
      SELECT match_rating_comparison("hello", "helloo");
          -- Outputs 1

  See [the Jellyfish documentation](https://jellyfish.readthedocs.io/en/latest/) for an explanation of each of these functions.

readme_html:
  <div id="readme" class="md" data-path="README.md"><article class="markdown-body entry-content container-lg" itemprop="text"><div class="markdown-heading" dir="auto"><h1 class="heading-element" dir="auto">datasette-jellyfish</h1><a id="user-content-datasette-jellyfish" class="anchor" aria-label="Permalink: datasette-jellyfish" href="#datasette-jellyfish"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto"><a href="https://pypi.org/project/datasette-jellyfish/" rel="nofollow"><img src="https://img.shields.io/pypi/v/datasette-jellyfish.svg" alt="PyPI" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/9d9a8466663760cffae546ecb1ea34e3fb15a789efb19a51c1a1355a7d13c0ee/68747470733a2f2f696d672e736869656c64732e696f2f707970692f762f6461746173657474652d6a656c6c79666973682e737667"></a>
  <a href="https://github.com/simonw/datasette-jellyfish/releases"><img src="https://img.shields.io/github/v/release/simonw/datasette-jellyfish?include_prereleases&label=changelog" alt="Changelog" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/574cb554aee752a1985538f1b1c83e9c12b05fa3096c5b0ecf1c930a0e9d30c6/68747470733a2f2f696d672e736869656c64732e696f2f6769746875622f762f72656c656173652f73696d6f6e772f6461746173657474652d6a656c6c79666973683f696e636c7564655f70726572656c6561736573266c6162656c3d6368616e67656c6f67"></a>
  <a href="https://github.com/simonw/datasette-jellyfish/actions?query=workflow%3ATest"><img src="https://github.com/simonw/datasette-jellyfish/workflows/Test/badge.svg" alt="Tests" style="max-width: 100%;"></a>
  <a href="https://github.com/simonw/datasette-jellyfish/blob/main/LICENSE"><img src="https://img.shields.io/badge/license-Apache%202.0-blue.svg" alt="License" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/c355f200ea90fddaa407b6eaab303663a669248ea3ca7b1fcf77dbe04ff5f48c/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6c6963656e73652d417061636865253230322e302d626c75652e737667"></a></p>
  <p dir="auto">Datasette plugin that adds custom SQL functions for fuzzy string matching, built on top of the <a href="https://github.com/jamesturk/jellyfish">Jellyfish</a> Python library by James Turk and Michael Stephens.</p>
  <p dir="auto">Interactive demos:</p>
  <ul dir="auto">
  <li><a href="https://latest-with-plugins.datasette.io/fixtures?sql=SELECT%0D%0A++++soundex%28%3As%29%2C+%0D%0A++++metaphone%28%3As%29%2C+%0D%0A++++nysiis%28%3As%29%2C+%0D%0A++++match_rating_codex%28%3As%29&s=demo" rel="nofollow">soundex, metaphone, nysiis, match_rating_codex comparison</a>.</li>
  <li><a href="https://latest-with-plugins.datasette.io/fixtures?sql=SELECT%0D%0A++++levenshtein_distance%28%3As1%2C+%3As2%29%2C%0D%0A++++damerau_levenshtein_distance%28%3As1%2C+%3As2%29%2C%0D%0A++++hamming_distance%28%3As1%2C+%3As2%29%2C%0D%0A++++jaro_similarity%28%3As1%2C+%3As2%29%2C%0D%0A++++jaro_winkler_similarity%28%3As1%2C+%3As2%29%2C%0D%0A++++match_rating_comparison%28%3As1%2C+%3As2%29%3B&s1=barrack+obama&s2=barrack+h+obama" rel="nofollow">distance functions comparison</a></li>
  </ul>
  <p dir="auto">Examples:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="SELECT soundex("hello");
      -- Outputs H400
  SELECT metaphone("hello");
      -- Outputs HL
  SELECT nysiis("hello");
      -- Outputs HAL
  SELECT match_rating_codex("hello");
      -- Outputs HLL
  SELECT levenshtein_distance("hello", "hello world");
      -- Outputs 6
  SELECT damerau_levenshtein_distance("hello", "hello world");
      -- Outputs 6
  SELECT hamming_distance("hello", "hello world");
      -- Outputs 6
  SELECT jaro_similarity("hello", "hello world");
      -- Outputs 0.8181818181818182
  SELECT jaro_winkler_similarity("hello", "hello world");
      -- Outputs 0.890909090909091
  SELECT match_rating_comparison("hello", "helloo");
      -- Outputs 1"><pre class="notranslate"><code>SELECT soundex("hello");
      -- Outputs H400
  SELECT metaphone("hello");
      -- Outputs HL
  SELECT nysiis("hello");
      -- Outputs HAL
  SELECT match_rating_codex("hello");
      -- Outputs HLL
  SELECT levenshtein_distance("hello", "hello world");
      -- Outputs 6
  SELECT damerau_levenshtein_distance("hello", "hello world");
      -- Outputs 6
  SELECT hamming_distance("hello", "hello world");
      -- Outputs 6
  SELECT jaro_similarity("hello", "hello world");
      -- Outputs 0.8181818181818182
  SELECT jaro_winkler_similarity("hello", "hello world");
      -- Outputs 0.890909090909091
  SELECT match_rating_comparison("hello", "helloo");
      -- Outputs 1
  </code></pre></div>
  <p dir="auto">See <a href="https://jellyfish.readthedocs.io/en/latest/" rel="nofollow">the Jellyfish documentation</a> for an explanation of each of these functions.</p>
  </article></div>


node_id: MDEwOlJlcG9zaXRvcnkxNzUzMjE0OTc=
name: csv-diff
full_name: simonw/csv-diff
html_url: https://github.com/simonw/csv-diff
description: Python CLI tool and library for diffing CSV and JSON files
created_at: 2019-03-13T01:11:26Z
updated_at: 2025-04-15T07:21:25Z
pushed_at: 2024-09-06T05:20:04Z
language: Python
license: apache-2.0
topics: ["click", "csv", "csv-diff", "datasette-io", "datasette-tool", "diff", "git-scraping", "tsv-diff"]
visibility: public
default_branch: main
permissions: {"admin": false, "maintain": false, "push": false, "triage": false, "pull": true}
readme:
  # csv-diff

  [![PyPI](https://img.shields.io/pypi/v/csv-diff.svg)](https://pypi.org/project/csv-diff/)
  [![Changelog](https://img.shields.io/github/v/release/simonw/csv-diff?include_prereleases&label=changelog)](https://github.com/simonw/csv-diff/releases)
  [![Tests](https://github.com/simonw/csv-diff/workflows/Test/badge.svg)](https://github.com/simonw/csv-diff/actions?query=workflow%3ATest)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/csv-diff/blob/main/LICENSE)

  Tool for viewing the difference between two CSV, TSV or JSON files. See [Generating a commit log for San Francisco’s official list of trees](https://simonwillison.net/2019/Mar/13/tree-history/) (and the [sf-tree-history repo commit log](https://github.com/simonw/sf-tree-history/commits)) for background information on this project.

  ## Installation

      pip install csv-diff

  ## Usage

  Consider two CSV files:

  `one.csv`

      id,name,age
      1,Cleo,4
      2,Pancakes,2

  `two.csv`

      id,name,age
      1,Cleo,5
      3,Bailey,1

  `csv-diff` can show a human-readable summary of differences between the files:

      $ csv-diff one.csv two.csv --key=id
      1 row changed, 1 row added, 1 row removed

      1 row changed

        Row 1
          age: "4" => "5"

      1 row added

        id: 3
        name: Bailey
        age: 1

      1 row removed

        id: 2
        name: Pancakes
        age: 2

  The `--key=id` option means that the `id` column should be treated as the unique key, to identify which records have changed.

  The tool will automatically detect if your files are comma- or tab-separated. You can over-ride this automatic detection and force the tool to use a specific format using `--format=tsv` or `--format=csv`.

  You can also feed it JSON files, provided they are a JSON array of objects where each object has the same keys. Use `--format=json` if your input files are JSON.

  Use `--show-unchanged` to include full details of the unchanged values for rows with at least one change in the diff output:

      % csv-diff one.csv two.csv --key=id --show-unchanged
      1 row changed

        id: 1
          age: "4" => "5"

          Unchanged:
            name: "Cleo"

  ### JSON output

  You can use the `--json` option to get a machine-readable difference:

      $ csv-diff one.csv two.csv --key=id --json
      {
          "added": [
              {
                  "id": "3",
                  "name": "Bailey",
                  "age": "1"
              }
          ],
          "removed": [
              {
                  "id": "2",
                  "name": "Pancakes",
                  "age": "2"
              }
          ],
          "changed": [
              {
                  "key": "1",
                  "changes": {
                      "age": [
                          "4",
                          "5"
                      ]
                  }
              }
          ],
          "columns_added": [],
          "columns_removed": []
      }

  ### Adding templated extras

  You can specify additional keys to be displayed in the human-readable format using the `--extra` option:

      --extra name "Python format string with {id} for variables"

  For example, to output a link to `https://news.ycombinator.com/latest?id={id}` for each item with an ID, you could use this:

  ```bash
  csv-diff one.csv two.csv --key=id \
    --extra latest "https://news.ycombinator.com/latest?id={id}"
  ```
  These extras display something like this:
  ```
  1 row changed

    id: 41459472
      points: "24" => "25"
      numComments: "5" => "6"
    extras:
      latest: https://news.ycombinator.com/latest?id=41459472
  ```

  ## As a Python library

  You can also import the Python library into your own code like so:

      from csv_diff import load_csv, compare
      diff = compare(
          load_csv(open("one.csv"), key="id"),
          load_csv(open("two.csv"), key="id")
      )

  `diff` will now contain the same data structure as the output in the `--json` example above.

  If the columns in the CSV have changed, those added or removed columns will be ignored when calculating changes made to specific rows.

  ## As a Docker container

  ### Build the image

      $ docker build -t csvdiff .

  ### Run the container

      $ docker run --rm -v $(pwd):/files csvdiff

  Suppose current directory contains two csv files : one.csv two.csv

      $ docker run --rm -v $(pwd):/files csvdiff one.csv two.csv
      
  ## Alternatives

  - [csvdiff](https://github.com/aswinkarthik/csvdiff) is a "fast diff tool for comparing CSV files" - you may get better results from this than from `csv-diff` against larger files.

readme_html:
  <div id="readme" class="md" data-path="README.md"><article class="markdown-body entry-content container-lg" itemprop="text"><div class="markdown-heading" dir="auto"><h1 class="heading-element" dir="auto">csv-diff</h1><a id="user-content-csv-diff" class="anchor" aria-label="Permalink: csv-diff" href="#csv-diff"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto"><a href="https://pypi.org/project/csv-diff/" rel="nofollow"><img src="https://img.shields.io/pypi/v/csv-diff.svg" alt="PyPI" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/4e48d4bbbcd4e506c19f8cc384ea4bdd8b7646e8b67f06cb0904c2a373e92444/68747470733a2f2f696d672e736869656c64732e696f2f707970692f762f6373762d646966662e737667"></a>
  <a href="https://github.com/simonw/csv-diff/releases"><img src="https://img.shields.io/github/v/release/simonw/csv-diff?include_prereleases&label=changelog" alt="Changelog" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/d29036a7d947794fa7ae5e35a5a873a051c9c02fd47c432bd642572f279aa025/68747470733a2f2f696d672e736869656c64732e696f2f6769746875622f762f72656c656173652f73696d6f6e772f6373762d646966663f696e636c7564655f70726572656c6561736573266c6162656c3d6368616e67656c6f67"></a>
  <a href="https://github.com/simonw/csv-diff/actions?query=workflow%3ATest"><img src="https://github.com/simonw/csv-diff/workflows/Test/badge.svg" alt="Tests" style="max-width: 100%;"></a>
  <a href="https://github.com/simonw/csv-diff/blob/main/LICENSE"><img src="https://img.shields.io/badge/license-Apache%202.0-blue.svg" alt="License" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/c355f200ea90fddaa407b6eaab303663a669248ea3ca7b1fcf77dbe04ff5f48c/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6c6963656e73652d417061636865253230322e302d626c75652e737667"></a></p>
  <p dir="auto">Tool for viewing the difference between two CSV, TSV or JSON files. See <a href="https://simonwillison.net/2019/Mar/13/tree-history/" rel="nofollow">Generating a commit log for San Francisco’s official list of trees</a> (and the <a href="https://github.com/simonw/sf-tree-history/commits">sf-tree-history repo commit log</a>) for background information on this project.</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Installation</h2><a id="user-content-installation" class="anchor" aria-label="Permalink: Installation" href="#installation"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="pip install csv-diff"><pre class="notranslate"><code>pip install csv-diff
  </code></pre></div>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Usage</h2><a id="user-content-usage" class="anchor" aria-label="Permalink: Usage" href="#usage"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">Consider two CSV files:</p>
  <p dir="auto"><code>one.csv</code></p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="id,name,age
  1,Cleo,4
  2,Pancakes,2"><pre class="notranslate"><code>id,name,age
  1,Cleo,4
  2,Pancakes,2
  </code></pre></div>
  <p dir="auto"><code>two.csv</code></p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="id,name,age
  1,Cleo,5
  3,Bailey,1"><pre class="notranslate"><code>id,name,age
  1,Cleo,5
  3,Bailey,1
  </code></pre></div>
  <p dir="auto"><code>csv-diff</code> can show a human-readable summary of differences between the files:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ csv-diff one.csv two.csv --key=id
  1 row changed, 1 row added, 1 row removed

  1 row changed

    Row 1
      age: "4" => "5"

  1 row added

    id: 3
    name: Bailey
    age: 1

  1 row removed

    id: 2
    name: Pancakes
    age: 2"><pre class="notranslate"><code>$ csv-diff one.csv two.csv --key=id
  1 row changed, 1 row added, 1 row removed

  1 row changed

    Row 1
      age: "4" => "5"

  1 row added

    id: 3
    name: Bailey
    age: 1

  1 row removed

    id: 2
    name: Pancakes
    age: 2
  </code></pre></div>
  <p dir="auto">The <code>--key=id</code> option means that the <code>id</code> column should be treated as the unique key, to identify which records have changed.</p>
  <p dir="auto">The tool will automatically detect if your files are comma- or tab-separated. You can over-ride this automatic detection and force the tool to use a specific format using <code>--format=tsv</code> or <code>--format=csv</code>.</p>
  <p dir="auto">You can also feed it JSON files, provided they are a JSON array of objects where each object has the same keys. Use <code>--format=json</code> if your input files are JSON.</p>
  <p dir="auto">Use <code>--show-unchanged</code> to include full details of the unchanged values for rows with at least one change in the diff output:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="% csv-diff one.csv two.csv --key=id --show-unchanged
  1 row changed

    id: 1
      age: "4" => "5"

      Unchanged:
        name: "Cleo""><pre class="notranslate"><code>% csv-diff one.csv two.csv --key=id --show-unchanged
  1 row changed

    id: 1
      age: "4" => "5"

      Unchanged:
        name: "Cleo"
  </code></pre></div>
  <div class="markdown-heading" dir="auto"><h3 class="heading-element" dir="auto">JSON output</h3><a id="user-content-json-output" class="anchor" aria-label="Permalink: JSON output" href="#json-output"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">You can use the <code>--json</code> option to get a machine-readable difference:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ csv-diff one.csv two.csv --key=id --json
  {
      "added": [
          {
              "id": "3",
              "name": "Bailey",
              "age": "1"
          }
      ],
      "removed": [
          {
              "id": "2",
              "name": "Pancakes",
              "age": "2"
          }
      ],
      "changed": [
          {
              "key": "1",
              "changes": {
                  "age": [
                      "4",
                      "5"
                  ]
              }
          }
      ],
      "columns_added": [],
      "columns_removed": []
  }"><pre class="notranslate"><code>$ csv-diff one.csv two.csv --key=id --json
  {
      "added": [
          {
              "id": "3",
              "name": "Bailey",
              "age": "1"
          }
      ],
      "removed": [
          {
              "id": "2",
              "name": "Pancakes",
              "age": "2"
          }
      ],
      "changed": [
          {
              "key": "1",
              "changes": {
                  "age": [
                      "4",
                      "5"
                  ]
              }
          }
      ],
      "columns_added": [],
      "columns_removed": []
  }
  </code></pre></div>
  <div class="markdown-heading" dir="auto"><h3 class="heading-element" dir="auto">Adding templated extras</h3><a id="user-content-adding-templated-extras" class="anchor" aria-label="Permalink: Adding templated extras" href="#adding-templated-extras"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">You can specify additional keys to be displayed in the human-readable format using the <code>--extra</code> option:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="--extra name "Python format string with {id} for variables""><pre class="notranslate"><code>--extra name "Python format string with {id} for variables"
  </code></pre></div>
  <p dir="auto">For example, to output a link to <code>https://news.ycombinator.com/latest?id={id}</code> for each item with an ID, you could use this:</p>
  <div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="csv-diff one.csv two.csv --key=id \
    --extra latest "https://news.ycombinator.com/latest?id={id}""><pre>csv-diff one.csv two.csv --key=id \
    --extra latest <span class="pl-s"><span class="pl-pds">"</span>https://news.ycombinator.com/latest?id={id}<span class="pl-pds">"</span></span></pre></div>
  <p dir="auto">These extras display something like this:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="1 row changed

    id: 41459472
      points: "24" => "25"
      numComments: "5" => "6"
    extras:
      latest: https://news.ycombinator.com/latest?id=41459472"><pre class="notranslate"><code>1 row changed

    id: 41459472
      points: "24" => "25"
      numComments: "5" => "6"
    extras:
      latest: https://news.ycombinator.com/latest?id=41459472
  </code></pre></div>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">As a Python library</h2><a id="user-content-as-a-python-library" class="anchor" aria-label="Permalink: As a Python library" href="#as-a-python-library"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">You can also import the Python library into your own code like so:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="from csv_diff import load_csv, compare
  diff = compare(
      load_csv(open("one.csv"), key="id"),
      load_csv(open("two.csv"), key="id")
  )"><pre class="notranslate"><code>from csv_diff import load_csv, compare
  diff = compare(
      load_csv(open("one.csv"), key="id"),
      load_csv(open("two.csv"), key="id")
  )
  </code></pre></div>
  <p dir="auto"><code>diff</code> will now contain the same data structure as the output in the <code>--json</code> example above.</p>
  <p dir="auto">If the columns in the CSV have changed, those added or removed columns will be ignored when calculating changes made to specific rows.</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">As a Docker container</h2><a id="user-content-as-a-docker-container" class="anchor" aria-label="Permalink: As a Docker container" href="#as-a-docker-container"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <div class="markdown-heading" dir="auto"><h3 class="heading-element" dir="auto">Build the image</h3><a id="user-content-build-the-image" class="anchor" aria-label="Permalink: Build the image" href="#build-the-image"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ docker build -t csvdiff ."><pre class="notranslate"><code>$ docker build -t csvdiff .
  </code></pre></div>
  <div class="markdown-heading" dir="auto"><h3 class="heading-element" dir="auto">Run the container</h3><a id="user-content-run-the-container" class="anchor" aria-label="Permalink: Run the container" href="#run-the-container"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ docker run --rm -v $(pwd):/files csvdiff"><pre class="notranslate"><code>$ docker run --rm -v $(pwd):/files csvdiff
  </code></pre></div>
  <p dir="auto">Suppose current directory contains two csv files : one.csv two.csv</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ docker run --rm -v $(pwd):/files csvdiff one.csv two.csv"><pre class="notranslate"><code>$ docker run --rm -v $(pwd):/files csvdiff one.csv two.csv
  </code></pre></div>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Alternatives</h2><a id="user-content-alternatives" class="anchor" aria-label="Permalink: Alternatives" href="#alternatives"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <ul dir="auto">
  <li><a href="https://github.com/aswinkarthik/csvdiff">csvdiff</a> is a "fast diff tool for comparing CSV files" - you may get better results from this than from <code>csv-diff</code> against larger files.</li>
  </ul>
  </article></div>


node_id: MDEwOlJlcG9zaXRvcnkxNzU1NTAxMjc=
name: yaml-to-sqlite
full_name: simonw/yaml-to-sqlite
html_url: https://github.com/simonw/yaml-to-sqlite
description: Utility for converting YAML files to SQLite
created_at: 2019-03-14T04:49:08Z
updated_at: 2025-02-28T21:18:55Z
pushed_at: 2022-04-08T18:11:14Z
language: Python
license: apache-2.0
topics: ["datasette-io", "datasette-tool", "sqlite", "yaml"]
visibility: public
default_branch: main
permissions: {"admin": false, "maintain": false, "push": false, "triage": false, "pull": true}
readme:
  # yaml-to-sqlite

  [![PyPI](https://img.shields.io/pypi/v/yaml-to-sqlite.svg)](https://pypi.org/project/yaml-to-sqlite/)
  [![Changelog](https://img.shields.io/github/v/release/simonw/yaml-to-sqlite?include_prereleases&label=changelog)](https://github.com/simonw/yaml-to-sqlite/releases)
  [![Tests](https://github.com/simonw/yaml-to-sqlite/workflows/Test/badge.svg)](https://github.com/simonw/yaml-to-sqlite/actions?query=workflow%3ATest)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/yaml-to-sqlite/blob/main/LICENSE)

  Load the contents of a YAML file into a SQLite database table.

  ```
  $ yaml-to-sqlite --help
  Usage: yaml-to-sqlite [OPTIONS] DB_PATH TABLE YAML_FILE

    Convert YAML files to SQLite

  Options:
    --version             Show the version and exit.
    --pk TEXT             Column to use as a primary key
    --single-column TEXT  If YAML file is a list of values, populate this column
    --help                Show this message and exit.
  ```
  ## Usage

  Given a `news.yml` file containing the following:
  ```yaml
  - date: 2021-06-05
    body: |-
      [Datasette 0.57](https://docs.datasette.io/en/stable/changelog.html#v0-57) is out with an important security patch.
  - date: 2021-05-10
    body: |-
      [Django SQL Dashboard](https://simonwillison.net/2021/May/10/django-sql-dashboard/) is a new tool that brings a useful authenticated subset of Datasette to Django projects that are built on top of PostgreSQL.
  ```
  Running this command:
  ```bash
  $ yaml-to-sqlite news.db stories news.yml
  ```
  Will create a database file with this schema:
  ```bash
  $ sqlite-utils schema news.db
  CREATE TABLE [stories] (
     [date] TEXT,
     [body] TEXT
  );
  ```
  The `--pk` option can be used to set a column as the primary key for the table:

  ```bash
  $ yaml-to-sqlite news.db stories news.yml --pk date
  $ sqlite-utils schema news.db
  CREATE TABLE [stories] (
     [date] TEXT PRIMARY KEY,
     [body] TEXT
  );
  ```
  ## Single column YAML lists

  The `--single-column` option can be used when the YAML file is a list of values, for example a file called `dogs.yml` containing the following:

  ```yaml
  - Cleo
  - Pancakes
  - Nixie
  ```
  Running this command:
  ```bash
  $ yaml-to-sqlite dogs.db dogs.yaml --single-column=name
  ```
  Will create a single `dogs` table with a single `name` column that is the primary key:

  ```bash
  $ sqlite-utils schema dogs.db
  CREATE TABLE [dogs] (
     [name] TEXT PRIMARY KEY
  );
  $ sqlite-utils dogs.db 'select * from dogs' -t
  name
  --------
  Cleo
  Pancakes
  Nixie
  ```

readme_html:
  <div id="readme" class="md" data-path="README.md"><article class="markdown-body entry-content container-lg" itemprop="text"><div class="markdown-heading" dir="auto"><h1 class="heading-element" dir="auto">yaml-to-sqlite</h1><a id="user-content-yaml-to-sqlite" class="anchor" aria-label="Permalink: yaml-to-sqlite" href="#yaml-to-sqlite"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto"><a href="https://pypi.org/project/yaml-to-sqlite/" rel="nofollow"><img src="https://img.shields.io/pypi/v/yaml-to-sqlite.svg" alt="PyPI" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/027b22b0f0ee5ab8a480133a63c6cc2cf37c667a0be256030896c6c0539f4430/68747470733a2f2f696d672e736869656c64732e696f2f707970692f762f79616d6c2d746f2d73716c6974652e737667"></a>
  <a href="https://github.com/simonw/yaml-to-sqlite/releases"><img src="https://img.shields.io/github/v/release/simonw/yaml-to-sqlite?include_prereleases&label=changelog" alt="Changelog" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/29b7a6cafcce074d7fb91c0f7db2df5c95dbb720a8e395fcebda945f3bd367ac/68747470733a2f2f696d672e736869656c64732e696f2f6769746875622f762f72656c656173652f73696d6f6e772f79616d6c2d746f2d73716c6974653f696e636c7564655f70726572656c6561736573266c6162656c3d6368616e67656c6f67"></a>
  <a href="https://github.com/simonw/yaml-to-sqlite/actions?query=workflow%3ATest"><img src="https://github.com/simonw/yaml-to-sqlite/workflows/Test/badge.svg" alt="Tests" style="max-width: 100%;"></a>
  <a href="https://github.com/simonw/yaml-to-sqlite/blob/main/LICENSE"><img src="https://img.shields.io/badge/license-Apache%202.0-blue.svg" alt="License" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/c355f200ea90fddaa407b6eaab303663a669248ea3ca7b1fcf77dbe04ff5f48c/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6c6963656e73652d417061636865253230322e302d626c75652e737667"></a></p>
  <p dir="auto">Load the contents of a YAML file into a SQLite database table.</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ yaml-to-sqlite --help
  Usage: yaml-to-sqlite [OPTIONS] DB_PATH TABLE YAML_FILE

    Convert YAML files to SQLite

  Options:
    --version             Show the version and exit.
    --pk TEXT             Column to use as a primary key
    --single-column TEXT  If YAML file is a list of values, populate this column
    --help                Show this message and exit."><pre class="notranslate"><code>$ yaml-to-sqlite --help
  Usage: yaml-to-sqlite [OPTIONS] DB_PATH TABLE YAML_FILE

    Convert YAML files to SQLite

  Options:
    --version             Show the version and exit.
    --pk TEXT             Column to use as a primary key
    --single-column TEXT  If YAML file is a list of values, populate this column
    --help                Show this message and exit.
  </code></pre></div>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Usage</h2><a id="user-content-usage" class="anchor" aria-label="Permalink: Usage" href="#usage"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">Given a <code>news.yml</code> file containing the following:</p>
  <div class="highlight highlight-source-yaml notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="- date: 2021-06-05
    body: |-
      [Datasette 0.57](https://docs.datasette.io/en/stable/changelog.html#v0-57) is out with an important security patch.
  - date: 2021-05-10
    body: |-
      [Django SQL Dashboard](https://simonwillison.net/2021/May/10/django-sql-dashboard/) is a new tool that brings a useful authenticated subset of Datasette to Django projects that are built on top of PostgreSQL."><pre>- <span class="pl-ent">date</span>: <span class="pl-c1">2021-06-05</span>
    <span class="pl-ent">body</span>: <span class="pl-s">|-</span>
  <span class="pl-s">    [Datasette 0.57](https://docs.datasette.io/en/stable/changelog.html#v0-57) is out with an important security patch.</span>
  <span class="pl-s"></span>- <span class="pl-ent">date</span>: <span class="pl-c1">2021-05-10</span>
    <span class="pl-ent">body</span>: <span class="pl-s">|-</span>
  <span class="pl-s">    [Django SQL Dashboard](https://simonwillison.net/2021/May/10/django-sql-dashboard/) is a new tool that brings a useful authenticated subset of Datasette to Django projects that are built on top of PostgreSQL.</span></pre></div>
  <p dir="auto">Running this command:</p>
  <div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="$ yaml-to-sqlite news.db stories news.yml"><pre>$ yaml-to-sqlite news.db stories news.yml</pre></div>
  <p dir="auto">Will create a database file with this schema:</p>
  <div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="$ sqlite-utils schema news.db
  CREATE TABLE [stories] (
     [date] TEXT,
     [body] TEXT
  );"><pre>$ sqlite-utils schema news.db
  CREATE TABLE [stories] (
     [date] TEXT,
     [body] TEXT
  )<span class="pl-k">;</span></pre></div>
  <p dir="auto">The <code>--pk</code> option can be used to set a column as the primary key for the table:</p>
  <div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="$ yaml-to-sqlite news.db stories news.yml --pk date
  $ sqlite-utils schema news.db
  CREATE TABLE [stories] (
     [date] TEXT PRIMARY KEY,
     [body] TEXT
  );"><pre>$ yaml-to-sqlite news.db stories news.yml --pk date
  $ sqlite-utils schema news.db
  CREATE TABLE [stories] (
     [date] TEXT PRIMARY KEY,
     [body] TEXT
  )<span class="pl-k">;</span></pre></div>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Single column YAML lists</h2><a id="user-content-single-column-yaml-lists" class="anchor" aria-label="Permalink: Single column YAML lists" href="#single-column-yaml-lists"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">The <code>--single-column</code> option can be used when the YAML file is a list of values, for example a file called <code>dogs.yml</code> containing the following:</p>
  <div class="highlight highlight-source-yaml notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="- Cleo
  - Pancakes
  - Nixie"><pre>- <span class="pl-s">Cleo</span>
  - <span class="pl-s">Pancakes</span>
  - <span class="pl-s">Nixie</span></pre></div>
  <p dir="auto">Running this command:</p>
  <div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="$ yaml-to-sqlite dogs.db dogs.yaml --single-column=name"><pre>$ yaml-to-sqlite dogs.db dogs.yaml --single-column=name</pre></div>
  <p dir="auto">Will create a single <code>dogs</code> table with a single <code>name</code> column that is the primary key:</p>
  <div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="$ sqlite-utils schema dogs.db
  CREATE TABLE [dogs] (
     [name] TEXT PRIMARY KEY
  );
  $ sqlite-utils dogs.db 'select * from dogs' -t
  name
  --------
  Cleo
  Pancakes
  Nixie"><pre>$ sqlite-utils schema dogs.db
  CREATE TABLE [dogs] (
     [name] TEXT PRIMARY KEY
  )<span class="pl-k">;</span>
  $ sqlite-utils dogs.db <span class="pl-s"><span class="pl-pds">'</span>select * from dogs<span class="pl-pds">'</span></span> -t
  name
  --------
  Cleo
  Pancakes
  Nixie</pre></div>
  </article></div>


node_id: MDEwOlJlcG9zaXRvcnkxODQxNjg4NjQ=
name: datasette-render-html
full_name: simonw/datasette-render-html
html_url: https://github.com/simonw/datasette-render-html
description: Plugin for selectively rendering the HTML is specific columns
created_at: 2019-04-30T01:21:25Z
updated_at: 2025-03-18T21:43:26Z
pushed_at: 2023-03-09T02:46:43Z
homepage: https://datasette.io/plugins/datasette-render-html
language: Python
topics: ["datasette", "datasette-io", "datasette-plugin"]
visibility: public
default_branch: master
permissions: {"admin": false, "maintain": false, "push": false, "triage": false, "pull": true}
readme:
  # datasette-render-html

  [![PyPI](https://img.shields.io/pypi/v/datasette-render-html.svg)](https://pypi.org/project/datasette-render-html/)
  [![Changelog](https://img.shields.io/github/v/release/simonw/datasette-render-html?include_prereleases&label=changelog)](https://github.com/simonw/datasette-render-html/releases)
  [![Tests](https://github.com/simonw/datasette-render-html/workflows/Test/badge.svg)](https://github.com/simonw/datasette-render-html/actions?query=workflow%3ATest)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/datasette-render-html/blob/main/LICENSE)

  This Datasette plugin lets you configure Datasette to render specific columns as HTML in the table and row interfaces.

  This means you can store HTML in those columns and have it rendered as such on those pages.

  If you have a database called `docs.db` containing a `glossary` table and you want the `definition` column in that table to be rendered as HTML, you would use a `metadata.json` file that looks like this:

      {
          "databases": {
              "docs": {
                  "tables": {
                      "glossary": {
                          "plugins": {
                              "datasette-render-html": {
                                  "columns": ["definition"]
                              }
                          }
                      }
                  }
              }
          }
      }

  ## Installation

  Install this plugin in the same environment as Datasette.

      $ datasette install datasette-render-html

  ## Usage

  With the plugin installed, launch Datasette with the above `metadata.json` configuration:

      $ datasette -m metadata.json docs.db

  ## Security

  This plugin allows HTML to be rendered exactly as it is stored in the database. As such, you should be sure only to use this against columns with content that you trust - otherwise you could open yourself up to an [XSS attack](https://owasp.org/www-community/attacks/xss/).

  It's possible to configure this plugin to apply to columns with specific names across whole databases or the full Datasette instance, but doing so is not safe. It could open you up to XSS vulnerabilities where an attacker composes a SQL query that results in a column containing unsafe HTML.

  As such, you should only use this plugin against specific columns in specific tables, as shown in the example above.

readme_html:
  <div id="readme" class="md" data-path="README.md"><article class="markdown-body entry-content container-lg" itemprop="text"><div class="markdown-heading" dir="auto"><h1 class="heading-element" dir="auto">datasette-render-html</h1><a id="user-content-datasette-render-html" class="anchor" aria-label="Permalink: datasette-render-html" href="#datasette-render-html"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto"><a href="https://pypi.org/project/datasette-render-html/" rel="nofollow"><img src="https://img.shields.io/pypi/v/datasette-render-html.svg" alt="PyPI" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/450a68e82a87550c1d612840d405a7b59f8e0afc85cdbd1d074b521bef84de90/68747470733a2f2f696d672e736869656c64732e696f2f707970692f762f6461746173657474652d72656e6465722d68746d6c2e737667"></a>
  <a href="https://github.com/simonw/datasette-render-html/releases"><img src="https://img.shields.io/github/v/release/simonw/datasette-render-html?include_prereleases&label=changelog" alt="Changelog" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/3cf242a7692a30591261a80ef342f7fde25d95a889eb9fd25e57cc4f541180c6/68747470733a2f2f696d672e736869656c64732e696f2f6769746875622f762f72656c656173652f73696d6f6e772f6461746173657474652d72656e6465722d68746d6c3f696e636c7564655f70726572656c6561736573266c6162656c3d6368616e67656c6f67"></a>
  <a href="https://github.com/simonw/datasette-render-html/actions?query=workflow%3ATest"><img src="https://github.com/simonw/datasette-render-html/workflows/Test/badge.svg" alt="Tests" style="max-width: 100%;"></a>
  <a href="https://github.com/simonw/datasette-render-html/blob/main/LICENSE"><img src="https://img.shields.io/badge/license-Apache%202.0-blue.svg" alt="License" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/c355f200ea90fddaa407b6eaab303663a669248ea3ca7b1fcf77dbe04ff5f48c/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6c6963656e73652d417061636865253230322e302d626c75652e737667"></a></p>
  <p dir="auto">This Datasette plugin lets you configure Datasette to render specific columns as HTML in the table and row interfaces.</p>
  <p dir="auto">This means you can store HTML in those columns and have it rendered as such on those pages.</p>
  <p dir="auto">If you have a database called <code>docs.db</code> containing a <code>glossary</code> table and you want the <code>definition</code> column in that table to be rendered as HTML, you would use a <code>metadata.json</code> file that looks like this:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="{
      "databases": {
          "docs": {
              "tables": {
                  "glossary": {
                      "plugins": {
                          "datasette-render-html": {
                              "columns": ["definition"]
                          }
                      }
                  }
              }
          }
      }
  }"><pre class="notranslate"><code>{
      "databases": {
          "docs": {
              "tables": {
                  "glossary": {
                      "plugins": {
                          "datasette-render-html": {
                              "columns": ["definition"]
                          }
                      }
                  }
              }
          }
      }
  }
  </code></pre></div>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Installation</h2><a id="user-content-installation" class="anchor" aria-label="Permalink: Installation" href="#installation"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">Install this plugin in the same environment as Datasette.</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ datasette install datasette-render-html"><pre class="notranslate"><code>$ datasette install datasette-render-html
  </code></pre></div>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Usage</h2><a id="user-content-usage" class="anchor" aria-label="Permalink: Usage" href="#usage"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">With the plugin installed, launch Datasette with the above <code>metadata.json</code> configuration:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ datasette -m metadata.json docs.db"><pre class="notranslate"><code>$ datasette -m metadata.json docs.db
  </code></pre></div>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Security</h2><a id="user-content-security" class="anchor" aria-label="Permalink: Security" href="#security"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">This plugin allows HTML to be rendered exactly as it is stored in the database. As such, you should be sure only to use this against columns with content that you trust - otherwise you could open yourself up to an <a href="https://owasp.org/www-community/attacks/xss/" rel="nofollow">XSS attack</a>.</p>
  <p dir="auto">It's possible to configure this plugin to apply to columns with specific names across whole databases or the full Datasette instance, but doing so is not safe. It could open you up to XSS vulnerabilities where an attacker composes a SQL query that results in a column containing unsafe HTML.</p>
  <p dir="auto">As such, you should only use this plugin against specific columns in specific tables, as shown in the example above.</p>
  </article></div>


node_id: MDEwOlJlcG9zaXRvcnkxODkzMjE2NzE=
name: datasette-jq
full_name: simonw/datasette-jq
html_url: https://github.com/simonw/datasette-jq
description: Datasette plugin that adds a custom SQL function for executing jq expressions against JSON values
created_at: 2019-05-30T01:06:31Z
updated_at: 2024-11-15T22:53:54Z
pushed_at: 2020-04-09T05:43:43Z
language: Python
license: apache-2.0
topics: ["datasette", "datasette-io", "datasette-plugin", "jq"]
visibility: public
default_branch: master
permissions: {"admin": false, "maintain": false, "push": false, "triage": false, "pull": true}
readme:
  # datasette-jq

  [![PyPI](https://img.shields.io/pypi/v/datasette-jq.svg)](https://pypi.org/project/datasette-jq/)
  [![CircleCI](https://circleci.com/gh/simonw/datasette-jq.svg?style=svg)](https://circleci.com/gh/simonw/datasette-jq)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/datasette-jq/blob/master/LICENSE)

  Datasette plugin that adds custom SQL functions for executing [jq](https://stedolan.github.io/jq/) expressions against JSON values.

  Install this plugin in the same environment as Datasette to enable the `jq()` SQL function.

  Usage:

      select jq(
          column_with_json,
          "{top_3: .classifiers[:3], v: .version}"
      )

  See [the jq manual](https://stedolan.github.io/jq/manual/#Basicfilters) for full details of supported expression syntax.

  ## Interactive demo

  You can try this plugin out at [datasette-jq-demo.datasette.io](https://datasette-jq-demo.datasette.io/)

  Sample query:

      select package, "https://pypi.org/project/" || package || "/" as url,
      jq(info, "{summary: .info.summary, author: .info.author, versions: .releases|keys|reverse}")
      from packages

  [Try this query out](https://datasette-jq-demo.datasette.io/demo?sql=select+package%2C+%22https%3A%2F%2Fpypi.org%2Fproject%2F%22+%7C%7C+package+%7C%7C+%22%2F%22+as+url%2C%0D%0Ajq%28info%2C+%22%7Bsummary%3A+.info.summary%2C+author%3A+.info.author%2C+versions%3A+.releases%7Ckeys%7Creverse%7D%22%29%0D%0Afrom+packages) in the interactive demo.

readme_html:
  <div id="readme" class="md" data-path="README.md"><article class="markdown-body entry-content container-lg" itemprop="text"><div class="markdown-heading" dir="auto"><h1 class="heading-element" dir="auto">datasette-jq</h1><a id="user-content-datasette-jq" class="anchor" aria-label="Permalink: datasette-jq" href="#datasette-jq"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto"><a href="https://pypi.org/project/datasette-jq/" rel="nofollow"><img src="https://img.shields.io/pypi/v/datasette-jq.svg" alt="PyPI" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/a0d601b9df707dcfd7047fc20ab74400f00cb038afe7f59cd742fbd29b1cadbc/68747470733a2f2f696d672e736869656c64732e696f2f707970692f762f6461746173657474652d6a712e737667"></a>
  <a href="https://circleci.com/gh/simonw/datasette-jq" rel="nofollow"><img src="https://circleci.com/gh/simonw/datasette-jq.svg?style=svg" alt="CircleCI" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/93fca742694aebc774476c8d4463e70af4929c6f8705512e8cc112da6733a44a/68747470733a2f2f636972636c6563692e636f6d2f67682f73696d6f6e772f6461746173657474652d6a712e7376673f7374796c653d737667"></a>
  <a href="https://github.com/simonw/datasette-jq/blob/master/LICENSE"><img src="https://img.shields.io/badge/license-Apache%202.0-blue.svg" alt="License" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/c355f200ea90fddaa407b6eaab303663a669248ea3ca7b1fcf77dbe04ff5f48c/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6c6963656e73652d417061636865253230322e302d626c75652e737667"></a></p>
  <p dir="auto">Datasette plugin that adds custom SQL functions for executing <a href="https://stedolan.github.io/jq/" rel="nofollow">jq</a> expressions against JSON values.</p>
  <p dir="auto">Install this plugin in the same environment as Datasette to enable the <code>jq()</code> SQL function.</p>
  <p dir="auto">Usage:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="select jq(
      column_with_json,
      "{top_3: .classifiers[:3], v: .version}"
  )"><pre class="notranslate"><code>select jq(
      column_with_json,
      "{top_3: .classifiers[:3], v: .version}"
  )
  </code></pre></div>
  <p dir="auto">See <a href="https://stedolan.github.io/jq/manual/#Basicfilters" rel="nofollow">the jq manual</a> for full details of supported expression syntax.</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Interactive demo</h2><a id="user-content-interactive-demo" class="anchor" aria-label="Permalink: Interactive demo" href="#interactive-demo"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">You can try this plugin out at <a href="https://datasette-jq-demo.datasette.io/" rel="nofollow">datasette-jq-demo.datasette.io</a></p>
  <p dir="auto">Sample query:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="select package, "https://pypi.org/project/" || package || "/" as url,
  jq(info, "{summary: .info.summary, author: .info.author, versions: .releases|keys|reverse}")
  from packages"><pre class="notranslate"><code>select package, "https://pypi.org/project/" || package || "/" as url,
  jq(info, "{summary: .info.summary, author: .info.author, versions: .releases|keys|reverse}")
  from packages
  </code></pre></div>
  <p dir="auto"><a href="https://datasette-jq-demo.datasette.io/demo?sql=select+package%2C+%22https%3A%2F%2Fpypi.org%2Fproject%2F%22+%7C%7C+package+%7C%7C+%22%2F%22+as+url%2C%0D%0Ajq%28info%2C+%22%7Bsummary%3A+.info.summary%2C+author%3A+.info.author%2C+versions%3A+.releases%7Ckeys%7Creverse%7D%22%29%0D%0Afrom+packages" rel="nofollow">Try this query out</a> in the interactive demo.</p>
  </article></div>


node_id: MDEwOlJlcG9zaXRvcnkxOTA5NTA3ODE=
name: datasette-bplist
full_name: simonw/datasette-bplist
html_url: https://github.com/simonw/datasette-bplist
description: Datasette plugin for working with Apple's binary plist format
created_at: 2019-06-09T01:15:01Z
updated_at: 2024-04-08T10:27:44Z
pushed_at: 2023-02-17T15:56:46Z
language: Python
license: apache-2.0
topics: ["bplist", "datasette", "datasette-io", "datasette-plugin"]
visibility: public
default_branch: master
permissions: {"admin": false, "maintain": false, "push": false, "triage": false, "pull": true}
readme:
  # datasette-bplist

  [![PyPI](https://img.shields.io/pypi/v/datasette-bplist.svg)](https://pypi.org/project/datasette-bplist/)
  [![CircleCI](https://circleci.com/gh/simonw/datasette-bplist.svg?style=svg)](https://circleci.com/gh/simonw/datasette-bplist)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/datasette-bplist/blob/master/LICENSE)

  Datasette plugin for working with Apple's [binary plist](https://en.wikipedia.org/wiki/Property_list) format.

  This plugin adds two features: a display hook and a SQL function.

  The display hook will detect any database values that are encoded using the binary plist format. It will decode them, convert them into JSON and display them pretty-printed in the Datasette UI.

  The SQL function `bplist_to_json(value)` can be used inside a SQL query to convert a binary plist value into a JSON string. This can then be used with SQLite's `json_extract()` function or with the [datasette-jq](https://github.com/simonw/datasette-jq) plugin to further analyze that data as part of a SQL query.

  Install this plugin in the same environment as Datasette to enable this new functionality:

      pip install datasette-bplist

  ## Trying it out

  If you use a Mac you already have plenty of SQLite databases that contain binary plist data.

  One example is the database that powers the Apple Photos app.

  This database tends to be locked, so you will need to create a copy of the database in order to run queries against it:

      cp ~/Pictures/Photos\ Library.photoslibrary/database/photos.db /tmp/photos.db

  The database also makes use of custom SQLite extensions which prevent it from opening in Datasette.

  You can work around this by exporting the data that you want to experiment with into a new SQLite file.

  I recommend trying this plugin against the `RKMaster_dataNote` table, which contains plist-encoded EXIF metadata about the photos you have taken.

  You can export that table into a fresh database like so:

      sqlite3 /tmp/photos.db ".dump RKMaster_dataNote" | sqlite3 /tmp/exif.db

  Now run `datasette /tmp/exif.db` and you can start trying out the plugin.

  ## Using the bplist_to_json() SQL function

  Once you have the `exif.db` demo working, you can try the `bplist_to_json()` SQL function.

  Here's a query that shows the camera lenses you have used the most often to take photos:

      select
          json_extract(
              bplist_to_json(value),
              "$.{Exif}.LensModel"
          ) as lens,
          count(*) as n
      from RKMaster_dataNote
      group by lens
      order by n desc;

  If you have a large number of photos this query can take a long time to execute, so you may need to increase the SQL time limit enforced by Datasette like so:

      $ datasette /tmp/exif.db \
          --config sql_time_limit_ms:10000

  Here's another query, showing the time at which you took every photo in your library which is classified as as screenshot:

      select
          attachedToId,
          json_extract(
              bplist_to_json(value),
              "$.{Exif}.DateTimeOriginal"
          )
      from RKMaster_dataNote
      where
          json_extract(
              bplist_to_json(value),
              "$.{Exif}.UserComment"
          ) = "Screenshot"

  And if you install the [datasette-cluster-map](https://github.com/simonw/datasette-cluster-map) plugin, this query will show you a map of your most recent 1000 photos:

      select
          *, 
          json_extract(
              bplist_to_json(value),
              "$.{GPS}.Latitude"
          ) as latitude,
          -json_extract(
              bplist_to_json(value),
              "$.{GPS}.Longitude"
          ) as longitude,
          json_extract(
              bplist_to_json(value),
              "$.{Exif}.DateTimeOriginal"
          ) as datetime
      from
          RKMaster_dataNote
      where
          latitude is not null
      order by
          attachedToId desc

readme_html:
  <div id="readme" class="md" data-path="README.md"><article class="markdown-body entry-content container-lg" itemprop="text"><div class="markdown-heading" dir="auto"><h1 class="heading-element" dir="auto">datasette-bplist</h1><a id="user-content-datasette-bplist" class="anchor" aria-label="Permalink: datasette-bplist" href="#datasette-bplist"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto"><a href="https://pypi.org/project/datasette-bplist/" rel="nofollow"><img src="https://img.shields.io/pypi/v/datasette-bplist.svg" alt="PyPI" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/bc728f100110ba61f365e2315f1aaa38f49c56c58da9b8fe4a1770e3a8ae6778/68747470733a2f2f696d672e736869656c64732e696f2f707970692f762f6461746173657474652d62706c6973742e737667"></a>
  <a href="https://circleci.com/gh/simonw/datasette-bplist" rel="nofollow"><img src="https://circleci.com/gh/simonw/datasette-bplist.svg?style=svg" alt="CircleCI" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/eb6376ad0af308a8d0c21c94bf450734e5b4ed63edb8901bdc45ae39ffcde84b/68747470733a2f2f636972636c6563692e636f6d2f67682f73696d6f6e772f6461746173657474652d62706c6973742e7376673f7374796c653d737667"></a>
  <a href="https://github.com/simonw/datasette-bplist/blob/master/LICENSE"><img src="https://img.shields.io/badge/license-Apache%202.0-blue.svg" alt="License" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/c355f200ea90fddaa407b6eaab303663a669248ea3ca7b1fcf77dbe04ff5f48c/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6c6963656e73652d417061636865253230322e302d626c75652e737667"></a></p>
  <p dir="auto">Datasette plugin for working with Apple's <a href="https://en.wikipedia.org/wiki/Property_list" rel="nofollow">binary plist</a> format.</p>
  <p dir="auto">This plugin adds two features: a display hook and a SQL function.</p>
  <p dir="auto">The display hook will detect any database values that are encoded using the binary plist format. It will decode them, convert them into JSON and display them pretty-printed in the Datasette UI.</p>
  <p dir="auto">The SQL function <code>bplist_to_json(value)</code> can be used inside a SQL query to convert a binary plist value into a JSON string. This can then be used with SQLite's <code>json_extract()</code> function or with the <a href="https://github.com/simonw/datasette-jq">datasette-jq</a> plugin to further analyze that data as part of a SQL query.</p>
  <p dir="auto">Install this plugin in the same environment as Datasette to enable this new functionality:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="pip install datasette-bplist"><pre class="notranslate"><code>pip install datasette-bplist
  </code></pre></div>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Trying it out</h2><a id="user-content-trying-it-out" class="anchor" aria-label="Permalink: Trying it out" href="#trying-it-out"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">If you use a Mac you already have plenty of SQLite databases that contain binary plist data.</p>
  <p dir="auto">One example is the database that powers the Apple Photos app.</p>
  <p dir="auto">This database tends to be locked, so you will need to create a copy of the database in order to run queries against it:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="cp ~/Pictures/Photos\ Library.photoslibrary/database/photos.db /tmp/photos.db"><pre class="notranslate"><code>cp ~/Pictures/Photos\ Library.photoslibrary/database/photos.db /tmp/photos.db
  </code></pre></div>
  <p dir="auto">The database also makes use of custom SQLite extensions which prevent it from opening in Datasette.</p>
  <p dir="auto">You can work around this by exporting the data that you want to experiment with into a new SQLite file.</p>
  <p dir="auto">I recommend trying this plugin against the <code>RKMaster_dataNote</code> table, which contains plist-encoded EXIF metadata about the photos you have taken.</p>
  <p dir="auto">You can export that table into a fresh database like so:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="sqlite3 /tmp/photos.db ".dump RKMaster_dataNote" | sqlite3 /tmp/exif.db"><pre class="notranslate"><code>sqlite3 /tmp/photos.db ".dump RKMaster_dataNote" | sqlite3 /tmp/exif.db
  </code></pre></div>
  <p dir="auto">Now run <code>datasette /tmp/exif.db</code> and you can start trying out the plugin.</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Using the bplist_to_json() SQL function</h2><a id="user-content-using-the-bplist_to_json-sql-function" class="anchor" aria-label="Permalink: Using the bplist_to_json() SQL function" href="#using-the-bplist_to_json-sql-function"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">Once you have the <code>exif.db</code> demo working, you can try the <code>bplist_to_json()</code> SQL function.</p>
  <p dir="auto">Here's a query that shows the camera lenses you have used the most often to take photos:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="select
      json_extract(
          bplist_to_json(value),
          "$.{Exif}.LensModel"
      ) as lens,
      count(*) as n
  from RKMaster_dataNote
  group by lens
  order by n desc;"><pre class="notranslate"><code>select
      json_extract(
          bplist_to_json(value),
          "$.{Exif}.LensModel"
      ) as lens,
      count(*) as n
  from RKMaster_dataNote
  group by lens
  order by n desc;
  </code></pre></div>
  <p dir="auto">If you have a large number of photos this query can take a long time to execute, so you may need to increase the SQL time limit enforced by Datasette like so:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ datasette /tmp/exif.db \
      --config sql_time_limit_ms:10000"><pre class="notranslate"><code>$ datasette /tmp/exif.db \
      --config sql_time_limit_ms:10000
  </code></pre></div>
  <p dir="auto">Here's another query, showing the time at which you took every photo in your library which is classified as as screenshot:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="select
      attachedToId,
      json_extract(
          bplist_to_json(value),
          "$.{Exif}.DateTimeOriginal"
      )
  from RKMaster_dataNote
  where
      json_extract(
          bplist_to_json(value),
          "$.{Exif}.UserComment"
      ) = "Screenshot""><pre class="notranslate"><code>select
      attachedToId,
      json_extract(
          bplist_to_json(value),
          "$.{Exif}.DateTimeOriginal"
      )
  from RKMaster_dataNote
  where
      json_extract(
          bplist_to_json(value),
          "$.{Exif}.UserComment"
      ) = "Screenshot"
  </code></pre></div>
  <p dir="auto">And if you install the <a href="https://github.com/simonw/datasette-cluster-map">datasette-cluster-map</a> plugin, this query will show you a map of your most recent 1000 photos:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="select
      *, 
      json_extract(
          bplist_to_json(value),
          "$.{GPS}.Latitude"
      ) as latitude,
      -json_extract(
          bplist_to_json(value),
          "$.{GPS}.Longitude"
      ) as longitude,
      json_extract(
          bplist_to_json(value),
          "$.{Exif}.DateTimeOriginal"
      ) as datetime
  from
      RKMaster_dataNote
  where
      latitude is not null
  order by
      attachedToId desc"><pre class="notranslate"><code>select
      *, 
      json_extract(
          bplist_to_json(value),
          "$.{GPS}.Latitude"
      ) as latitude,
      -json_extract(
          bplist_to_json(value),
          "$.{GPS}.Longitude"
      ) as longitude,
      json_extract(
          bplist_to_json(value),
          "$.{Exif}.DateTimeOriginal"
      ) as datetime
  from
      RKMaster_dataNote
  where
      latitude is not null
  order by
      attachedToId desc
  </code></pre></div>
  </article></div>


node_id: MDEwOlJlcG9zaXRvcnkxOTEwMjI5Mjg=
name: datasette-render-binary
full_name: simonw/datasette-render-binary
html_url: https://github.com/simonw/datasette-render-binary
description: Datasette plugin for rendering binary data
created_at: 2019-06-09T15:25:52Z
updated_at: 2024-02-22T14:56:22Z
pushed_at: 2022-12-10T06:44:29Z
language: Python
license: apache-2.0
topics: ["datasette", "datasette-io", "datasette-plugin"]
visibility: public
default_branch: main
permissions: {"admin": false, "maintain": false, "push": false, "triage": false, "pull": true}
readme:
  # datasette-render-binary

  [![PyPI](https://img.shields.io/pypi/v/datasette-render-binary.svg)](https://pypi.org/project/datasette-render-binary/)
  [![Changelog](https://img.shields.io/github/v/release/simonw/datasette-render-binary?include_prereleases&label=changelog)](https://github.com/simonw/datasette-render-binary/releases)
  [![Tests](https://github.com/simonw/datasette-render-binary/workflows/Test/badge.svg)](https://github.com/simonw/datasette-render-binary/actions?query=workflow%3ATest)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/datasette-render-binary/blob/main/LICENSE)

  Datasette plugin for rendering binary data.

  Install this plugin in the same environment as Datasette to enable this new functionality:

      pip install datasette-render-binary

  Binary data in cells will now be rendered as a mixture of characters and octets.

  ![Screenshot](https://raw.githubusercontent.com/simonw/datasette-render-binary/master/example.png)

readme_html:
  <div id="readme" class="md" data-path="README.md"><article class="markdown-body entry-content container-lg" itemprop="text"><div class="markdown-heading" dir="auto"><h1 class="heading-element" dir="auto">datasette-render-binary</h1><a id="user-content-datasette-render-binary" class="anchor" aria-label="Permalink: datasette-render-binary" href="#datasette-render-binary"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto"><a href="https://pypi.org/project/datasette-render-binary/" rel="nofollow"><img src="https://img.shields.io/pypi/v/datasette-render-binary.svg" alt="PyPI" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/963be0ba98dca1f574b39db8ae90a0ac85227767c89d2ee7f8e490e3c3c75a28/68747470733a2f2f696d672e736869656c64732e696f2f707970692f762f6461746173657474652d72656e6465722d62696e6172792e737667"></a>
  <a href="https://github.com/simonw/datasette-render-binary/releases"><img src="https://img.shields.io/github/v/release/simonw/datasette-render-binary?include_prereleases&label=changelog" alt="Changelog" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/f1befbae39af69d371f3a938447746980c2759070b196072944632899ffbc789/68747470733a2f2f696d672e736869656c64732e696f2f6769746875622f762f72656c656173652f73696d6f6e772f6461746173657474652d72656e6465722d62696e6172793f696e636c7564655f70726572656c6561736573266c6162656c3d6368616e67656c6f67"></a>
  <a href="https://github.com/simonw/datasette-render-binary/actions?query=workflow%3ATest"><img src="https://github.com/simonw/datasette-render-binary/workflows/Test/badge.svg" alt="Tests" style="max-width: 100%;"></a>
  <a href="https://github.com/simonw/datasette-render-binary/blob/main/LICENSE"><img src="https://img.shields.io/badge/license-Apache%202.0-blue.svg" alt="License" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/c355f200ea90fddaa407b6eaab303663a669248ea3ca7b1fcf77dbe04ff5f48c/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6c6963656e73652d417061636865253230322e302d626c75652e737667"></a></p>
  <p dir="auto">Datasette plugin for rendering binary data.</p>
  <p dir="auto">Install this plugin in the same environment as Datasette to enable this new functionality:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="pip install datasette-render-binary"><pre class="notranslate"><code>pip install datasette-render-binary
  </code></pre></div>
  <p dir="auto">Binary data in cells will now be rendered as a mixture of characters and octets.</p>
  <p dir="auto"><a target="_blank" rel="noopener noreferrer nofollow" href="https://raw.githubusercontent.com/simonw/datasette-render-binary/master/example.png"><img src="https://raw.githubusercontent.com/simonw/datasette-render-binary/master/example.png" alt="Screenshot" style="max-width: 100%;"></a></p>
  </article></div>


node_id: MDEwOlJlcG9zaXRvcnkxOTUwODcxMzc=
name: datasette-auth-github
full_name: simonw/datasette-auth-github
html_url: https://github.com/simonw/datasette-auth-github
description: Datasette plugin that authenticates users against GitHub
created_at: 2019-07-03T16:02:53Z
updated_at: 2026-03-25T01:17:00Z
pushed_at: 2025-11-16T00:15:34Z
homepage: https://datasette-auth-github-demo.datasette.io/
language: Python
license: apache-2.0
topics: ["asgi", "datasette", "datasette-io", "datasette-plugin"]
visibility: public
default_branch: main
permissions: {"admin": false, "maintain": false, "push": false, "triage": false, "pull": false}
pull_request_creation_policy: all
readme:
  # datasette-auth-github

  [![PyPI](https://img.shields.io/pypi/v/datasette-auth-github.svg)](https://pypi.org/project/datasette-auth-github/)
  [![Changelog](https://img.shields.io/github/v/release/simonw/datasette-auth-github?include_prereleases&label=changelog)](https://github.com/simonw/datasette-auth-github/releases)
  [![Tests](https://github.com/simonw/datasette-auth-github/workflows/Test/badge.svg)](https://github.com/simonw/datasette-auth-github/actions?query=workflow%3ATest)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/datasette-auth-github/blob/main/LICENSE)

  Datasette plugin that authenticates users against GitHub.

  <!-- toc -->

  - [Setup instructions](#setup-instructions)
  - [The authenticated actor](#the-authenticated-actor)
  - [Restricting access to specific users](#restricting-access-to-specific-users)
  - [Restricting access to specific GitHub organizations or teams](#restricting-access-to-specific-github-organizations-or-teams)
  - [What to do if a user is removed from an organization or team](#what-to-do-if-a-user-is-removed-from-an-organization-or-team)

  <!-- tocstop -->

  ## Setup instructions

  * Install the plugin: `datasette install datasette-auth-github`
  * Create a GitHub OAuth app: https://github.com/settings/applications/new
  * Set the Authorization callback URL to `http://127.0.0.1:8001/-/github-auth-callback`
  * Create a `metadata.json` file with the following structure:

  ```json
  {
      "title": "datasette-auth-github demo",
      "plugins": {
          "datasette-auth-github": {
              "client_id": {"$env": "GITHUB_CLIENT_ID"},
              "client_secret": {"$env": "GITHUB_CLIENT_SECRET"}
          }
      }
  }
  ```

  Now you can start Datasette like this, passing in the secrets as environment variables:

      $ GITHUB_CLIENT_ID=XXX GITHUB_CLIENT_SECRET=YYY datasette \
          fixtures.db -m metadata.json

  Note that hard-coding secrets in `metadata.json` is a bad idea as they will be visible to anyone who can navigate to `/-/metadata`. Instead, we use Datasette's mechanism for [adding secret plugin configuration options](https://docs.datasette.io/en/stable/plugins.html#secret-configuration-values).

  By default anonymous users will still be able to interact with Datasette. If you wish all users to have to sign in with a GitHub account first, add this to your ``metadata.json``:

  ```json
  {
      "allow": {
          "id": "*"
      },
      "plugins": {
          "datasette-auth-github": {
              "...": "..."
          }
      }
  }
  ```
  ## The authenticated actor

  Visit `/-/actor` when signed in to see the shape of the authenticated actor. It should look something like this:

  ```json
  {
      "actor": {
          "display": "simonw",
          "gh_id": "9599",
          "gh_name": "Simon Willison",
          "gh_login": "simonw",
          "gh_email": "...",
          "gh_orgs": [
              "dogsheep",
              "datasette-project"
          ],
          "gh_teams": [
              "dogsheep/test"
          ]
      }
  }
  ```

  The `gh_orgs` and `gh_teams` properties will only be present if you used `load_teams` or `load_orgs`, documented below.

  ## Restricting access to specific users

  You can use Datasette's [permissions mechanism](https://docs.datasette.io/en/stable/authentication.html) to specify which user or users are allowed to access your instance. Here's how to restrict access to just GitHub user `simonw`:

  ```json
  {
      "allow": {
          "gh_login": "simonw"
      },
      "plugins": {
          "datasette-auth-github": {
              "...": "..."
          }
      }
  }
  ```

  This `"allow"` block can be positioned at the database, table or query level instead: see [Configuring permissions in metadata.json](https://docs.datasette.io/en/stable/authentication.html#configuring-permissions-in-metadata-json) for details.

  Note that GitHub allows users to change their username, and it is possible for other people to claim old usernames. If you are concerned that your users may change their usernames you can key the allow blocks against GitHub user IDs instead, which do not change:

  ```json
  {
      "allow": {
          "gh_id": "9599"
      }
  }
  ```

  ## Restricting access to specific GitHub organizations or teams

  You can also restrict access to users who are members of a specific GitHub organization.

  You'll need to configure the plugin to check if the user is a member of that organization when they first sign in. You can do that using the `"load_orgs"` plugin configuration option.

  Then you can use `"allow": {"gh_orgs": [...]}` to specify which organizations are allowed access.

  ```json
  {
      "plugins": {
          "datasette-auth-github": {
              "...": "...",
              "load_orgs": ["your-organization"]
          }
      },
      "allow": {
          "gh_orgs": "your-organization"
      }
  }
  ```

  If your organization is [arranged into teams](https://help.github.com/en/articles/organizing-members-into-teams) you can restrict access to a specific team like this:

  ```json
  {
      "plugins": {
          "datasette-auth-github": {
              "...": "...",
              "load_teams": [
                  "your-organization/staff",
                  "your-organization/engineering",
              ]
          }
      },
      "allows": {
          "gh_team": "your-organization/engineering"
      }
  }
  ```

  ## What to do if a user is removed from an organization or team

  A user's organization and team memberships are checked once, when they first sign in. Those teams and organizations are then persisted in the user's signed `ds_actor` cookie.

  This means that if a user is removed from an organization or team but still has a Datasette cookie, they will still be able to access that Datasette instance.

  You can remedy this by rotating the `DATASETTE_SECRET` environment variable any time you make changes to your GitHub organization members.

  Changing this value will cause all of your existing users to be signed out, by invalidating their cookies. When they sign back in again their new memberships will be recorded in a new cookie.

  See [Configuring the secret](https://docs.datasette.io/en/stable/settings.html?highlight=secret#configuring-the-secret) in the Datasette documentation for more details.

readme_html:
  <div id="readme" class="md" data-path="README.md"><article class="markdown-body entry-content container-lg" itemprop="text"><div class="markdown-heading" dir="auto"><h1 class="heading-element" dir="auto">datasette-auth-github</h1><a id="user-content-datasette-auth-github" class="anchor" aria-label="Permalink: datasette-auth-github" href="#datasette-auth-github"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto"><a href="https://pypi.org/project/datasette-auth-github/" rel="nofollow"><img src="https://img.shields.io/pypi/v/datasette-auth-github.svg" alt="PyPI" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/4395dab5dc19c503996915fb14d29cccdfa028aa579541685b16125d873d7550/68747470733a2f2f696d672e736869656c64732e696f2f707970692f762f6461746173657474652d617574682d6769746875622e737667"></a>
  <a href="https://github.com/simonw/datasette-auth-github/releases"><img src="https://img.shields.io/github/v/release/simonw/datasette-auth-github?include_prereleases&label=changelog" alt="Changelog" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/4d3d97a67f0b78d5fdbcb7f0bcdd912d673bfc34c2b0a3f66f1256dcd021d37b/68747470733a2f2f696d672e736869656c64732e696f2f6769746875622f762f72656c656173652f73696d6f6e772f6461746173657474652d617574682d6769746875623f696e636c7564655f70726572656c6561736573266c6162656c3d6368616e67656c6f67"></a>
  <a href="https://github.com/simonw/datasette-auth-github/actions?query=workflow%3ATest"><img src="https://github.com/simonw/datasette-auth-github/workflows/Test/badge.svg" alt="Tests" style="max-width: 100%;"></a>
  <a href="https://github.com/simonw/datasette-auth-github/blob/main/LICENSE"><img src="https://img.shields.io/badge/license-Apache%202.0-blue.svg" alt="License" style="max-width: 100%;" data-camo-src="https://camo.githubusercontent.com/c355f200ea90fddaa407b6eaab303663a669248ea3ca7b1fcf77dbe04ff5f48c/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6c6963656e73652d417061636865253230322e302d626c75652e737667"></a></p>
  <p dir="auto">Datasette plugin that authenticates users against GitHub.</p>

  <ul dir="auto">
  <li><a href="#setup-instructions">Setup instructions</a></li>
  <li><a href="#the-authenticated-actor">The authenticated actor</a></li>
  <li><a href="#restricting-access-to-specific-users">Restricting access to specific users</a></li>
  <li><a href="#restricting-access-to-specific-github-organizations-or-teams">Restricting access to specific GitHub organizations or teams</a></li>
  <li><a href="#what-to-do-if-a-user-is-removed-from-an-organization-or-team">What to do if a user is removed from an organization or team</a></li>
  </ul>

  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Setup instructions</h2><a id="user-content-setup-instructions" class="anchor" aria-label="Permalink: Setup instructions" href="#setup-instructions"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <ul dir="auto">
  <li>Install the plugin: <code>datasette install datasette-auth-github</code></li>
  <li>Create a GitHub OAuth app: <a href="https://github.com/settings/applications/new">https://github.com/settings/applications/new</a></li>
  <li>Set the Authorization callback URL to <code>http://127.0.0.1:8001/-/github-auth-callback</code></li>
  <li>Create a <code>metadata.json</code> file with the following structure:</li>
  </ul>
  <div class="highlight highlight-source-json notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="{
      "title": "datasette-auth-github demo",
      "plugins": {
          "datasette-auth-github": {
              "client_id": {"$env": "GITHUB_CLIENT_ID"},
              "client_secret": {"$env": "GITHUB_CLIENT_SECRET"}
          }
      }
  }"><pre>{
      <span class="pl-ent">"title"</span>: <span class="pl-s"><span class="pl-pds">"</span>datasette-auth-github demo<span class="pl-pds">"</span></span>,
      <span class="pl-ent">"plugins"</span>: {
          <span class="pl-ent">"datasette-auth-github"</span>: {
              <span class="pl-ent">"client_id"</span>: {<span class="pl-ent">"$env"</span>: <span class="pl-s"><span class="pl-pds">"</span>GITHUB_CLIENT_ID<span class="pl-pds">"</span></span>},
              <span class="pl-ent">"client_secret"</span>: {<span class="pl-ent">"$env"</span>: <span class="pl-s"><span class="pl-pds">"</span>GITHUB_CLIENT_SECRET<span class="pl-pds">"</span></span>}
          }
      }
  }</pre></div>
  <p dir="auto">Now you can start Datasette like this, passing in the secrets as environment variables:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ GITHUB_CLIENT_ID=XXX GITHUB_CLIENT_SECRET=YYY datasette \
      fixtures.db -m metadata.json"><pre class="notranslate"><code>$ GITHUB_CLIENT_ID=XXX GITHUB_CLIENT_SECRET=YYY datasette \
      fixtures.db -m metadata.json
  </code></pre></div>
  <p dir="auto">Note that hard-coding secrets in <code>metadata.json</code> is a bad idea as they will be visible to anyone who can navigate to <code>/-/metadata</code>. Instead, we use Datasette's mechanism for <a href="https://docs.datasette.io/en/stable/plugins.html#secret-configuration-values" rel="nofollow">adding secret plugin configuration options</a>.</p>
  <p dir="auto">By default anonymous users will still be able to interact with Datasette. If you wish all users to have to sign in with a GitHub account first, add this to your <code>metadata.json</code>:</p>
  <div class="highlight highlight-source-json notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="{
      "allow": {
          "id": "*"
      },
      "plugins": {
          "datasette-auth-github": {
              "...": "..."
          }
      }
  }"><pre>{
      <span class="pl-ent">"allow"</span>: {
          <span class="pl-ent">"id"</span>: <span class="pl-s"><span class="pl-pds">"</span>*<span class="pl-pds">"</span></span>
      },
      <span class="pl-ent">"plugins"</span>: {
          <span class="pl-ent">"datasette-auth-github"</span>: {
              <span class="pl-ent">"..."</span>: <span class="pl-s"><span class="pl-pds">"</span>...<span class="pl-pds">"</span></span>
          }
      }
  }</pre></div>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">The authenticated actor</h2><a id="user-content-the-authenticated-actor" class="anchor" aria-label="Permalink: The authenticated actor" href="#the-authenticated-actor"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">Visit <code>/-/actor</code> when signed in to see the shape of the authenticated actor. It should look something like this:</p>
  <div class="highlight highlight-source-json notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="{
      "actor": {
          "display": "simonw",
          "gh_id": "9599",
          "gh_name": "Simon Willison",
          "gh_login": "simonw",
          "gh_email": "...",
          "gh_orgs": [
              "dogsheep",
              "datasette-project"
          ],
          "gh_teams": [
              "dogsheep/test"
          ]
      }
  }"><pre>{
      <span class="pl-ent">"actor"</span>: {
          <span class="pl-ent">"display"</span>: <span class="pl-s"><span class="pl-pds">"</span>simonw<span class="pl-pds">"</span></span>,
          <span class="pl-ent">"gh_id"</span>: <span class="pl-s"><span class="pl-pds">"</span>9599<span class="pl-pds">"</span></span>,
          <span class="pl-ent">"gh_name"</span>: <span class="pl-s"><span class="pl-pds">"</span>Simon Willison<span class="pl-pds">"</span></span>,
          <span class="pl-ent">"gh_login"</span>: <span class="pl-s"><span class="pl-pds">"</span>simonw<span class="pl-pds">"</span></span>,
          <span class="pl-ent">"gh_email"</span>: <span class="pl-s"><span class="pl-pds">"</span>...<span class="pl-pds">"</span></span>,
          <span class="pl-ent">"gh_orgs"</span>: [
              <span class="pl-s"><span class="pl-pds">"</span>dogsheep<span class="pl-pds">"</span></span>,
              <span class="pl-s"><span class="pl-pds">"</span>datasette-project<span class="pl-pds">"</span></span>
          ],
          <span class="pl-ent">"gh_teams"</span>: [
              <span class="pl-s"><span class="pl-pds">"</span>dogsheep/test<span class="pl-pds">"</span></span>
          ]
      }
  }</pre></div>
  <p dir="auto">The <code>gh_orgs</code> and <code>gh_teams</code> properties will only be present if you used <code>load_teams</code> or <code>load_orgs</code>, documented below.</p>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Restricting access to specific users</h2><a id="user-content-restricting-access-to-specific-users" class="anchor" aria-label="Permalink: Restricting access to specific users" href="#restricting-access-to-specific-users"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">You can use Datasette's <a href="https://docs.datasette.io/en/stable/authentication.html" rel="nofollow">permissions mechanism</a> to specify which user or users are allowed to access your instance. Here's how to restrict access to just GitHub user <code>simonw</code>:</p>
  <div class="highlight highlight-source-json notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="{
      "allow": {
          "gh_login": "simonw"
      },
      "plugins": {
          "datasette-auth-github": {
              "...": "..."
          }
      }
  }"><pre>{
      <span class="pl-ent">"allow"</span>: {
          <span class="pl-ent">"gh_login"</span>: <span class="pl-s"><span class="pl-pds">"</span>simonw<span class="pl-pds">"</span></span>
      },
      <span class="pl-ent">"plugins"</span>: {
          <span class="pl-ent">"datasette-auth-github"</span>: {
              <span class="pl-ent">"..."</span>: <span class="pl-s"><span class="pl-pds">"</span>...<span class="pl-pds">"</span></span>
          }
      }
  }</pre></div>
  <p dir="auto">This <code>"allow"</code> block can be positioned at the database, table or query level instead: see <a href="https://docs.datasette.io/en/stable/authentication.html#configuring-permissions-in-metadata-json" rel="nofollow">Configuring permissions in metadata.json</a> for details.</p>
  <p dir="auto">Note that GitHub allows users to change their username, and it is possible for other people to claim old usernames. If you are concerned that your users may change their usernames you can key the allow blocks against GitHub user IDs instead, which do not change:</p>
  <div class="highlight highlight-source-json notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="{
      "allow": {
          "gh_id": "9599"
      }
  }"><pre>{
      <span class="pl-ent">"allow"</span>: {
          <span class="pl-ent">"gh_id"</span>: <span class="pl-s"><span class="pl-pds">"</span>9599<span class="pl-pds">"</span></span>
      }
  }</pre></div>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">Restricting access to specific GitHub organizations or teams</h2><a id="user-content-restricting-access-to-specific-github-organizations-or-teams" class="anchor" aria-label="Permalink: Restricting access to specific GitHub organizations or teams" href="#restricting-access-to-specific-github-organizations-or-teams"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">You can also restrict access to users who are members of a specific GitHub organization.</p>
  <p dir="auto">You'll need to configure the plugin to check if the user is a member of that organization when they first sign in. You can do that using the <code>"load_orgs"</code> plugin configuration option.</p>
  <p dir="auto">Then you can use <code>"allow": {"gh_orgs": [...]}</code> to specify which organizations are allowed access.</p>
  <div class="highlight highlight-source-json notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="{
      "plugins": {
          "datasette-auth-github": {
              "...": "...",
              "load_orgs": ["your-organization"]
          }
      },
      "allow": {
          "gh_orgs": "your-organization"
      }
  }"><pre>{
      <span class="pl-ent">"plugins"</span>: {
          <span class="pl-ent">"datasette-auth-github"</span>: {
              <span class="pl-ent">"..."</span>: <span class="pl-s"><span class="pl-pds">"</span>...<span class="pl-pds">"</span></span>,
              <span class="pl-ent">"load_orgs"</span>: [<span class="pl-s"><span class="pl-pds">"</span>your-organization<span class="pl-pds">"</span></span>]
          }
      },
      <span class="pl-ent">"allow"</span>: {
          <span class="pl-ent">"gh_orgs"</span>: <span class="pl-s"><span class="pl-pds">"</span>your-organization<span class="pl-pds">"</span></span>
      }
  }</pre></div>
  <p dir="auto">If your organization is <a href="https://help.github.com/en/articles/organizing-members-into-teams">arranged into teams</a> you can restrict access to a specific team like this:</p>
  <div class="highlight highlight-source-json notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="{
      "plugins": {
          "datasette-auth-github": {
              "...": "...",
              "load_teams": [
                  "your-organization/staff",
                  "your-organization/engineering",
              ]
          }
      },
      "allows": {
          "gh_team": "your-organization/engineering"
      }
  }"><pre>{
      <span class="pl-ent">"plugins"</span>: {
          <span class="pl-ent">"datasette-auth-github"</span>: {
              <span class="pl-ent">"..."</span>: <span class="pl-s"><span class="pl-pds">"</span>...<span class="pl-pds">"</span></span>,
              <span class="pl-ent">"load_teams"</span>: [
                  <span class="pl-s"><span class="pl-pds">"</span>your-organization/staff<span class="pl-pds">"</span></span>,
                  <span class="pl-s"><span class="pl-pds">"</span>your-organization/engineering<span class="pl-pds">"</span></span>,
              ]
          }
      },
      <span class="pl-ent">"allows"</span>: {
          <span class="pl-ent">"gh_team"</span>: <span class="pl-s"><span class="pl-pds">"</span>your-organization/engineering<span class="pl-pds">"</span></span>
      }
  }</pre></div>
  <div class="markdown-heading" dir="auto"><h2 class="heading-element" dir="auto">What to do if a user is removed from an organization or team</h2><a id="user-content-what-to-do-if-a-user-is-removed-from-an-organization-or-team" class="anchor" aria-label="Permalink: What to do if a user is removed from an organization or team" href="#what-to-do-if-a-user-is-removed-from-an-organization-or-team"><svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg></a></div>
  <p dir="auto">A user's organization and team memberships are checked once, when they first sign in. Those teams and organizations are then persisted in the user's signed <code>ds_actor</code> cookie.</p>
  <p dir="auto">This means that if a user is removed from an organization or team but still has a Datasette cookie, they will still be able to access that Datasette instance.</p>
  <p dir="auto">You can remedy this by rotating the <code>DATASETTE_SECRET</code> environment variable any time you make changes to your GitHub organization members.</p>
  <p dir="auto">Changing this value will cause all of your existing users to be signed out, by invalidating their cookies. When they sign back in again their new memberships will be recorded in a new cookie.</p>
  <p dir="auto">See <a href="https://docs.datasette.io/en/stable/settings.html?highlight=secret#configuring-the-secret" rel="nofollow">Configuring the secret</a> in the Datasette documentation for more details.</p>
  </article></div>


