docs: add sqlite documentation

Signed-off-by: Eric Callahan <arksine.code@gmail.com>

docs/changelog.md

@@ -61,15 +61,15 @@ The format is based on [Keep a Changelog].
 
 ### Changed
 
-- **build**: Bumped apprise to version `1.7.0`.
+- **build**: Bumped apprise to version `1.8.0`.
 - **build**: Bumped lmdb to version `1.4.1`
 - **build**: Bumped tornado to version `6.4.0`
-- **build**: Bumped jinja2 to version `3.1.3`
+- **build**: Bumped jinja2 to version `3.1.4`
 - **build**: Bumped zeroconf to version `0.131.0`
 - **build**: Bumped libnacl to version `2.1.0`
 - **build**: Bumped distro to version `1.9.0`
 - **build**: Bumped pillow to version `10.3.0`
-- **build**: Bumped streaming-form-data to version `1.13.0`
+- **build**: Bumped streaming-form-data to version `1.15.0`
 - **machine**: Added `ratos-configurator` to list of default allowed services
 - **update_manager**:  It is now required that an application be "allowed"
   for Moonraker to restart it after an update.
@@ -91,6 +91,9 @@ The format is based on [Keep a Changelog].
 - **gpio**:  Migrate from libgpiod to python-periphery
 - **authorization**:  The authorization module is now loaded as part of Moonraker's
   core.
+- **database**: Migrated the underlying database from LMDB to Sqlite.
+- **history**: Use dedicated SQL tables to store job history and job totals.
+- **authorization**: Use a dedicated SQL table to store user data.
 
 ## [0.8.0] - 2023-02-23
 

docs/installation.md

@@ -202,8 +202,7 @@ structure using the default data path of `$HOME/printer_data`.
 │   ├── moonraker.conf
 │   └── printer.cfg
 ├── database
-│   ├── data.mdb
+│   └── moonraker-sql.db
-│   └── lock.mdb
 ├── gcodes
 │   ├── test_gcode_one.gcode
 │   └── test_gcode_two.gcode
@@ -216,7 +215,7 @@ structure using the default data path of `$HOME/printer_data`.
 └── moonraker.asvc
 ```
 
-If it is not desirible for the files and folders to exist in these specific
+If it is not desirable for the files and folders to exist in these specific
 locations it is acceptable to use symbolic links.  For example, it is common
 for the gcode folder to be located at `$HOME/gcode_files`.  Rather than
 reconfigure Klipper's `virtual_sdcard` it may be desirable to create a
@@ -566,14 +565,29 @@ Retrieve the API Key via the browser from a trusted client:
 
         {"result": "8ce6ae5d354a4365812b83140ed62e4b"}
 
-### LMDB Database Backup and Restore
+### Database Backup and Restore
 
-Moonraker uses a [LMDB Database](http://www.lmdb.tech/doc/) for persistent
+Moonraker stores persistent data using an Sqlite database.  By default
-storage of procedurally generated data.  LMDB database files are platform
+the database file is located at `<data_folder>/database/moonraker-sql.db`.
-dependent, and thus cannot be easily transferred between different machines.
+API Endpoints are available to backup and restore the database.  All
-A file generated on a Raspberry Pi cannot be directly transferred to an x86
+backups are stored at `<data_folder>/backup/database/<backup_name>` and
-machine.  Likewise, a file generated on a 32-bit version of Linux cannot
+restored from the same location.  Database files may contain sensitive
-be transferred to a 64-bit machine.
+information, therefore they are not served by Moonraker.  Another protocol
+such as SCP, SMB, etc is required to transfer a backup off of the host.
+
+Alternatively it is possible to perform a manual backup by copying the
+existing database file when the Moonraker service has been stopped.
+Restoration can be performed by stopping the Moonraker service and
+overwriting the existing database with the backup.
+
+#### LDMB Database (deprecated)
+
+Previous versions of Moonraker used a [LMDB Database](http://www.lmdb.tech/doc/)
+for persistent storage of procedurally generated data.  LMDB database files are
+platform dependent, and thus cannot be easily transferred between different
+machines. A file generated on a Raspberry Pi cannot be directly transferred
+to an x86 machine.  Likewise, a file generated on a 32-bit version of Linux
+cannot be transferred to a 64-bit machine.
 
 Moonraker includes two scripts, `backup-database.sh` and `restore-database.sh`
 to help facilitate database backups and transfers.

docs/web_api.md

@@ -3555,8 +3555,10 @@ the request.  The entire settings object could be accessed by providing
 may be read by omitting the `key` argument, however as explained below it
 is not possible to modify a namespace without specifying a key.
 
-#### List namespaces
+#### List Database Info
-Lists all available namespaces.
+
+Lists all namespaces with read and/or write access.  Also lists database
+backup files.
 
 HTTP request:
 ```http
@@ -3574,14 +3576,21 @@ JSON-RPC request:
 
 Returns:
 
-An object containing an array of namespaces in the following format:
+An object containing an array of namespaces and an array of backup files.
 ```json
 {
     "namespaces": [
         "gcode_metadata",
-        "history",
+        "webcams",
-        "moonraker",
+        "update_manager",
-        "test_namespace"
+        "announcements",
+        "database",
+        "moonraker"
+    ],
+    "backups": [
+        "sqldb-backup-20240513-134542.db",
+        "testbackup.db",
+        "testbackup2.db"
     ]
 }
 ```
@@ -3701,6 +3710,180 @@ deleted item.
 }
 ```
 
+#### Compact Database
+
+Compacts and defragments the the sqlite database using the `VACUUM` command.
+This API cannot be requested when Klipper is printing.
+
+HTTP request:
+```http
+POST /server/database/compact
+```
+
+JSON-RPC request:
+```json
+{
+    "jsonrpc": "2.0",
+    "method": "server.database.compact",
+    "id": 4654
+}
+```
+Returns:
+An object containing the size of the database on disk before and after
+the database is compacted.
+```json
+{
+    "previous_size": 139264,
+    "new_size": 122880
+}
+```
+
+#### Backup Database
+
+Creates a backup of the current database.  The backup will be
+created in the `<data_path>/backup/database/<filename>`.
+
+This API cannot be requested when Klipper is printing.
+
+HTTP request:
+```http
+POST /server/database/backup
+Content-Type: application/json
+
+{
+    "filename": "sql-db-backup.db"
+}
+```
+
+JSON-RPC request:
+```json
+{
+    "jsonrpc": "2.0",
+    "method": "server.database.post_backup",
+    "params": {
+        "filename": "sql-db-backup.db"
+    },
+    "id": 4654
+}
+```
+
+Parameters:
+
+- `filename`: An optional file name for the backup file.  The default
+   is `sqldb-backup-<year><month><day>-<hour><minute><second>`.
+
+
+Returns:
+An object containing the path on disk to the backup.
+```json
+{
+    "backup_path": "/home/test/printer_data/backup/database/sql-db-backup.db"
+}
+```
+
+#### Delete a backup
+
+Deletes a previously backed up database.
+
+HTTP request:
+```http
+DELETE /server/database/backup
+Content-Type: application/json
+
+{
+    "filename": "sql-db-backup.db"
+}
+```
+
+JSON-RPC request:
+```json
+{
+    "jsonrpc": "2.0",
+    "method": "server.database.delete_backup",
+    "params": {
+        "filename": "sql-db-backup.db"
+    },
+    "id": 4654
+}
+```
+
+Parameters:
+
+- `filename`: The name of the backup file to delete.  Must be a valid
+  filename reported in by the [database list](#list-database-info) API.
+  This parameter must be provided.
+
+Returns:
+An object containing the path on disk to the backup file that was removed.
+```json
+{
+    "backup_path": "/home/test/printer_data/backup/database/sql-db-backup.db"
+}
+```
+
+#### Restore Database
+
+Restores a previously backed up sqlite database file. The backup
+must be located at `<data_path>/backup/database/<filename>`. The
+`<filename>` must be a valid filename reported in by the
+[database list](#list-database-info) API.
+
+This API cannot be requested when Klipper is printing.
+
+!!! Note
+    Moonraker will restart immediately after this request is processed.
+
+HTTP request:
+```http
+POST /server/database/restore
+Content-Type: application/json
+
+{
+    "filename": "sql-db-backup.db"
+}
+```
+
+JSON-RPC request:
+```json
+{
+    "jsonrpc": "2.0",
+    "method": "server.database.restore",
+    "params": {
+        "filename": "sql-db-backup.db"
+    },
+    "id": 4654
+}
+```
+
+Parameters:
+
+- `filename`: The name of the database file to restore.  Must be a valid
+  filename reported in by the [database list](#list-database-info) API.
+  This parameter must be provided.
+
+Returns:
+An object containing a list of restored namespaces and restored tables.
+```json
+{
+    "restored_tables": [
+        "table_registry",
+        "namespace_store",
+        "authorized_users",
+        "job_history",
+        "job_totals"
+    ],
+    "restored_namespaces": [
+        "database",
+        "fluidd",
+        "gcode_metadata",
+        "mainsail",
+        "moonraker",
+        "update_manager",
+        "webcams"
+    ]
+}
+```
+
 ### Job Queue APIs
 
 The following endpoints may be used to manage Moonraker's job queue.
@@ -7123,7 +7306,7 @@ receive the following:
 ### Debug APIs
 
 The APIs in this section are available when Moonraker the debug argument
-(`-g`) has been supplied via the command line.  Some API may also depend
+(`-g`) has been supplied via the command line.  Some APIs may also depend
 on Moonraker's configuration, ie: an optional component may choose to
 register a debug API.
 
@@ -7131,11 +7314,11 @@ register a debug API.
     Debug APIs may expose security vulnerabilities.  They should only be
     enabled by developers on secured machines.
 
-#### List Database Namespaces (debug)
+#### List Database Info (debug)
 
-Debug version of [List Namespaces](#list-namespaces). Return value includes
+Debug version of [List Database Info](#list-database-info). Returns
-namespaces exlusively reserved for Moonraker. Only availble when Moonraker's
+all namespaces, including those exlusively reserved for Moonraker.
-debug features are enabled.
+In addition, registered SQL tables are reported.
 
 
 HTTP request:
@@ -7152,14 +7335,42 @@ JSON-RPC request:
 }
 ```
 
+Returns:
+
+An object containing an array of namespaces, an array of tables, and
+an array of backup files.
+```json
+{
+    "namespaces": [
+        "gcode_metadata",
+        "webcams",
+        "update_manager",
+        "announcements",
+        "database",
+        "moonraker"
+    ],
+    "backups": [
+        "sqldb-backup-20240513-134542.db",
+        "testbackup.db",
+        "testbackup2.db"
+    ],
+    "tables": [
+        "job_history",
+        "job_totals",
+        "namespace_store",
+        "table_registry",
+        "authorized_users"
+    ]
+}
+```
+
 #### Get Database Item (debug)
 
 Debug version of [Get Database Item](#get-database-item).  Keys within
-protected and forbidden namespaces are accessible. Only availble when
+protected and forbidden namespaces are accessible.
-Moonraker's debug features are enabled.
 
 !!! Warning
-    Moonraker's forbidden namespaces include items such as user credentials.
+    Moonraker's forbidden namespaces may include items such as user credentials.
     This endpoint should NOT be implemented in front ends directly.
 
 HTTP request:
@@ -7182,8 +7393,7 @@ JSON-RPC request:
 #### Add Database Item (debug)
 
 Debug version of [Add Database Item](#add-database-item).  Keys within
-protected and forbidden namespaces may be added. Only availble when
+protected and forbidden namespaces may be added.
-Moonraker's debug features are enabled.
 
 !!! Warning
     This endpoint should be used for testing/debugging purposes only.
@@ -7219,8 +7429,7 @@ JSON-RPC request:
 #### Delete Database Item (debug)
 
 Debug version of [Delete Database Item](#delete-database-item).  Keys within
-protected and forbidden namespaces may be removed. Only availble when
+protected and forbidden namespaces may be removed.
-Moonraker's debug features are enabled.
 
 !!! Warning
     This endpoint should be used for testing/debugging purposes only.
@@ -7247,6 +7456,93 @@ JSON-RPC request:
 }
 ```
 
+#### Get Database Table
+
+Requests all the contents of a specified table.
+
+HTTP request:
+```http
+GET /debug/database/table?table=job_history
+```
+
+JSON-RPC request:
+```json
+{
+    "jsonrpc": "2.0",
+    "method": "debug.database.table",
+    "params": {
+        "table": "job_history"
+    },
+    "id": 4654
+}
+```
+
+Parameters:
+
+- `table`:  The name of the table to request.  This parameter must
+  be provided.
+
+Returns:
+
+An object with the table's name and a list of all rows contained
+within the table.  The `rowid` will always be included for each
+row, however it may be represented by an alias.  In the example
+below the alias for `rowid` is `job_id`.
+
+```json
+{
+    "table_name": "job_history",
+    "rows": [
+        {
+            "job_id": 1,
+            "user": "No User",
+            "filename": "active_test.gcode",
+            "status": "completed",
+            "start_time": 1690749153.2661753,
+            "end_time": 1690749173.076986,
+            "print_duration": 0.0,
+            "total_duration": 19.975574419135228,
+            "filament_used": 0.0,
+            "metadata": {
+                "size": 211,
+                "modified": 1635771217.0,
+                "uuid": "627371e0-faa5-4ced-8bb4-7017d29226fa",
+                "slicer": "Unknown",
+                "gcode_start_byte": 8,
+                "gcode_end_byte": 211
+            },
+            "auxiliary_data": [],
+            "instance_id": "default"
+        },
+        {
+            "job_id": 2,
+            "user": "No User",
+            "filename": "active_test.gcode",
+            "status": "completed",
+            "start_time": 1701262034.9242446,
+            "end_time": 1701262054.7332363,
+            "print_duration": 0.0,
+            "total_duration": 19.990913168992847,
+            "filament_used": 0.0,
+            "metadata": {
+                "size": 211,
+                "modified": 1635771217.0,
+                "uuid": "627371e0-faa5-4ced-8bb4-7017d29226fa",
+                "slicer": "Unknown",
+                "gcode_start_byte": 8,
+                "gcode_end_byte": 211
+            },
+            "auxiliary_data": {
+                "spool_ids": [
+                    2
+                ]
+            },
+            "instance_id": "default"
+        }
+    ]
+}
+```
+
 #### Test a notifier (debug)
 
 You can trigger a notifier manually using this endpoint.