s, s, and | s are. The html output mode is envisioned as being useful for CGI.
6. Writing results to a file
By default, sqlite3 sends query results to standard output. You can change this using the ".output" and ".once" commands. Just put the name of an output file as an argument to .output and all subsequent query results will be written to that file. Or use the .once command instead of .output and output will only be redirected for the single next command before reverting to the console. Use .output with no arguments to begin writing to standard output again. For example:
sqlite> .mode list
sqlite> .separator |
sqlite> .output test_file_1.txt
sqlite> select * from tbl1;
sqlite> .exit
$ cat test_file_1.txt
hello|10
goodbye|20
$
If the first character of the ".output" or ".once" filename is a pipe symbol ("|") then the remaining characters are treated as a command and the output is sent to that command. This makes it easy to pipe the results of a query into some other process. For example, the "open -f" command on a Mac opens a text editor to display the content that it reads from standard input. So to see the results of a query in a text editor, one could type:
sqlite3> .once '|open -f'
sqlite3> SELECT * FROM bigTable;
If the ".output" or ".once" commands have an argument of "-e" then output is collected into a temporary file and the system text editor is invoked on that text file. Thus, the command ".once -e" achieves the same result as ".once '|open -f'" but with the benefit of being portable across all systems.
If the ".output" or ".once" commands have a "-x" argument, that causes them to accumulate output as Comma-Separated-Values (CSV) in a temporary file, then invoke the default system utility for viewing CSV files (usually a spreadsheet program) on the result. This is a quick way of sending the result of a query to a spreadsheet for easy viewing:
sqlite3> .once -x
sqlite3> SELECT * FROM bigTable;
The ".excel" command is an alias for ".once -x". It does exactly the same thing.
6.1. File I/O Functions
The command-line shell adds two application-defined SQL functions that facilitate reading content from a file into a table column, and writing the content of a column into a file, respectively.
The readfile(X) SQL function reads the entire content of the file named X and returns that content as a BLOB. This can be used to load content into a table. For example:
sqlite> CREATE TABLE images(name TEXT, type TEXT, img BLOB);
sqlite> INSERT INTO images(name,type,img)
...> VALUES('icon','jpeg',readfile('icon.jpg'));
The writefile(X,Y) SQL function write the blob Y into the file named X and returns the number of bytes written. Use this function to extract the content of a single table column into a file. For example:
sqlite> SELECT writefile('icon.jpg',img) FROM images WHERE name='icon';
Note that the readfile(X) and writefile(X,Y) functions are extension functions and are not built into the core SQLite library. These routines are available as a loadable extension in the ext/misc/fileio.c source file in the SQLite source code repositories.
6.2. The edit() SQL funtion
The CLI has another build-in SQL function named edit(). Edit() takes one or two arguments. The first argument is a value - usually a large multi-line string to be edited. The second argument is the name of a text editor. If the second argument is omitted, the VISUAL environment variable is used. The edit() function writes its first argument into a temporary file, invokes the editor on the temporary file, rereads the file back into memory after the editor is done, then returns the edited text.
The edit() function can be used to make changes to large text values. For example:
sqlite> UPDATE docs SET body=edit(body) WHERE name='report-15';
In this example, the content of the docs.body field for the entry where docs.name is "report-15" will be sent to the editor. After the editor returns, the result will be written back into the docs.body field.
The default operation of edit() is to invoke a text editor. But by using an alternative edit program in the second argument, you can also get it to edit images or other non-text resources. For example, if you want to modify a JPEG image that happens to be stored in a field of a table, you could run:
sqlite> UPDATE pics SET img=edit(img,'gimp') WHERE id='pic-1542';
The edit program can also be used as a viewer, by simply ignoring the return value. For example, to merely look at the image above, you might run:
sqlite> SELECT length(edit(img,'gimp')) WHERE id='pic-1542';
7. Querying the database schema
The sqlite3 program provides several convenience commands that are useful for looking at the schema of the database. There is nothing that these commands do that cannot be done by some other means. These commands are provided purely as a shortcut.
For example, to see a list of the tables in the database, you can enter ".tables".
sqlite> .tables
tbl1
tbl2
sqlite>
The ".tables" command is similar to setting list mode then executing the following query:
SELECT name FROM sqlite_master
WHERE type IN ('table','view') AND name NOT LIKE 'sqlite_%'
ORDER BY 1
But the ".tables" command does more. It queries the sqlite_master table for all attached databases, not just the primary database. And it arranges its output into neat columns.
The ".indexes" command works in a similar way to list all of the indexes. If the ".indexes" command is given an argument which is the name of a table, then it shows just indexes on that table.
The ".schema" command shows the complete schema for the database, or for a single table if an optional tablename argument is provided:
sqlite> .schema
create table tbl1(one varchar(10), two smallint)
CREATE TABLE tbl2 (
f1 varchar(30) primary key,
f2 text,
f3 real
)
sqlite> .schema tbl2
CREATE TABLE tbl2 (
f1 varchar(30) primary key,
f2 text,
f3 real
)
sqlite>
The ".schema" command is roughly the same as setting list mode, then entering the following query:
SELECT sql FROM sqlite_master
ORDER BY tbl_name, type DESC, name
As with ".tables", the ".schema" command shows the schema for all attached databases. If you only want to see the schema for a single database (perhaps "main") then you can add an argument to ".schema" to restrict its output:
sqlite> .schema main.*
The ".schema" command can be augmented with the "--indent" option, in which case it tries to reformat the various CREATE statements of the schema so that they are more easily readable by humans.
The ".databases" command shows a list of all databases open in the current connection. There will always be at least 2. The first one is "main", the original database opened. The second is "temp", the database used for temporary tables. There may be additional databases listed for databases attached using the ATTACH statement. The first output column is the name the database is attached with, and the second column is the filename of the external file.
sqlite> .databases
The ".fullschema" dot-command works like the ".schema" command in that it displays the entire database schema. But ".fullschema" also includes dumps of the statistics tables "sqlite_stat1", "sqlite_stat3", and "sqlite_stat4", if they exist. The ".fullschema" command normally provides all of the information needed to exactly recreate a query plan for a specific query. When reporting suspected problems with the SQLite query planner to the SQLite development team, developers are requested to provide the complete ".fullschema" output as part of the trouble report. Note that the sqlite_stat3 and sqlite_stat4 tables contain samples of index entries and so might contain sensitive data, so do not send the ".fullschema" output of a proprietary database over a public channel.
8. CSV Import
Use the ".import" command to import CSV (comma separated value) data into an SQLite table. The ".import" command takes two arguments which are the name of the disk file from which CSV data is to be read and the name of the SQLite table into which the CSV data is to be inserted.
Note that it is important to set the "mode" to "csv" before running the ".import" command. This is necessary to prevent the command-line shell from trying to interpret the input file text as some other format.
sqlite> .mode csv
sqlite> .import C:/work/somedata.csv tab1
There are two cases to consider: (1) Table "tab1" does not previously exist and (2) table "tab1" does already exist.
In the first case, when the table does not previously exist, the table is automatically created and the content of the first row of the input CSV file is used to determine the name of all the columns in the table. In other words, if the table does not previously exist, the first row of the CSV file is interpreted to be column names and the actual data starts on the second row of the CSV file.
For the second case, when the table already exists, every row of the CSV file, including the first row, is assumed to be actual content. If the CSV file contains an initial row of column labels, that row will be read as data and inserted into the table. To avoid this, make sure that table does not previously exist.
9. CSV Export
To export an SQLite table (or part of a table) as CSV, simply set the "mode" to "csv" and then run a query to extract the desired rows of the table.
sqlite> .header on
sqlite> .mode csv
sqlite> .once c:/work/dataout.csv
sqlite> SELECT * FROM tab1;
sqlite> .system c:/work/dataout.csv
In the example above, the ".header on" line causes column labels to be printed as the first row of output. This means that the first row of the resulting CSV file will contain column labels. If column labels are not desired, set ".header off" instead. (The ".header off" setting is the default and can be omitted if the headers have not been previously turned on.)
The line ".once FILENAME" causes all query output to go into the named file instead of being printed on the console. In the example above, that line causes the CSV content to be written into a file named "C:/work/dataout.csv".
The final line of the example (the ".system c:/work/dataout.csv") has the same effect as double-clicking on the c:/work/dataout.csv file in windows. This will typically bring up a spreadsheet program to display the CSV file.
That command only works as written on Windows. The equivalent line on a Mac would be:
sqlite> .system open dataout.csv
On Linux and other unix systems you will need to enter something like:
sqlite> .system xdg-open dataout.csv
9.1. Export to Excel
To simplify export to a spreadsheet, the CLI provides the ".excel" command which captures the output of a single query and sends that output to the default spreadsheet program on the host computer. Use it like this:
sqlite> .excel
sqlite> SELECT * FROM tab;
The command above writes the output of the query as CSV into a temporary file, invokes the default handler for CSV files (usually the preferred spreadsheet program such as Excel or LibreOffice), then deletes the temporary file. This is essentially a short-hand method of doing the sequence of ".csv", ".once", and ".system" commands described above.
The ".excel" command is really an alias for ".once -x". The -x option to .once causes it to writes results as CSV into a temporary file that is named with a ".csv" suffix, then invoke the systems default handler for CSV files.
There is also a ".once -e" command which works similarly, except that it names the temporary file with a ".txt" suffix so that the default text editor for the system will be invoked, instead of the default spreadsheet.
10. Converting An Entire Database To An ASCII Text File
Use the ".dump" command to convert the entire contents of a database into a single ASCII text file. This file can be converted back into a database by piping it back into sqlite3.
A good way to make an archival copy of a database is this:
$ sqlite3 ex1 .dump | gzip -c >ex1.dump.gz
This generates a file named ex1.dump.gz that contains everything you need to reconstruct the database at a later time, or on another machine. To reconstruct the database, just type:
$ zcat ex1.dump.gz | sqlite3 ex2
The text format is pure SQL so you can also use the .dump command to export an SQLite database into other popular SQL database engines. Like this:
$ createdb ex2
$ sqlite3 ex1 .dump | psql ex2
11. Loading Extensions
You can add new custom application-defined SQL functions, collating sequences, virtual tables, and VFSes to the command-line shell at run-time using the ".load" command. First, convert the extension in to a DLL or shared library (as described in the Run-Time Loadable Extensions document) then type:
sqlite> .load /path/to/my_extension
Note that SQLite automatically adds the appropriate extension suffix (".dll" on windows, ".dylib" on Mac, ".so" on most other unixes) to the extension filename. It is generally a good idea to specify the full pathname of the extension.
SQLite computes the entry point for the extension based on the extension filename. To override this choice, simply add the name of the extension as a second argument to the ".load" command.
Source code for several useful extensions can be found in the ext/misc subdirectory of the SQLite source tree. You can use these extensions as-is, or as a basis for creating your own custom extensions to address your own particular needs.
12. Cryptographic Hashes Of Database Content
The ".sha3sum" dot-command computes a SHA3 hash of the content of the database. To be clear, the hash is computed over the database content, not its representation on disk. This means, for example, that a VACUUM or similar data-preserving transformation does not change the hash.
The ".sha3sum" command supports options "--sha3-224", "--sha3-256", "--sha3-384", and "--sha3-512" to define which variety of SHA3 to use for the hash. The default is SHA3-256.
The database schema (in the sqlite_master table) is not normally included in the hash, but can be added by the "--schema" option.
The ".sha3sum" command takes a single optional argument which is a LIKE pattern. If this option is present, only tables whose names match the LIKE pattern will be hashed.
The ".sha3sum" command is implemented with the help of the extension function "sha3_query()" that is included with the command-line shell.
13. Database Content Self-Tests
The ".selftest" command attempts to verify that a database is intact and is not corrupt. The .selftest command looks for a table in schema named "selftest" and defined as follows:
CREATE TABLE selftest(
tno INTEGER PRIMARY KEY, -- Test number
op TEXT, -- 'run' or 'memo'
cmd TEXT, -- SQL command to run, or text of "memo"
ans TEXT -- Expected result of the SQL command
);
The .selftest command reads the rows of the selftest table in selftest.tno order. For each 'memo' row, it writes the text in 'cmd' to the output. For each 'run' row, it runs the 'cmd' text as SQL and compares the result to the value in 'ans', and shows an error message if the results differ.
If there is no selftest table, the ".selftest" command runs PRAGMA integrity_check.
The ".selftest --init" command creates the selftest table if it does not already exists, then appends entries that check the SHA3 hash of the content of all tables. Subsequent runs of ".selftest" will verify that the database has not been changed in any way. To generates tests to verify that a subset of the tables are unchanged, simply run ".selftest --init" then DELETE the selftest rows that refer to tables that are not constant.
14. SQLAR Archive Support
The ".archive" dot-command (often abbreviated as ".ar") provides built-in support for the SQLite ARchive format. The interface is similar to that of the "tar" command on unix systems. Each invocation of the ".ar" command must specify a single command option. The following commands are available:
Option Long Option Purpose
-c --create Create a new archive containing specified files.
-x --extract Extract specified files from archive.
-t --list List the files in the archive.
-u --update Add files to existing archive.
As well as the command option, each invocation of ".ar" may specify one or more modifier options. Some modifier options require an argument, some do not. The following modifier options are available:
Option Long Option Purpose
-v --verbose List each file as it is processed.
-f FILE --file FILE If specified, use file FILE as the archive. Otherwise, assume that the current "main" database is the archive to be operated on.
-a FILE --append FILE Like --file, use file FILE as the archive, but open the file using the apndvfs VFS so that the archive will be appended to the end of FILE if FILE already exists.
-C DIR --directory DIR If specified, interpret all relative paths as relative to DIR, instead of the current working directory.
-n --dryrun Show the SQL that would be run to carry out the archive operation, but do not actually change anything.
-- -- All subsequent command line words are command arguments, not options.
Long and short style options may be mixed. For example, the following are equivalent:
-- Two ways to create a new archive named "new_archive.db" containing
-- files "file1", "file2" and "file3".
.ar -c --file new_archive.db file1 file2 file3
.ar -f new_archive.db --create file1 file2 file3
Alternatively, the first argument following to ".ar" may be the concatenation of the short form of all required options (without the "-" characters). In this case arguments for options requiring them are read from the command line next, and any remaining words are considered command arguments. For example:
-- Create a new archive "new_archive.db" containing files "file1" and
-- "file2" from directory "dir1".
.ar cCf dir1 new_archive.db file1 file2 file3
14.1. SQLAR Create Command
Create a new archive, overwriting any existing archive (either in the current "main" db or in the file specified by a --file option). Each argument following the options is a file to add to the archive. Directories are imported recursively. See above for examples.
14.2. SQLAR Extract Command
Extract files from the archive (either to the current working directory or to the directory specified by a --directory option). If there are no arguments following the options all files are extracted from the archive. Or, if there are arguments, they are the names of files to extract from the archive. Any specified directories are extracted recursively. It is an error if any specified files are not part of the archive.
-- Extract all files from the archive in the current "main" db to the
-- current working directory. List files as they are extracted.
.ar --extract --verbose
-- Extract file "file1" from archive "ar.db" to directory "dir1".
.ar fCx ar.db dir1 file1
14.3. SQLAR List Command
List the contents of the archive. If no arguments are specified, then all files are listed. Otherwise, only those specified as arguments are. Currently, the --verbose option does not change the behaviour of this command. That may change in the future.
-- List contents of archive in current "main" db..
.ar --list
14.4. SQLAR Update Command
This command works the same way as the --create command, except that it does not delete the current archive before commencing. New versions of files silently replace existing files with the same names, but otherwise the initial contents of the archive (if any) remain intact.
14.5. Operations On ZIP Archives
If FILE is a ZIP archive rather than an SQLAR, the ".archive" command still works, except that only --extract and --list operations are supported. ZIP archives are currently read-only to SQLite. (This limitation may be relaxed in a future release.)
14.6. SQL Used To Implement SQLAR Operations
The various SQLAR Archive commands are implemented using SQL statements. Application developers can easily add SQLAR Archive reading and writing support to their own projects by running the appropriate SQL.
To see what SQL statements are used to implement an SQLAR Archive operation, add the --dryrun or -n option. This causes the SQL to be displayed but inhibits the execution of the SQL.
The SQL statements used to implement SQLAR operations make use of various loadable extensions. These extensions are all available in the SQLite source tree in the ext/misc/ subfolder. The extensions needed for full SQLAR support include:
fileio.c — This extension adds SQL functions readfile() and writefile() for reading and writing content from files on disk. The fileio.c extension also includes fsdir() table-valued function for listing the contents of a directory and the lsname() function for converting numeric st_mode integers from the stat() system call into human-readable strings after the fashion of the "ls -l" command.
sqlar.c — This extension adds the sqlar_compress() and sqlar_uncompress() functions that are needed to compress and uncompress file content as it is insert and extracted from an SQLAR.
zipfile.c — This extension implements the "zipfile(FILE)" table-valued function which is used to read ZIP archives. This extension is only needed when reading ZIP archives instead of SQLAR archives.
appendvfs.c — This extension implements a new VFS that allows an SQLite database to be appended to some other file, such as an executable. This extension is only needed if the --append option to the .archive command is used.
15. Index Recommendations (SQLite Expert)
Note: This command is experimental. It may be removed or the interface modified in incompatible ways at some point in the future.
For most non-trivial SQL databases, the key to performance is creating the right SQL indexes. In this context "the right SQL indexes" means those that cause the queries that an application needs to optimize run fast. The ".expert" command can assist with this by proposing indexes that might assist with specific queries, were they present in the database.
The ".expert" command is issued first, followed by the SQL query on a separate line. For example, consider the following session:
sqlite> CREATE TABLE x1(a, b, c); -- Create table in database
sqlite> .expert
sqlite> SELECT * FROM x1 WHERE a=? AND b>?; -- Analyze this SELECT
CREATE INDEX x1_idx_000123a7 ON x1(a, b);
0|0|0|SEARCH TABLE x1 USING INDEX x1_idx_000123a7 (a=? AND b>?)
sqlite> CREATE INDEX x1ab ON x1(a, b); -- Create the recommended index
sqlite> .expert
sqlite> SELECT * FROM x1 WHERE a=? AND b>?; -- Re-analyze the same SELECT
(no new indexes)
0|0|0|SEARCH TABLE x1 USING INDEX x1ab (a=? AND b>?)
In the above, the user creates the database schema (a single table - "x1"), and then uses the ".expert" command to analyze a query, in this case "SELECT * FROM x1 WHERE a=? AND b>?". The shell tool recommends that the user create a new index (index "x1_idx_000123a7") and outputs the plan that the query would use in EXPLAIN QUERY PLAN format. The user then creates an index with an equivalent schema and runs the analysis on the same query again. This time the shell tool does not recommend any new indexes, and outputs the plan that SQLite will use for the query given the existing indexes.
The ".expert" command accepts the following options:
Option Purpose
--verbose If present, output a more verbose report for each query analyzed.
--sample PERCENT By default, the ".expert" command recommends indexes based on the query and database schema alone. This is similar to the way the SQLite query planner selects indexes for queries if the user has not run the ANALYZE command on the database to generate data distribution statistics.
If this option is passed a non-zero argument, the ".expert" command generates similar data distribution statistics for all indexes considered based on PERCENT percent of the rows currently stored in each database table. For databases with unusual data distributions, this may lead to better index recommendations, particularly if the application intends to run ANALYZE.
For small databases and modern CPUs, there is usually no reason not to pass "--sample 100". However, gathering data distribution statistics can be expensive for large database tables. If the operation is too slow, try passing a smaller value for the --sample option.
Th functionality described in this section may be integrated into other applications or tools using the SQLite expert extension code.
16. Other Dot Commands
There are many other dot-commands available in the command-line shell. See the ".help" command for a complete list for any particular version and build of SQLite.
17. Using sqlite3 in a shell script
One way to use sqlite3 in a shell script is to use "echo" or "cat" to generate a sequence of commands in a file, then invoke sqlite3 while redirecting input from the generated command file. This works fine and is appropriate in many circumstances. But as an added convenience, sqlite3 allows a single SQL command to be entered on the command line as a second argument after the database name. When the sqlite3 program is launched with two arguments, the second argument is passed to the SQLite library for processing, the query results are printed on standard output in list mode, and the program exits. This mechanism is designed to make sqlite3 easy to use in conjunction with programs like "awk". For example:
$ sqlite3 ex1 'select * from tbl1' |
> awk '{printf " |