Documentation Source Text

For command-line usage, add the short style command-line options immediately
following the "-A", without an intervening space.  All subsequent arguments
are considered to be part of the .archive command.  For example, the following 
commands are equivalent:

<codeblock>
sqlite3 new_archive.db -Acv file1 file2 file3
sqlite3 new_archive.db ".ar -cv file1 file2 file3"
</codeblock>

<p>
Long and short style options may be mixed. For example, the following are
equivalent:

<codeblock>

<title>Measuring and Reducing CPU Usage in SQLite</title>
<tcl>hd_keywords {CPU cycles used} {CPU performance measurement}</tcl>

<table_of_contents>

<h1>Overview</h1>

<p>The graph below shows the number of CPU cycles used by SQLite on a
standard workload, for all versions of SQLite going back about 9 years.
Recent versions of SQLite use less than a third of the CPU cycles 
compared to older versions.

<p>
This article describes how the SQLite developers measure CPU usage,
what those measurements actually mean, and the techniques used by
SQLite developers on their continuing quest to further reduce the
CPU usage of the SQLite library.

<p>Beginning with SQLite [version 3.10.0] ([dateof:3.10.0]), 
the unix OS interface will
attempt to resolve symbolic links and open the database file by its
canonical name.  Prior to version 3.10.0, opening a database file 
through a symbolic link was similar to opening a database file
that had multiple hard links and resulted in undefined behavior.</p>

<tcl>hd_fragment fork</tcl>
<h2> Carrying an open database connection across a fork() </h2>

<p> Do not open an SQLite database connection, then fork(), then try
to use that database connection in the child process.  All kinds of
locking problems will result and you can easily end up with a corrupt
database.  SQLite is not designed to support that kind of behavior.
Any database connection that is used in a child process must be opened
in the child process, not inherited from the parent.

<p> Do not even call [sqlite3_close()] on a database connection from a
child process if the connection was opened in the parent.  It is safe
to close the underlying file descriptor, but the [sqlite3_close()]
interface might invoke cleanup activities that will delete content out
from under the parent, leading to errors and perhaps even database
corruption.

<h1> Failure to sync</h1>

<p>In order to guarantee that database files are always consistent, SQLite
will occasionally ask the operating system to flush all pending writes to
persistent storage then wait for that flush to complete.  This is 
accomplished using the <tt>fsync()</tt> system call under unix and

limited to the value set by
[sqlite3_limit]([SQLITE_LIMIT_COLUMN],...).)^</p>

<tcl>hd_fragment uniqueidx {unique index}</tcl>
<p>^If the UNIQUE keyword appears between CREATE and INDEX then duplicate
index entries are not allowed.  ^Any attempt to insert a duplicate entry
will result in an error.  ^For the purposes of unique indices, all NULL values
are considered different from all other NULL values and are thus unique.
This is one of the two possible interpretations of the SQL-92 standard
(the language in the standard is ambiguous) and is the interpretation
followed by PostgreSQL, MySQL, Firebird, and Oracle.  Informix and
Microsoft SQL Server follow the other interpretation of the standard.</p>

<p>^If the optional IF NOT EXISTS clause is present and another index
with the same name already exists, then this command becomes a no-op.</p>

<center><table border=1>
  <tr><th>Expression Affinity   <th>Column Declared Type
  <tr><td>TEXT                  <td>"TEXT"
  <tr><td>NUMERIC               <td>"NUM"
  <tr><td>INTEGER               <td>"INT"
  <tr><td>REAL                  <td>"REAL"
  <tr><td>BLOB (a.k.a "NONE")   <td>"" (empty string)
</table></center>)^

<p>^(A table created using CREATE TABLE AS has no PRIMARY KEY and no
constraints of any kind. The default value of each column is NULL. The default
collation sequence for each column of the new table is BINARY.)^

<p>^Tables created using CREATE TABLE AS are initially populated with the

<p>There are many benefits to this approach, including improved
performance, reduced cost and complexity, and
improved reliability.  See technical notes
[file-format benefits|"aff_short.html"] and 
[application file-format|"appfileformat.html"] and
[faster than the filesystem|"fasterthanfs.html"] for more information.
This use case is closely related to the
<a href="#wireproto">data transfer format</a> and
<a href="#container">data container</a> use cases below.
</li>

<tcl>hd_fragment website {using SQLite for websites}</tcl>
<li><p><b>Websites</b></p>

<p>SQLite works great as the database engine for most low to
medium traffic websites (which is to say, most websites).
The amount of web traffic that SQLite can handle depends

SQL database engine in this scenario.
Database requests are serialized by the server, so concurrency is not
an issue.  Concurrency is also improved by "database sharding":
using separate database files for different subdomains.  For
example, the server might have a separate SQLite database for each
user, so that the server can handle hundreds or thousands of simultaneous
connections, but each SQLite database is only used by one connection.</p>
</li>

<tcl>hd_fragment wireproto {data transfer format}</tcl>
<li><p><b>Data transfer format</b><p>

<p>Because an SQLite database is a single compact file in a
[file format|well-defined cross-platform format], it is often used
as a container for transfering content from one system to another.
The sender gathers content into an SQLite database file, transfers
that one file to the receiver, then the receiver uses SQL to extract
the content as needed.

<p>An SQLite database facilitates data transfer between systems even
when the endpoints have different word sizes and/or byte orders.
The data can be a complex mix of large binary blobs, text, and small
numeric or boolean values.  The data format can be easily extended
by adding new tables and/or columns, without breaking legacy receivers.
The SQL query language means that receivers are not required to parse
the entire transfer all at once, but can instead query the
received content as needed.  The data format is "transparent" in the
sense that it is easily decoded for human viewing using 
a variety of universally available, open-source tools, from multiple
vendors.
</li>

<tcl>hd_fragment container {data container}</tcl>
<li><p><b>File archive and/or data container</b></p>

<p>
The [SQLite Archive] idea shows how
SQLite can be used as a substitute for ZIP archives or Tarballs.
An archive of files stored in SQLite is only very slightly larger, and
in some cases actually smaller, than the equivalent ZIP archive.

</li>

<li><p><b>High Concurrency</b></p>

<p>
SQLite supports an unlimited number of simultaneous readers, but it 
will only allow one writer at any instant in time.
For many situations, this is not a problem.  Writers queue up. Each application
does its database work quickly and moves on, and no lock lasts for more
than a few dozen milliseconds. But there are some applications that require
more concurrency, and those applications may need to seek a different
solution.
</p>
</li>

<title>Why SQLite Does Not Use Git</title>

<table_of_contents>

<h1>Introduction</h1>

<p>
SQLite does not use the
[https://git-scm.org|Git] version control system.
SQLite uses
[https://fossil-scm.org/|Fossil] instead.
Fossil and Git are both block-chain version-control systems.
They are both "distributed".  They both store content has a 
sequence of immutable check-ins identified by a cryptographic
hash.  Git is wildly popular, to the point that many younger
developers are familiar with nothing else.  And yet, the developers
of SQLite prefer Fossil.  This article tries to explain why.

<h1>A Few Reasons Why SQLite Does Not Use Git</h1>

<ol>
<li><p>
With Git, it is very difficult to find the successors (decendents) of
a check-in.

<p>
Git allows you to go backwards in time.  Given the latest
check-in on a branch, Git lets you see all the ancestors of that
check-in.  But Git makes it very difficult to move in the other
direction.  Given some historical check-in, it is quite challenging
in Git to find out what came next.  It can be done, but it is sufficiently
difficult and slow that nobody ever does it.  There is no button in
GitHub that shows the descendents of a check-in.

<p>
This is a deal-breaker, a show-stopper.  The principle maintainer
of SQLite cannot function effectively without being able to view
the successors of a check-in.  This one issue is sufficient reason
to not use Git, in the view of the designer of SQLite.

<li><p>
The mental model for Git is needlessly complex and consequently
distracts attention from software under development.  A user of Git
needs to keep all of the following in mind:
<ol type='a'>
<li> The working directory
<li> The "index" or staging area
<li> The local head
<li> The local copy of the remote head
<li> The actual remote head
</ol>
<p>
Git contains commands (or options on commands) for moving and
comparing content between all of these locations. 

<p>In contrast,
Fossil users only need to think about their working directory and
the check-in they are working on.  That is 60% less distraction.
Every developer has a finite number of brain-cycles.  Fossil
requires fewer brain-cycles to operate, thus freeing up 
intellectual resources to focus on the software under development.

<li><p>
Git does not track branch history.  This makes review of historical
branches tedious.

<p>
As an example, consider display of a single historical
branch of SQLite as rendered by GitHub and by Fossil:

<ul>
<li><b>GitHub:</b> [https://github.com/mackyle/sqlite/commits/prefer-coroutine-sort-subquery]
<li><b>Fossil:</b> [https://sqlite.org/src/timeline?r=prefer-coroutine-sort-subquery]
</ul>

<p>
The Fossil view clearly shows that the branch was eventually merged back into
trunk.  It shows where the branch started, and it shows two occasions where changes
on trunk were merged into the branch.  GitHub shows none of this.  In fact, the
GitHub display is mostly useless in trying to figure out what happened.

<li><p>
Git lacks native wiki and bug tracking.
If you want these essential features, you have to install additional software
such as GitLab, or else use a third-party service such as GitHub.  And even
then, the wiki and bug reports are centralized, not distributed.

<li><p>
Git requires more administrative support.
Setting up a website for a project to use Git requires a lot more software,
and a lot more work, than setting up a similar site with an integrated package
like Fossil.

<li><p>Nobody really understands Git.
<img src="https://www.fossil-scm.org/fossil/doc/trunk/www/xkcd-git.gif"><br>
(credit: [https://xkcd.com/1597/])
</ol>

<h2>See Also</h2>

<p>Other pages that talk about Fossil and Git include:
<ul>
<li><p>[https://fossil-scm.org/fossil/doc/trunk/www/fossil-v-git.wiki|Fossil vs. Git]
<li><p>[https://www.fossil-scm.org/fossil/doc/trunk/www/quotes.wiki|What others say about Fossil and Git]
</ul>

<h1>A Git-User's Guide To Accessing SQLite Source Code</h1>

<p>
If you are a devoted Git user and have no intention of changing,
you can still easily access SQLite.  This section gives some hints
on how to do so.

<h2>GitHub Mirrors</h2>

<p>
There is a mirror of the SQLite source tree on GitHub at
[https://github.com/mackyle/sqlite].  This mirror is maintained
by user "mackyle" who is unaffiliated with, and unknown to,
the official SQLite development team.  We do not know mackyle,
but we observe that he does a terrific job of keeping his mirror
current, and so if you want to access the SQLite source code on
GitHub, his mirror is the recommended source.

<h2>Web Access</h2>

<p>
The [https://sqlite.org/src/timeline|SQLite Fossil Repository] contains links
for downloading  a Tarball, ZIP Archive, or [SQLite Archive] for any
historical version of SQLite.  The URLs for these downloads are
simple and can be incorporated easily into automated tools.  The format is:

<center>
<tt>https://sqlite.org/src/tarball/</tt><i>VERSION</i><tt>/sqlite.tar.gz</tt>
</center>

<p>
Simply replace <i>VERSION</i> with some description of the version to be
downloaded.  The <i>VERSION</i> can be a prefix of the cryptographic hash
name of a specific check-in, or the name of a branch (in which case the
most recent version of the branch is fetched) or a tag for a specific
check-in like "version-3.23.0".

<h2>Fossil Access</h2>

<p>
Fossil is really easy to install and use.  Here are the steps for unix.
(Windows is similar.)

<ol>
<li>
Download the self-contained Fossil executable from
[https://fossil-scm.org/fossil/uv/download.html] and put the executable
somewhere on your $PATH.
<li><tt>mkdir ~/fossils</tt>
<li><tt>fossil clone https://fossil-scm.org/fossil ~/fossils/sqlite.fossil</tt>
<li><tt>mkdir ~/sqlite; cd ~/sqlite</tt>
<li><tt>fossil open ~/sqlite.fossil .</tt>
</ol>

<p>
Subsequently, to update your check-out, simply type:

<blockquote>
<tt>fossil up </tt><i>VERSION</i>
</blockquote>

<p>
Use "trunk" for version to get the latest trunk version of SQLite.
Or use a prefix of a cryptographic hash name, or the name of some branch
or tag.

<p>
Use the "<tt>fossil ui</tt>" command from within the ~/sqlite checkout to
bring up a local copy of the website.