The format of a cell depends on which kind of b-tree page the cell appears on. The following table shows the elements of a cell, in order of appearance, for the various b-tree page types.
-
- +
Table B-Tree Leaf Cell:
Table B-Tree Leaf Cell (header 0x0d):
- -
- A varint which is the total number of bytes of payload, including any overflow
- A varint which is the integer key, a.k.a. "[rowid]"
- The initial portion of the payload that does not spill to overflow @@ -667,27 +667,27 @@ pages.
- A 4-byte big-endian integer page number for the first page of the overflow page list - omitted if all payload fits on the b-tree page.
- +
Table B-Tree Interior Cell:
Table B-Tree Interior Cell (header 0x05):
- -
- A 4-byte big-endian page number which is the left child pointer.
- A varint which is the integer key
- +
Index B-Tree Leaf Cell:
Index B-Tree Leaf Cell (header 0x0a):
- -
- A varint which is the total number of bytes of key payload, including any overflow
- The initial portion of the payload that does not spill to overflow pages.
- A 4-byte big-endian integer page number for the first page of the overflow page list - omitted if all payload fits on the b-tree page.
- +
Index B-Tree Interior Cell:
Index B-Tree Interior Cell (header 0x02):
- A 4-byte big-endianpage number which is the left child pointer.
- A varint which is the total number of bytes of key payload, including any overflow
- The initial portion of the payload that does not spill to overflow @@ -704,14 +704,14 @@ B-tree Cell Format
Datatype Appears in... Description - Table Leaf - Table Interior - Index Leaf - Index Interior + Table Leaf (0x0d) + Table Interior (0x05) + Index Leaf (0x0a) + Index Interior (0x02) 4-byte integer ✔ ✔ Index: pages/fts5.in ================================================================== --- pages/fts5.in +++ pages/fts5.in @@ -161,19 +161,23 @@
By enclosing it in double quotes ("). Within a string, any embedded double quote characters may be escaped SQL-style - by adding a second double-quote character. -
As a bareword that includes no whitespace or reserved characters, - and is not "AND", "OR" or "NOT" (case sensitive). Reserved characters - are: -
- : ~ ! @ # $ % ^ & * ( ) + , = -- In other words, the top row of a regular US keyboard, the plus sign, - comma and colon characters. Strings that include any of these - characters must be quoted. +As an FTS5 bareword that is not "AND", "OR" or "NOT" (case sensitive). + An FTS5 bareword is a string of one or more consecutive characters that + are all either: + +
+
+ + Strings that include any other characters must be quoted.- Non-ASCII range characters (i.e. unicode codepoints greater + than 127), or +
- One of the 52 upper and lower case ASCII characters, or +
- One of the 10 decimal digit ASCII characters, or +
- The underscore character (unicode codepoint 96). +
FTS queries are made up of phrases. A phrase is an ordered list of one or more tokens. A string is transformed into a phrase by passing it to Index: pages/index.in ================================================================== --- pages/index.in +++ pages/index.in @@ -70,12 +70,13 @@
Current Status
-
- Version 3.8.10.1 -of SQLite is recommended for all new development. +
- Version 3.8.10.2 +of SQLite is recommended for all new development. Upgrading +from all prior releases is recommended.
Common Links
Index: pages/lang.in ================================================================== --- pages/lang.in +++ pages/lang.in @@ -402,11 +402,14 @@ RecursiveBubbleDiagram attach-stmt^The ATTACH DATABASE statement adds another database file to the current [database connection]. -^The filename for the database to be attached is the value of +Database files that were previously attached can be removed using +the [DETACH DATABASE] command. + +
^The filename for the database to be attached is the value of the expression that occurs before the AS keyword. ^The filename of the database follows the same semantics as the filename argument to [sqlite3_open()] and [sqlite3_open_v2()]; the special name "[:memory:]" results in an [in-memory database] and an empty string results in a new temporary database. Index: pages/malloc.in ================================================================== --- pages/malloc.in +++ pages/malloc.in @@ -360,11 +360,11 @@
The name "memsys5" used for the zero-malloc memory allocator implies that there are several additional memory allocators available, and indeed there are. The default memory allocator is "memsys1". The debugging memory allocator is "memsys2". Those have already been covered.
-If SQLite is compiled with [SQLITE_ENABLE_MEMSYS3] than another +
If SQLite is compiled with [SQLITE_ENABLE_MEMSYS3] then another zero-malloc memory allocator, similar to memsys5, is included in the source tree. The memsys3 allocator, like memsys5, must be activated by a call to [sqlite3_config]([SQLITE_CONFIG_HEAP],...). Memsys3 uses the memory buffer supplied as its source for all memory allocations. The difference between memsys3 and memsys5 is that memsys3 uses a Index: pages/news.in ================================================================== --- pages/news.in +++ pages/news.in @@ -16,10 +16,26 @@ regsub -all {[Tt]icket #(\d+)} $txt \ {\0} txt hd_resolve "
$txt" hd_puts "
" } + +newsitem {2015-05-20} {Release 3.8.10.2} { +Yikes! Index corruption after a sequence of valid SQL statements! +
It has been many years since anything like + [https://www.sqlite.org/src/info/34cd55d6|this bug] has snuck into + an official SQLite release. But for the pasts seven months + ([version 3.8.7] through [version 3.8.10.1]) + if you do an INSERT into a carefully + crafted schema in which there are two nested triggers that convert + an index key value from TEXT to INTEGER and then back + to TEXT again, the INTEGER value might get inserted as the index + key instead of the correct TEXT, resulting in index corruption. + This patch release adds a single line of code to fix the problem. +
If you do actually encounter this problem, running [REINDEX] on the + damaged indexes will clear it. +} newsitem {2015-05-09} {Release 3.8.10.1} {
The 3.8.10 release did not add the new [SQLITE_ENABLE_DBSTAT_VTAB] compile-time option to the [sqlite3_compileoption_used()] interface. This patch release fixes that omission. And while we are at it, Index: pages/pragma.in ================================================================== --- pages/pragma.in +++ pages/pragma.in @@ -1538,11 +1538,12 @@
Notes:
- Pragmas whose names are marked through in the list above -are deprecated that are maintained for historical compatibility only. +are deprecated. They are not maintained. They continue to exist +for historical compatibility only. Do not use the deprecated pragmas in new applications. Remove deprecated pragmas from existing applications at your earliest opportunity.
Note that running SQLite using gcov is not a test of SQLite — +
Note that running SQLite with gcov is not a test of SQLite — it is a test of the test suite. The gcov run does not test SQLite because the -fprofile-args and -ftest-coverage options cause the compiler to generate different code. The gcov run merely verifies that the test suite provides 100% branch test coverage. The gcov run is a test of the test - a meta-test. @@ -683,11 +683,11 @@ then the test program is recompiled using delivery compiler options (without the special -fprofile-arcs and -ftest-coverage options) and the test program is rerun. This second run is the actual test of SQLite. -
It is important to verify that both the gcov test run +
It is important to verify that the gcov test run
and the second real test run both give the same output. Any
differences in output indicate either the use of undefined or
indeterminate behavior in the SQLite code (and hence a bug),
or a bug in the compiler.
Note that SQLite has, over the previous decade, encountered bugs
@@ -697,17 +697,27 @@
The developers of SQLite have found that full coverage testing is an -extremely productive method for preventing the introduction of new bugs -as the system evolves. Because every single branch +extremely effective method for locating and preventing bugs. +Because every single branch instruction in SQLite core code is covered by test cases, the developers -can be confident that changes they make in one part of the code +can be confident that changes made in one part of the code do not have unintended consequences in other parts of the code. -It would be extremely difficult maintain the quality of SQLite without such -assurances.
+The many new features and performance improvements that have been +added to SQLite in recent years would not have been possible without +the availability full-coverage testing. + +Maintaining 100% MC/DC is laborious and time-consuming.
+The level of effort needed to maintain full-coverage testing
+is probably not cost effective for a typical application.
+However, we think that full-coverage testing is justified for a
+[most widely deployed|very widely deployed] infrastructure library
+like SQLite, and especially for a database library which by its very
+nature "remembers" past mistakes.
+
Dynamic analysis refers to internal and external checks on the Index: pages/th3.in ================================================================== --- pages/th3.in +++ pages/th3.in @@ -33,40 +33,40 @@
TH3 was originally written for validation testing only, but has subsequently been used for development testing and debugging as well, and has proven very helpful in those roles. A full-coverage -test takes less than three minutes on a workstation and hence +test takes less than five minutes on a workstation and hence serves as a fast regression test during day-to-day maintenance of the SQLite code base.
TH3 is a test program generator. The output of TH3 is a program -written in ANSI-C and intended to be +implemented in C-code and intended to be linked against the SQLite library under test. The generated test program is compiled and run on the target platform in order to verify correct operation of SQLite on that platform.
The inputs to TH3 are test modules written in C or SQL and small configuration files that determine how to initialize SQLite. The -TH3 package includes over one thousand test -modules and several dozen configuration files. +TH3 package includes over eleven hundred test +modules and more than three dozen configuration files. New modules and configurations can be added to customize TH3 for specialized applications. Each time TH3 is run, it reads a subset of the available test modules and configuration files to generate a custom C program that performs all of the specified tests under all -configurations. A complete test of SQLite normally involves running +specified configurations. A complete test of SQLite normally involves running TH3 multiple times to generate multiple test programs covering different aspects of SQLite's operation, then linking all test programs against a common SQLite library and running them separately on the target platform. SQLite will be found to work if all test programs pass.
There are no arbitrary limits in TH3. One could generate a -single test program that contained all test modules and configuration files. +single test program that contained all test modules and all configuration files. However, such a test program might be too large to deploy on embedded platforms. Hence, TH3 provides the ability to break the library of test modules up into smaller, more easily digested pieces.
Each individual test module might contain dozens, hundreds, or thousands @@ -119,16 +119,176 @@ appear on standard output. (Alternative output arrangements can be made using a compile-time option for embedded devices that lack a standard output channel.) The program returns zero if there are no errors and non-zero if any problems were detected.
-TH3 comes with additional TCL scripts (for example: -"th3make" and "fulltest.tcl") -which help to automate the process of running mkth3.tcl, compiling th3.c and -sqlite3.c, running the resulting test programs, and analyzing the results. -These other scripts are for convenience only and are not required in order -to make use of TH3. +
Typical output from a single TH3 test program run looks like this: + +
+ ++With SQLite 3.8.11 2015-05-15 04:13:15 56ef98a04765c34c1c2f3ed7a6f03a732f3b886e +-DSQLITE_COVERAGE_TEST +-DSQLITE_NO_SYNC +-DSQLITE_SYSTEM_MALLOC +-DSQLITE_THREADSAFE=1 +Config-begin c1. +Begin c1.pager08 +End c1.pager08 +Begin c1.build33 +End c1.build33 +Begin c1.orderby01 +End c1.orderby01 +... 15014 lines of output omitted .... +Begin 64k.syscall01 +End 64k.syscall01 +Begin 64k.build01 +End 64k.build01 +Begin 64k.auth01 +End 64k.auth01 +Config-end 64k. TH3 memory used: 6373738 +Config-begin wal1. +Begin wal1.wal37 +End wal1.wal37 +Config-end wal1. TH3 memory used: 100961 +All 226 VDBE coverage points reached +th3: 0 errors out of 1442264 tests in 213.741 seconds. 64-bit little-endian +th3: SQLite 3.8.11 2015-05-15 04:13:15 56ef98a04765c34c1c2f3ed7a6f03a732f3b886e +
The output begins with a report of the [SQLITE_SOURCE_ID] +(cross-checked again [sqlite3_sourceid()]) for the +SQLite under test and the compile-time options used as reported +by [sqlite3_compileoption_get()]. The output concludes with a summary +of the test results and a repeat of the [SQLITE_SOURCE_ID]. If any +errors are detected, additional lines detail the problem. The error +reporting lines always begin with a single space character so that they +can be quickly extracted from large output files using: + +
+ ++grep "^ " +
The default output shows the beginning and end of each configuration +and test module combination. In the example above "c1" and "64k" are +configurations and "pager08", "build33", "orderby01", etc. are test modules. +Compile-time and run-time options are available to increase or decrease +the amount of output. +The output can be increased by showing each test case within each +test module. The output can be decreased +by degrees: omitting test modules starts and stops, +omitting configuration starts and stops, and finally by omitting all output. +
TH3 comes with additional TCL scripts that help automate the testing +process on workstations. The "th3make" script automatically runs "mkth3.tcl" +and "gcc" and then runs the resulting test program and checks the results. +Arguments to th3make include all of the "*.test" test modules and +"*.cfg" configurations that are to be included in the test. Additional +options to th3make can cause the test program to be compiled using different +compilers (GCC, Clang, MSVC), to use different output verbosity levels, to +run the test program under valgrind, to check the output for coverage using +gcov, and so forth. The th3make script also accepts "*.rc" filenames as +arguments. These *.rc files are just collections of other arguments that +are commonly used together for a single purpose. For example, the "quick.rc" +file contains a set of eight arguments to th3make that run a fast (3-minute) +full-coverage test. This allows the operator to type "./th3make quick.rc" as +a short-cut to typing out all of the required command-line options. The +following are a few of the more than 40 available *.rc files: + +
The TH3 repository also includes the "multitest.tcl" script, another +TCL script used to automate TH3 testing on workstations. Multitest.tcl +automatically compiles SQLite, then +runs ./th3make repeatedly with a variety of alignments, and captures +the output in a succinct summary screen. A typical multitest.tcl run +generates output that looks like this: + +
+ ++file mkdir sqlite3bld +cd sqlite3bld +exec sh /home/drh/sqlite/sqlite/configure +file copy -force config.h ../config.h +exec make clean sqlite3.c +file rename sqlite3.c ../sqlite3.c +aa4f0f90c9c77424943e026a2ecee4a6c7f9e0d3 ../sqlite3.c +file rename sqlite3.h ../sqlite3.h +exec make clean sqlite3.c OPTS=-DSQLITE_ENABLE_UPDATE_DELETE_LIMIT=1 +file rename sqlite3.c ../sqlite3udl.c +0d3bbc92c433f940253bb2c7c19de7783133929d ../sqlite3udl.c +exec make clean sqlite3.c OPTS=-DSQLITE_SMALL_STACK=1 +file rename sqlite3.c ../sqlite3ss.c +fcf6963e94096324461076d3b9e9dc1888e066e1 ../sqlite3ss.c +cd .. +******************************************************************************* +t01: quick.rc.................................................. Ok (00:04:00) +t02: cov.rc.................................................... Ok (00:04:40) +t03: quick.rc extensions.rc -D_HAVE_SQLITE_CONFIG_H............ Ok (00:05:22) +t04: cov.rc -DSQLITE_ENABLE_STAT4 -D_HAVE_SQLITE_CONFIG_H...... Ok (00:05:20) +t05: test.rc ../th3private/*.test.............................. Ok (00:00:17) +t06: test.rc ../th3private/*.test -DSQLITE_ENABLE_STAT4........ Ok (00:00:43) +t07: quick.rc -DSQLITE_TEST_REALLOC_STRESS -funsigned-char..... Ok (00:04:56) +t08: quick.rc -DSQLITE_THREADSAFE=0 -fsigned-char.............. Ok (00:03:12) +t09: quick.rc sqlite3ss.c -DSQLITE_MAX_ATTACHED=125............ Ok (00:04:04) +t10: quick.rc -DSQLITE_RUNTIME_BYTEORDER....................... Ok (00:03:58) +t11: quick.rc -DSQLITE_DIRECT_OVERFLOW_READ.................... Ok (00:04:01) +t12: fast.rc................................................... Ok (00:14:19) +t13: fast.rc alignment1.rc -m32................................ Ok (00:20:51) +t14: fast.rc alignment2.rc sqlite3udl.c........................ Ok (00:16:06) +t15: fast.rc alignment4.rc..................................... Ok (00:12:55) +t16: fast.rc alignment5.rc..................................... Ok (00:14:58) +t17: fast.rc alignment6.rc..................................... Ok (00:14:31) +t18: fast.rc alignment7.rc..................................... Ok (00:16:06) +t19: fast.rc alignment8.rc sqlite3udl.c........................ Ok (00:24:09) +t20: test.rc alignment1.rc..................................... Ok (00:49:27) +t21: test.rc alignment2.rc sqlite3udl.c........................ Ok (00:38:43) +t22: test.rc alignment4.rc -m32 CC=clang....................... Ok (00:39:49) +t23: test.rc alignment5.rc..................................... Ok (00:36:33) +t24: test.rc alignment6.rc..................................... Ok (00:33:53) +t25: test.rc alignment7.rc..................................... Ok (00:42:16) +t26: test.rc alignment8.rc sqlite3udl.c........................ Ok (01:05:22) +t27: memdebug.rc extensions.rc................................. Ok (01:35:56) +t28: fast.rc -fsanitize=undefined.............................. Ok (00:15:09) +t29: min.rc -O3 -valgrind...................................... Ok (01:26:10) +t30: min.rc -O3 -valgrind extensions.rc........................ Ok (01:47:12) +t31: test-ex.rc................................................ Ok (03:20:18) +******************************************************************************* +0 failures on 31 th3make runs and 166721387 tests in (16:25:29) +SQLite 3.8.10 2015-05-05 18:52:54 04afa3febee32854fbb09ef8d4ffffd432119716 +
As can be seen above, a single run +of multitest.tcl invokes th3make dozens times and takes between 12 and 24 +hours. The middle section of the output shows the arguments to each individual +th3make run and the result and elapse time for that th3make. +All build products and output for the separate th3make runs are +captures in subdirectories for post-test analysis. +The two-line summary at the bottom shows the total number of errors and tests +over all th3make runs and the total elapse time, together with the +[SQLITE_SOURCE_ID] information for the version of SQLite that was +tested. This summary information is recorded in the +release +checklist during testing.
Using one particular subset of the available TH3 test modules (the "cov1" tests) SQLite obtained Index: search/search.tcl ================================================================== --- search/search.tcl +++ search/search.tcl @@ -194,10 +194,20 @@ # occurences, add a " to the end of the query so that fts3 can parse # it without error. if {[regexp -all \x22 $::A(q)] % 2} { append ::A(q) \x22 } set ::TITLE "Results for: \"[htmlize $::A(q)]\"" + + # Set nRes to the total number of documents that the user's query matches. + # + set rc [catch { + set nRes [db one { SELECT count(*) FROM page WHERE page MATCH $::A(q) }] + }] + if {$rc} { + set ::A(q) "\"$::A(q)\"" + set nRes [db one { SELECT count(*) FROM page WHERE page MATCH $::A(q) }] + } #db func rank rank #db func erank erank # If the user has clicked the "Lucky" button and the query returns one or @@ -212,21 +222,18 @@ ORDER BY rank(matchinfo(page), nk, nt, nc) DESC }] if {$url != ""} { cgi_redirect $url } } - set score 0 - catch {set score $::A(score)} - - # Set nRes to the total number of documents that the users query matches. - # If nRes is 0, then the users query returned zero results. Return a short + # If nRes is 0, then the user's query returned zero results. Return a short # message to that effect. # - set nRes [db one { SELECT count(*) FROM page WHERE page MATCH $::A(q) }] if {$nRes == 0} { return [subst { No results for: [htmlize $::A(q)] }] } + set score 0 + catch {set score $::A(score)} # Set iStart to the index of the first result to display. Results are # indexed starting at zero from most to least relevant. # set iStart [expr {([info exists ::A(i)] ? $::A(i) : 0)*10}]