Documentation Source Text

Check-in [c0a1fede37]
Login

Many hyperlinks are disabled.
Use anonymous login to enable hyperlinks.

Overview
Comment:Changes to walformat.html document.
Downloads: Tarball | ZIP archive | SQL archive
Timelines: family | ancestors | descendants | both | trunk
Files: files | file ages | folders
SHA3-256:c0a1fede37dad73254e5534712a4df5ba84b72ff899bcad9da667909d82dca08
User & Date: drh 2017-12-12 01:16:57
Context
2017-12-16
17:20
Add documentation for the "swarmvtab" extension. check-in: 9cc758b326 user: dan tags: trunk
2017-12-12
01:16
Changes to walformat.html document. check-in: c0a1fede37 user: drh tags: trunk
2017-12-09
15:44
Add documentation for shell tool SQLAR archive support. check-in: 809f9fa181 user: dan tags: trunk
Changes
Hide Diffs Unified Diffs Ignore Whitespace Patch

Changes to pages/walformat.in.

170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
...
220
221
222
223
224
225
226
227
228




229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
...
634
635
636
637
638
639
640



641
642
643



644
645
646
647
648
649
650






651
652
653
654
655
656
657
658
659
660
661


662
663
































</tr>
<tr>
<td>14..15</td><td>szPage</td>
<td>The database page size in bytes, or 1 if the page size is 65536.
</tr>
<tr>
<td>16..19</td><td>mxFrame</td>
<td>Number of valid frames in the WAL file.
</tr>
<tr>
<td>20..23</td><td>nPage</td>
<td>Size of the database file in pages.
</tr>
<tr>
<td>24..31</td><td>aFrameCksum</td>
................................................................................
</tr>
</table>
</center>

<tcl>hd_fragment mxframe mxFrame</tcl>
<h3>The mxFrame field</h3>

<p>The 32-bit unisgned integer at offset 16 (and repeated at offset 64)
is the number of valid frames in the WAL.




<p>
When this field is zero, it indicates that the WAL is empty and that
all content should be obtained directly from the database file.
<p>
When mxFrame is equal to [nBackfill], that indicates that all content
in the WAL has been written back into the database.
In that case, all content can be read directly from the database.
Furthermore, the next writer is free to [reset the WAL] if no other
connections are holding locks on WAL_READ_LOCK(N) for N>0.
<p>
The mxFrame value is always greater than or equal to both
[nBackfill] and nBackfillAttempted.


<tcl>hd_fragment nbackfill nBackfill</tcl>
<h3>The nBackfill field</h3>
................................................................................
If the WAL and WAL-index files are missing, then a process that
lacks permission to create and initialize those files will not be
able to open the database, unless the database is designated
as immutable using the [immutable query parameter].

<li><p><b>Reconstruct the global shared WAL-index during [recovery]</b>






<li><p><b>Append a new transaction to the end of the WAL</b>




<li><p><b>Read content from the database and WAL as part of a transaction</b>

<li><p><b>Run a checkpoint</b>

<li><p><b>Reset the WAL file</b>







</ul>

<tcl>hd_fragment recovery recovery {WAL-mode crash recovery}</tcl>
<h1>Crash Recovery</h1>

<p>
After an application crash, or a system crash, or a power failure, the
database might be left in an inconsistent state.  When that happens,
the first connection to the database will need to run a recovery
procedure to restore the database to a consistent state before it begins
using the database.  This section describes that procedure.



<p><i>TBD...</i></p>







































|







 







|
|
>
>
>
>

|






|







 







>
>
>

<

>
>
>







>
>
>
>
>
>



|


<
<
<
<
<
>
>

<
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
...
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
...
638
639
640
641
642
643
644
645
646
647
648

649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671





672
673
674

675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
</tr>
<tr>
<td>14..15</td><td>szPage</td>
<td>The database page size in bytes, or 1 if the page size is 65536.
</tr>
<tr>
<td>16..19</td><td>mxFrame</td>
<td>Number of valid and committed frames in the WAL file.
</tr>
<tr>
<td>20..23</td><td>nPage</td>
<td>Size of the database file in pages.
</tr>
<tr>
<td>24..31</td><td>aFrameCksum</td>
................................................................................
</tr>
</table>
</center>

<tcl>hd_fragment mxframe mxFrame</tcl>
<h3>The mxFrame field</h3>

<p>The 32-bit unsigned integer at offset 16 (and repeated at offset 64)
is the number of valid frames in the WAL.  Because WAL frame are numbered
starting with 1, mxFrame is also the index of the last valid commit frame
in the WAL. A commit frame is a frame that has a non-zero "size of database"
value in bytes 4 through 7 of the frame header, and that indicates the end
of a transaction.
<p>
When mxFrame field is zero, it indicates that the WAL is empty and that
all content should be obtained directly from the database file.
<p>
When mxFrame is equal to [nBackfill], that indicates that all content
in the WAL has been written back into the database.
In that case, all content can be read directly from the database.
Furthermore, the next writer is free to [reset the WAL] if no other
connections hold locks on WAL_READ_LOCK(N) for N>0.
<p>
The mxFrame value is always greater than or equal to both
[nBackfill] and nBackfillAttempted.


<tcl>hd_fragment nbackfill nBackfill</tcl>
<h3>The nBackfill field</h3>
................................................................................
If the WAL and WAL-index files are missing, then a process that
lacks permission to create and initialize those files will not be
able to open the database, unless the database is designated
as immutable using the [immutable query parameter].

<li><p><b>Reconstruct the global shared WAL-index during [recovery]</b>

<p>All of the WAL-index locks, except for WAL_READ_LOCK(0),
are held exclusively while reconstructing the global shared WAL-index
during [recovery].  


<li><p><b>Append a new transaction to the end of the WAL</b>

<p>An exclusive lock is held on WAL_WRITE_LOCK while adding new
frame onto the end of a WAL file.  

<li><p><b>Read content from the database and WAL as part of a transaction</b>

<li><p><b>Run a checkpoint</b>

<li><p><b>Reset the WAL file</b>

<p>A [WAL reset] means to rewind the WAL and start adding new frames
at the beginning.  This occurs while appending new frames to a WAL
that has [mxFrame] equal to [nBackfill] and which has no locks on
WAL_READ_LOCK(1) through WAL_READ_LOCK(4).  The WAL_WRITE_LOCK is
held.

</ul>

<tcl>hd_fragment recovery recovery {WAL-mode crash recovery}</tcl>
<h1>Recovery</h1>

<p>





Recovery is the process of rebuilding the WAL-index so that it is
synchronized with the WAL.


<p>
Recovery is run by the first thread to connect to a WAL-mode database.
Recovery restores the WAL-index so that it accurately describes the
WAL file.  If there is no WAL file present when the first thread connects
to the database, there is nothing to recover, but the recovery process
still runs to initialize the WAL-index.

<p>
If the WAL-index is implemented as a memory-mapped file and that file is
read-only to the first thread to connect, then that thread creates an
private heap-memory ersazt WAL-index and runs the recovery routine to
populate that private WAL-index.  The same data results, but it is held
privately rather that being written into the public shared memory area.

<p>
Recovery works by doing a single pass over the WAL, from beginning to end.
The checksums are verified on each frame of the WAL as it is read.  The
scan stops at the end of the file or at the first invalid checksum.
The [mxFrame] field is set to the index of the last valid commit frame
in WAL.  Since WAL frame numbers are indexed starting with 1, mxFrame is
also the number of valid frames in the WAL.  A "commit frame" is a frame
that has a non-zero value in bytes 4 through 7 of the frame header.
Since the recovery procedure has no way of knowing how many frames of the
WAL might have previously been copied back into the database, it initializes
the [nBackfill] value to zero.

<p>
During recovery of the global shared-memory WAL-index, exclusive locks are
held on WAL_WRITE_LOCK, WAL_CKPT_LOCK, WAL_RECOVER_LOCK, and WAL_READ_LOCK(1) through WAL_READ_LOCK(4).  In other words, all locks associated with the
WAL-index except for WAL_READ_LOCK(0) are held exclusively.  This prevents
any other thread from writing the database and from reading any transactions
that are held in the WAL, until the recovery is complete.