Roll src/third_party/sqlite/src/ 5e8c30a1e..0324bd3ef (222 commits)

https://chromium.googlesource.com/chromium/deps/sqlite.git/+log/5e8c30a1e0e0..0324bd3ef1af

$ git log 5e8c30a1e..0324bd3ef --date=short --no-merges --format='%ad %ae %s'
2020-08-19 huangdarwin Amalgamations for release 3.33.0
2020-08-14 drh Version 3.33.0
2020-08-13 dan Fix "make test" handling of environment variable QUICKTEST_OMIT so that it can be used to exclude test files in other than the main test directory.
2020-08-12 drh Improvement on the previous fix.
2020-08-12 drh Fix an assertion() fault in ALTER TABLE found by OSSFuzz.  Test case in TH3.
2020-08-11 drh Fix harmless USAN warnings from gcc9.
2020-08-11 drh New test cases for the use of the ieee754 and decimal extensions in the CLI.
2020-08-11 dan Modify a test for corruption within the wal checkpoint code to account for the pending-byte page. And for the fact that test configurations might move the pending-byte page.
2020-08-11 drh Minor comment fixes.  No changes to code.
2020-08-11 drh Remove an unused #define from sqliteInt.h.
2020-08-10 drh Simplify #ifdefs associated with Parse.eParseMode.  Fix an #ifdef error associated with SQLITE_OMIT_AUTOVACUUM.
2020-08-10 dan Fix a problem causing test failures in corruptL.test for some permutations.
2020-08-10 dan Fix a problem building fts3 separately from the amalgamation.
2020-08-10 dan Fix a shell tool build error caused by some combinations of options.
2020-08-10 drh Fix harmless compiler warnings that surface in newer versions of GCC.
2020-08-10 dan Fix another test script problem in walvfs.test.
2020-08-10 dan Fix minor test script problems.
2020-08-09 drh Fix a harmless compiler warning.
2020-08-08 drh Fix the check-in at [41474548ef3f7454] so that it computes the pointer in time for error checking at the end of the routine in the case of a non-OOM error.
2020-08-08 dan Fix test script busy2.test so that it works with the inmemory-journal permutation.
2020-08-08 dan Changes to busy2.test, corruptL.test and fkey5.test so that new test cases pass with all test permutations.
2020-08-08 dan Fix a test script problem causing an error for SQLITE_ENABLE_OVERSIZE_CELL_CHECK builds in corruptL.test.
2020-08-08 drh Move a pointer computation until after OOM checks to avoid a nuisance USAN warning.
2020-08-08 dan Change the name of sqlite3SelectTrace to sqlite3_unsupported_selecttrace.
2020-08-08 drh Update requirement marks due to wording improvements in the documentation.
2020-08-08 drh Reorder declarations in the decimal extension for C89.
2020-08-07 drh Do the oversize-WAL corruption test before the size hint is issued.
2020-08-07 dan Fix a file-descriptor leak in test script corruptL.test.
2020-08-07 dan Return an SQLITE_CORRUPT error if the final expected size of the database when checkpointing is not reasonable - where reasonable is defined (basically) as the sum of the sizes of the database and wal files.
2020-08-07 drh Add the --checkpoint option to speedtest1.
2020-08-06 drh Fix the columnar output modes in the CLI so that they work with parameters. See [https://sqlite.org/forum/forumpost/17ba6aac24] for details of the problem fixed.
2020-08-04 mistachkin Fix compilation issues with MSVC.
2020-07-31 drh Back out a NEVER() that turns out to be reachable.
2020-07-31 drh Remove an ALWAYS() that turns out to be reachable.
2020-07-30 drh Test for schema corruption is reachable after all.
2020-07-30 drh Provide an alternative "guaranteed-safe" method for overwriting the WAL index on recovery, in case some platform is found for which memcpy() cannot do this safely.
2020-07-30 drh Fix compiler warnings in MSVC.
2020-07-30 drh Fix unreachable branches.
2020-07-29 drh Fix signed/unsigned compiler warnings.
2020-07-28 drh Earlier detection of out-of-range page numbers in the btree layer.
2020-07-28 drh Add an sqlite3FaultSim() to make an OOM case more accessible and remove the ALWAYS() on the conditional that is false when the OOM actually occurs.
2020-07-27 drh On recovery, always overwrite the old with the new, even if they are the same. Add ALWAYS() macros on branches currently thought to be unreachable, pending additional testing.
2020-07-27 dan Fix a couple of test scripts to match the new wal recovery behaviour on this branch.
2020-07-27 drh Improved error reporting if walLockExclusive() fails.
2020-07-25 dan Allow a wal mode recovery to proceed even if there are readers.
2020-07-24 drh Remove a surplus space from a comment
2020-07-24 drh Fix other potentiall pointer aliasing problems associated with subclassing of the sqlite3_file object for various VFS implementations.
2020-07-24 drh Fix pointer aliasing problem in the in-memory journal code. Ref: [https://sqlite.org/forum/forumpost/d44eb2fc44|forum post d44eb2fc44]
2020-07-23 drh Add the OMIT_ZLIB compile-time option to sessionfuzz.c.  (Originally checked into the wrong branch.)
2020-07-23 drh Fix a typo in an error message.
(...)
2020-06-10 dan Ensure that the "push-down" optimization does not push constraints down into compound queries if any of the component queries uses window functions.
2020-06-10 drh Disable AggInfo consistency checks when unwinding after an OOM.
2020-06-09 drh Mark an always-true conditional with ALWAYS().
2020-06-09 dan Ensure that aggregate functions that (a) are part of SELECT statements with no FROM clause and (b) have one or more scalar sub-selects as arguments are assigned to the correct aggregate context.
2020-06-09 dan Modify a test file to avoid causing Tcl to allocate too much memory.
2020-06-09 drh Give the expression pointer fields of AggInfo distinctive names in order to simplify tracking of all their uses.
2020-06-09 drh Improved tree-view debugging output for aggregate functions.
2020-06-08 dan Fix a case where a corrupted fts3 record could cause an assert() failure, or spurious SQLITE_NOMEM error in builds with assert() disabled.
2020-06-07 drh Fix minor OOM problems.
2020-06-07 drh AggInfo objects might be referenced even after the sqlite3Select() function that created them has exited.  So AggInfo cannot be a stack variable.  And it must not be freed until the Parse object is destroyed.
2020-06-07 drh Alternative fix to ticket [c8d3b9f0a750a529]:  Prior to deleting or modifying an Expr not that is referenced by an AggInfo, modify the AggInfo to get its own copy of the original Expr.
2020-06-05 drh In the debugging treeview output, change the name of "SELECT-expr" expression nodes to be "subquery-expr", so as to not confuse them with actual SELECT nodes.
2020-06-05 drh Always use ?...? to indicate optional arguments in the output of ".help" in the CLI.  Change ".mode column" so that it automatically activates ".headers on" if headers have not been previously turned on or off.
2020-06-04 drh Add support for "box" mode in the CLI:  Like "table" except that it uses unicode box-drawing characters instead of ascii-art.
2020-06-04 drh Improved display of ".mode table" output for empty result sets.
2020-06-04 dan Use __has_extension(c_atomic) instead of __has_feature(c_atomic) to detect support for atomic load and store operations with clang.
2020-06-04 dan Use AtomicStore() to set values in the wal-index hash table.
2020-06-04 drh Work around a bug in clang-11.0.0.
2020-06-03 drh Fix for ticket [810dc8038872e212].  Thank to user "Maxulite" for tracking down the problem!
2020-06-03 drh Simplification to the interrupt handling logic in sqlite3VdbeExec() saves a few bytes of code space.
2020-06-03 drh Improve the query planner so that it is better able to find full index scan plan when there is an INDEXED BY clause.
2020-05-30 drh Draw the dashes below the headers in "explain" mode in the CLI.
2020-05-30 drh Improved VDBE comments on the ANALYZE code generator.  This change also fixes a harmless use of an uninitialized integer variable as an input to the %d format on a VDBE comment.
2020-05-29 mistachkin Enhancements to the incremental build support for MSVC.
2020-05-29 drh Remove a stray "&" character in the CLI, detected by a clang warning.
2020-05-29 drh Add the "shelltest" target to the MSVC makefile as well.
2020-05-29 drh Fix the ".import" command of the CLI to clean up better after errors. Add the new "shelltest" makefile target on unix platforms.
2020-05-29 drh Improvements to help text for the CLI.
2020-05-29 drh Fix a memory leak in the CLI when an unknown or unrecognized argument is given to the ".dump" command.
2020-05-29 drh Improvements to columnar output in the CLI.  Columns automatically expand to contain the largest row.
2020-05-29 drh Space to hold the ".width" of columns in the CLI is now obtained from malloc() and hence is not limited in the number of columns supported.
2020-05-29 drh Incremental improvements to tabular output modes in the CLI.  The "markdown" and "table" modes no have headers turned on by default.
2020-05-29 dan Expand upon a comment in os_unix.c. No changes to code.
2020-05-29 drh In the json output mode of the CLI, do correct quoting of escape characters. Also, show BLOBs as JSON strings, possibly with embedded \u0000 bytes.
2020-05-28 drh Progress toward adding new output modes to the CLI:  json, table, and markdown.
2020-05-28 drh Enhance the ".quote" mode in the shell so that it honors .separator.
2020-05-28 drh When the sqlite_stat1 data is missing for some indexes of a table but is present for the table itself or for other indexes in the same table, then do not let the estimated number of rows in that table get too small, as doing so can deceive the query planner into ignoring a perfectly good index.
2020-05-27 drh Small performance improvement and size reduction in the expression code generator.
2020-05-27 drh Change a datatype from i16 to int to appease Converity and help eliminate a false-positive.
2020-05-26 drh Fix the cksumvfs extension so that it will not register itself more than once.
2020-05-26 drh Performance optimization in the transfer of error messages from statements to connections.
2020-05-26 drh Increase the version number to 3.33.0 to begin the next release cycle.
2020-05-26 drh Innocuous changes to help Coverity avoid false-positives.
2020-05-25 drh Attempt to work around a false-positive warning in the CGo compiler.
2020-05-01 dan Fix problems with UPDATE...FROM statements that modify rowid or primary-key values.
2020-04-30 dan Add OOM tests for the new code on this branch.
2020-04-30 dan Report an error if an UPDATE...FROM statement has an ORDER BY but no LIMIT clause. Add tests for multi-column primary keys.
2020-04-29 dan Fix problems with using LIMIT and FROM clauses as part of single UPDATE statement.
2020-04-29 dan Fix various bugs in new feature on this branch.
2020-04-27 dan Allow a FROM clause in UPDATE statements.

Created with:
  roll-dep src/third_party/sqlite/src

Bug: 1117367
Change-Id: Ie37ce39b80489b01ea8569eafea79e5bf8c54d5e
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/2364964
Commit-Queue: Darwin Huang <huangdarwin@chromium.org>
Reviewed-by: Chris Mumford <cmumford@google.com>
Cr-Commit-Position: refs/heads/master@{#800336}
GitOrigin-RevId: 9c2848796cc438e7d5a622f0f8ae2b4bf3bc3f50
2 files changed
tree: 624799161b76a2fb506a07a96de17f3f43194eb2
  1. dev/
  2. fuzz/
  3. scripts/
  4. .gitignore
  5. BUILD.gn
  6. DEPS
  7. LICENSE
  8. OWNERS
  9. PRESUBMIT.py
  10. README.chromium
  11. README.md
  12. sqlite3.h
  13. sqlite3_shim.c
  14. sqlite3_shim_fixups.h
  15. sqlite_chromium_configuration_flags.gni
  16. sqlite_common_configuration_flags.gni
  17. sqlite_dev_configuration_flags.gni
  18. sqlite_shell_icu_helper.cc
  19. sqlite_shell_icu_helper.h
  20. sqlite_shell_shim.c
README.md

This is the top folder for Chromium's SQLite. The actual SQLite source is not in this repository, but instead cloned into the src directory from https://chromium.googlesource.com/chromium/deps/sqlite.

The directory structure is as follows. Files common to all third_party projects (BUILD.GN, OWNERS, LICENSE) are omitted.

  • src/ The Chromium fork of SQLite (cloned via top level DEPS file).
  • scripts/ Scripts that generate the files in the amalgamations in src/.
  • sqlite.h The header used by the rest of Chromium to include SQLite. This forwards to src/amalgamation/sqlite3.h
  • fuzz/ Google OSS-Fuzz (ClusterFuzz) testing for Chromium's SQLite build This directory contains:

The SQLite amalgamation is committed to the SQLite Chromium repository (in src), but it is created by a script that lives in the Chromium repository. This is because the configuration variables for building and amalgamation generation are shared.

There are two amalgamations. The one in //third_party/sqlite/src/amalgamation is used by Chromium. A second, located at //third_party/sqlite/src/amalgamation_dev is used for some Chromium developer tools and is not distributed.

Upgrade to a new SQLite release.

Note SQLite tags all releases version-<release number>, e.g. version-3.33.0. The Chromium project prefixes all tags/branches with “chromium-”, e.g. chromium-version-3.33.0.

  1. Create new release branch

    Use the SQLite commit ID when creating a branch. For example “562fd18b9dc27216191c0a6477bba9b175f7f0d2” corresponds to the 3.33.0 release. The commit is used, instead of the tag name, because we do not mirror the SQLite tags along with the commits. The correct commit ID can be found at sqlite/releases.

    Create the branch at Gerrit/branches.

  2. Checkout the new Chromium release branch.

    cd //third_party/sqlite/src
    git fetch origin
    export VERSION=3.33.0
    git checkout -b chromium-version-$VERSION \
        --track origin/chromum-version-$VERSION
    
  3. Generate and commit the SQLite amalgamations.

    ../scripts/generate_amalgamation.py
    git add amalgamation
    git add amalgamation_dev
    git commit -m "Amalgamations for release $VERSION"
    
  4. Run local tests.

    Follow steps in Running Tests below to execute all verifications and tests.

  5. Upload the new release branch for review.

    git cl upload
    
  6. Roll the Chromium DEPS file.

    Once review above has merged:

    1. Roll the chromium/src/DEPS file to reference that new commit ID.
      roll-dep src/third_party/sqlite/src --roll-to <git hash of merged CL>
      
    2. Update the version in //third_party/sqlite/README.chromium. Append the commit created by roll-dep above.

Cherry-pick unreleased commit from SQLite.

Sometimes critical fixes land in SQLite's master, but are not yet in a release. If these need to be brought into the current release branch do the following:

  1. Checkout the current release branch.

    export VERSION=3.33.0
    cd //third_party/sqlite/src
    git checkout -b chromium-version-$VERSION \
      --track origin/chromium-version-$VERSION
    
  2. Cherry-pick the change

    Git can be used to cherry pick upstream changes into a release branch but the sqlite_cherry_picker.py script is preferred. This script automates a few tasks such as:

    • Identifying the correct Git commit ID to use if given the Fossil commit ID.
    • Automatically calculating Fossil manifest hashes.
    • Skipping conflicted binary files.
    • Generating the amalgamations.

    Cherry-pick the commit:

    ../scripts/sqlite_cherry_picker.py <full git or fossil commit id>
    

    If there is a conflict that the script cannot resolve then, like git-cherry-pick, the script will exit and leave you to resolve the conflicts. Once resolved run the script a second time:

    ../scripts/sqlite_cherry_picker.py --continue
    
  3. Run local tests.

    Follow steps in Running Tests below to execute all verifications and tests.

  4. Upload cherry-picked change (with amalgamations) for review.

    git cl upload
    
  5. Update the Chromium DEPS file.

    Once review above has merged, roll the chromium/src/DEPS file to reference that new commit ID.

    roll-dep src/third_party/sqlite/src --roll-to <git hash of merged CL>
    

Running Tests

Build all desktop targets:

Check that extract_sqlite_api.py added “chrome_” to all exported symbols. Only “_fini” and “_init” should be unprefixed, but are conditionally exported by the linker and may be omitted.

autoninja -C out/Default
nm -B out/Default/libchromium_sqlite3.so | cut -c 18- | sort | grep '^T'

Running unit tests

out/Default/sql_unittests

Running web tests

third_party/blink/tools/run_web_tests.py -t Default storage/websql/

Running SQLite's TCL test suite within the Chromium checkout.

This is one of the SQLite test suites. They take approximately 3 minutes to build and run on a fast workstation.

cd //third_party/sqlite
./scripts/generate_amalgamation.py --testing
make --directory=src test | tee /tmp/test.log

Note: Tests may fail on Chromium release branches. This is because some tests rely on SQLite databases (binary files) which are committed to the source and are likely not merged down when cherry picked. It is safe to ignore these errors which should be reasonably easy to identify based on the cherry picked upstream changes.

Show error'ed tests:

egrep 'errors out of' /tmp/test.log

Show broken tests:

egrep 'Failures on these tests:' /tmp/test.log

Broken tests will also show lines ending in “...” instead of “... Ok”.

When done cleanup the SQLite repository:

cd src
make clean
git clean -i  # and delete everything
rm -rf testdir
git checkout amalgamation/sqlite3.c
git checkout amalgamation_dev/sqlite3.c