Roll src/third_party/sqlite/src/ 0324bd3ef..d9581878f (171 commits)

https://chromium.googlesource.com/chromium/deps/sqlite.git/+log/0324bd3ef1af..d9581878fcf8

$ git log 0324bd3ef..d9581878f --date=short --no-merges --format='%ad %ae %s'
2020-12-22 huangdarwin Fix SQLITE_OMIT_WINDOWFUNC builds by moving declaration of sqlite3ExpandSubquery out of "ifndef SQLITE_OMIT_WINDOWFUNC" block.
2020-12-22 huangdarwin Amalgamations for release 3.34.0
2020-12-01 drh Version 3.34.0
2020-11-30 Dan Kennedy Update the tea/win/makefile.vc file that ships as part of the autoconf package to use "sqlite3" instead of "tclsqlite3" for the installed dll filename.
2020-11-30 drh Do not reuse the obsolete SQLITE_TESTCTRL_PRNG_RESET value for SQLITE_TESTCTRL_SEEK_COUNT.  Give SEEK_COUNT its own unique value. This avoids incompatibility with legacy test code.
2020-11-27 Dan Kennedy Fix a compiler warning in fts5_aux.c.
2020-11-27 Dan Kennedy Add missing cast to fts5 bm25() code.
2020-11-27 Dan Kennedy Remove a redundant "if( rc==SQLITE_OK )" from the fts5 bm25() code.
2020-11-27 Dan Kennedy Move an "if( rc==SQLITE_OK )" to outside a loop body in the fts5 bm25() code.
2020-11-27 Dan Kennedy Fix a case in the FTS5 integrity check where a corrupt database could cause a buffer overread.
2020-11-26 Dan Kennedy Update mkunicode.tcl to match the change erroneously made to machine generated file fts5_unicode2.c in [b7b7bde9].
2020-11-25 drh Update a requirement mark in the e_expr.test script.
2020-11-25 drh Fix harmless compiler warnings about unused function parameters.
2020-11-25 drh Fix a potential NULL pointer dereference in the geopoly_overlap() routine of the GeoPoly extension.
2020-11-25 drh Fix the shell1.test test script so that it works on windows.
2020-11-25 drh Change the -init option to the command-line shell to honor the -bail option. Also change it to report an error if the filename mentioned cannot be opened.
2020-11-24 drh Fix a typo in the sqlite3_txn_state() documentation.
2020-11-24 drh Add the new sqlite3_txn_state() interface to the set of routines accessible by run-time loadables extensions.
2020-11-24 drh Fix a harmless typo in a comment.
2020-11-24 dan Fix a bug in OOM handling code introduced by [6a28713d].
2020-11-24 dan Ensure that super-journal and other journal filenames passed by SQLite to an sqlite3_vfs.xOpen() implementation may be safely passed to sqlite3_uri_parameter() and similar functions.
2020-11-24 drh Remove the VERSION_NUMBER macro from configure.ac as it has been unused since 2009 check-in [7f4810747b086498].  See also [forum:/forumpost/bb2c634fcd|forum post bb2c634fcd].
2020-11-23 drh Syntactic changes to work around a bug in gcov 9.3.0.  No changes to the generated code for release builds.
2020-11-23 drh In os_unix.c, put ALWAYS() on unreachable branches associated with pathname normalization.
2020-11-23 dan Fix a test script problem causing crash8.test to fail when run with leak-sanitizer.
2020-11-23 drh Allow "substring()" as an alias for "substr()" for compatibility with other SQL engines.
2020-11-20 dan Add file doc/vdbesort-memory.md, containing notes on the way vdbesort.c uses heap memory.
2020-11-19 drh More test case changes so that shared-cache mode tests all work with normalized database names.
2020-11-19 drh Fix the tkt3793 test case so that work even when filenames are normalized.
2020-11-18 drh Improvements to the mechanism that attempts to report SQLITE_CORRUPT if pread() says that the underlying filesystem is corrupt.
2020-11-18 dan Update cksumvfs.c so that if SQLITE_AMALGAMATION is defined, SQLITE_CKSUMVFS_STATIC is implied.
2020-11-18 dan Add test infrastructure for cksumvfs. And update cksumvfs so that it works in concert with version 2 VFSs. No changes to core SQLite.
2020-11-18 dan Fix a problem causing sqlite3_carray_bind() to return an undefined value.
2020-11-18 dan Update test code to fix a problem with SQLITE_OMIT_VIRTUAL_TABLE builds.
2020-11-18 drh On unix, for certain error codes of read()/pread() return SQLITE_IOERR_CORRUPTFS instead of SQLITE_IOERR_READ.  And then convert this error into SQLITE_CORRUPT prior to returning back to the application.
2020-11-17 drh Claw back most of the performance lost in the previous commit.
2020-11-17 dan Fix trivial memory leaks in the shell and sqldiff programs.
2020-11-17 dan Fix test file busy2.test so that it works with the "inmemory_journal" permutation.
2020-11-17 drh Add a single-argument form to the CARRAY table-valued function, with content bound using the sqlite3_carray_bind() interface that is included with the extension.
2020-11-16 drh Enhance the unix VFS so that it removes extra "/", "/./" and "/../" from the database filename.
2020-11-14 drh Modify UPDATE so that two-pass updates on a rowid table use an ephemeral table to store rowids rather than a RowSet.  This uses less memory, though it is slower.
2020-11-12 drh Improved diagnostics output with ".wheretrace 0x800".  No changes to non-debug builds.
2020-11-11 drh Fix a typo in a comment.  No changes to code.
2020-11-08 drh Use NEVER and ALWAYS macros to confirm that the return value from sqlite3ExprSkipCollateAndLikely() is never NULL in some of its use cases.
2020-11-06 drh Add ALWAYS() to conditionals associated with SHM locking that are always true.
2020-11-05 drh Suppress errors associated with TEMP triggers that reference objects in non-TEMP databases.  This is a continuation of the fix for ticket #3810 shown in check-in [ba1afc040171810d] from [/timeline?c=trunk:200908061743|2009-08-06], based on a bug report in [forum:/forumpost/157dc791df|forum post 157dc791df]
2020-11-05 drh Remove unused variable from speedtest1.c
2020-11-02 drh Fix an another OSS-Fuzz discovered assertion fault due to ALTER TABLE and strange triggers.
2020-10-31 drh Add the -tabs command-line option to the CLI.
2020-10-31 dan Update the ALTER TABLE command to correctly handle UPDATE ... FROM statements within trigger programs.
(...)
2020-09-15 drh Do not invoke usleep() for more than 999999 microseconds.
2020-09-11 dan Catch fts5 index corruption caused by issuing 'delete' commands with incorrect data earlier in some cases. Also fix a couple of test script problems.
2020-09-10 drh Try again to fix the typo in the sqlite3_txn_state() documentation.
2020-09-10 drh Fix typo in the documentation for the new sqlite3_txn_state() interface.
2020-09-10 drh If an amalgamation is created using SQLITE_ENABLE_UPDATE_DELETE_LIMIT, ensure that it can still be compiled without that option by setting the SQLITE_UDL_CAPABLE_PARSER flag in the preamble.
2020-09-07 dan Fix another (harmless in practice) tsan error in shared-cache mode.
2020-09-06 drh Remove an artifical limitation on the length of columns in the "box" output mode of the command-line shell.
2020-09-05 drh Fix a bug in Fossil that might cause it to crash if there is a multi-terminal token with a space following the "|" separator. This does not affect SQLite.
2020-09-04 dan Fix a tsan error that could occur when using shared-cache mode.
2020-09-04 dan Ensure that the busy-handler count is reset at the end of each sqlite3_file_control() and sqlite3_prepare() (and _v2() and _v3()).
2020-09-01 mistachkin Fix harmless compiler warning.
2020-09-01 drh In the Lemon output, add a prefix comment that explains that the output file is automatically generated and shows the name of the source file.
2020-09-01 drh Lemon updates:  (1) include the #defines for all tokens in the generated C file, so that the C-file can be stand-alone.  (2) If the grammar begins with a %include {...} directive on line one, make that directive the header for the generated C file.  (3) Enhance the lemon.html documentation.
2020-09-01 drh Improvements to the IN-early-out optimization so that it works more efficiently when there are two or more indexed IN clauses on a single table.
2020-09-01 drh Fix a harmless compiler warning.
2020-08-31 drh New test-control that returns the number of calls to sqlite3BtreeMovetoUnpacked() on the main database and then resets the counter.  This only works for SQLITE_DEBUG builds.
2020-08-31 drh Fix the documentation for the OP_IdxGT family of opcodes to show that the P5 operand is not used.
2020-08-29 dan Fix a sanitizer warning in zipfile.c.
2020-08-28 dan Fix handling of an xShmLock(SHARED, UNLOCK) call when the caller does not hold any lock on the specified slot, but another connection in the same process holds an EXCLUSIVE.
2020-08-28 dan Modify the unixShmLock() function to avoid iterating through the (possibly large) set of connections to the same database file.
2020-08-28 mistachkin Add the miscellaneous 'series' extension to the shell.
2020-08-28 drh Update Lemon documentation.  Patches from sgbeal.
2020-08-28 drh Fix a couple of unreachable branches.
2020-08-28 dan Add fts5 test to confirm that for a table with columns a, b, c and d, "{a b}" and "-{c d}" are handled similarly.
2020-08-27 drh Remove a (harmless) redundant variable from the CLI implementation.
2020-08-27 drh Include the original text of the CHECK constraint in the error message for anonymous CHECK constraints.
2020-08-27 drh Apple the same fix for ticket [9eda2697f5cc1aba] to text-to-integer conversions that was done for floating point conversions by check-in [1c76f1d8ec0937a2].
2020-08-27 dan Do not attempt to run test script analyze3.test as part of the "prepare" permutation.
2020-08-26 drh Enhance the ".databases" command in the CLI so that it shows the result of sqlite3_db_readonly() and sqlite3_txn_state() for each database file.
2020-08-26 drh If the argument to the ".read" command in the CLI begins with "|" then run the remainder of the argument as a command and read input from the output of that command.
2020-08-25 drh Add support for the sqlite3_txn_state() interface.
2020-08-24 drh Reduce the N in the logN term for the sorting cost associated with computing DISTINCT by B-Tree by half, under the assumption that the DISTINCT will eliminate about half the rows of output.  This is really a wild guess.  But we do not have any better way of estimating what the row-count reduction due to DISTINCT will actually be.
2020-08-24 dan Correctly handle expressions like "x IS (not) true/false" within the rhs of IN() expressions. Fix for [f3ff1472].
2020-08-20 dan Fix a crash that could occur in SQLITE_MAX_EXPR_DEPTH=0 builds when processing SQL containing syntax errors.
2020-08-20 dan Avoid a buffer overrun in test code that could occur if certain test functions were passed a hex-string containing an odd number of digits.
2020-08-19 drh Try to make SQLite easier to compiler for Mac Catalyst. See [https://sqlite.org/forum/forumpost/803387a1c5|forum post 803387a1c5].
2020-08-19 drh Do not skip over TK_IF_NULL_ROW operators when bypassing TK_COLLATE operators. Fix to check-in [ac31edd3eeafcef4] which was itself a fix for ticket [45f4bf4eb4ffd788].
2020-08-19 dan Update releasetest_data.tcl so that the "Fast-One" configuration runs bigmmap.test as part of release testing.
2020-08-19 dan Fix a problem in test file bigmmap.test.
2020-08-17 drh Fix the -quote option on the CLI to set the correct column separator.
2020-08-16 drh Optimization to sqlite3BeginWriteOperation().
2020-08-15 drh Optimization of the sqlite3SrcListDelete() routine.
2020-08-15 drh Remove more unnecessary sqlite3GetVdbe() calls, replacing them with assert()s.
2020-08-15 drh Avoid unnecessary calls to the sqlite3GetVdbe() routine.  Add assert() statements to prove each call is unnecessary.
2020-08-15 drh Omit the unnecessary not-NULL check on the upper bound of a forward index range scan.  The subsequent OP_IdxGT will always catch the NULL.  Similarly, omit the not-NULL check on the lower bound of a reverse index range scan, as the SeekLE opcode will disallow the NULL.
2020-08-14 drh Update the version number to 3.34.0 for the next development cycle.
2020-08-14 drh Optimizations to the logic that converts main table accesses into equivalent index accesses.  Code is now slightly smaller and faster than trunk.
2020-08-14 drh For UPDATE and DELETE, use OP_DeferredSeek always.  If the seek must later be resolved, add the OP_FinishSeek opcode after all WHERE clause terms have been processed.  This obviates the need for the WHERE_SEEK_TABLE and WHERE_SEEK_UNIQ_TABLE flags to sqlite3WhereBegin() and the ensuing complication, and it allows the covering index optimization to be used further into WHERE clause processing.
2020-08-14 drh Experimental change to try to get some DELETE operations to access values using the index rather than the main table, so as to avoid unnecessary main table seeks.
2020-08-14 dan Fix an inconsequential memory leak in sqldiff. Update the "Sanitize" case in wapptest.tcl to use -fsanitize=address,undefined, and to run more tests.

Created with:
  roll-dep src/third_party/sqlite/src

Bug: 1161048
Change-Id: Ieb8f0154930d6cc0597b5aee9444678cf021035b
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/2600334
Reviewed-by: Chris Mumford <cmumford@google.com>
Commit-Queue: Darwin Huang <huangdarwin@chromium.org>
Cr-Commit-Position: refs/heads/master@{#839609}
GitOrigin-RevId: b8631d09aaba64ee30d492e47e6ccf43860e0cac
1 file changed
tree: dcc9b0eaf2e09774a424e66462baae2c08d43299
  1. dev/
  2. fuzz/
  3. scripts/
  4. .gitignore
  5. BUILD.gn
  6. DEPS
  7. DIR_METADATA
  8. LICENSE
  9. OWNERS
  10. PRESUBMIT.py
  11. README.chromium
  12. README.md
  13. sqlite3.h
  14. sqlite3_shim.c
  15. sqlite3_shim_fixups.h
  16. sqlite_chromium_configuration_flags.gni
  17. sqlite_common_configuration_flags.gni
  18. sqlite_dev_configuration_flags.gni
  19. sqlite_shell_icu_helper.cc
  20. sqlite_shell_icu_helper.h
  21. sqlite_shell_shim.c
README.md

This is the top folder for Chromium's SQLite. The actual SQLite source is not in this repository, but instead cloned into the src directory from https://chromium.googlesource.com/chromium/deps/sqlite.

The directory structure is as follows. Files common to all third_party projects (BUILD.GN, OWNERS, LICENSE) are omitted.

  • src/ The Chromium fork of SQLite (cloned via top level DEPS file).
  • scripts/ Scripts that generate the files in the amalgamations in src/.
  • sqlite.h The header used by the rest of Chromium to include SQLite. This forwards to src/amalgamation/sqlite3.h
  • fuzz/ Google OSS-Fuzz (ClusterFuzz) testing for Chromium's SQLite build This directory contains:

The SQLite amalgamation is committed to the SQLite Chromium repository (in src), but it is created by a script that lives in the Chromium repository. This is because the configuration variables for building and amalgamation generation are shared.

There are two amalgamations. The one in //third_party/sqlite/src/amalgamation is used by Chromium. A second, located at //third_party/sqlite/src/amalgamation_dev is used for some Chromium developer tools and is not distributed.

Upgrade to a new SQLite release.

Note SQLite tags all releases version-<release number>, e.g. version-3.33.0. The Chromium project prefixes all tags/branches with “chromium-”, e.g. chromium-version-3.33.0.

  1. Create new release branch

    Use the SQLite commit ID when creating a branch. For example “562fd18b9dc27216191c0a6477bba9b175f7f0d2” corresponds to the 3.33.0 release. The commit is used, instead of the tag name, because we do not mirror the SQLite tags along with the commits. The correct commit ID can be found at sqlite/releases.

    Create the branch at Gerrit/branches.

  2. Checkout the new Chromium release branch.

    cd //third_party/sqlite/src
    git fetch origin
    export VERSION=3.33.0
    git checkout -b chromium-version-$VERSION \
        --track origin/chromum-version-$VERSION
    
  3. Generate and commit the SQLite amalgamations.

    ../scripts/generate_amalgamation.py
    git add amalgamation
    git add amalgamation_dev
    git commit -m "Amalgamations for release $VERSION"
    
  4. Run local tests.

    Follow steps in Running Tests below to execute all verifications and tests.

  5. Upload the new release branch for review.

    git cl upload
    
  6. Roll the Chromium DEPS file.

    Once review above has merged:

    1. Roll the chromium/src/DEPS file to reference that new commit ID.
      roll-dep src/third_party/sqlite/src --roll-to <git hash of merged CL>
      
    2. Update the version in //third_party/sqlite/README.chromium. Append the commit created by roll-dep above.

Cherry-pick unreleased commit from SQLite.

Sometimes critical fixes land in SQLite's master, but are not yet in a release. If these need to be brought into the current release branch do the following:

  1. Checkout the current release branch.

    export VERSION=3.33.0
    cd //third_party/sqlite/src
    git checkout -b chromium-version-$VERSION \
      --track origin/chromium-version-$VERSION
    
  2. Cherry-pick the change

    Git can be used to cherry pick upstream changes into a release branch but the sqlite_cherry_picker.py script is preferred. This script automates a few tasks such as:

    • Identifying the correct Git commit ID to use if given the Fossil commit ID.
    • Automatically calculating Fossil manifest hashes.
    • Skipping conflicted binary files.
    • Generating the amalgamations.

    Cherry-pick the commit:

    ../scripts/sqlite_cherry_picker.py <full git or fossil commit id>
    

    If there is a conflict that the script cannot resolve then, like git-cherry-pick, the script will exit and leave you to resolve the conflicts. Once resolved run the script a second time:

    ../scripts/sqlite_cherry_picker.py --continue
    
  3. Run local tests.

    Follow steps in Running Tests below to execute all verifications and tests.

  4. Upload cherry-picked change (with amalgamations) for review.

    git cl upload
    
  5. Update the Chromium DEPS file.

    Once review above has merged, roll the chromium/src/DEPS file to reference that new commit ID.

    roll-dep src/third_party/sqlite/src --roll-to <git hash of merged CL>
    

Running Tests

Build all desktop targets:

Check that extract_sqlite_api.py added “chrome_” to all exported symbols. Only “_fini” and “_init” should be unprefixed, but are conditionally exported by the linker and may be omitted.

autoninja -C out/Default
nm -B out/Default/libchromium_sqlite3.so | cut -c 18- | sort | grep '^T'

Running unit tests

out/Default/sql_unittests

Running web tests

third_party/blink/tools/run_web_tests.py -t Default storage/websql/

Running SQLite's TCL test suite within the Chromium checkout.

This is one of the SQLite test suites. They take approximately 3 minutes to build and run on a fast workstation.

cd //third_party/sqlite
./scripts/generate_amalgamation.py --testing
make --directory=src test | tee /tmp/test.log

Note: Tests may fail on Chromium release branches. This is because some tests rely on SQLite databases (binary files) which are committed to the source and are likely not merged down when cherry picked. It is safe to ignore these errors which should be reasonably easy to identify based on the cherry picked upstream changes.

Show error'ed tests:

egrep 'errors out of' /tmp/test.log

Show broken tests:

egrep 'Failures on these tests:' /tmp/test.log

Broken tests will also show lines ending in “...” instead of “... Ok”.

When done cleanup the SQLite repository:

cd src
make clean
git clean -i  # and delete everything
rm -rf testdir
git checkout amalgamation/sqlite3.c
git checkout amalgamation_dev/sqlite3.c