Google Git
Sign in
chromium / chromium / src / third_party / sqlite / refs/heads/main
commit725b2db213272482eeb9f4214589239dc683c29f[log] [tgz]
authorEvan Stade <estade@chromium.org>Tue Mar 26 22:12:16 2024
committerCopybara-Service <copybara-worker@google.com>Tue Mar 26 22:17:53 2024
treea3021b02f46af1f89bf628c25834cb1cdd532e52
parent0de8ddb83d29e5a892bb7f2f36f361e7a57850a4 [diff]
Roll src/third_party/sqlite/src/ f51e3fa40..25d86f930 (33 commits)

https://chromium.googlesource.com/chromium/deps/sqlite.git/+log/f51e3fa404f5..25d86f9308e8

$ git log f51e3fa40..25d86f930 --date=short --no-merges --format='%ad %ae %s'
2024-03-26 estade A formerly impossible case has now become possible because of the new ExprIsConstant enhancements.  So deal with that.
2024-03-26 estade Amalgamations for release 3.45.2
2024-03-12  Version 3.45.2
2024-03-11  Fix the ".import" command in the CLI so that it works correctly with tables that contain computed columns.
2024-03-09  Reenable the use of SQLITE_OMIT_VIRTUALTABLE by fixing a misplaced #endif caused by the merge at [c080560c2d0d753c].
2024-03-09  Fix the rtreeJ.test module to align it with [cd017c28d516399e].
2024-03-09  Handle an OOM case in fts5 code to query tokendata=1 tables.
2024-03-09  The NOT NULL strength reduction optimization should be applied to the WHERE clause only.
2024-03-08  Silently ignore redundant ON CONFLICT clauses in an UPSERT.
2024-03-06  Fix a memory leak in new memdb1.test test cases that were added by [e638d5e408ea2e18].  No changes to SQLite itself.
2024-03-06 mistachkin Fix harmless compiler warning seen with MSVC.
2024-03-06 mistachkin Fix harmless compiler warnings seen with MSVC.
2024-03-06 mistachkin Fix a minor test file locking issue on Windows.
2024-03-06 mistachkin Work around obscure floating point issue seen with older versions of MSVC.
2024-03-06 mistachkin Fix two -Werror=lto-type-mismatch warnings reported in [forum:ef62b57bd5|forum post ef62b57bd5].
2024-03-06 mistachkin The quote() SQL function should convert +Inf into 9.0e+999 and -Inf into -9.0e+999.  See [forum:/forumpost/6675b25108|forum post 6675b25108].
2024-03-06 mistachkin Make sure key comprisons are done correctly if the index key contains NaN values that have not been shifted into NULLs. That can only happen due to database corruption, but we need to deal with it nevertheless.
2024-03-06 mistachkin When inserting a 3-byte cell into a btree, ensure that the extra padding byte is 0x00.  This is not necessary for security, as far as I can tell, but it seems like a reasonable precaution.
2024-03-06 mistachkin Rig sqlite3_serialize() so that it will initialize a previously uninitialized database prior to serializing it, so that it does not have a zero-byte size and does not return NULL (except for OOM).  [forum:/forumpost/498777780e16880a|Forum thread 498777780e16880a].
2024-03-06 mistachkin When doing a text-affinity comparison between two values where one or both have both a text and a numeric type, make sure the numeric type does not confuse the answer.  This is a deeper fix to the problem observed by [forum:/forumpost/3776b48e71|forum post 3776b48e71].  The problem bisects to [25f2246be404f38b] on 2014-08-24, prior to version 3.8.7.
2024-03-06 mistachkin Ensure that the replace() SQL function always returns a TEXT value even when its first argument is numeric and its second argument is an empty string.  Fix for the issue reported by [forum:/forumpost/3776b48e71|forum post 3776b48e71].
2024-03-05 stephan Reformulate [34439fe3aeea7cbb] slightly to resolve a false-positive OOM reported in [forum:2eadfe94e3|forum post 2eadfe94e3].
2024-03-05  The value returned by the json_each.json field needs to survive longer than the json_each virtual table itself, in the event that the value is used in an aggregate expression.  dbsqlfuzz 5120bd0b3bb0b73eebbcb79ac37c6b0663fccad6
2024-02-17  Fix rounding in zero-precision %f and %g printf conversions.
2024-02-16  Fix the subtype on the value column from json_each/json_tree for cases when the value is an array or object.
2024-02-13  Fix a bug in PRAGMA integrity_check introduced by enhancement (8a) in release 3.42.0.
2024-02-12  Have rtree avoid keeping a blob handle open following an error.
2024-02-11  Cherrypick multiple fixes for trifling faults from trunk, so that all 834 dbsqlfuzz run clean with ASAN on branch-3.45. dbsqlfuzz.
2024-02-10 larrybr Further clarify sqlite3_exec() callback parameter lifetime.
2024-02-08 larrybr Mention limited lifetime of string arrays passed to sqlite3_exec() callback.
2024-02-07  Detect and respond to an OOM on the jsonStringTerminate() call of jsonReturnStringAsBlob() routine.
2024-02-06  Return SQLITE_ABORT if the underlying shadow tables change in the middle of an rtree query in such a way as to invalidate an rtree internal priority queue entry. This replaces the SQLITE_ABORT_ROLLBACK mechanism added in [97cffff331b].
2024-02-03  Following a ROLLBACK that reverts changes to an RTREE, any pending queries against that same RTREE abort with code SQLITE_ABORT_ROLLBACK.

Created with:
  roll-dep src/third_party/sqlite/src

Bug: 328481092
Change-Id: I2006bb852066d09307c8c1451e51eabfd2b85f6c
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/5398460
Reviewed-by: Austin Sullivan <asully@chromium.org>
Commit-Queue: Austin Sullivan <asully@chromium.org>
Cr-Commit-Position: refs/heads/main@{#1278637}
NOKEYCHECK=True
GitOrigin-RevId: a4c810cb534b8efb092222cdbe3307432e70cf25
1 file changed
tree: a3021b02f46af1f89bf628c25834cb1cdd532e52
  1. dev/
  2. fuzz/
  3. scripts/
  4. .gitignore
  5. BUILD.gn
  6. DEPS
  7. DIR_METADATA
  8. LICENSE
  9. OWNERS
  10. PRESUBMIT.py
  11. README.chromium
  12. README.md
  13. sqlite3.h
  14. sqlite3_shim.c
  15. sqlite3_shim_fixups.h
  16. sqlite_chromium_configuration_flags.gni
  17. sqlite_common_configuration_flags.gni
  18. sqlite_dev_configuration_flags.gni
  19. sqlite_shell_icu_helper.cc
  20. sqlite_shell_icu_helper.h
  21. sqlite_shell_shim.c
README.md

Chromium SQLite.

This is the top folder for Chromium's SQLite. The actual SQLite source is not in this repository, but instead cloned into the src directory from https://chromium.googlesource.com/chromium/deps/sqlite.

The directory structure is as follows. Files common to all third_party projects (ex. BUILD.GN, OWNERS, LICENSE) are omitted.

  • src/ The Chromium fork of SQLite (cloned via top level DEPS file).
  • scripts/ Scripts that generate the files in the amalgamations in src/.
  • sqlite.h The header used by the rest of Chromium to include SQLite. This forwards to src/amalgamation/sqlite3.h
  • fuzz/ Google OSS-Fuzz (ClusterFuzz) testing for Chromium's SQLite build.

Amalgamations

SQLite amalgamations are committed to the SQLite Chromium repository (in src), but are created by a script that lives in the Chromium repository. This is because the configuration variables for building and amalgamation generation are shared.

There are two amalgamations:

  • //third_party/sqlite/src/amalgamation is shipped, tested, and Fuzzed by Chromium.
  • //third_party/sqlite/src/amalgamation_dev is not distributed or tested by Chromium. It is used for some developer tools (either only for local development, or only on trusted input).

//third_party/sqlite/src repository.

CLs in this repository cannot be submitted through the commit queue (ex. CQ+2), because there is no commit queue / try bot support for this repository. Please use the “Submit” button (in Gerrit's 3-dot menu on the top right) to submit CLs in this repository instead.

Playbook

Upgrade to a new SQLite release.

SQLite should be upgraded as soon as possible whenever a new version is available. This is because new versions often contain security and stability improvements, and frequent upgrades allow Chromium to have minimal cherry-pick diffs when requesting investigation for SQLite bugs discovered by Chromium Fuzzers. New versions may be viewed here, and bugs for these upgrades may look like this example.

Historically, Chromium fuzzers often find issues within 2 weeks after upgrading to new SQLite versions. Avoid upgrading SQLite within 1-2 weeks of a Chromium branch point to allow fuzzers time to run. However, if the new SQLite release contains known security or stability fixes, upgrade once available and monitor fuzzers more closely.

SQLite version upgrades tend to be extremely large changes (example), for which the diffs are not possible to thoroughly review.

Note SQLite tags all releases version-<release number>, e.g. version-3.40.0. The Chromium project prefixes all tags/branches with “chromium-”, e.g. chromium-version-3.40.0.

  1. Create new release branch

    Create the branch at Gerrit/branches. The branch name should look like chromium-version-3.40.0 and the initial revision will look something like refs/tags/upstream/version-3.40.0.

    Note: To create a release branch, you must be listed as a member in the sqlite-owners Gerrit group

  2. Checkout the new Chromium release branch.

    Get the version from the README.chromium.

    cd third_party/sqlite/src  # from //chromium/src
git fetch origin
export VERSION=3.40.0
git checkout -b chromium-version-$VERSION \
    --track origin/chromium-version-$VERSION

  3. Generate and commit the SQLite amalgamations.

    ./../scripts/generate_amalgamation.py
git add amalgamation amalgamation_dev
git commit -m "Amalgamations for release $VERSION"

  4. Run local tests.

    Follow steps in Running Tests below to execute all verifications and tests.

  5. Upload the new release branch for review.

    git cl upload

  6. Roll the Chromium DEPS file.

    Once review above has merged:

    1. Roll the chromium/src/DEPS file to reference that new commit hash.
      roll-dep src/third_party/sqlite/src --roll-to <git hash of merged CL>
    2. Update the version in //third_party/sqlite/README.chromium. Amend the commit created by roll-dep above.

Cherry-pick unreleased commit from SQLite.

Sometimes critical fixes land in SQLite's master, but are not yet in a release. This may occur when other SQLite embedders find critical security or stability issues that SQLite authors then fix, but are often detected by Chromium ClusterFuzz as well.

If you're triaging a ClusterFuzz bug, an internal playbook on how to triage and fix ClusterFuzz bugs is available at go/sqlite-clusterfuzz-bug-process.

If changes need to be brought into the current release branch, please do the following:

  1. Checkout the current release branch.

    Get the version from the README.chromium.

    export VERSION=3.40.0
cd third_party/sqlite/src  # from //chromium/src
git checkout -b chromium-version-$VERSION \
  --track origin/chromium-version-$VERSION

  2. Cherry-pick the change

    Git can be used to cherry pick upstream changes into a release branch but the sqlite_cherry_picker.py script is preferred. This script automates a few tasks such as:

    • Identifying the correct Git commit hash to use if given the Fossil commit hash. note this is currently broken and a Git hash must be provided
    • Automatically calculating Fossil manifest hashes.
    • Skipping conflicted binary files.
    • Generating the amalgamations.

    Cherry-pick the commit:

    ../scripts/sqlite_cherry_picker.py <full git commit hash>

    If there is a conflict that the script cannot resolve then, like git cherry-pick, the script will exit and leave you to resolve the conflicts. Once resolved run the script a second time:

    ../scripts/sqlite_cherry_picker.py --continue

    If you have access to the SQLite fossil commit hash, and would like to map this to the corresponding git hash, you can use GitHub search. As SQLite‘s git repository’s commits include the fossil hash, you can search for the fossil hash, using the following query with the fossil commit hash appended (example search): https://github.com/sqlite/sqlite/search?type=commits&q=

    If the cherry-picking script is unable to cherry-pick a commit, like in https://crbug.com/1162100, manually apply the change from a SQLite or git, in //third_party/sqlite/src's files modified in the SQLite tracker, like at https://sqlite.org/src/info/a0bf931bd712037e. From there, run ../scripts/generate_amalgamation.py to propagate these changes over to the amalgamation files. sqlite_cherry_picker.py should generally be preferred, as it updates hashes and simplifies tracking.

  3. Run local tests.

    Follow steps in Running Tests below to execute all verifications and tests.

  4. Upload cherry-picked change (with amalgamations) for review.

If the relevant bug is a security bug, make sure that the reviewers are cc‘ed. Otherwise, they may not know what/why they’re reviewing.

```sh
git cl upload
```

  1. Update the Chromium DEPS file.

    Once review above has merged, roll the chromium/src/DEPS file to reference that new commit hash.

    roll-dep src/third_party/sqlite/src --roll-to <git hash of merged CL>

Running Tests

Build all desktop targets:

Check that extract_sqlite_api.py added “chrome_” to all exported symbols. Only “_fini” and “_init” should be unprefixed, but are conditionally exported by the linker and may be omitted.

autoninja -C out/Default
nm -B out/Default/libchromium_sqlite3.so | cut -c 18- | sort | grep '^T'

Running unit tests

out/Default/sql_unittests

Running web tests

third_party/blink/tools/run_web_tests.py -t Default storage/websql/

Running SQLite's TCL test suite within the Chromium checkout.

This is one of the SQLite test suites. They take approximately 3 minutes to build and run on a fast workstation.

Note: Tests currently fail both locally and on Chromium release branches. They fail on release branches because some tests rely on SQLite databases (binary files) which are committed to the source and are likely not merged down when cherry picked. It is safe to ignore these errors which should be reasonably easy to identify based on the cherry picked upstream changes. Until these tests are fixed, it is safe to ignore these tests when running SQLite test suites.

cd third_party/sqlite  # from //chromium/src
./scripts/generate_amalgamation.py --testing
make --directory=src test | tee /tmp/test.log

Show tests with errors:

egrep 'errors out of' /tmp/test.log

Show broken tests:

egrep 'Failures on these tests:' /tmp/test.log

Broken tests will also show lines ending in “...” instead of “... Ok”.

When done, clean up the SQLite repository:

cd src
make clean
git clean -i  # and delete everything
rm -rf testdir
git checkout amalgamation/sqlite3.c
git checkout amalgamation_dev/sqlite3.c