commit | 371ed9197931e7b3e267b0b460060d5abb5fe1bb | [log] [tgz] |
---|---|---|
author | Austin Sullivan <asully@chromium.org> | Wed Feb 22 23:00:09 2023 |
committer | Copybara-Service <copybara-worker@google.com> | Wed Feb 22 23:21:54 2023 |
tree | 11ce25185b4d10950a448078c385353439b1e2da | |
parent | 9728ae56f974f56ca014c99576f1a5526b186266 [diff] |
sqlite: Upgrade to 3.41.0 Roll src/third_party/sqlite/src/ 469aae811..88742a546 (536 commits) https://chromium.googlesource.com/chromium/deps/sqlite.git/+log/469aae8118e1..88742a54683b $ git log 469aae811..88742a546 --date=short --no-merges --format='%ad %ae %s' 2023-02-22 asully Amalgamations for release 3.41.0 2023-02-21 Version 3.41.0 2023-02-21 Disable DQS by default in the MSVC makefile for the amalgamation tarball. 2023-02-21 Yet another comment typo fix. No changes to working code. 2023-02-21 Fix a minor typo in a comment. No changes to working code. 2023-02-21 Fix formatting and improved documentation on the various sqlite3_file_control() opcodes. Other minor documentation changes. No changes to working code. 2023-02-20 Update the TEA version number to 3.41.0 2023-02-20 Fix an incorrect #ifdef in the CLI. 2023-02-19 Fix a harmless compiler warning. 2023-02-18 Fix stale requirement marks and fix a typo in the documentation for sqlite3_preupdate_hook(). 2023-02-18 Fix a harmless UBSAN warning in debugging code of the new unhex() function. 2023-02-17 Fix a harmless typo in the test case added by [29fc06465efb948f]. 2023-02-17 Do not allow the COUNTOFVIEW optimization to run if the count() contains a FILTER clause. dbsqlfuzz 4f8e0de6e272bbbb3e1b41cb5aea31e0b47297e3 2023-02-14 Dan Kennedy Update testrunner.tcl to run zipvfs test scripts on unix. 2023-02-13 Ignore extra parentheses around a subquery on the RHS of an IN operator, because that is what PostgreSQL does. 2023-02-13 Do not allow WHERE clause terms to match constant string index terms, which can happen if DQS_DDL is enabled. Follow-up to [44200596aa943963]. dbsqlfuzz 54c9db85ed4af7055f5fd0d50877875c82b11d46. 2023-02-13 Dan Kennedy Fix compile time option SQLITE_DEFAULT_SYNCHRONOUS so that it works consistently. 2023-02-13 Dan Kennedy Allow vector-IN expressions like "(a, b) IN ( (?,?), (?,?) )" to use an index. 2023-02-13 In the LIKE optimization, do not analyze the new virtual WHERE clause terms until both have been added, since they are expected to be consecutive and the analysis might add complementary terms. This fixes a problem caused by [44200596aa943963] and discovered by dbsqlfuzz and recorded as case 7e3b5983727d843b910b2d9ab556e4afcd777cfb. 2023-02-11 Change a variable from 32 to 64-bits to avoid a harmless compiler warning in Xcode. [forum:/forumpost/402d733c22|Forum post 402d733c22]. 2023-02-10 Do a better job of detecting when a WHERE clause term might be useful to an expression index. Fix for performance regression reported by [forum:/forumpost/e65800d8cb|forum thread e65800d8cb]. 2023-02-10 Dan Kennedy Fix a problem with the fts5 trigram tokenizer and LIKE or GLOB patterns for which contain runs of 2 or fewer non-wildcard characters that are 3 or more bytes when encoded as utf-8. 2023-02-10 Ensure that the valueFromFunction() routine does not clear a prior parser error. dbsqlfuzz 6fa816f20cf5b62260d635d110b88f38e29d8fe1. 2023-02-10 stephan Minor text-only updates to wasm demo/test HTML and license header. 2023-02-10 stephan Fix ext/wasm/fiddle build, which was silently broken by recent build refactoring. 2023-02-09 Disable the double-quoted string misfeature by default in CLI builds. DQS can be reenabled at run-time using the ".dbconfig dqs_dml" and ".dbconfig dqs_ddl" dot-commands. 2023-02-09 stephan Squelch two harmless signedness comparison warnings in shell.c.in. 2023-02-09 New test cases added to fuzzdata8.db. 2023-02-09 The "flexnum" affinity that was added by [44135d6ea84f7ba6] needs to also be added to the output of the affinity() built-in function. dbsqlfuzz d309eaa5fe492c9606a8be876c2bc7dedb29d3d8 2023-02-08 Back out the 'txn' enhancement to date/time functions. The duration of a "transaction" is confused and needs to be straightened out prior to moving forward with this change. 2023-02-08 Always use 64-bit integers for stats associated with STAT1 and STAT4. 2023-02-08 Dan Kennedy Better fix the problem where optimizing an fts5 table too often causes it to become unreadable (first attempt was [35bed981]). 2023-02-08 larrybr Cause gcc warning suppression in shell.c to be nice in other project(s). 2023-02-08 Dan Kennedy Update an assert() in the stat4 code that is only true for a well-formed database. 2023-02-08 Fix an incorrect assert() in STAT4 logic added just a few days ago on [2023-02-01|/info/55a26c67ed4a3a93]. 2023-02-08 Test cases to show that CURRENT_TIMESTAMP and similar hold the same value across a transaction. 2023-02-08 Revert the behavior of date/time functions with no arguments so that they once again work like 'now', even while CURRENT_TIMESTAMP and similar work like 'txn'. 2023-02-08 Change the behavior of date-time functions without any arguments (and thus the CURRENT_TIME, CURRENT_DATE, and CURRENT_TIMESTAMP variables) so that they work like 'txn' instead of like 'now'. This is an incompatibility with legacy, but brings SQLite into conformance with all other SQL systems. 2023-02-07 Simplify the code and add test cases. 2023-02-07 Add support for the 'txn' argument to date/time functions that works like 'now' but keeps the same time for the entire transaction. 2023-02-07 Dan Kennedy Fix a test script problem causing "make test" to fail for non-RBU builds. 2023-02-07 Fix a typo in example documentation code for sqlite3_vtab_in_next(). 2023-02-06 stephan Roll back part of [c54f29d8] which attempted to use symbols which that worker does not have access to. 2023-02-06 Dan Kennedy Fix RBU test scripts to avoid attempting to copy or delete a locked file. 2023-02-06 Fix the build after the previous change 2023-02-06 Remove the long obsolete "client/server" mode tests. 2023-02-06 Dan Kennedy Update test scripts speed1.test and speed1p.test so they may be run by testrunner.tcl. 2023-02-06 Shorten the status line for testrunner.tcl so that it fits on an 80-character terminal. 2023-02-06 Fix error in the releasetest makefile target from the previous check-in. 2023-02-06 Update "releasetest" makefile targets to use testrunner.tcl. (...) 2022-11-22 This is the first in what is anticipated to be a long sequence of incremental changes aimed at improving aggregate query processing, and in particular helping aggregate queries take better advantage of indexes on expression. The end goal is to resolve ticket [99378177930f87bd], though it remains to be seen whether or not I can get there with this approach. 2022-11-22 Convert an ALWAYS() in sqlite3DbSpanDup() into an assert(), for a performance increase and size reduction. 2022-11-21 Split out the debugging code that dumps an AggInfo object into a separate subroutine called printAggInfo() so that it can be invoked interactively during debugging. No changes to production code. 2022-11-21 Change the name of the Parse.pIdxExpr field to pIdxEpr so that the name is distinct from all other identifiers in the code and thus less confusing. 2022-11-21 stephan OPFS contention test: give each worker a distinct recognizable name instead of a random one. 2022-11-21 Performance optimization and size reduction in insertCell() by omitting the "pRC" argument and instead returning the result code as an integer. 2022-11-21 Performance optimization to sqlite3BtreeInsert(). 2022-11-21 Small performance improvement in sqlite3BtreeTransferRow(). 2022-11-21 stephan Add new JS tests dir to those pushed to the test server. 2022-11-21 stephan More tweaking of OPFS concurrency measures and the related test app. 2022-11-21 stephan Resolve missing SQLITE_LOCKED result code which triggered a new (since last checkin) exception in the OPFS VFS. Improve output of the OPFS contention tester app. 2022-11-21 stephan Add test app for experimenting with multi-worker OPFS concurrency. Tweak OPFS VFS to significantly improve the otherwise "unfortunate" concurrency situation. 2022-11-21 larrybr Get clean builds for MS tools. Add build examples for extension glommer (basexx.c) Make sure extensions build for dynamic load and built into shell. 2022-11-21 larrybr Fix a base64 decode bug. Provide for convenient inclusion of extension(s) built into the CLI, to simplify testing and for its own sake. Improve comments. Cure collision between base64.c and base85.c when both are in the same translation unit. 2022-11-20 stephan Remove check for WASM_WASI macro when detecting wasi compilation mode, as that macro is project-specific. Rely only on __wasi__ (exposed by clang) to detect wasi compilation mode. 2022-11-20 stephan js dist: account for a file rename in the previous checkin. 2022-11-20 stephan Further minor cleanups in the JS build related to vanilla vs ESM. 2022-11-20 stephan Remove an obsolete reference to WASMFS from ext/wasm/index.html. 2022-11-20 stephan Generic cleanups and doc additions in the wasm build files. 2022-11-20 stephan Add 'snapshot' target to create distinctly-named snapshot builds of the wasm deliverables zip file. 2022-11-20 stephan Minor cleanups in the ESM-related preprocessor filtering. 2022-11-19 New assert()s confirm that BTREE_PREFORMAT must be the same value as OPFLAG_PREFORMAT. 2022-11-19 Performance optimization in sqlite3BtreeInsert(). 2022-11-19 Performance improvement and size reduction in balance_nonroot(). 2022-11-19 Small performance improvement and size reduction in btree. 2022-11-19 stephan Get tester1.js working via an ES6 worker module and add that variant to the dist zipfile. 2022-11-19 stephan Elide a wasi-incompatible shell.c block in SQLITE_WASI builds. 2022-11-19 Improved comments on the pageFreeArray() routine of btree.c. No changes to code. 2022-11-19 Streamline the decodeFlags() routine in btree.c for a small performance increase. 2022-11-19 stephan Default to SQLITE_THREADSAFE=0 and SQLITE_OMIT_LOAD_EXTENSION for wasi builds. 2022-11-19 stephan Account for lack of mmap(), getpid(), and shared memory APIs in wasi. 2022-11-19 stephan Preliminary patches to get sqlite3.c building as-is in WASI environments. 2022-11-19 stephan Add build of sqlite3.mjs (ES6 module), add a test app for it, and include it in the dist build. 2022-11-19 stephan More work towards creation of a ES6 JS module. 2022-11-19 larrybr New extensions for base85 and base64 conversion UDFs 2022-11-19 larrybr Create new branch named "base_convert" 2022-11-19 Small performance optimization in btree.c. 2022-11-19 Databases created using sqlite3_deserialize() should report their filename as an empty string, not as "x". Fix for ticket [53043c9793715f08]. 2022-11-18 Update the version number to 3.41.0 to begin the next development cycle. 2022-11-18 stephan Replace use of cpp with the fit-to-purpose c-pp to avoid cpp's C-centric/JS-unfriendly quirks. 2022-11-17 Fix corner cases in UTF8 handling in the REGEXP extension. [forum:/forumpost/3ffe058b04|Forum post 3ffe058b04]. 2022-11-17 Use the log10() and log2() functions from the standard C library to implement the equivalent SQL functions, in the hope that this will prevent reported precision problems. See [forum:/forumpost/cfceb1230bdcfd84|forum post cfceb1230bdcfd84] and the surrounding thread. 2022-11-17 Split out the documentation for sqlite3_value_encoding() into its own page and make it clear that this interface is intended for testing and debugging only. [forum:/forumpost/c9f445453da950ad|Forum thread c9f445453da950ad]. Comment changes only - no changes to code. 2022-11-17 Raise an error if an attempt is made to compile the CLI with the SQLITE_OMIT_COMPLETE option, since sqlite3_complete() really is necessary for the CLI to operate sanely. 2022-11-16 stephan Change a self.X JS reference to X to account for a symbol resolution discrepancy between vanilla JS and ES6 modules, as explained in [forum:801d8f77e5115141|forum post 801d8f77e5115141]. 2022-11-16 stephan wasm build: include FTS5 instead of FTS4, per /chat and HN discussions. 2022-11-16 Fix harmless typo in comment, reported by Debian in Fossil [https://fossil-scm.org/forum/forumpost/15f7327318]. 2022-11-10 Dan Kennedy Avoid returning SQLITE_SCHEMA if the first query run on a connection is "SELECT nosuchcolumn" or similar. Forum post <https://sqlite.org/forum/forumpost/c1798f77ef>. 2022-11-06 stephan shell.c.in: on non-Windows platforms, check for $XDG_CONFIG_HOME/sqlite3/sqliterc before ~/.sqliterc, per request in [forum:7a16582b1e403c81|forum post 7a16582b1e403c81]. 2022-11-03 stephan Experimentally use clang's C preprocessor to filter #ifdef's out of the generated sqlite3-api.js, the goal being to see if we can feasibly use cpp to include ES6 module-specific code in the main code base and conditionally filter it out. Created with: roll-dep src/third_party/sqlite/src Fixed: 1418203 Change-Id: I2c8cea2b76eba316a0b5db9c6c2348fe9e5008c7 Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/4284338 Reviewed-by: Nathan Memmott <memmott@chromium.org> Commit-Queue: Nathan Memmott <memmott@chromium.org> Auto-Submit: Austin Sullivan <asully@chromium.org> Commit-Queue: Austin Sullivan <asully@chromium.org> Cr-Commit-Position: refs/heads/main@{#1108637} NOKEYCHECK=True GitOrigin-RevId: bf33a39373d178f7ecc136c5495120c6cc1493a3
This is the top folder for Chromium's SQLite. The actual SQLite source is not in this repository, but instead cloned into the src
directory from https://chromium.googlesource.com/chromium/deps/sqlite.
The directory structure is as follows. Files common to all third_party projects (ex. BUILD.GN, OWNERS, LICENSE) are omitted.
src/
The Chromium fork of SQLite (cloned via top level DEPS file).scripts/
Scripts that generate the files in the amalgamations in src/.sqlite.h
The header used by the rest of Chromium to include SQLite. This forwards to src/amalgamation/sqlite3.hfuzz/
Google OSS-Fuzz (ClusterFuzz) testing for Chromium's SQLite build.SQLite amalgamations are committed to the SQLite Chromium repository (in src
), but are created by a script that lives in the Chromium repository. This is because the configuration variables for building and amalgamation generation are shared.
There are two amalgamations:
CLs in this repository cannot be submitted through the commit queue (ex. CQ+2), because there is no commit queue / try bot support for this repository. Please use the “Submit” button (in Gerrit's 3-dot menu on the top right) to submit CLs in this repository instead.
SQLite should be upgraded as soon as possible whenever a new version is available. This is because new versions often contain security and stability improvements, and frequent upgrades allow Chromium to have minimal cherry-pick diffs when requesting investigation for SQLite bugs discovered by Chromium Fuzzers. New versions may be viewed at https://www.sqlite.org/news.html, and bugs for these upgrades may look like this example.
Historically, Chromium fuzzers often find issues within 2 weeks after upgrading to new SQLite versions. Avoid upgrading SQLite within 1-2 weeks of a Chromium branch point to allow fuzzers time to run. However, if the new SQLite release contains known security or stability fixes, upgrade once available and monitor fuzzers more closely.
SQLite version upgrades tend to be extremely large changes (example), for which the diffs are not possible to thoroughly review.
Note SQLite tags all releases version-<release number>
, e.g. version-3.40.0
. The Chromium project prefixes all tags/branches with “chromium-”, e.g. chromium-version-3.40.0
.
Create new release branch
Use the SQLite git commit hash for the release, found at sqlite/tags, when creating a new release branch. For example, “562fd18b9dc27216191c0a6477bba9b175f7f0d2” corresponds to the 3.31.1 release. The commit is used instead of the tag name because we do not mirror the SQLite tags along with the commits.
Create the branch at Gerrit/branches.
Note: To create a release branch, you must be listed as a member in the sqlite-owners Gerrit group
Checkout the new Chromium release branch.
Get the version from the README.chromium.
cd third_party/sqlite/src # from //chromium/src git fetch origin export VERSION=3.40.0 git checkout -b chromium-version-$VERSION \ --track origin/chromium-version-$VERSION
Generate and commit the SQLite amalgamations.
./../scripts/generate_amalgamation.py git add amalgamation amalgamation_dev git commit -m "Amalgamations for release $VERSION"
Run local tests.
Follow steps in Running Tests below to execute all verifications and tests.
Upload the new release branch for review.
git cl upload
Roll the Chromium DEPS file.
Once review above has merged:
chromium/src/DEPS
file to reference that new commit hash.roll-dep src/third_party/sqlite/src --roll-to <git hash of merged CL>
Sometimes critical fixes land in SQLite's master, but are not yet in a release. This may occur when other SQLite embedders find critical security or stability issues that SQLite authors then fix, but are often detected by Chromium ClusterFuzz as well.
If you're triaging a ClusterFuzz bug, an internal playbook on how to triage and fix ClusterFuzz bugs is available at go/sqlite-clusterfuzz-bug-process.
If changes need to be brought into the current release branch, please do the following:
Checkout the current release branch.
Get the version from the README.chromium.
export VERSION=3.40.0 cd third_party/sqlite/src # from //chromium/src git checkout -b chromium-version-$VERSION \ --track origin/chromium-version-$VERSION
Cherry-pick the change
Git can be used to cherry pick upstream changes into a release branch but the sqlite_cherry_picker.py script is preferred. This script automates a few tasks such as:
Cherry-pick the commit:
../scripts/sqlite_cherry_picker.py <full git or fossil commit hash>
If there is a conflict that the script cannot resolve then, like git cherry-pick
, the script will exit and leave you to resolve the conflicts. Once resolved run the script a second time:
../scripts/sqlite_cherry_picker.py --continue
If you have access to the SQLite fossil commit hash, and would like to map this to the corresponding git hash, you can use GitHub search. As SQLite‘s git repository’s commits include the fossil hash, you can search for the fossil hash, using the following query with the fossil commit hash appended (example search): https://github.com/sqlite/sqlite/search?type=commits&q=
If the cherry-picking script is unable to cherry-pick a commit, like in https://crbug.com/1162100, manually apply the change from a SQLite or git, in //third_party/sqlite/src's files modified in the SQLite tracker, like at https://sqlite.org/src/info/a0bf931bd712037e. From there, run ../scripts/generate_amalgamation.py
to propagate these changes over to the amalgamation files. sqlite_cherry_picker.py should generally be preferred, as it updates hashes and simplifies tracking.
Run local tests.
Follow steps in Running Tests below to execute all verifications and tests.
Upload cherry-picked change (with amalgamations) for review.
If the relevant bug is a security bug, make sure that the reviewers are cc‘ed. Otherwise, they may not know what/why they’re reviewing.
```sh git cl upload ```
Update the Chromium DEPS file.
Once review above has merged, roll the chromium/src/DEPS
file to reference that new commit hash.
roll-dep src/third_party/sqlite/src --roll-to <git hash of merged CL>
Build all desktop targets:
Check that extract_sqlite_api.py
added “chrome_” to all exported symbols. Only “_fini” and “_init” should be unprefixed, but are conditionally exported by the linker and may be omitted.
autoninja -C out/Default nm -B out/Default/libchromium_sqlite3.so | cut -c 18- | sort | grep '^T'
out/Default/sql_unittests
third_party/blink/tools/run_web_tests.py -t Default storage/websql/
This is one of the SQLite test suites. They take approximately 3 minutes to build and run on a fast workstation.
Note: Tests currently fail both locally and on Chromium release branches. They fail on release branches because some tests rely on SQLite databases (binary files) which are committed to the source and are likely not merged down when cherry picked. It is safe to ignore these errors which should be reasonably easy to identify based on the cherry picked upstream changes. Until these tests are fixed, it is safe to ignore these tests when running SQLite test suites.
cd third_party/sqlite # from //chromium/src ./scripts/generate_amalgamation.py --testing make --directory=src test | tee /tmp/test.log
Show tests with errors:
egrep 'errors out of' /tmp/test.log
Show broken tests:
egrep 'Failures on these tests:' /tmp/test.log
Broken tests will also show lines ending in “...” instead of “... Ok”.
When done, clean up the SQLite repository:
cd src make clean git clean -i # and delete everything rm -rf testdir git checkout amalgamation/sqlite3.c git checkout amalgamation_dev/sqlite3.c