blob: af2cd26fc9542aeaf2d7b76111edd364509a8925 [file] [log] [blame]
Name: sqlite
URL: https://sqlite.org/
Version: 3.27.1
Included In Release: Yes
Security Critical: Yes
License: Public domain
1) Managing differences between SQLite core and Chromium's version.
2) Making changes to Chromium SQLite.
3) Import new release of SQLite.
4) Running SQLite's test suite within the Chromium checkout.
---
1) Managing differences between SQLite core and Chromium's version.
Chromium maintains some differences WRT SQLite, for reasons beyond this
document's remit. Some differences are bugs we have found and fixed (and
hopefully upstreamed), some are fixes we've backported from a later version of
SQLite, and some our local changes unlikely to ever be upstreamed. New versions
of SQLite are imported every year or two, at which point the changes need to be
reviewed for continued applicability, and sometimes adjusted to reflect upstream
code changes.
To this end, the repository contains a reference copy of the SQLite source code
as of the last import, plus a series of patches which can be applied to
re-create the current trunk code. These patches are generated and processed by
git, with the intention of re-creating a commit series so that importers can use
their regular revision-control knowledge to manage import merges.
The directory structure is as follows. Files common to all third_party projects
(BUILD.GN, OWNERS, LICENSE) are omitted.
* sqlite-src-*/ - Upstream source code, without any modifications. The number
after src- is a release version or a snapshot number (which is
a commit time). See https://www.sqlite.org/cgi/src/timeline
* patches/ - Our patches to the currently used release, formatted by git using
the UNIX mailbox format. The patches can be applied with git am,
and created with git format-patch.
* src/ - The currently used source code, with our patches applied.
* amalgamation/ - The supported method of using SQLite is via an amalgamation
build, which merges all the code in one .c file and one .h
file. See https://www.sqlite.org/amalgamation.html
* amalgamation/config.h - Linux build configuration
* scripts/ - Scripts that generate the files in the amalgamation
* sqlite.h - The header used by the rest of Chromium to include SQLite. This
forwards to amalgamation/sqlite3.h
* fuzz/ - Google OSS-Fuzz (ClusterFuzz) testing for Chromium's SQLite build
---
2) Making changes to Chromium SQLite.
third_party/sqlite/src is the patched source from SQLite. This is used to
generate the amalgamation, a concatenation of all of the files into a giant
sqlite3.c. To prototype, edit in src/, then call
./scripts/generate_amalgamation.sh
to regenerate sqlite3.c. The code in src/ is much easier to edit, and the
SQLite test framework can easily be run. During development it may be
convenient to modify BUILD.gn based on src/main.mk to just pull in the src/
files rather than sqlite3.c.
Once your patch is complete, squash it down into a reasonable CL, then
re-generate the patches. This is a truncated version of the import flow. The
following is written like a shell script to allow copy/paste to a shell, ignore
comments and change the obvious lines. These instructions should work on Linux
or OSX. They may assume a modern version of git (I'm using 2.2.1).
# The steps below are easier if done in the SQLite directory.
cd third_party/sqlite
# Must match the version in //third_party/sqlite/sqlite-src-xxxxxxx.
# This is SQLite's version number, and uses upstream's convention.
export BASE=3250300
export GNU_SED=sed # OSX: "brew install gnu-sed", then use "gsed" here.
#### Create a reference branch.
git new-branch sqlite-base
git rm -rf src
cp -r sqlite-src-${BASE}/ src
# Clean up trailing whitespace and CRLF so any patches look clean.
find src/ -type f -not -iname "*.db" -not -iname "*.eps" -not -iname "*.ico" \
-not -iname "*.jpg" -not -iname "*.pfx" -not -iname "*.png" \
-not -iname "*.tiff" -not -iname "*.vsix" \
-exec ${GNU_SED} --in-place 's/[[:space:]]\+$//' {} \+
git add src/
git clean -i -d -x src # Make sure no file is git-ignored.
git commit -m "Squash: Reset SQLite src/ to sqlite-src-${BASE}."
# This branch will not build. It will be used for rebasing, then deleted.
#### Create a reference branch with patches applied.
git new-branch --upstream-current sqlite-dev
git am --keep-non-patch --ignore-space-change patches/*.patch
git diff origin/master src/
# This branch should be identical to master.
#### Develop and validate the change, or cherry-pick it from a dev branch.
# The goal is to have a set of reasonably-independent CLs which can be
# understood separately, so that future importers can sensibly determine how to
# handle conflicts. So use git-rebase and slipstream fixups back into existing
# patches, or add a new patch.
./scripts/generate_amalgamation.sh
git cl format amalgamation/rename_exports.h
cd ../..
ninja -C out/Default
# Check that extract_sqlite_api.py added chrome_ to all exported symbols.
# Only "_fini" and "_init" should be unprefixed.
nm -B out/Default/libchromium_sqlite3.so | cut -c 18- | sort | grep '^T'
out/Default/sql_unittests
third_party/blink/tools/run_web_tests.py -t Default storage/websql/
cd third_party/sqlite
#### Create the review.
# Rebuild the patch set.
git rm patches/*
git format-patch --output-directory=patches --zero-commit \
sqlite-base..sqlite-dev
git add amalgamation/
git add patches/
git commit -m "Squash: regenerate amalgamation and patches."
git branch --set-upstream-to=origin/master
git cl upload --squash
---
3) Import a new SQLite release.
Importing a new SQLite involves merging our local changes with SQLite's changes.
Like any other merge, this may involve modifying some commits and dropping
others. The basic idea below is to generate git branches to work with:
* sqlite-new-upstream - new release code archived in a separate directory
* sqlite-old-base - current release without patches
* sqlite-old - current release with patches mapped to git commits
* sqlite-new-base - new release without patches
* sqlite-new - new release with patches mapped to git commits
* sqlite-new-cl - new release in one git commit, for git cl upload
We will upload sqlite-new-upstream as a massive (800k LOC+) CL that cannot
possibly be reviewed, but is generated in an automated fashion. We will then
squash sqlite-new to sqlite-new-upstream and obtain one CL that only contains
diffs. The second CL is still large, but it's a fraction of the first
(automated) CL.
# The steps below are easier if done in the SQLite directory.
cd third_party/sqlite
# The numbers below are SQLite version numbers, and use upstream's convention
# for tagging release binaries and source zipballs.
export OLD=3270100
export NEW=3280000
export GNU_SED=sed # OSX: "brew install gnu-sed", then use "gsed" here.
#### Download and unpack the new SQLite release.
git new-branch sqlite-new-upstream
# URL from "Alternative Source Code Formats" at https://sqlite.org/download.html
curl https://sqlite.org/2019/sqlite-src-${NEW}.zip > upstream.zip
mkdir sqlite-src-${NEW}
unzip ./upstream.zip -d sqlite-src-${NEW}
rm ./upstream.zip
mv sqlite-src-${NEW}/sqlite-*/* sqlite-src-${NEW}/
rm -r sqlite-src-${NEW}/sqlite-*/.fossil-settings
rmdir sqlite-src-${NEW}/sqlite-*/
rm -r sqlite-src-${NEW}/compat
xdg-open sqlite-src-${NEW} # Make sure everything looks right.
#### Add the new release code in a separate CL, for code review sanity.
git add sqlite-src-${NEW} # Committing the code as downloaded, on purpose.
git clean -i -d -x sqlite-src-${NEW} # Make sure no file is git-ignored.
git commit -m "sqlite: Add code for release ${NEW}"
git cl upload # Have the new code in a separate (impossible to review) CL.
#### Create a branch for the old SQLite release's upstream version.
git new-branch sqlite-old-base
git rm -rf src
cp -r sqlite-src-${OLD}/ src
# Clean up trailing whitespace and CRLF so any patches look clean.
find src/ -type f -not -iname "*.db" -not -iname "*.eps" -not -iname "*.ico" \
-not -iname "*.jpg" -not -iname "*.pfx" -not -iname "*.png" \
-not -iname "*.tiff" -not -iname "*.vsix" \
-exec ${GNU_SED} --in-place 's/[[:space:]]\+$//' {} \+
git add src/
git clean -i -d -x src # Make sure no file is git-ignored.
git commit -m "Squash: Reset SQLite src/ to sqlite-src-${OLD}."
# This branch will not build. It will be used for rebasing, then deleted.
#### Create a branch for our old SQLite code, with patches mapped to commits.
git new-branch --upstream-current sqlite-old
git am --keep-non-patch --ignore-space-change patches/*.patch
git diff origin/master src/
# This branch should be identical to master.
#### Create a branch for the new SQLite release's upstream version.
git checkout sqlite-old-base
git new-branch --upstream-current sqlite-new-base
git rm -rf src
git checkout sqlite-new-upstream -- sqlite-src-${NEW}/
git mv sqlite-src-${NEW}/ src
# Clean up trailing whitespace and CRLF so any patches look clean.
find src/ -type f -not -iname "*.db" -not -iname "*.eps" -not -iname "*.ico" \
-not -iname "*.jpg" -not -iname "*.pfx" -not -iname "*.png" \
-not -iname "*.tiff" -not -iname "*.vsix" \
-exec ${GNU_SED} --in-place 's/[[:space:]]\+$//' {} \+
git add src/
git clean -i -d -x src # Make sure no file is git-ignored.
git commit -m "Squash: Reset SQLite src/ to sqlite-src-${NEW}."
# This branch will not build. It will be used for rebasing, then deleted.
#### Create a branch for updating our patches.
git checkout sqlite-old
git new-branch --upstream-current sqlite-new
# Rebase our patches, which are mapped to separate commits, onto the new
# release. There will be merge conflicts that must be fixed. This is the
# interesting part of the work.
git rebase sqlite-new-base
#### Finally, create the branch that we'll upload.
git new-branch --upstream-current sqlite-new-cl
./scripts/generate_amalgamation.sh
git cl format amalgamation/rename_exports.h
#### Copy any new entries to the seed-corpus for dbfuzz2.
# We use a set of seed databases for fuzzing SQLite's resilience to database
# corruption. Sometimes, new seed databases are added upstream.
# Find any files of the pattern src/test/dbfuzz2-seed*.db, and copy them to
# the fuzz/db_corpus directory.
cp --no-clobber src/test/dbfuzz2-seed*.db fuzz/db_corpus
#### Validate the upgrade.
# The goal is to have a set of reasonably-independent CLs which can be
# understood separately, so that future importers can sensibly determine how to
# handle conflicts. So use git-rebase and slipstream fixups back into their
# original CL until everything builds and works.
cd ../..
autoninja -C out/Default
# Check that extract_sqlite_api.py added chrome_ to all exported symbols.
# Only "_fini" and "_init" should be unprefixed.
nm -B out/Default/libchromium_sqlite3.so | cut -c 18- | sort | grep '^T'
out/Default/sql_unittests
third_party/blink/tools/run_web_tests.py -t Default storage/websql/
cd third_party/sqlite
#### Create the review.
# Rebuild the patch set.
git rm patches/*
git format-patch --output-directory=patches --zero-commit \
sqlite-new-base..sqlite-new
git add amalgamation/
git add patches/
git commit -m "Squash: regenerate amalgamation and patches."
git branch --set-upstream-to=origin/master
git cl upload --squash
# Edit the commit message appropriately to reflect anything from
# <https://www.sqlite.org/changes.html> which might be deemed important. Don't
# enumerate all of the patch messages, those are assumed, but do reference any
# material changes made.
# TODO(pwnall) Describe an appropriate comment style. Seems like it should at
# least include the SQLite version number. Meanwhile, look in the logs for past
# commits to model things on.
#### Drop the old version of SQLite.
git new-branch sqlite-rm-old
git rm -r sqlite-src-${OLD}
git commit -m "sqlite: Remove source code for old release ${OLD}."
git cl upload
Note that things can be broken down differently, if you prefer. For instance,
adding the new version of the SQLite distro and removing the old one can be
distinct CLs.
--------------------------------------------
4) Running SQLite's test suite within the Chromium checkout.
TODO(pwnall): This hasn't been tried out for at least a year.
Prerequisites: The test suite requires tcl-dev and libicu-dev. Install those on
Ubuntu like:
sudo apt-get install tcl8.6-dev libicu-dev
On macOS, I use Homebrew:
brew install icu4c tcl-tk
export PATH="$(brew --prefix icu4c)/bin:$(brew --prefix tcl-tk)/bin:$PATH"
Run the commands in scripts/generate_amalgamation.sh, but replace the "make"
command with "make test".
make test > /tmp/test.log
egrep 'errors out of' /tmp/test.log
# Show broken tests:
egrep 'Failures on these tests:' /tmp/test.log
# Broken tests will also show lines ending in "..." instead of "... Ok".
In version 3.10.2 on OSX 10.11.2, I see:
6 errors out of 139819 tests
The failed tests are:
pager4-1.3 pager4-1.4 pager4-1.5 pager4-1.9 pager4-1.10 pager4-1.11
This is due to the change in os_unix.c fileHasMoved() to support WebDatabase.
Commenting out the early return allows them to succeed.
In version 3.10.2 on Ubuntu 14.04.3 I see:
9 errors out of 140309 tests
The failed tests are:
oserror-1.1.1 oserror-1.1.2 oserror-1.1.3
pager4-1.3 pager4-1.4 pager4-1.5 pager4-1.9 pager4-1.10 pager4-1.11
The oserror tests fail because there are too many fds available, and can be
fixed by running "ulimit -n 1024" before the test. The pager4 tests are failing
for the same reason as for OSX.
--
NOTE(pwnall): On Ubuntu it is possible to run the tests in a tmpfs something
like:
TMPFS=/dev/shm/sqlite_build
BUILD=$PWD
mkdir $TMPFS
(cd $TMPFS ; $BUILD/testfixture $BUILD/../test/veryquick.test >/tmp/test.log)
This is faster, but it is plausible that different things are being tested than
real-world use.