| Name: sqlite |
| URL: https://sqlite.org/ |
| Version: 3.24.0 |
| Included In Release: Yes |
| Security Critical: Yes |
| License: Public domain |
| |
| 1) Managing differences between SQLite core and Chromium's version. |
| 2) Making changes to Chromium SQLite. |
| 3) Import new release of SQLite. |
| 4) Running SQLite's test suite within the Chromium checkout. |
| |
| --- |
| |
| 1) Managing differences between SQLite core and Chromium's version. |
| |
| Chromium maintains some differences WRT SQLite, for reasons beyond this |
| document's remit. Some differences are bugs we have found and fixed (and |
| hopefully upstreamed), some are fixes we've backported from a later version of |
| SQLite, and some our local changes unlikely to ever be upstreamed. New versions |
| of SQLite are imported every year or two, at which point the changes need to be |
| reviewed for continued applicability, and sometimes adjusted to reflect upstream |
| code changes. |
| |
| To this end, the repository contains a reference copy of the SQLite source code |
| as of the last import, plus a series of patches which can be applied to |
| re-create the current trunk code. These patches are generated and processed by |
| git, with the intention of re-creating a commit series so that importers can use |
| their regular revision-control knowledge to manage import merges. |
| |
| The directory structure is as follows. Files common to all third_party projects |
| (BUILD.GN, OWNERS, LICENSE) are omitted. |
| |
| * sqlite-src-*/ - Upstream source code, without any modifications. The number |
| after src- is a release version or a snapshot number (which is |
| a commit time). See https://www.sqlite.org/cgi/src/timeline |
| * patches/ - Our patches to the currently used release, formatted by git using |
| the UNIX mailbox format. The patches can be applied with git am, |
| and created with git format-patch. |
| * src/ - The currently used source code, with our patches applied. |
| * amalgamation/ - The supported method of using SQLite is via an amalgamation |
| build, which merges all the code in one .c file and one .h |
| file. See https://www.sqlite.org/amalgamation.html |
| * amalgamation/config.h - Linux build configuration |
| * scripts/ - Scripts that generate the files in the amalgamation |
| * sqlite.h - The header used by the rest of Chromium to include SQLite. This |
| forwards to amalgamation/sqlite3.h |
| * fuzz/ - Google OSS-Fuzz (ClusterFuzz) testing for Chromium's SQLite build |
| |
| --- |
| |
| 2) Making changes to Chromium SQLite. |
| |
| third_party/sqlite/src is the patched source from SQLite. This is used to |
| generate the amalgamation, a concatenation of all of the files into a giant |
| sqlite3.c. To prototype, edit in src/, then call |
| ./scripts/generate_amalgamation.sh |
| to regenerate sqlite3.c. The code in src/ is much easier to edit, and the |
| SQLite test framework can easily be run. During development it may be |
| convenient to modify BUILD.gn based on src/main.mk to just pull in the src/ |
| files rather than sqlite3.c. |
| |
| Once your patch is complete, squash it down into a reasonable CL, then |
| re-generate the patches. This is a truncated version of the import flow. The |
| following is written like a shell script to allow copy/paste to a shell, ignore |
| comments and change the obvious lines. These instructions should work on Linux |
| or OSX. They may assume a modern version of git (I'm using 2.2.1). |
| |
| # Everything based in sqlite subdir. |
| cd third_party/sqlite |
| |
| BASE=3080704 |
| |
| #### Create a reference branch. |
| git checkout -b sqlite_${BASE} master |
| git rm -rf src |
| rm -rf src # In case there are things git doesn't know in there. |
| cp -a sqlite-src-${BASE} src |
| # -f includes ignored files, of which there are a couple. |
| git add -f src/ |
| git commit -m "Reset to sqlite-src-${BASE}" |
| # This branch is unlikely to build. |
| |
| #### Create a reference branch with patches applied. |
| git checkout -b sqlite_${BASE}_patched master |
| git rebase sqlite_${BASE} |
| git am --keep-non-patch patches/*.patch |
| git diff master |
| # This branch should be identical to master, unless someone forgot to export |
| # their changes into a patch. If so, do that as a separate CL and start over. |
| |
| #### Cherry-pick your change. |
| git cherry-pick <your change> |
| # This branch should be identical to your development branch, except |
| # amalgamation. |
| |
| # Rebuild the patch set. |
| git rm patches/* |
| git format-patch --output-directory=patches sqlite_${BASE}..HEAD |
| git add patches/*.patch |
| git commit -m "Rebuild patches for sqlite_${BASE}" |
| |
| # Re-generate the amalgamation. |
| ./scripts/generate_amalgamation.sh |
| git commit -m './scripts/generate_amalgamation.sh' amalgamation/ |
| # At this point everything should build and work. |
| |
| # Do a squash upload. This should add your single patch to patches/, and apply |
| # the changes your patch represents to src/ and amalgamation/. Other patches |
| # will have hash changes. A sensible check-in comment would be something like |
| # the patch's checkin comment, plus "regenerate amalgamation and generate patch |
| # file." |
| # TODO(pwnall): Should hash changes be checked in, or backed out? |
| |
| # Find unsuspecting victim and send review. |
| |
| --- |
| |
| 3) Import a new SQLite release. |
| |
| Importing a new SQLite involves merging our local changes with SQLite's changes. |
| Like any other merge, this may involve modifying some commits and dropping |
| others. The basic idea below is to generate git branches to work with: |
| |
| * sqlite-new-upstream - new release code archived in a separate directory |
| * sqlite-old-base - current release without patches |
| * sqlite-old - current release with patches mapped to git commits |
| * sqlite-new-base - new release without patches |
| * sqlite-new - new release with patches mapped to git commits |
| * sqlite-new-cl - new release in one git commit, for git cl upload |
| |
| We will upload sqlite-new-upstream as a massive (800k LOC+) CL that cannot |
| possibly be reviewed, but is generated in an automated fashion. We will then |
| squash sqlite-new to sqlite-new-upstream and obtain one CL that only contains |
| diffs. The second CL is still large, but it's a fraction of the first |
| (automated) CL. |
| |
| # The steps below are easier if done in the SQLite directory. |
| cd third_party/sqlite |
| |
| export OLD=3240000 |
| export NEW=3250000 |
| export GNU_SED=sed # OSX: "brew install gnu-sed", then use "gsed" here. |
| |
| #### Download and unpack the new SQLite release. |
| git new-branch sqlite-new-upstream |
| # URL from "Alternative Source Code Formats" at https://sqlite.org/download.html |
| curl https://sqlite.org/2018/sqlite-src-${NEW}.zip > upstream.zip |
| mkdir sqlite-src-$NEW |
| unzip ./upstream.zip -d sqlite-src-$NEW |
| rm ./upstream.zip |
| mv sqlite-src-$NEW/sqlite-*/* sqlite-src-$NEW/ |
| rmdir sqlite-src-$NEW/sqlite-*/ |
| xdg-open sqlite-src-$NEW # Make sure everything looks right. |
| |
| #### Add the new release code in a separate CL, for code review sanity. |
| git add sqlite-src-$NEW # Committing the code as downloaded, on purpose. |
| git clean -i -d -x sqlite-src-$NEW # Make sure no file is git-ignored. |
| git commit -m "sqlite: Add code for release 3.25.0" |
| git cl upload # Have the new code in a separate (impossible to review) CL. |
| |
| #### Create a branch for the old SQLite release's upstream version. |
| git new-branch --upstream-current sqlite-old-base |
| git rm -rf src |
| cp -r sqlite-src-${OLD}/ src |
| # Clean up trailing whitespace and CRLF so any patches look clean. |
| find src/ -type f -not -iname "*.db" -not -iname "*.eps" -not -iname "*.ico" \ |
| -not -iname "*.jpg" -not -iname "*.pfx" -not -iname "*.png" \ |
| -not -iname "*.tiff" -not -iname "*.vsix" \ |
| -exec $GNU_SED --in-place 's/[[:space:]]\+$//' {} \+ |
| git add src/ |
| git clean -i -d -x sqlite-src-$NEW # Make sure no file is git-ignored. |
| git commit -m "Squash: Reset SQLite src/ to sqlite-src-${OLD}." |
| # This branch will not build. It will be used for rebasing, then deleted. |
| |
| #### Create a branch for our old SQLite code, with patches mapped to commits. |
| git new-branch --upstream-current sqlite-old |
| git am --keep-non-patch --ignore-space-change patches/*.patch |
| git diff --ignore-space-change origin/master src/ |
| # This branch should be identical to master. |
| |
| #### Create a branch for the new SQLite release's upstream version. |
| git checkout sqlite-old-base |
| git new-branch --upstream-current sqlite-new-base |
| git rm -rf src |
| cp -r sqlite-src-${NEW}/ src |
| # Clean up trailing whitespace and CRLF so any patches look clean. |
| find src/ -type f -not -iname "*.db" -not -iname "*.eps" -not -iname "*.ico" \ |
| -not -iname "*.jpg" -not -iname "*.pfx" -not -iname "*.png" \ |
| -not -iname "*.tiff" -not -iname "*.vsix" \ |
| -exec $GNU_SED --in-place 's/[[:space:]]\+$//' {} \+ |
| git add src/ |
| git clean -i -d -x sqlite-src-$NEW # Make sure no file is git-ignored. |
| git commit -m "Squash: Reset SQLite src/ to sqlite-src-${NEW}." |
| # This branch will not build. It will be used for rebasing, then deleted. |
| |
| #### Create a branch for updating our patches. |
| git checkout sqlite-old |
| git new-branch --upstream-current sqlite-new |
| # Rebase our patches, which are mapped to separate commits, onto the new |
| # release. There will be merge conflicts that must be fixed. This is the |
| # interesting part of the work. |
| git rebase sqlite-new-base |
| |
| #### Finally, create the branch that we'll upload. |
| git new-branch --upstream-current sqlite-new-cl |
| ./scripts/generate_amalgamation.sh |
| git cl format amalgamation/rename_exports.h |
| |
| #### Validate the upgrade. |
| # The goal is to have a set of reasonably-independent CLs which can be |
| # understood separately, so that future importers can sensibly determine how to |
| # handle conflicts. So use git-rebase and slipstream fixups back into their |
| # original CL until everything builds and works. |
| cd ../.. |
| ninja -C out/Default |
| # Check that extract_sqlite_api.py added chrome_ to all exported symbols. |
| # Only "_fini" and "_init" should be unprefixed. |
| nm -B out/Default/libchromium_sqlite3.so | cut -c 18- | sort | grep '^T' |
| out/Default/sql_unittests |
| third_party/blink/tools/run_web_tests.py -t Default storage/websql/* |
| cd third_party/sqlite |
| |
| #### Create the review. |
| # Rebuild the patch set. |
| git rm patches/* |
| git format-patch --output-directory=patches --ignore-space-change \ |
| sqlite-new-base..sqlite-new |
| git add amalgamation/ |
| git add patches/*.patch |
| git commit -m "Squash: regenerate amalgamation and patches." |
| git branch --set-upstream-to=sqlite-new-upstream |
| git cl upload --squash |
| # Edit the commit message appropriately to reflect anything from |
| # <https://www.sqlite.org/changes.html> which might be deemed important. Don't |
| # enumerate all of the patch messages, those are assumed, but do reference any |
| # material changes made. |
| # TODO(pwnall) Describe an appropriate comment style. Seems like it should at |
| # least include the SQLite version number. Meanwhile, look in the logs for past |
| # commits to model things on. |
| |
| #### Drop the old version of SQLite. |
| git new-branch sqlite-rm-old |
| git rm -r sqlite_${BASE} |
| git commit -m "sqlite: Remove source code for old release ${OLD}." |
| |
| Note that things can be broken down differently, if you prefer. For instance, |
| adding the new version of the SQLite distro and removing the old one can be |
| distinct CLs. |
| |
| -------------------------------------------- |
| |
| 4) Running SQLite's test suite within the Chromium checkout. |
| |
| TODO(pwnall): This hasn't been tried out for at least a year. |
| |
| Prerequisites: The test suite requires tcl-dev and libicu-dev. Install those on |
| Ubuntu like: |
| sudo apt-get install tcl8.6-dev libicu-dev |
| On OSX, I use MacPorts: |
| sudo port install tcl |
| |
| cd third_party/sqlite/src |
| mkdir build |
| cd build |
| # This sometimes gives integer-size warnings on the differences between |
| # Tcl_WideInt and sqlite3_int64 and int64. Usually this is easily fixed by |
| # changing a variable to Tcl_WideInt. |
| make -j -f ../Makefile.linux-gcc testfixture sqlite3 sqlite3_analyzer sqldiff |
| make -f ../Makefile.linux-gcc test > /tmp/test.log |
| egrep 'errors out of' /tmp/test.log |
| # Show broken tests: |
| egrep 'Failures on these tests:' /tmp/test.log |
| # Broken tests will also show lines ending in "..." instead of "... Ok". |
| |
| In version 3.10.2 on OSX 10.11.2, I see: |
| 6 errors out of 139819 tests |
| The failed tests are: |
| pager4-1.3 pager4-1.4 pager4-1.5 pager4-1.9 pager4-1.10 pager4-1.11 |
| This is due to the change in os_unix.c fileHasMoved() to support WebDatabase. |
| Commenting out the early return allows them to succeed. |
| |
| In version 3.10.2 on Ubuntu 14.04.3 I see: |
| 9 errors out of 140309 tests |
| The failed tests are: |
| oserror-1.1.1 oserror-1.1.2 oserror-1.1.3 |
| pager4-1.3 pager4-1.4 pager4-1.5 pager4-1.9 pager4-1.10 pager4-1.11 |
| The oserror tests fail because there are too many fds available, and can be |
| fixed by running "ulimit -n 1024" before the test. The pager4 tests are failing |
| for the same reason as for OSX. |
| |
| -- |
| |
| NOTE(pwnall): On Ubuntu it is possible to run the tests in a tmpfs something |
| like: |
| |
| TMPFS=/dev/shm/sqlite_build |
| BUILD=$PWD |
| mkdir $TMPFS |
| (cd $TMPFS ; $BUILD/testfixture $BUILD/../test/veryquick.test >/tmp/test.log) |
| |
| This is faster, but it is plausible that different things are being tested than |
| real-world use. |