chrome/common/extensions/docs/server2/README

TODO(devlin): MD docs seem to be the new hotness; this should probably be
converted.

--------
Overview

This is a Google App Engine server which serves the documentation for Chrome
apps and extensions. At time of this writing, the primary URL is:
http://developer.chrome.com/.


---------------------
Developing the Server

You shouldn't need app engine locally to develop the server, preview.py should
be sufficient. If for some reason you want to test against the app engine SDK:

  1. Download the python Google App Engine SDK from:
     https://developers.google.com/appengine/downloads

  2. Run './start_dev_server.py <path/to/dev_appserver.py>'
     (dev_appserver.py is part of the App Engine)

  3. View docs at http://localhost:8080/(apps|extensions)/<doc_name>

NOTE: The dev server will not work right way: you need to populate its
Datastore. You will need a local datastore cache in the working directory where
you started the dev server, and you will need to manually instruct the dev
server to pull data from that file:

  1. Run './update_cache.py --no-push --save-file=FOOCACHE'. This will take
     a very long time (30-40 minutes). It is advisable that you keep a copy
     of this file around if you plan use the dev server often. It can be
     updated much faster (< 3 minutes) in that case by also including
     --load-file=FOOCACHE on subsequent update_cache.py runs.

     You MUST have branch heads fetched in your local repo in order for your
     local data set to be populated correctly. You can accomplish this by
     running:

       gclient sync --with_branch_heads
       git fetch origin

     You may also specify --commit=<commitish> when running update_cache.py in
     order to update the cache from a specific commit. This may be a commit ID,
     or a partial commit ID, or a local branch ref, etc. To test local changes,
     you MUST commit them locally and use the local commit to update your cache.

  2. Once you have a cache (e.g. FOOCACHE) in the working directory of your
     dev server, visit the URL:

       http://localhost:8080/_update_cache/FOOCACHE

     If you see errors about the FOOCACHE file not existing, the server may be
     looking for the file in the root chromium directory (under src/), even if
     you launched the server from the server2 directory. Try moving the FOOCACHE
     file there.

     The server should take about a minute to fully populate its Datastore
     from the data in your FOOCACHE file. Now you have a working dev server!


------------------------------------------------------------
Using Google Cloud Storage content providers with preview.py

  1. create a directory "[...]/server2/local_debug/gcs/<bucketname>" for every
     gcs bucket referenced in content_providers.json

  2. copy files to the respective local bucket directories. Preview.py has
     no access to the real Google Cloud Storage.


--------------------
Deploying the Server

You will need to have access to the http://chrome-apps-doc.appspot.com app.
Contact rdevlin.cronin@chromium.org or another extensions team member to obtain
access.

Once you have access:

1. Increment the version in app.yaml so we can roll back if the update breaks.

2. Land your patch in chromium. Resync/pull.

3. Run build_server.py. This copies some depenencies from /third_party into the
   server directory so that they get uploaded to App Engine.

4. Run appcfg.py (supplied with the App Engine SDK) to upload the server code:

    appcfg.py update .

5. When prompted for your credentials, enter the information for the account
   that has access to the production app.

Note: Steps 1 - 5 can be completed (mostly) synchronously. However, once the
version is pushed, the docserver will need to repopulate its cache, which runs
from a cron job. (Theoretically, this job runs a few times a day - in practice,
it has been observered to be late.)

6. Go to http://www.appspot.com, select the docs project, click "versions" in
   the sidebar. CHECK THE VERSION YOU DEPLOYED WORKS. You can do this by
   clicking the version with the "popout" link, which should send you to a link
   like "https://3-50-3-dot-chrome-apps-doc.appspot.com". This should work, and
   you should be able to see documentation by going to /extensions,
   /extensions/webRequest, /extensions/contentScripts, etc. If you see a
   "Not found" error, this means the version does not work. Likely, this is
   because the cache hasn't been fully updated.

   If the newly-deployed version works, migrate traffic to the new version. This
   happens nearly instantly, so you should immediately double-check the live
   site at developer.chrome.com/extensions.

   If you get an error about too many versions when deploying, go into this
   view and delete the version which was deployed the longest time ago.  Then
   try to deploy again.


----------------------
Updating the Datastore

Even when the server code hasn't changed, new data is constantly flowing into
the chromium repo and some of that data includes changes to new or existing
content hosted by the server. In order for the front-end to reflect these
changes, new data must be pushed into the project's Datastore.

This is done periodically by a Compute Engine instance running the
update_cache.py tool. While it is possible to push from other hosts if you
get all the right credentials in all the right places, it is strongly
discouraged and also therefore not documented.

To force a push safely from the VM, navigate to the developer console under
Compute -> Compute Engine -> VM instances and open SSH for the "git-processor"
instance. From within the SSH session, first switch to user "git-processor":

  sudo su - git-processor

You can run './update-docs.sh' from the home directory there. This will
automatically fetch any new objects from the upstream repository and then
perform a full update if there are pending changes.

It's almost never necessary to use this tool. For one interesting example,
consider the case where no new commits are landing in the chrome repository
but a change was just pushed to one of the GCS providers (like say, the
chromedocs-multidevice bucker). In this case, you may want to force an update
of only the content_providers data.

  ./update_docs.sh --data-source=content_providers --force

This will safely do the push for you, ensuring that the automated job
does not collide with your own.

----------------------
Server architecture

The server is composed of two parts:

1) A compute engine VM instance which runs a cron job to periodicially pull the
   chromium repo and uses the update_cache.py script to push the new data to the
   cloud datastore and clear the existing memcache for the App engine instance
   to ensure stale data is not served. All the keys in the cloud datastore are
   indexed by the app version specified in app.yaml.

2) A python app engine app which reads the data in the cloud datastore and
   serves the website.

Hence, even if we were to stop the compute engine VM instance, the website
should still keep running. But it won't update, since the cloud datastore won't
get updated.

-----------------------------------------------
Updating Docs served from Google Cloud Storage.

Some docs (e.g. webstore docs) are served using Google Cloud Storage buckets. To
update files in these docs, follow these steps:

1) Go the project homepage and then to the Storage section.
2) Go to the relevant storage bucket. For example, to update webstore docs, go
   to the page for "chromedocs-webstore".
3) Modify an existing file or upload a new file. Edit file permissions if
   necessary to ensure these are accessible to Public. (Ideally these should be
   only accessible to the cron job running on the compute engine VM instance.)
4) After the next cron job on the compute engine completes, the modifications
   should be live.
5) In order to expedite 4), a manual datastore update can be performed by
   running "./update_docs.sh --data-source=content_providers --force" on the VM
   instance.

Note: these steps shouldn't require a server version bump. If they do, please
file a bug.

---------------------------------------
VM: Some notes and troubleshooting tips

Q1: How is the datastore populated in the VM?
A:  /home/git-processor/update-docs-full.sh
    This script is run every 2 hours through cron to sync the chromium git repo
    in this machine. Once it has done so, it will populate docserver data store
    and write that in /home/git-processor/.docserver_cache. Docserver uses this
    file as data sources to populate its documentation.

Q2: Is chromium repo in the VM in sync?
A:  Check the git logs on the repo.

    $ sudo su git-processor
    $ cd ~/chrome/src
    $ git log

    Verify that chromium checkout is up-to-date by looking at the latest git log
    results.

Q3: Did the last run to populate .docserver_cache succeed?
A:  Check the logs.

    Logs can be checked from the Cloud dashboard. Go to the project homepage and
    select Logging > Logs. Choose the VM instance and "syslog" from the filter
    options.

    Alternatively, run
    $ sudo -s # You need to be root.
    $ vi /var/log/user.log

Jan 19 21:51:39 git-processor logger: Acquiring docserver update lock...
Jan 19 21:51:39 git-processor logger: INFO:root:Starting refresh from commit 1ee51d53b10f8d2688a225d6cbd33dfdf8965403...
...
    Specifically you want to see some logs about writing keys to the
    cache file, e.g.:
Jan 19 21:33:21 git-processor logger: INFO:root:Saved 644 keys to /home/git-processor/.docserver_cache.

Q4: Docs stopped updating for a while, what to do?
A:  Start by checking whether the appengine instance contains datastore *close*
    to HEAD. You can do this by visiting

    https://developer.chrome.com/_query_commit/master

    Then check the commit hash. If the commit hash is behind by more than few
    days from HEAD then this means that last run(s) to populate .docserver_cache
    failed in the VM.  An example of this happening can be found in:
    https://crbug.com/682376.

    Start checking the logs (Q3). If you see that there are no logs after
    "Acquiring docserver update lock..", then the script might have hung. The
    script uses /var/lock/docserver-update.lock lock file with flock. The
    process/script probably is still running (I don't know why that happens
    yet). You'd need to verify that this is the case.
    For example, I ran:

    $ ps -e -o  lstart,pid,ppid,cmd | grep update-docs

Sat Oct  1 15:00:00 2016 10230 10229 /bin/sh -c /usr/bin/env bash /home/git-processor/update-docs-full.sh 2>&1 | /usr/bin>
Sat Oct  1 15:00:00 2016 10231 10230 bash /home/git-processor/update-docs-full.sh
Wed Dec 14 19:50:01 2016 10744 10743 /bin/sh -c /usr/bin/env bash /home/git-processor/update-docs-full.sh 2>&1 | /usr/bin>
Wed Dec 14 19:50:01 2016 10745 10744 bash /home/git-processor/update-docs-full.sh
    Once you're sure that one or more of these processes are running for a
    while, you'd want to kill them, which should be "safe" (endorsed by rockot@)
    in most of the cases. If that doesn't work, then delete the lock file. This
    will let subsequent cron runs to update docserver.

    Also ensure that the App engine's current version is in sync with the
    version specified in app.yaml on ToT.

    Some other miscellaneous debugging tips:
    - Viewing updates made by last cron job
      To view the contents of the data persisted by the last cron job, inspect
      the .docserver_cache file on the Compute Engine instance. See
      /home/git-processor/tools/dump_keys.py for an example of how to do so.

    - Viewing actual datastore entries
      Go to the project homepage and then move to the "Datastore" section. From
      the dropdowns, choose "Namespace" = [default], "Kind"=
      "PersistentObjectStoreItem" and click on "Filter Entities" and then enter
      the key to inspect. Example search:
      Key(PersistentObjectStoreItem, 'class=APIListDataSource&category=LocalGitFileSystem@master&app_version=3-65-0/api_data')
      The data here should be the same as the one in .docserver_cache.

    - Debugging live App Engine instance
      Go the Project homepage, and head over to the "Debug" section. Using
      Stackdriver debugging, it's possible to take live snapshots and inspect
      the state of the live App Engine instance. E.g. it allows you to inspect
      variables, see the stack trace etc. by setting a "logpoint".


Q5: How to migrate to a new VM instance? (Say to update the operating system).
    ROUGH steps:
    1) Stop the old VM.
    2) Backup the cloud datastore to a cloud storage bucket (optional).
    3) Clone the persistent disk (not the boot disk) from the old VM. Else just
       create a new disk and copy necessary data (in /home/git-processor) later.
    4) Create a new VM instance (Use configuration from the old VM) and attach
       the new persistent disk to the new VM. Make sure that new VM has the same
       API access scopes as the old one (for e.g. it can write to the cloud
       datastore).
    5) Mount the persistent disk at /home/git-processor (Modify /etc/fstab so
       that the disk is automatically mounted on restarts).
    6) Ensure the data in /home/git-processor is the same as the old VM (if you
       had not cloned the disk, use some other tool to copy the data).
    7) Increment the app engine version in the chromium repo and commit the
       change (so that we don't somehow affect the currently serving production
       version).
    8) On the VM, fetch chrome in /home/git-processor with all the branch heads.
    9) Ensure app.yaml in the chrome repo has the new app version.
    10) Install any necessary dependencies. For e.g. /home/git-processor/gcd is
        a python virtual environment used by update-docs-full.sh (which is run
        as part of the cron job). Hence pip, virtualenv need to be installed and
        it needs to be ensured that the virtualenv at /home/git-processor/gcd is
        correctly configured. Requirements file generated by pip can be used for
        this purpose.
    11) Run update-docs-full.sh and ensure it works fine. (After it runs, the
        new app engine version should work fine and have the latest changes).
    12) If it does, add a cronjob for the git-processor user to regularly run
        update-docs-full.sh.
    13) Ensure the cron job is running fine and serve all the traffic from the
        new app engine version.
    14) Delete the old VM, persistent disk after some time.
    15) (Optional) Install logging agent on the VM so that the syslog from the
        VM can be seen from the cloud dashboard.