tree: 5eae5a50c4fee1466b3e62562a86b946812854e2 [path history] [tgz]
  1. .coveragerc
  2. .expect_tests.cfg
  3. .expect_tests_pretest.py
  4. .gitignore
  5. Makefile
  6. OWNERS
  7. README.md
  8. api/
  9. app.yaml
  10. appengine_config.py
  11. benchmark/
  12. businesslogic/
  13. codereview.settings
  14. cron.yaml
  15. doc/
  16. dos.yaml
  17. features/
  18. framework/
  19. module-besearch.yaml
  20. monorailapp.py
  21. project/
  22. proto/
  23. queue.yaml
  24. registerpages.py
  25. schema/
  26. search/
  27. services/
  28. settings.py
  29. sitewide/
  30. static/
  31. templates/
  32. testing/
  33. third_party/
  34. tools/
  35. tracker/
appengine/monorail/README.md

Monorail Issue Tracker

Monorail is the Issue Tracker used by the Chromium project and other related projects. It is hosted at bugs.chromium.org.

If you wish to file a bug against Monorail itself, please do so in our self-hosting tracker. We also discuss development of Monorail at infra-dev@chromium.org.

Getting started with Monorail development

Here's how to run Monorail locally for development on any unix system (not under google3):

  1. You need to get the Chrome Infra depot_tools commands to check out the source code and all its related dependencies and to be able to send changes for review.
  2. Check out the Monorail source code
    1. cd /path/to/empty/workdir
    2. fetch infra
    3. cd infra/appengine/monorail
  3. Make sure you have the AppEngine SDK:
    1. It should be fetched for you by step 1 above (during runhooks)
    2. Otherwise, you can download it from https://developers.google.com/appengine/downloads#Google_App_Engine_SDK_for_Python
  4. Install MySQL v5.6. Either download directly, or if you're on an Ubuntu derivative:
    1. Do not download v5.7 (as of April 2016)
    2. sudo apt-get install mysql-server mysql-client
  5. Get the database backend running and use the command-line to create a database named “monorail”:
    1. sudo /usr/local/mysql/bin/mysqld_safe
    2. mysql --user=root --password=<pw>
    3. CREATE DATABASE monorail;
  6. Install Python MySQLdb. Either:
    1. sudo apt-get install python-mysqldb
    2. Or, download from http://sourceforge.net/projects/mysql-python/
      1. Follow instructions to install.
      2. If needed, add these lines to your ~/.profile file and restart on MacOS 10.8:
        1. setenv DYLD_LIBRARY_PATH /usr/local/mysql/lib/
        2. setenv VERSIONER_PYTHON_PREFER_64_BIT no
        3. setenv VERSIONER_PYTHON_PREFER_32_BIT yes
      3. In Mac OSX 10.11.1, if you see errors about failing to import MySQLdb or that _mysql.so references an untrusted relative path, then run: sudo install_name_tool -change libmysqlclient.18.dylib
        /usr/local/mysql/lib/libmysqlclient.18.dylib
        /Library/Python/2.7/site-packages/_mysql.so
  7. Set up one master SQL database. (You can keep the same sharding options in settings.py that you have configured for production.).
    1. mysql --user=root monorail < schema/framework.sql
    2. mysql --user=root monorail < schema/project.sql
    3. mysql --user=root monorail < schema/tracker.sql
  8. Configure the site defaults in settings.py. You can leave it as-is for now.
  9. Run the app:
    1. You can use ‘make serve’, or
    2. Run the app with AppEngine dev_appserver.py with the command: ../../../google_appengine/dev_appserver.py --mysql_user=root app.yaml module-besearch.yaml
  10. Browse the app at localhost:8080 your browser.
  11. Optional: Create/modify your Monorail User row in the database and make that user a site admin.
    1. UPDATE User SET is_site_admin = TRUE WHERE email = 'YOUR@EMAIL';
    2. Restart your local dev appserver.

Here's how to run unit tests from the command-line:

  1. cd infra
  2. ./test.py test appengine/monorail
  3. Or, in the monorail directory, use 'make test'

Here's how to deploy a new version to an instance that has already been set up:

  1. You need the master and replica databases set up on the Developer API console
  2. Make sure that the app ID is correct in app.yaml.
  3. In the monorail directory, run the command make deploy_staging
  4. Make any needed schema changes by looking at the end of sql/alter-table-log.txt. Be sure to connect to the staging master DB.
  5. On console.cloud.google.com, try out the new version using a version specific URL:
    1. Test some of the expected changes
    2. Add a comment to an issue
    3. Enter a new issue and CC your personal account
    4. Verify that you got an email (at the “all” email address specified in settings.py)
    5. Try doing a query that is not cached, then repeat it to test the cached case
  6. When everything looks good, make the new version the default on staging.
  7. If you updated the staging schema, disconnect from the staging master DB so that command prompt is not left open in a terminal window.
  8. Repeat the process on prod. Be sure to repeat the same schema changes on the prod database.
  9. If you updated the prod schema, disconnect from the prod master DB so that command prompt is not left open in a terminal window.

Here's an outline of what would be needed to set up a new Monorail instance:

  1. Create new GAE apps for production and staging.
  2. Configure GCP billing.
  3. Create new master DBs and 10 read replicas for prod and staging.
    1. Set up IP address and configure admin password and allowed IP addr. Instructions.
    2. Set up backups on master. The first backup must be created before you can configure replicas.
  4. Fork settings.py and configure every part of it, especially trusted domains and “all” email settings.
  5. You might want to also update /_constants.py files.
  6. Set up log saving to bigquery or something.
  7. Set up monitoring and alerts.
  8. Set up attachment storage in GCS.
  9. Set up spam data and train models.
  10. Fork and customize some of HTML in templates/framework/master-header.ezt, master-footer.ezt, and some CSS to give the instance a visually different appearance.
  11. Get From-address whitelisted so that the “View issue” link in Gmail/Inbox works.
  12. Set up a custom domain with SSL and get that configured into GAE. Make sure to have some kind of reminder system set up so that you know before cert expire.
  13. Configure the API. Details? Allowed clients are now configured through luci-config, so that is a whole other thing to set up. (Or, maybe decide not to offer any API access.)
  14. Gain permission to sync GGG user groups. Set up borgcron job to sync user groups. Configure that job to hit the API for your instance. (Or, maybe decide not to sync any user groups.)
  15. Monorail does not not access any internal APIs, so no whitelisting is required.
  16. For projects on code.google.com, coordinate with that team to set flags to do per-issue redirects from old project to new site. As each project is imported, set it's moved-to field.

Troubleshooting

  • **TypeError: connect() got an unexpected keyword argument ‘charset’ This error occurs when dev_appserver cannot find the MySQLdb library. Try installing it via sudo apt-get install python-mysqldb.
  • TypeError: connect() argument 6 must be string, not None This occurs when your mysql server is not running. Check if it is running with ps aux | grep mysqld. Start it up with /etc/init.d/mysqld start on linux, or just mysqld.
  • dev_appserver says OSError: [Errno 24] Too many open files and then lists out all source files dev_appserver wants to reload source files that you have changed in the editor, however that feature does not seem to work well with multiple GAE modules and instances running in different processes. The workaround is to control-C or kill the dev_appserver processes and restart them.