Google Git
Sign in
chromium / infra / infra / e932ace4e465dddb32a5c9685ea1175c47981eb0
commite932ace4e465dddb32a5c9685ea1175c47981eb0[log] [tgz]
authorRoberto Carrillo <robertocn@google.com>Fri Apr 20 18:37:02 2018
committerCommit Bot <commit-bot@chromium.org>Fri Apr 20 22:29:19 2018
treeeda8043c310be2fd418fffc01292c200245bc0bd
parenta875517db7b8099c7f2bfdb54c5120e7501a27f0 [diff]
[cr-audit-commits] Use a taskqueue to run audits.

Also, use a single cron job to schedule the audit tasks.

To be clear, this new approach involves two phases (Scheduler Cron Job,
and Auditor Task Queue)

Phase I - Scheduler Cron Job
A new handler called Scheduler is added to iterate over the available
configs and schedule needed audit tasks, creating new RepoState
datastore entries as needed.  Note that the URL to each ref is passed
to the taskqueue task (Auditor handler below), and that the name of the
corresponding config has been added as a field to the RepoState entity.

Phase II - Auditor Task Queue
This change replaces the CommitScanner and CommitAuditor handlers with a
single handler called Auditor, intended to be run as a taskqueue task
when scheduled by the 'Scheduler' cron job above. This task will scan
the git ref, whose url is given as a parameter and using the matching
configuration (from the hardcoded RuleMap) it will create RelevantCommit
entities when appropriate, followed by an audit of such commits and
notifications for those that are found to be in violation of any of the
rules.

The following design docs contain the discussions about this change:
   - go/audit-configuration-for-dynamic-refs
   - go/syncing-audit-app-scheduler-with-audit-task

R=stgao,chanli
BUG=803314

Change-Id: I1b7f52fa68b41db9cc5575f2a42041afde36443c
Reviewed-on: https://chromium-review.googlesource.com/1010976
Commit-Queue: Roberto Carrillo <robertocn@chromium.org>
Reviewed-by: Shuotao Gao <stgao@chromium.org>
14 files changed
tree: eda8043c310be2fd418fffc01292c200245bc0bd
  1. appengine/
  2. appengine_module/
  3. bootstrap/
  4. build/
  5. cipd/
  6. config/
  7. crdx/
  8. data/
  9. doc/
  10. docker/
  11. glyco/
  12. go/
  13. infra/
  14. infra_libs/
  15. node/
  16. packages/
  17. recipes/
  18. test/
  19. utils/
  20. .gitattributes
  21. .gitignore
  22. codereview.settings
  23. CONTRIBUTING.md
  24. DEPS
  25. LICENSE
  26. navbar.md
  27. OWNERS
  28. PRESUBMIT.py
  29. README.md
  30. roll-deps.py
  31. run.py
  32. test.py
  33. WHITESPACE
README.md

infra.git repository

Welcome to the Chrome Infra repository!

Wondering where to start? Check out General Chrome Infrastructure documentation. In particular, to check out this repo and the rest of the infrastructure code, follow the instructions here. The rest of this page is specific to this repo.

Entry points

  • run.py: wrapper script to run programs contained in subdirectories without having to deal with sys.path modifications.
  • test.py: multi-purpose script to run tests.
  • infra_libs/: generally useful functions and classes
  • infra/services/: standalone programs intended to be run as daemons.
  • infra/tools: command-line tools, intended to be run by developers.
  • appengine/: many Chrome-infra-managed AppEngine applications
  • infra/experimental: for, well, experimental stuff. Once they are stabilized and reviewed, they should be moved in a more permanent place.

Miscellaneous technical stuff

  • bootstrap/: utilities to set up a proper Python virtual environment.
  • infra/path_hacks: submodules of this modules give access to modules in the build/ repository. from infra.path_hacks.common import <stg> is actually getting <stg> from build/scripts/common.
  • utils/: purpose? utils?
  • Need to bump infra/deployed to pick up changes?
    • git push origin <updated hash>:deployed
    • mail chrome-troopers@, include:
      • previously deployed hash (for quick rollback)
      • the hash you just pushed
      • the list of CLs that made this push necessary
      • the output of the git push command

Integrating tests with test.py

If you've added a new module, integrate your tests with test.py:

  1. Create a .coveragerc file in the root directory of the module you want to test. Take a look at another .coveragerc to see what to include in that.
  2. Create a “test” directory in the root directory of the module you want to test. More your *_test.py files to this directory.

Double-check that your tests are getting picked up when you want them to be: ./test.py test <path-to-package>.

Tests still not getting picked up by test.py? Double-check to make sure you have init.py files in each directory of your module so Python recognizes it as a package.