metadata: use os.walk to speedup metadata file discovery

os.walk is more efficient than the current handwritten traversal.

Measured the time to scan reduced from 30s+ to 8s on p920 on
chromium/src.

`followlinks=True` is set to preserve behavior that os.path.isdir
returns True for symlink to directories, and the current traversal code
will descend into those.

Change-Id: I941eec9105a46d6538ca484fbb5249a75888e38a
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5379945
Reviewed-by: Anne Redulla <aredulla@google.com>
Commit-Queue: Jiewei Qian <qjw@chromium.org>
1 file changed
tree: e7e60d25ea13cba0544512389e9757a5b71455a8
  1. bootstrap/
  2. fetch_configs/
  3. hooks/
  4. infra/
  5. man/
  6. metadata/
  7. python-bin/
  8. python2-bin/
  9. recipes/
  10. testing_support/
  11. tests/
  12. third_party/
  13. win_toolchain/
  14. zsh-goodies/
  15. .cipd_impl.ps1
  16. .flake8
  17. .gitattributes
  18. .gitignore
  19. .isort.cfg
  20. .style.yapf
  21. .vpython3
  22. auth.py
  23. autoninja
  24. autoninja.bat
  25. autoninja.py
  26. autosiso
  27. autosiso.bat
  28. autosiso.py
  29. bazel.py
  30. bb
  31. bb.bat
  32. black
  33. bootstrap_python3
  34. breakpad.py
  35. BUILD_OWNERS
  36. cipd
  37. cipd.bat
  38. cipd_bin_setup.bat
  39. cipd_bin_setup.sh
  40. cipd_client_version
  41. cipd_client_version.digests
  42. cipd_manifest.txt
  43. cipd_manifest.versions
  44. cipd_manifest_cros_python2.txt
  45. cipd_manifest_cros_python2.versions
  46. clang-format
  47. clang-format.bat
  48. clang_format.py
  49. codereview.settings
  50. compile_single_file
  51. compile_single_file.bat
  52. compile_single_file.py
  53. cpplint.bat
  54. cpplint.py
  55. cpplint_chromium.py
  56. cros
  57. CROS_OWNERS
  58. cros_python2.vpython
  59. crosjobs
  60. detect_host_arch.py
  61. DIR_METADATA
  62. dirmd
  63. dirmd.bat
  64. download_from_google_storage
  65. download_from_google_storage.bat
  66. download_from_google_storage.py
  67. ensure_bootstrap
  68. fetch
  69. fetch.bat
  70. fetch.py
  71. gclient
  72. gclient-new-workdir.py
  73. gclient.bat
  74. gclient.py
  75. gclient_completion.sh
  76. gclient_eval.py
  77. gclient_paths.py
  78. gclient_scm.py
  79. gclient_utils.py
  80. gerrit_client.py
  81. gerrit_util.py
  82. git-cache
  83. git-cl
  84. git-crrev-parse
  85. git-drover
  86. git-find-releases
  87. git-footers
  88. git-freeze
  89. git-gs
  90. git-hyper-blame
  91. git-map
  92. git-map-branches
  93. git-mark-merge-base
  94. git-migrate-default-branch
  95. git-nav-downstream
  96. git-nav-upstream
  97. git-new-branch
  98. git-number
  99. git-rebase-update
  100. git-rename-branch
  101. git-reparent-branch
  102. git-retry
  103. git-runhooks
  104. git-squash-branch
  105. git-thaw
  106. git-upstream-diff
  107. git_cache.py
  108. git_cl.py
  109. git_cl_completion.sh
  110. git_common.py
  111. git_completion.sh
  112. git_dates.py
  113. git_find_releases.py
  114. git_footers.py
  115. git_freezer.py
  116. git_hyper_blame.py
  117. git_map.py
  118. git_map_branches.py
  119. git_mark_merge_base.py
  120. git_migrate_default_branch.py
  121. git_nav_downstream.py
  122. git_new_branch.py
  123. git_number.py
  124. git_rebase_update.py
  125. git_rename_branch.py
  126. git_reparent_branch.py
  127. git_retry.py
  128. git_squash_branch.py
  129. git_upstream_diff.py
  130. gn
  131. gn.bat
  132. gn.py
  133. goma_auth
  134. goma_auth.bat
  135. goma_ctl
  136. goma_ctl.bat
  137. gsutil.py
  138. gsutil.py.bat
  139. gsutil.vpython3
  140. infra_to_superproject
  141. infra_to_superproject.bat
  142. infra_to_superproject.py
  143. isort
  144. led
  145. led.bat
  146. LICENSE
  147. lockfile.py
  148. luci-auth
  149. luci-auth.bat
  150. LUCI_OWNERS
  151. lucicfg
  152. lucicfg.bat
  153. mac_toolchain
  154. metrics.py
  155. metrics.README.md
  156. metrics_utils.py
  157. my_activity.py
  158. ninja
  159. ninja.bat
  160. ninja.py
  161. ninja_reclient.py
  162. ninjalog.README.md
  163. ninjalog_uploader.py
  164. ninjalog_uploader_wrapper.py
  165. OWNERS
  166. owners_client.py
  167. owners_finder.py
  168. pinpoint
  169. pinpoint.bat
  170. post_build_ninja_summary.py
  171. PRESUBMIT.py
  172. presubmit_canned_checks.py
  173. presubmit_diff.py
  174. presubmit_support.py
  175. profile.xml
  176. prpc
  177. prpc.bat
  178. pylint-2.17
  179. pylint-2.17.bat
  180. pylint-2.6
  181. pylint-2.6.bat
  182. pylint-2.7
  183. pylint-2.7.bat
  184. pylint_main.py
  185. pylintrc
  186. pylintrc-2.17
  187. python_runner.sh
  188. rdb
  189. rdb.bat
  190. rdb_wrapper.py
  191. README.gclient.md
  192. README.git-cl.md
  193. README.md
  194. reclient_helper.py
  195. reclient_metrics
  196. reclient_metrics.bat
  197. reclient_metrics.py
  198. reclient_metrics.README.md
  199. RECLIENT_OWNERS
  200. reclientreport
  201. reclientreport.bat
  202. reclientreport.py
  203. repo
  204. repo_launcher
  205. roll-dep
  206. roll-dep.bat
  207. roll_dep.py
  208. rustfmt.py
  209. scm.py
  210. setup_color.py
  211. siso
  212. siso.bat
  213. siso.py
  214. split_cl.py
  215. subcommand.py
  216. subprocess2.py
  217. swift-format
  218. swift_format.py
  219. update_depot_tools
  220. update_depot_tools.bat
  221. update_depot_tools_toggle.py
  222. upload_metrics.py
  223. upload_to_google_storage.py
  224. utils.py
  225. vpython3
  226. vpython3.bat
  227. WATCHLISTS
  228. watchlists.py
  229. weekly
  230. whitespace.txt
  231. win32imports.py
  232. wtf
  233. yapf
  234. yapf.bat
README.md

depot_tools

Tools for working with Chromium development. It requires python 3.8.

Tools

The most important tools are:

  • fetch: A gclient wrapper to checkout a project. Use fetch --help for more details.
  • gclient: A meta-checkout tool. Think repo or git submodules, except that it support OS-specific rules, e.g. do not checkout Windows only dependencies when checking out for Android. Use gclient help for more details and README.gclient.md.
  • git cl: A code review tool to interact with Rietveld or Gerrit. Use git cl help for more details and README.git-cl.md.
  • roll-dep: A gclient dependency management tool to submit a dep roll, updating a dependency to a newer revision.

There are a lot of git utilities included.

Updating

depot_tools updates itself automatically when running gclient tool. To disable auto update, set the environment variable DEPOT_TOOLS_UPDATE=0 or run ./update_depot_tools_toggle.py --disable.

To update package manually, run update_depot_tools.bat on Windows, or ./update_depot_tools on Linux or Mac.

On Windows only, running gclient will install git and python.

Contributing

To contribute change for review:

git new-branch <somename>
# Hack
git add .
git commit -a -m "Fixes goat teleporting"
# find reviewers
git cl owners
git log -- <yourfiles>

# Request a review.
git cl upload -r reviewer1@chromium.org,reviewer2@chromium.org --send-mail

# Edit change description if needed.
git cl desc

# If change is approved, flag it to be committed.
git cl set-commit

# If change needs more work.
git rebase-update
...
git cl upload -t "Fixes goat teleporter destination to be Australia"

See also open bugs, open reviews, forum or report problems.

cpplint.py

Until 2018, our cpplint.py was a copy of the upstream version at https://github.com/google/styleguide/tree/gh-pages/cpplint. Unfortunately, that repository is not maintained any more. If you want to update cpplint.py in depot_tools, just upload a patch to do so. We will figure out a long-term strategy via issue https://crbug.com/916550.

Note that the cpplint.py here is also used by the Tricium analyzer, so if the cpplint.py here changes, we should also update the copy used there.