Google Git
Sign in
chromium / website / 6212a1c32f19061f556d01d2149c087e2cbb0d91
commit6212a1c32f19061f556d01d2149c087e2cbb0d91[log] [tgz]
authorRaphael Kubo da Costa <raphael.kubo.da.costa@intel.com>Fri Sep 16 14:01:38 2022
committerchromium-website-scoped@luci-project-accounts.iam.gserviceaccount.com <chromium-website-scoped@luci-project-accounts.iam.gserviceaccount.com>Fri Sep 16 16:21:53 2022
tree6bc74f75ebad89264eda9a9e0d33ae966dca0e24
parent7e899845af747ef1b8f5b8ace107dd79d39c3a1b [diff]
fetch_lobs.py: Use requests instead of urllib3

Or rather, use requests and let it call urllib3 behind the scenes.
urllib3 requires one to manually instantiate a ProxyManager class to use
an http proxy, which would make this script very cumbersome. The
requests module handles this transparently behinds the scene while still
using urllib3 and its pooling capabilities.

Testing on my laptop shows no significant regression in the time it took
to fetch all LOBs:

* requests + http proxy
  Fetched 1162 LOBs (454.4MB) in 91.876 seconds (39.6Mbps).

* requests + no proxy
  Fetched 1162 LOBs (454.4MB) in 73.608 seconds (49.4Mbps).

* urllib3 + no proxy
  Fetched 1162 LOBs (454.4MB) in 74.916 seconds (48.5Mbps).

Bug: 1266070
Change-Id: I8b4f2b6aa05c535b9470da7e449dc3dc268a8ef0
Reviewed-on: https://chromium-review.googlesource.com/c/website/+/3899080
Reviewed-by: Struan Shrimpton <sshrimp@google.com>
Commit-Queue: Struan Shrimpton <sshrimp@google.com>
Auto-Submit: Raphael Kubo Da Costa <raphael.kubo.da.costa@intel.com>
2 files changed
tree: 6bc74f75ebad89264eda9a9e0d33ae966dca0e24
  1. docs/
  2. infra/
  3. scripts/
  4. site/
  5. third_party/
  6. .eleventy.js
  7. .firebaserc
  8. .gclient
  9. .gitignore
  10. .vpython3
  11. AUTHORS
  12. CONTENT_OWNERS
  13. DEPS
  14. DIR_METADATA
  15. node_modules.tar.gz.sha1
  16. npmw
  17. originals.tar.gz.sha1
  18. OWNERS
  19. package-lock.json
  20. package.json
  21. PRESUBMIT.cfg
  22. PRESUBMIT.py
  23. README.md
README.md

Source code for www.chromium.org

This Git repo contains the source content for www.chromium.org.

NOTE: 2021-11-15. This is not (yet) actually true. This is a work-in-progress repo that we plan to move to.

The website is implemented by serving static content (mostly Markdown files) that is translated / built ahead of time into HTML using the Eleventy static site generator (which is written in Node/JS) and deployed onto Firebase Cloud Hosting.

See //docs/CONTRIBUTING.md if you wish to contribute to the site.