Google Git
Sign in
chromium / external / github.com / luci / luci-py / e3fbb1f5dafa59a2c57cf3a9fe3708f4309ab653 / . / appengine / isolate
tree: 303e99f8e61e91fc5cb2c756cef7e5e7a845c524 [path history] [tgz]
  1. doc/
  2. static/
  3. templates/
  4. tools/
  5. cloudstorage ⇨ ../third_party/cloudstorage
  6. components ⇨ ../components/components
  7. gae_ts_mon ⇨ ../third_party/gae_ts_mon
  8. graphy ⇨ ../third_party/graphy
  9. gviz ⇨ ../third_party/gviz
  10. mapreduce ⇨ ../third_party/mapreduce
  11. pipeline ⇨ ../third_party/pipeline
  12. test_support ⇨ ../components/test_support
  13. tool_support ⇨ ../components/tool_support
  14. acl.py
  15. app.yaml
  16. appengine_config.py
  17. config.py
  18. cron.yaml
  19. gcs.py
  20. handlers_backend.py
  21. handlers_endpoints_v1.py
  22. handlers_endpoints_v1_test.py
  23. handlers_frontend.py
  24. handlers_stats.py
  25. handlers_test.py
  26. index.yaml
  27. main_backend.py
  28. main_frontend.py
  29. mapreduce.yaml
  30. mapreduce_jobs.py
  31. model.py
  32. model_test.py
  33. module-backend.yaml
  34. PRESUBMIT.py
  35. queue.yaml
  36. README.md
  37. stats.py
  38. stats_test.py
  39. template.py
  40. test_env.py
appengine/isolate/README.md

Isolate Server

An AppEngine service to efficiently cache large set of large files over the internet with high level of duplication. The cache is content addressed and uses Cloud Storage as its backing store.

Isolate enables sending temporary files around. It is a pure cache, files will be deleted.

Isolate can be used standalone when only files need to be transfered but no task scheduler is needed.

Documentation

Setting up

  • Visit http://console.cloud.google.com and create a project. Replace <appid> below with your project id.
  • Visit Google Cloud Console
    • IAM & Admin, click Add Member and add someone else so you can safely be hit by a bus.
    • IAM & Admin, Service accounts, click Create service account:
      • Name it “server@<appid>.iam.gserviceaccount.com”.
      • Check Furnish a new private key and select P12.
        • TODO(vadimsh): switch to JSON keys.
      • Click Create.
    • Storage, click Create bucket, name it with the same . Do not use any pre-created bucket, they won't work.
      • Click on Browser on the left.
      • Click the 3 dots on the right of your new bucket and select Edit bucket permission.
      • Click Add item, chose User, “server@<appid>.iam.gserviceaccount.com”, Writer.
      • Click Save.
    • Pub/Sub, click Enable API.
  • Upload the code with: ./tools/gae upl -x -A <appid>
  • If you plan to use an auth_service,
  • else
    • Visit “https://<appid>.appspot.com/auth/bootstrap” and click Proceed.
  • Visit “https://<appid>.appspot.com/auth/groups”:
    • Create access groups as relevant. Visit the “IP Whitelists” tab and add bot external IP addresses if needed.
  • Visit “https://<appid>.appspot.com/restricted/config”
    • Set “API access client email address” to “server@<appid>.iam.gserviceaccount.com”.
    • Follow the on-screen instructions to generate the base64 encoded DER private key.
    • Click Submit.
  • Tweak settings:
    • Visit Google Cloud Console
      • App Engine, Memcache, click Change:
        • Chose Dedicated.
        • Set the cache to Dedicated 5Gb.
        • Wait a day of steady state usage.
        • Set the limit to be lower than the value read at “Total cache size” in “Memcache Viewer”.
    • App Engine, Settings, click Edit:
      • Set Google login Cookie expiration to: 2 weeks, click Save.