Isolate Server

An AppEngine service to efficiently cache large set of large files over the internet with high level of duplication. The cache is content addressed and uses Cloud Storage as its backing store.

Isolate enables sending temporary files around. It is a pure cache, files will be deleted.

Isolate can be used standalone when only files need to be transfered but no task scheduler is needed.

Documentation

Setting up

• Visit http://console.cloud.google.com and create a project. Replace <appid> below with your project id.
• Visit Google Cloud Console
  • App Engine
    • Enable it and choose us-central.
  • IAM & Admin > IAM
    • Click Add Member and add someone else so you can safely be hit by a bus.
  • IAM & Admin > Service accounts
    • Click Create service account.
    • Service account name server
    • Service account ID server
    • Click Create.
    • Click Continue, no need to add a permission.
    • Click Create key.
    • Select P12 and click Create.
      • TODO(vadimsh): switch to JSON keys.
    • Click Done.
  • Storage
    • Click Create bucket.
    • Name it with the same . Do not use any pre-created bucket, they won't work.
    • Choose Multi-Regional, United States.
    • Click Create.
    • Click Permissions tab.
    • Click Add members.
    • Enter the user server@<appid>.iam.gserviceaccount.com.
    • Select Roles in Storage Legacy group: Storage Legacy Bucket Writer and Storage Legacy Object Reader.
    • Click Add.
  • Pub/Sub
    • Click Enable API.
  • App Engine > Memcache
    • Click Change.
    • Chose Dedicated.
    • Set the cache to Dedicated 5Gb.
    • Wait a day of steady state usage.
    • Set the limit to be lower than the value read at “Total cache size” in “Memcache Viewer”.
  • App Engine > Settings
    • Click Edit:
    • Set Google login Cookie expiration to: 2 weeks.
    • Click Save.
• Upload the code with: ./tools/gae upl -x -A <appid>
• Run setup_bigquery.sh to create the BigQuery isolated.stat table and grant write access to the AppEngine app. The cron job will fail meanwhile.
• If you plan to use an auth_service,
  • Make sure it is setup already.
  • Follow instructions here.
• _Else visit “https://<appid>.appspot.com/auth/bootstrap” and click Proceed.
• Visit “https://<appid>.appspot.com/auth/groups”
  • Create access groups as relevant. Visit the “IP Whitelists” tab and add bot external IP addresses if needed.
• Visit “https://<appid>.appspot.com/restricted/config”
  • Follow the on-screen instructions to generate the base64 encoded DER private key.
  • Click Submit.
• If you plan to use a config service (the normal case):
  • Make sure it is setup already.
  • Make sure you set SettingsCfg.ui_client_id to be server@<appid>.iam.gserviceaccount.com.
  • Follow instruction here.
• If you are not using a config service, see Configuring using FS mode. You'll need to add an entry to settings.cfg like ui_client_id: "server@<appid>.iam.gserviceaccount.com"

Stats

Use prpc CLI client from https://go.chromium.org/luci/grpc/cmd/prpc:

echo '{"resolution":"MINUTE","limit":20}' | prpc call -verbose <host> isolated.Isolated.Stats