Test GCP Projects Setup Scripts

This directory contains the following scripts to create and configure GCP projects/storage etc. These GCP projects are where our CI builders run tests.

Host files generation scripts

generate_host_files.py: This script generates host files for different projects that share the same storage location. It will be used by our LUCI bot at the beginning of a test run to generate candidate host files to which to deploy.

Pool creation scripts

create_test_pool.py: It invokes the following 3 scripts to build a project pool.
create_gcp_project.py: This creates a project and associates it to a billing account.
create_gcp_storage.py: This script creates the shared storage on a given project (ex: for test assets).
configure_test_project.py: This script configures a gcp_project for E2E tests (Enable APIs & set ACLs)
configure_cron_stop_instances.py: This script schedules a job that stops instances older than 24h. We normally clean-up the deployment at the end of each test so we shouldn't need this, but this provides a safety net in case of unexpected failures. The deployment would get cleaned up automatically by cel_ctl next time we run a test on that project.

Usage examples:

  python generate_host_files.py --template examples/schema/host/example.host.textpb
       --projects="p1;p2;p3" --storage_bucket celab-ci-pool-assets
       --storage_prefix=tests-1234 --destination_dir /tmp/host_files

 python scripts/tests/create_test_pool.py --name celab-ci-pool
      --folder <XYZ> --billing <XYZ>
      --service_accounts="celab-ci-builder@...com"
      --size 5

 Example with real value:
 python create_test_pool.py --name celab-poros --folder 960844185733
 --billing 01771F-48826F-E305AD --index 11 --size 10 --service_accounts 'poros-dev@appspot.gserviceaccount.com;poros-prod@appspot.gserviceaccount.com'

Project configuration scripts

grant_project_iam.py
- Usage 1: It takes a list of project names and IAM email id to add to grant IAM access.
- Usage 2: It takes the service account postfix and grant the service account of the same project with certain IAM
- Usage 3: It takes target_project_id and prefix_type to bind service account from many projects to access one GCP project
- Usage 4: It takes the service account name and grants the service account of each project with a certain IAM role (in the format <name>@<project_id>.iam.gserviceaccount.com).

Usage 1 example:

python grant_project_iam.py \
  --project_names "celab-poros-001;celab-poros-002;celab-poros-003;celab-poros-004;celab-poros-005;celab-poros-006;celab-poros-007;celab-poros-008;celab-poros-009;celab-poros-010" \
  --principal_type "group" \
  --emailid "chrome-enterprise-lab@google.com" \
  --role "roles/editor"

Usage 2 example:

python grant_project_iam.py \
  --project_names "celab-chrome-try-021;celab-chrome-try-022;celab-chrome-try-023;celab-chrome-try-024;celab-chrome-try-025;celab-chrome-try-026;celab-chrome-try-027;celab-chrome-try-028;celab-chrome-try-029;celab-chrome-try-030;celab-chrome-try-031;celab-chrome-try-032;celab-chrome-try-033;celab-chrome-try-034;celab-chrome-try-035;celab-chrome-try-036;celab-chrome-try-037;celab-chrome-try-038;celab-chrome-try-039;celab-chrome-try-040;celab-chrome-try-041;celab-chrome-try-042;celab-chrome-try-043;celab-chrome-try-044;celab-chrome-try-045;celab-chrome-try-046;celab-chrome-try-047;celab-chrome-try-048;celab-chrome-try-049;celab-chrome-try-050" \
  --principal_type "serviceAccount" \
  --postfix "@appspot.gserviceaccount.com" \
  --role "roles/editor"

Usage 3 example:

python grant_project_iam.py \
  --project_names "celab-poros-031;celab-poros-032;celab-poros-033;celab-poros-034;celab-poros-035;celab-poros-036;celab-poros-037;celab-poros-038;celab-poros-039;celab-poros-040;celab-poros-041;celab-poros-042;celab-poros-043;celab-poros-044;celab-poros-045;celab-poros-046;celab-poros-047;celab-poros-048;celab-poros-049;celab-poros-050;celab-poros-051;celab-poros-052;celab-poros-053;celab-poros-054;celab-poros-055;celab-poros-056;celab-poros-057;celab-poros-058;celab-poros-059;celab-poros-060" \
  --principal_type "serviceAccount" \
  --prefix_type "project_number" \
  --postfix "@cloudservices.gserviceaccount.com" \
  --role "roles/compute.imageUser" \
  --target_project_id "celab-selfhost-instances"

Usage 4 example (grant Infrastructure Manager access):

python grant_project_iam.py \
  --project_names "celab-try-001;celab-try-002;celab-try-003;celab-try-004;celab-try-005" \
  --principal_type "serviceAccount" \
  --service_account_name "cel-instance-service" \
  --role "roles/config.admin"

Caveats

For test pool project, you need to grant the default app engine service account, eg. celab-chrome-try-011@appspot.gserviceaccount.com with project editor access. Otherwise, the deployment will fail during test run.