scripts/tests/README.md

# Test GCP Projects Setup Scripts

This directory contains the following scripts to create and configure GCP projects/storage etc.
  These GCP projects are where our CI builders run tests.

## Host files generation scripts

generate_host_files.py:
  This script generates host files for different projects that share the same
  storage location. It will be used by our LUCI bot at the beginning of a test
  run to generate candidate host files to which to deploy.

## Pool creation scripts

- create_test_pool.py:
  It invokes the following 3 scripts to build a project pool.
- create_gcp_project.py:
  This creates a project and associates it to a billing account.
- create_gcp_storage.py:
  This script creates the shared storage on a given project (ex: for test assets).
- configure_test_project.py:
  This script configures a gcp_project for E2E tests (Enable APIs & set ACLs)
- configure_cron_stop_instances.py:
  This script schedules a job that stops instances older than 24h.
  We normally clean-up the deployment at the end of each test so we shouldn't
  need this, but this provides a safety net in case of unexpected failures.
  The deployment would get cleaned up automatically by cel_ctl next time we
  run a test on that project.

Usage examples:

```bash
  python generate_host_files.py --template examples/schema/host/example.host.textpb
       --projects="p1;p2;p3" --storage_bucket celab-ci-pool-assets
       --storage_prefix=tests-1234 --destination_dir /tmp/host_files
 ```

 ```bash
  python scripts/tests/create_test_pool.py --name celab-ci-pool
       --folder <XYZ> --billing <XYZ>
       --service_accounts="celab-ci-builder@...com"
       --size 5

  Example with real value:
  python create_test_pool.py --name celab-poros --folder 960844185733
  --billing 01771F-48826F-E305AD --index 11 --size 10 --service_accounts 'poros-dev@appspot.gserviceaccount.com;poros-prod@appspot.gserviceaccount.com'

```

## Project configuration scripts

- grant_project_iam.py
  - Usage 1: It takes a list of project names and IAM email id to add to grant IAM access.
  - Usage 2: It takes the service account postfix and grant the service account of the same project with certain IAM
  - Usage 3: It takes target_project_id and prefix_type to bind service account from many projects to access one GCP project
  - Usage 4: It takes the service account name and grants the service account of each project with a certain IAM role (in the format `<name>@<project_id>.iam.gserviceaccount.com`).

Usage 1 example:

```bash
python grant_project_iam.py \
  --project_names "celab-poros-001;celab-poros-002;celab-poros-003;celab-poros-004;celab-poros-005;celab-poros-006;celab-poros-007;celab-poros-008;celab-poros-009;celab-poros-010" \
  --principal_type "group" \
  --emailid "chrome-enterprise-lab@google.com" \
  --role "roles/editor"
```

Usage 2 example:

```bash
python grant_project_iam.py \
  --project_names "celab-chrome-try-021;celab-chrome-try-022;celab-chrome-try-023;celab-chrome-try-024;celab-chrome-try-025;celab-chrome-try-026;celab-chrome-try-027;celab-chrome-try-028;celab-chrome-try-029;celab-chrome-try-030;celab-chrome-try-031;celab-chrome-try-032;celab-chrome-try-033;celab-chrome-try-034;celab-chrome-try-035;celab-chrome-try-036;celab-chrome-try-037;celab-chrome-try-038;celab-chrome-try-039;celab-chrome-try-040;celab-chrome-try-041;celab-chrome-try-042;celab-chrome-try-043;celab-chrome-try-044;celab-chrome-try-045;celab-chrome-try-046;celab-chrome-try-047;celab-chrome-try-048;celab-chrome-try-049;celab-chrome-try-050" \
  --principal_type "serviceAccount" \
  --postfix "@appspot.gserviceaccount.com" \
  --role "roles/editor"
```

Usage 3 example:

```bash
python grant_project_iam.py \
  --project_names "celab-poros-031;celab-poros-032;celab-poros-033;celab-poros-034;celab-poros-035;celab-poros-036;celab-poros-037;celab-poros-038;celab-poros-039;celab-poros-040;celab-poros-041;celab-poros-042;celab-poros-043;celab-poros-044;celab-poros-045;celab-poros-046;celab-poros-047;celab-poros-048;celab-poros-049;celab-poros-050;celab-poros-051;celab-poros-052;celab-poros-053;celab-poros-054;celab-poros-055;celab-poros-056;celab-poros-057;celab-poros-058;celab-poros-059;celab-poros-060" \
  --principal_type "serviceAccount" \
  --prefix_type "project_number" \
  --postfix "@cloudservices.gserviceaccount.com" \
  --role "roles/compute.imageUser" \
  --target_project_id "celab-selfhost-instances"
```

Usage 4 example (grant Infrastructure Manager access):

```bash
python grant_project_iam.py \
  --project_names "celab-try-001;celab-try-002;celab-try-003;celab-try-004;celab-try-005" \
  --principal_type "serviceAccount" \
  --service_account_name "cel-instance-service" \
  --role "roles/config.admin"
```

## Caveats

1. For test pool project, you need to grant the default app engine service account, eg. celab-chrome-try-011@appspot.gserviceaccount.com with project editor access. Otherwise, the deployment will fail during test run.