tree: d7e0997d118d85e50111b9a2018372dd46b258c5

go/src/infra/fleetconsole/README.md

Fleet Console Server

Googlers, for broad docs on the Fleet Console, see: go/fleet-console

This directory contains the Fleet Console backend server for managing fleet machines.

How to run locally

From the root directory for this repo.

With a local db

First start the db using docker:

docker compose up -d

It also runs pgadmin so if you want to connect to the db via pgadmin visit localhost:5050 (no login should be required).

If this is your first time you will have to run migrations

You can now run the web server:

make build
make run-local-db

Without Docker (macOS with Homebrew)

If you are unable to use Docker, you can set up a local PostgreSQL database using Homebrew.

Install PostgreSQL:
```
brew install postgresql
```
Start the PostgreSQL service:
```
brew services start postgresql
```
Set the password for the default postgres user: The application expects the password to be password.
```
psql -d postgres -c "ALTER USER postgres WITH PASSWORD 'password';"
```
Run database migrations: Follow the instructions in the Run migrations section.
Run the web server:
```
make build
make run-local-db
```

Connecting to the Local Database

Once your local PostgreSQL database is running, you can connect to it using the psql command-line tool.

psql --host=localhost --port=5432 --username=postgres --dbname=postgres

After connecting, you can run SQL queries directly against the database.

Podman Setup (Recommended for Ent Migrations & Tests)

Podman is required for running Ent migrations and containerized integration tests. Note that you need the Docker CLI (not Docker Engine) installed so that Podman can act as a drop-in Docker provider. (We are evaluating adding Docker CLI setup to make tools).

To set up Podman:

sudo apt install podman
sudo usermod --add-subuids 100000-165535 --add-subgids 100000-165535 <username>
podman system migrate
systemctl --user start podman.socket # OR podman system service --time=0 &

Set the environment variable for Testcontainers:

export DOCKER_HOST=unix://$XDG_RUNTIME_DIR/podman/podman.sock

These steps were taken from sources:

Hot reloading

To run the server in development mode with hot-reloading, first install the necessary tools:

make tools

Then, run the server with:

make run-hot-local-db

This will automatically rebuild and restart the server when you make changes to the code.

Connecting to the dev db

Create a tunnel to the dev alloydb vpc:

make setup-dev-db-tunnel

By default, this runs in the foreground. To run in the background, use:

make setup-dev-db-tunnel MODE=bg

You can now run the web server:

make build
make run-dev-db

Making calls to the local service

To make calls to your local service, you can use prpc command line tool.

prpc call localhost:8800 fleetconsole.FleetConsole.PingDB <<EOF
{}
EOF

Run the web client

How to manually test

This codebase includes the consoleadmin CLI tool to test the Fleet Console server.

To build / run the CLI, run:

go build ./cmd/consoleadmin
./consoleadmin

You can do a liveness check for the local Fleet Console backend like so:

./consoleadmin ping -local
{}

And you can check the connection to the database like so:

./consoleadmin ping-db -local
{}

To see more commands available in the CLI run:

./consoleadmin help

To populate your local database with test data run:

./consoleadmin repopulate-cache -local

To populate your local database with mock Android device data run:

./consoleadmin mock-update-android-devices -local

How to run tests

Run go test ./...

debug console server

Run make debug-server to build an executable with dlv debug options

After, start an instance of server. eg local sqldb connection pointing to ufs dev:

dlv exec ./fleetconsoleserver -- \
-sqldb-connection-url="pgx://postgres@localhost:5432/postgres" \
-sqldb-password-secret="devsecret-text://password" \
-ufs-addr ufs.api.cr.dev:443

Then add any necessary break points with b file:line or b file.function. Start the server with c

In a separate session, run your console server command. Go back to server instance and debug. Here are a few useful debugging commands in dlv: - c: continue until the next breakpoint - n: go to the next line - step: step into a function - locals: list all the local variables and their values - args: list all args and their values - p [variable]: list the value of variable

dlv is a forward debugger, meaning you can't step back in time once you execute a line. However, you could install necessary extensions to add revert step or step back in time.

Create migrations

To create migrations:

change the schema in internal/infra/database/ent/schema/
run make gen to regenerate the ent go structs
run make add-migration <migration-name> to create a migration

Refer to ENT's documentation for more information.

Run migrations

Make sure you have atlas installed. Run

make tools

To run migrations locally run:

make apply-migrations

Dev

make setup-dev-db-tunnel
make apply-migrations-dev

Prod

make setup-prod-db-tunnel
make apply-migrations-prod

If you get an error make sure you are logged in gcloud cli for your application-default

gcloud auth application-default login

Cloning Production Data

To verify the behavior of the application (e.g., merging logic) against real production data, you can clone the production database into your local instance.

Clone Prod DB: This script opens a tunnel to the prod DB, dumps it using pg_dump, and imports it into your local postgres database.
```
./scripts/clone_prod_db.sh
```
Ensure you have gcloud, pg_dump, and psql installed and are authenticated with access to fleet-console-prod.
Run Repopulation: After cloning the DB, you can run the repopulation command locally to test how your changes interact with the prod data.
```
make run-local-db # Start server in one terminal
./consoleadmin repopulate-android-cache -local # Run command in another
```

How to deploy

Deployment configs are hosted in the infradata repo

Terraform

If you need to deploy changes in infrastructure that is done via terraform.

from a piper workspace

alias terraform='/google/bin/releases/g3terraform/runner_main -tf_label=terraform_1_10_4'

cd google3/configs/cloud/gong/services/chrome_cloud/chrome_fleet/fleet_console/envs

cd dev # To deploy to dev
cd prod # To deploy to prod

terraform plan # Will show what changes will be applied
# Once you are sure you want to move forward
terraform apply

Backend

Automatic Deployment

Push to dev

Dev is deployed through push on green.

Push to prod

To push dev to prod, use the Fleet Console backend promoter and click the “Trigger” action.

Environment changes

If you need to make change to the deployment environment

cd /your/infra/directory/data/cloud-run/projects/fleet-console/

cd dev/ # For dev
cd prod/ # For prod

vim service.star # Make the changes you need
make gen # Generate needed files

git commit -am '...'
git cl upload

Latchkey config

From a piper workspace

cd google3/configs/cloud/gong/org_hierarchy/google.com/teams/chrome-teams/cros-test-infra/

cd dev/project.fleet-console-dev/ # To deploy to dev
cd prod/project.fleet-console-prod/ # To deploy to prod

From here you can simply make the required edits and submit a new cl

Troubleshooting

Port Conflicts

If a port (such as 5432 for PostgreSQL/AlloyDB tunnel or 8800 for the web server) is already in use:

Make sure other applications, previous make or run commands, or orphan server processes are not holding the port.
Note that Podman (or Docker) containers running in the background can also bind to and hold ports.

To inspect processes bound to a port (e.g. port 5432):

sudo lsof -i :5432

To list active Podman containers that might be holding ports:

podman ps

Authentication Errors

When running the server locally, you may encounter an authentication error like this:

error: failed to initialize auth: failed to initialize the token source: interactive login is required

To resolve this, run the following command to set up your credentials:

luci-auth login -scopes "https://www.googleapis.com/auth/cloud-platform https://www.googleapis.com/auth/userinfo.email"

Authentication

To ensure that the Fleet Console server uses the correct service accounts (service@fleet-console-dev.iam.gserviceaccount.com and service@fleet-console-prod.iam.gserviceaccount.com), it is important to use auth.AsSelf when creating authenticated clients. This will ensure that the server uses its own service account credentials to make authenticated calls to other services.

Here is an example of how to create an authenticated HTTP client using auth.AsSelf:

	t, err := auth.GetRPCTransport(ctx, auth.AsSelf)
	if err != nil {
		return nil, errors.Fmt("failed to get RPC transport: %w", err)
	}
	prpcClient := &prpc.Client{
		C:    &http.Client{Transport: t},
		Host: host,
		Options: &prpc.Options{
			Insecure:  false,
			UserAgent: "fleet-console",
		},
	}