commit | 6e57cd43ec13c2f538484bd60d9da133783193c6 | [log] [tgz] |
---|---|---|
author | Max Illfelder <illfelder@users.noreply.github.com> | Fri Dec 07 23:03:19 2018 |
committer | GitHub <noreply@github.com> | Fri Dec 07 23:03:19 2018 |
tree | 411d517a64bb7399452ce7bf8789b06396a14281 | |
parent | d6e6ec199564ee4a44baa5d8c16bb4001dc7e96c [diff] |
Update the Travis credentials. (#714) Generated using: travis encrypt your-password-here --add deploy.password
This repository stores the collection of packages installed on Google supported Compute Engine images.
Table of Contents
The Linux guest environment denotes the Google provided configuration and tooling inside of a Google Compute Engine (GCE) virtual machine. The metadata server is a communication channel for transferring information from a client into the guest. The Linux guest environment includes a set of scripts and daemons (long-running processes) that read the content of the metadata server to make a virtual machine run properly on our platform.
The guest environment is made up of the following components:
The Linux guest environment is written in Python and is version agnostic between Python 2.6 and 3.5. There is complete unittest coverage for every Python library and script. The design of various guest libraries, daemons, and scripts, are detailed in the sections below.
The Python libraries are shared with each of the daemons and the instance setup tools.
The guest environment relies upon retrieving content from the metadata server to configure the VM environment. A metadata watching library handles all communication with the metadata server.
The library exposes two functions:
Metadata server requests have custom retry logic for metadata server unavailability; by default, any request has one minute to complete before the request is cancelled. In case of a brief network outage where the metadata server is unavailable, there is a short delay between retries.
The Google added daemons and scripts write to the serial port for added transparency. A common logging library is a thin wrapper around the Python logging module. The library configures appropriate SysLog handlers, sets the logging formatter, and provides a debug options for added logging and console output.
A configuration file allows users to disable daemons and modify instance setup behaviors from a single location. Guest environment daemons and scripts need a mechanism to integrate user settings into the guest. A configuration management library retrieves and modifies these settings.
The library exposes the following functions:
Guest environment daemons and scripts use a common library for file management. The library provides the following functions:
flock
and releases the lock on exit.A network-utilities library retrieves information about a network interface. The library is used for IP forwarding and for setting up an Ethernet interface on boot. The library exposes a GetNetworkInterface
function that retrieves the network interface name associated with a MAC address.
The guest environment daemons import and use the common libraries described above. Each daemon reads the configuration file before execution. This allows a user to easily disable undesired functionality. Additional daemon behaviors are detailed below.
The accounts daemon is responsible for provisioning and deprovisioning user accounts. The daemon grants permissions to user accounts, and updates the list of authorized keys that have access to accounts based on metadata SSH key updates. User account creation is based on adding and remove SSH Keys stored in metadata.
The accounts management daemon has the following behaviors.
google-sudoers
Linux group.google-sudoers
group.The clock skew daemon is responsible for syncing the software clock with the hypervisor clock after a stop/start event or after a migration. Preventing clock skew may result in system time has changed
messages in VM logs.
The network daemon uses network interface metadata to manage the network interfaces in the guest by performing the following tasks:
66
. This ID is a namespace for daemon configured IP addresses.Instance setup runs during VM boot. The script configures the Linux guest environment by performing the following tasks.
boto
config for using Google Cloud Storage.The defaults configuration file incorporates any user provided setting in /etc/default/instance_configs.cfg.template
and does not override other conflicting settings. This allows package updates without overriding user configuration.
Metadata scripts implement support for running user provided startup scripts and shutdown scripts. The guest support for metadata scripts is implemented in Python with the following design details.
startup-script
and startup-script-url
) both are executed.startup-script
and startup-script-url
) a URL is executed first.Users of Google provided images may configure the guest environment behaviors using a configuration file. To make configuration changes, add settings to /etc/default/instance_configs.cfg.template
. If you are attempting to change the behavior of a running instance, run /usr/bin/google_instance_setup
before reloading the affected daemons.
Linux distributions looking to include their own defaults can specify settings in /etc/default/instance_configs.cfg.distro
. These settings will not override /etc/default/instance_configs.cfg.template
. This enables distribution settings that do not override user configuration during package update.
The following are valid user configuration options.
Section | Option | Value |
---|---|---|
Accounts | deprovision_remove | true makes deprovisioning a user destructive. |
Accounts | groups | Comma separated list of groups for newly provisioned users. |
Accounts | useradd_cmd | Command string to create a new user. |
Accounts | userdel_cmd | Command string to delete a user. |
Accounts | usermod_cmd | Command string to modify a user's groups. |
Accounts | gpasswd_add_cmd | Command string to add a user to a group. |
Accounts | gpasswd_remove_cmd | Command string to remove a user from a group. |
Accounts | groupadd_cmd | Command string to create a new group. |
Daemons | accounts_daemon | false disables the accounts daemon. |
Daemons | clock_skew_daemon | false disables the clock skew daemon. |
Daemons | ip_forwarding_daemon | false (deprecated) skips IP forwarding. |
Daemons | network_daemon | false disables the network daemon. |
InstanceSetup | host_key_types | Comma separated list of host key types to generate. |
InstanceSetup | optimize_local_ssd | false prevents optimizing for local SSD. |
InstanceSetup | network_enabled | false skips instance setup functions that require metadata. |
InstanceSetup | set_boto_config | false skips setting up a boto config. |
InstanceSetup | set_host_keys | false skips generating host keys on first boot. |
InstanceSetup | set_multiqueue | false skips multiqueue driver support. |
IpForwarding | ethernet_proto_id | Protocol ID string for daemon added routes. |
IpForwarding | ip_aliases | false disables setting up alias IP routes. |
IpForwarding | target_instance_ips | false disables internal IP address load balancing. |
MetadataScripts | default_shell | String with the default shell to execute scripts. |
MetadataScripts | run_dir | String base directory where metadata scripts are executed. |
MetadataScripts | startup | false disables startup script execution. |
MetadataScripts | shutdown | false disables shutdown script execution. |
NetworkInterfaces | setup | false skips network interface setup. |
NetworkInterfaces | ip_forwarding | false skips IP forwarding. |
NetworkInterfaces | dhclient_script | String path to a dhclient script used by dhclient. |
NetworkInterfaces | dhcp_command | String to execute to enable network interfaces. |
Setting network_enabled
to false
will skip setting up host keys and the boto
config in the guest. The setting may also prevent startup and shutdown script execution.
The guest Python code is packaged as a compliant PyPI Python package that can be used as a library or run independently. In addition to the Python package, deb and rpm packages are created with appropriate init configuration for supported GCE distros. The packages are targeted towards distribution provided Python versions.
Distro | Package Type | Python Version | Init System |
---|---|---|---|
Debian 8 | deb | 2.7 | systemd |
Debian 9 | deb | 3.5 or 2.7 | systemd |
CentOS 6 | rpm | 2.6 | upstart |
CentOS 7 | rpm | 2.7 | systemd |
RHEL 6 | rpm | 2.6 | upstart |
RHEL 7 | rpm | 2.7 | systemd |
Ubuntu 14.04 | deb | 2.7 | upstart |
Ubuntu 16.04 | deb | 3.5 or 2.7 | systemd |
SLES 11 | rpm | 2.6 | sysvinit |
SLES 12 | rpm | 2.7 | systemd |
We build the following packages for the Linux guest environment.
google-compute-engine
/usr/bin
.instance_setup
.python-google-compute-engine
python3-google-compute-engine
The package source for Debian and RPM specs for Enterprise Linux 6 and 7 are included in this project. There are also Daisy workflows for spinning up GCE VM's to automatically build the packages for Debian, Red Hat, and CentOS. See the README in the packaging directory for more details.
There are several places where package versions have to be updated and must match to successfully release an update.
setup.py
Update the version string with the Python package version. Used for entry points through the Python egg and PyPI.specs/google-compute-engine.spec
Update the version of the google-compute-engine
package for EL6 and EL7.specs/python-google-compute-engine.spec
Update the version string of the python-google-compute-engine
package for EL6 and EL7.debian/changelog
Update google-compute-image-packages (VERSION) stable
, the version of the Debian packages.package_version
when invoking the package build workflows.The deb and rpm packages used in some GCE images are published to Google Cloud repositories. Debian 8 and 9, CentOS 6 and 7, and RHEL 6 and 7 use these repositories to install and update the google-compute-engine
, and python-google-compute-engine
(and python3-google-compute-engine
for Python 3) packages. If you are creating a custom image, you can also use these repositories in your image.
For Debian 8, run the following commands as root:
Add the public repo key to your system:
curl https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add -
Add a source list file /etc/apt/sources.list.d/google-cloud.list
:
tee /etc/apt/sources.list.d/google-cloud.list << EOM deb http://packages.cloud.google.com/apt google-compute-engine-jessie-stable main deb http://packages.cloud.google.com/apt google-cloud-packages-archive-keyring-jessie main EOM
Install the packages to maintain the public key over time:
apt-get update; apt-get install google-cloud-packages-archive-keyring
Install the google-compute-engine
and python-google-compute-engine
packages:
apt-get update; apt-get install -y google-compute-engine python-google-compute-engine
For Debian 9, run the following commands as root:
Add the public repo key to your system:
curl https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add -
Add a source list file /etc/apt/sources.list.d/google-cloud.list
:
tee /etc/apt/sources.list.d/google-cloud.list << EOM deb http://packages.cloud.google.com/apt google-compute-engine-stretch-stable main deb http://packages.cloud.google.com/apt google-cloud-packages-archive-keyring-stretch main EOM
Install the packages to maintain the public key over time:
apt-get update; apt-get install google-cloud-packages-archive-keyring
Install the google-compute-engine
and python-google-compute-engine
packages:
apt-get update; apt-get install -y google-compute-engine python-google-compute-engine
For EL6 and EL7 based distributions, run the following commands as root:
Add the yum repo to a repo file /etc/yum.repos.d/google-cloud.repo
for either EL6 or EL7. Change DIST
to either 6 or 7 respectively:
DIST=7 tee /etc/yum.repos.d/google-cloud.repo << EOM [google-cloud-compute] name=Google Cloud Compute baseurl=https://packages.cloud.google.com/yum/repos/google-cloud-compute-el${DIST}-x86_64 enabled=1 gpgcheck=1 repo_gpgcheck=1 gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg EOM
Install the google-compute-engine
, python-google-compute-engine
packages:
yum install -y google-compute-engine python-google-compute-engine
Deprecated Packages
Deprecated Package | Replacement |
---|---|
google-compute-engine-jessie | google-compute-engine and python-google-compute-engine |
google-compute-engine-stretch | google-compute-engine and python-google-compute-engine |
google-compute-engine-init | google-compute-engine |
google-compute-engine-init-jessie | google-compute-engine |
google-compute-engine-init-stretch | google-compute-engine |
google-config | google-compute-engine |
google-config-jessie | google-compute-engine |
google-config-stretch | google-compute-engine |
google-compute-daemon | python-google-compute-engine |
google-startup-scripts | google-compute-engine |
An old CentOS 6 image fails to install the packages with an error on SCL
CentOS 6 images prior to v20160526
may fail to install the package with the error:
http://mirror.centos.org/centos/6/SCL/x86_64/repodata/repomd.xml: [Errno 14] PYCURL ERROR 22 - "The requested URL returned error: 404 Not Found"
Remove the stale repository file: sudo rm -f /etc/yum.repos.d/CentOS-SCL.repo
On some CentOS or RHEL 6 systems, extraneous python egg directories can cause the python daemons to fail.
In /usr/lib/python2.6/site-packages
look for google_compute_engine-2.4.1-py27.egg-info
directories and google_compute_engine-2.5.2.egg-info
directories and delete them if you run into this problem.
Using boto with virtualenv
Specific to running boto
inside of a Python virtualenv
, virtual environments are isolated from system site-packages. This includes the installed Linux guest environment libraries that are used to configure boto
credentials. There are two recommended solutions:
virtualenv venv --system-site-packages
.boto
via the Linux guest environment PyPI package using pip install google-compute-engine
.Have a patch that will benefit this project? Awesome! Follow these steps to have it accepted.
All files in this repository are under the Apache License, Version 2.0 unless noted otherwise.