[moblab] Copying mobmonitor code into platform/moblab
Copying code out of chromite, project-moblab into a central
place to make development and maintenance of mobmonitor easier
going forward. Removing all dependencies to chromite.
As a part of the move, re-implementing a few chromite util
functions and removing mobmoncli, rpc: they are not needed
for our use case.
BUG=chromium:813227
TEST=Rewrote unit tests to be chromite independent and continue
to provide test coverage. Updated ebuild files and created a
build to do manual testing on local device.
Change-Id: Iefc0016c5874188d901f4e6591c10575fbe07a8b
Reviewed-on: https://chromium-review.googlesource.com/933098
Commit-Ready: Matt Mallett <mattmallett@chromium.org>
Tested-by: Matt Mallett <mattmallett@chromium.org>
Reviewed-by: Keith Haddow <haddowk@chromium.org>
diff --git a/src/mobmonitor/README b/src/mobmonitor/README
new file mode 100644
index 0000000..ebdf104
--- /dev/null
+++ b/src/mobmonitor/README
@@ -0,0 +1,161 @@
+# Copyright 2018 The Chromium OS Authors. All rights reserved.
+# Use of this source code is governed by a BSD-style license that can be
+# found in the LICENSE file.
+----------------------------------
+----------------------------------
+Details on using the Mob* Monitor:
+----------------------------------
+----------------------------------
+
+
+Overview:
+---------
+
+The Mob* Monitor provides a way to monitor the health state of a particular
+service. Service health is defined by a set of satisfiable checks, called
+health checks.
+
+The Mob* Monitor executes health checks that are written for a particular
+service and collects information on the health state. Users can query
+the health state of a service via an RPC/RESTful interface.
+
+When a service is unhealthy, the Mob* Monitor can be requested to execute
+repair actions that are defined in the service's check file package.
+
+
+Check Files and Check File Packages:
+------------------------------------
+
+Check file packages are located in the check file directory. Each 'package'
+is a Python package.
+
+The layout of the checkfile directory is as follows:
+
+checkfile_directory:
+ service1:
+ __init__.py
+ service_actions.py
+ more_service_actions.py
+ easy_check.py
+ harder_check.py
+ ...
+ service2:
+ __init__.py
+ service2_actions.py
+ service_check.py
+ ....
+ .
+ .
+ .
+ serviceN:
+ ...
+
+Each service check file package should be flat, that is, no subdirectories will
+be walked to collect health checks.
+
+Check files define health checks and must end in '_check.py'. The Mob* Monitor
+does not enforce how or where in the package you define repair actions.
+
+
+Health Checks:
+--------------
+
+Health checks are the basic conditions that altogether define whether or not a
+service is healthy from the perspective of the Mob* Monitor.
+
+A health check is a python object that implements the following interface:
+
+ - Check()
+
+ Tests the health condition.
+
+ -> Returns 0 if the health check was completely satisfied.
+ -> Returns a positive integer if the check was successfuly, but could
+ have been better.
+ -> Returns a negative integer if the check was unsuccessful.
+
+ - Diagnose(errocode)
+
+ Maps an error code to a description and a set of actions that can be
+ used to repair or improve the condition.
+
+ -> Returns a tuple of (description, actions) where:
+ description is a string describing the state.
+ actions is a list of repair functions.
+
+
+Health checks can (optionally) also define the following attributes:
+
+ - CHECK_INTERVAL: Defines the interval (in seconds) between health check
+ executions. This defaults to 30 seconds if not defined.
+
+
+A check file may contain as many health checks as the writer feels is
+necessary. There is no restriction on what else may be included in the
+check file. The writer is free to write many health check files.
+
+
+Repair Actions:
+---------------
+
+Repair actions are used to repair or improve the health state of a service. The
+appropriate repair actions to take are returned in a health check's Diagnose
+method.
+
+Repair actions are functions and can be defined anywhere in the service check
+package.
+
+It is suggested that repair actions are defined in files ending in 'actions.py'
+which are imported by health check files.
+
+
+Health Check and Action Example:
+--------------------------------
+
+Suppose we have a service named 'myservice'. The check file package should have
+the following layout:
+
+checkdir:
+ myservice:
+ __init__.py
+ myservice_check.py
+ repair_actions.py
+
+
+The 'myservice_check.py' file should look like the following:
+
+ from myservice import repair_actions
+
+ def IsKeyFileInstalled():
+ """Checks if the key file is installed.
+
+ Returns:
+ True if USB key is plugged in, False otherwise.
+ """
+ ....
+ return result
+
+
+ class MyHealthCheck(object):
+
+ CHECK_INTERVAL = 10
+
+ def Check(self):
+ if IsKeyFileInstalled():
+ return 0
+
+ return -1
+
+ def Diagnose(self, errcode):
+ if -1 == errcode:
+ return ('Key file is missing.' [repair_actions.InstallKeyFile])
+
+ return ('Unknown failure.', [])
+
+
+And the 'repair_actions.py' file should look like:
+
+
+ def InstallKeyFile(**kwargs):
+ """Installs the key file."""
+ ...
diff --git a/src/mobmonitor/__init__.py b/src/mobmonitor/__init__.py
new file mode 100644
index 0000000..e69de29
--- /dev/null
+++ b/src/mobmonitor/__init__.py
diff --git a/src/mobmonitor/checkfile/__init__.py b/src/mobmonitor/checkfile/__init__.py
new file mode 100644
index 0000000..e69de29
--- /dev/null
+++ b/src/mobmonitor/checkfile/__init__.py
diff --git a/src/mobmonitor/checkfile/manager.py b/src/mobmonitor/checkfile/manager.py
new file mode 100644
index 0000000..7c34f1f
--- /dev/null
+++ b/src/mobmonitor/checkfile/manager.py
@@ -0,0 +1,507 @@
+# -*- coding: utf-8 -*-
+# Copyright 2015 The Chromium OS Authors. All rights reserved.
+# Use of this source code is governed by a BSD-style license that can be
+# found in the LICENSE file.
+
+"""Store and manage Mob* Monitor checkfiles."""
+
+from __future__ import print_function
+
+import cherrypy
+import collections
+import imp
+import inspect
+import os
+import time
+
+from cherrypy.process import plugins
+import logging
+
+
+LOGGER = logging.getLogger(__name__)
+
+
+HCEXECUTION_IN_PROGRESS = 0
+HCEXECUTION_COMPLETED = 1
+
+HCSTATUS_HEALTHY = 0
+
+IN_PROGRESS_DESCRIPTION = 'Health check is currently executing.'
+NULL_DESCRIPTION = ''
+EMPTY_ACTIONS = []
+HEALTHCHECK_STATUS = collections.namedtuple('healthcheck_status',
+ ['name', 'health', 'description',
+ 'actions'])
+
+HEALTH_CHECK_METHODS = ['Check', 'Diagnose']
+
+CHECK_INTERVAL_DEFAULT_SEC = 10
+HEALTH_CHECK_DEFAULT_ATTRIBUTES = {
+ 'CHECK_INTERVAL_SEC': CHECK_INTERVAL_DEFAULT_SEC}
+
+CHECKFILE_DIR = '/usr/local/moblab/mobmonitor/checkfiles/'
+CHECKFILE_ENDING = '_check.py'
+
+SERVICE_STATUS = collections.namedtuple('service_status',
+ ['service', 'health', 'healthchecks'])
+
+ACTION_INFO = collections.namedtuple('action_info',
+ ['action', 'info', 'args', 'kwargs'])
+
+
+class CollectionError(Exception):
+ """Raise when an error occurs during checkfile collection."""
+
+
+def MapHealthcheckStatusToDict(hcstatus):
+ """Map a manager.HEALTHCHECK_STATUS named tuple to a dictionary.
+
+ Args:
+ hcstatus: A HEALTHCHECK_STATUS object.
+
+ Returns:
+ A dictionary version of the HEALTHCHECK_STATUS object.
+ """
+ return {'name': hcstatus.name, 'health': hcstatus.health,
+ 'description': hcstatus.description,
+ 'actions': [a.__name__ for a in hcstatus.actions]}
+
+
+def MapServiceStatusToDict(status):
+ """Map a manager.SERVICE_STATUS named tuple to a dictionary.
+
+ Args:
+ status: A SERVICE_STATUS object.
+
+ Returns:
+ A dictionary version of the SERVICE_STATUS object.
+ """
+ hcstatuses = [
+ MapHealthcheckStatusToDict(hcstatus) for hcstatus in status.healthchecks]
+ return {'service': status.service, 'health': status.health,
+ 'healthchecks': hcstatuses}
+
+
+def MapActionInfoToDict(actioninfo):
+ return {'action': actioninfo.action, 'info': actioninfo.info,
+ 'args': actioninfo.args, 'kwargs': actioninfo.kwargs}
+
+
+def isHealthcheckHealthy(hcstatus):
+ """Test if a health check is perfectly healthy.
+
+ Args:
+ hcstatus: A HEALTHCHECK_STATUS named tuple.
+
+ Returns:
+ True if the hcstatus is perfectly healthy, False otherwise.
+ """
+ if not isinstance(hcstatus, HEALTHCHECK_STATUS):
+ return False
+ return hcstatus.health and not (hcstatus.description or hcstatus.actions)
+
+
+def isServiceHealthy(status):
+ """Test if a service is perfectly healthy.
+
+ Args:
+ status: A SERVICE_STATUS named tuple.
+
+ Returns:
+ True if the status is perfectly healthy, False otherwise.
+ """
+ if not isinstance(status, SERVICE_STATUS):
+ return False
+ return status.health and not status.healthchecks
+
+
+def DetermineHealthcheckStatus(hcname, healthcheck):
+ """Determine the healthcheck status.
+
+ Args:
+ hcname: A string. The name of the health check.
+ healthcheck: A healthcheck object.
+
+ Returns:
+ A HEALTHCHECK_STATUS named tuple.
+ """
+ try:
+ # Run the health check condition.
+ result = healthcheck.Check()
+
+ # Determine the healthcheck's status.
+ health = result >= HCSTATUS_HEALTHY
+
+ if result == HCSTATUS_HEALTHY:
+ return HEALTHCHECK_STATUS(hcname, health, NULL_DESCRIPTION,
+ EMPTY_ACTIONS)
+
+ description, actions = healthcheck.Diagnose(result)
+ return HEALTHCHECK_STATUS(hcname, health, description, actions)
+
+ except Exception as e:
+ # Checkfiles may contain all kinds of errors! We do not want the
+ # Mob* Monitor to fail, so catch generic exceptions.
+ LOGGER.error('Failed to execute health check %s: %s', hcname, e,
+ exc_info=True)
+ return HEALTHCHECK_STATUS(hcname, False,
+ 'Failed to execute the health check.'
+ ' Please review the health check file.',
+ EMPTY_ACTIONS)
+
+
+def IsHealthCheck(obj):
+ """A sanity check to see if a class implements the health check interface.
+
+ Args:
+ obj: A Python object.
+
+ Returns:
+ True if obj has 'check' and 'diagnose' functions.
+ False otherwise.
+ """
+ return all(callable(getattr(obj, m, None)) for m in HEALTH_CHECK_METHODS)
+
+
+def ApplyHealthCheckAttributes(obj):
+ """Set default values for health check attributes.
+
+ Args:
+ obj: A Python object.
+
+ Returns:
+ The same object with default attribute values set if they were not
+ already defined.
+ """
+ for attr, default in HEALTH_CHECK_DEFAULT_ATTRIBUTES.iteritems():
+ if not hasattr(obj, attr):
+ setattr(obj, attr, default)
+
+ return obj
+
+
+def ImportFile(service, modulepath):
+ """Import and collect health checks from the given module.
+
+ Args:
+ service: The name of the service this check module belongs to and
+ for which the objects to import belong to.
+ modulepath: The path of the module to import.
+
+ Returns:
+ A tuple containing the healthchecks defined in the module and the
+ time of the module's last modification.
+
+ Raises:
+ SyntaxError may be raised by imp.load_source if the python file
+ specified by modulepath has errors.
+ """
+ # Name and load the module from the module path. If service is 'testservice'
+ # and the module path is '/path/to/checkdir/testservice/test_check.py',
+ # the module name becomes 'testservice.test_check'.
+ objects = []
+ modname = '%s.%s' % (service,
+ os.path.basename(os.path.splitext(modulepath)[0]))
+ module = imp.load_source(modname, modulepath)
+
+ for name in dir(module):
+ obj = getattr(module, name)
+ if inspect.isclass(obj) and IsHealthCheck(obj):
+ objects.append(ApplyHealthCheckAttributes(obj()))
+
+ return objects, os.path.getmtime(modulepath)
+
+
+class CheckFileManager(object):
+ """Manage the health checks that are associated with each service."""
+
+ def __init__(self, interval_sec=1, checkdir=CHECKFILE_DIR):
+ if not os.path.exists(checkdir):
+ raise CollectionError('Check directory does not exist: %s' % checkdir)
+
+ self.interval_sec = interval_sec
+ self.checkdir = checkdir
+ self.monitor = None
+
+ # service_checks is a dict of the following form:
+ #
+ # {service_name: {hcname: (mtime, healthcheck)}}
+ #
+ # service_name: A string and is the name of the service.
+ # hcname: A string and is the name of the health check.
+ # mtime: The epoch time of the last modification of the check file.
+ # healthcheck: The health check object.
+ self.service_checks = {}
+
+ # service_check_results is a dict of the following form:
+ #
+ # {service_name: {hcname: (execution_status, exec_time,
+ # healthcheck_status)}}
+ #
+ # service_name: As above.
+ # hcname: As above.
+ # execution_status: An integer. This will be one of the HCEXECUTION
+ # variables defined at the top of the file.
+ # exec_time: The time of last execution.
+ # healthcheck_status: A HEALTHCHECK_STATUS named tuple.
+ self.service_check_results = {}
+
+ # service_states is dict of the following form:
+ #
+ # {service_name: service_status}
+ #
+ # service_name: As above.
+ # service_status: A SERVICE_STATUS named tuple.
+ self.service_states = {}
+
+ def Update(self, service, objects, mtime):
+ """Update the healthcheck objects for each service.
+
+ Args:
+ service: The service that the healthcheck corresponds to.
+ objects: A list of healthcheck objects.
+ mtime: The time of last modification of the healthcheck module.
+ """
+ for obj in objects:
+ name = obj.__class__.__name__
+ self.service_checks.setdefault(service, {})
+
+ stored_mtime, _ = self.service_checks[service].get(name, (None, None))
+ if stored_mtime is None or mtime > stored_mtime:
+ self.service_checks[service][name] = (mtime, obj)
+ LOGGER.info('Updated healthcheck "%s" for service "%s" at time "%s"',
+ name, service, mtime)
+
+ def Execute(self, force=False):
+ """Execute all health checks and collect healthcheck status information.
+
+ Args:
+ force: Ignore the health check interval and execute the health checks.
+ """
+ for service, healthchecks in self.service_checks.iteritems():
+ # Set default result dictionary if this is a new service.
+ self.service_check_results.setdefault(service, {})
+
+ for hcname, (_mtime, healthcheck) in healthchecks.iteritems():
+ # Update if the record is stale or non-existent.
+ etime = time.time()
+ _, exec_time, status = self.service_check_results[service].get(
+ hcname, (None, None, None))
+
+ if exec_time is None or force or (
+ etime > healthcheck.CHECK_INTERVAL_SEC + exec_time):
+ # Record the execution status.
+ status = HEALTHCHECK_STATUS(hcname, True, IN_PROGRESS_DESCRIPTION,
+ EMPTY_ACTIONS)
+ self.service_check_results[service][hcname] = (
+ HCEXECUTION_IN_PROGRESS, etime, status)
+
+ # TODO (msartori): Implement crbug.com/501959.
+ # This bug deals with handling slow health checks.
+
+ status = DetermineHealthcheckStatus(hcname, healthcheck)
+
+ # Update the execution and healthcheck status.
+ self.service_check_results[service][hcname] = (
+ HCEXECUTION_COMPLETED, etime, status)
+
+ def ConsolidateServiceStates(self):
+ """Consolidate health check results and determine service health states."""
+ for service, results in self.service_check_results.iteritems():
+ self.service_states.setdefault(service, {})
+
+ quasi_or_unhealthy_checks = []
+ for (_exec_status, _exec_stime, hcstatus) in results.itervalues():
+ if not isHealthcheckHealthy(hcstatus):
+ quasi_or_unhealthy_checks.append(hcstatus)
+
+ health = all([hc.health for hc in quasi_or_unhealthy_checks])
+
+ self.service_states[service] = SERVICE_STATUS(service, health,
+ quasi_or_unhealthy_checks)
+
+ def CollectionExecutionCallback(self):
+ """Callback for cherrypy Monitor. Collect checkfiles from the checkdir."""
+ # Find all service check file packages.
+ _, service_dirs, _ = next(os.walk(self.checkdir))
+ for service_name in service_dirs:
+ service_package = os.path.join(self.checkdir, service_name)
+
+ # Import the package.
+ try:
+ file_, path, desc = imp.find_module(service_name, [self.checkdir])
+ imp.load_module(service_name, file_, path, desc)
+ except Exception as e:
+ LOGGER.warning('Failed to import package %s: %s', service_name, e,
+ exc_info=True)
+ continue
+
+ # Collect all of the service's health checks.
+ for file_ in os.listdir(service_package):
+ filepath = os.path.join(service_package, file_)
+ if os.path.isfile(filepath) and file_.endswith(CHECKFILE_ENDING):
+ try:
+ healthchecks, mtime = ImportFile(service_name, filepath)
+ self.Update(service_name, healthchecks, mtime)
+ except Exception as e:
+ LOGGER.warning('Failed to import module %s.%s: %s',
+ service_name, file_[:-3], e,
+ exc_info=True)
+
+ self.Execute()
+ self.ConsolidateServiceStates()
+
+ def StartCollectionExecution(self):
+ # The Monitor frequency is mis-named. It's the time between
+ # each callback execution.
+ self.monitor = plugins.Monitor(cherrypy.engine,
+ self.CollectionExecutionCallback,
+ frequency=self.interval_sec)
+ self.monitor.subscribe()
+
+ def GetServiceList(self):
+ """Return a list of the monitored services.
+
+ Returns:
+ A list of the services for which we have checks defined.
+ """
+ return self.service_states.keys()
+
+ def GetStatus(self, service):
+ """Query the current health state of the service.
+
+ Args:
+ service: The name of service that we are querying the health state of.
+
+ Returns:
+ A SERVICE_STATUS named tuple which has the following fields:
+ service: A string. The service name.
+ health: A boolean. True if all checks passed, False if not.
+ healthchecks: A list of failed or quasi-healthy checks for the service.
+ Each member of the list is a HEALTHCHECK_STATUS and details the
+ appropriate repair actions for that particular health check.
+
+ If service is not specified, a list of all service states is returned.
+ """
+ if not service:
+ return self.service_states.values()
+
+ return self.service_states.get(service, SERVICE_STATUS(service, False, []))
+
+ def ActionInfo(self, service, healthcheck, action):
+ """Describes a currently valid action for the given service and healthcheck.
+
+ An action is valid if the following hold:
+ The |service| is recognized and is in an unhealthy or quasi-healthy state.
+ The |healthcheck| is recognized and is in an unhealthy or quasi-healthy
+ state and it belongs to |service|.
+ The |action| is one specified as a suitable repair action by the
+ Diagnose method of some non-healthy healthcheck of |service|.
+
+ Args:
+ service: A string. The name of a service being monitored.
+ healthcheck: A string. The name of a healthcheck belonging to |service|.
+ action: A string. The name of an action returned by some healthcheck's
+ Diagnose method.
+
+ Returns:
+ An ACTION_INFO named tuple which has the following fields:
+ action: A string. The given |action| string.
+ info: A string. The docstring of |action|.
+ args: A list of strings. The positional arguments for |action|.
+ kwargs: A dictionary representing the default keyword arguments
+ for |action|. The keys will be the kwarg names and the values
+ will be the default arguments.
+ """
+ status = self.service_states.get(service, None)
+ if not status:
+ return ACTION_INFO(action, 'Service not recognized.', [], {})
+ elif isServiceHealthy(status):
+ return ACTION_INFO(action, 'Service is healthy.', [], {})
+
+ hc = [x for x in status.healthchecks if x.name == healthcheck]
+ if not hc:
+ return ACTION_INFO(action, 'Healthcheck not recognized.', [], {})
+ hc = hc[0]
+ if isHealthcheckHealthy(hc):
+ return ACTION_INFO(action, 'Healthcheck is healthy.', [], {})
+
+ func = None
+ for a in hc.actions:
+ if a.__name__ == action:
+ func = a
+ break
+
+ if func is None:
+ return ACTION_INFO(action, 'Action not recognized.', [], {})
+
+ # Collect information on the repair action.
+ argspec = inspect.getargspec(func)
+ func_args = argspec.args or []
+ func_args = [x for x in func_args if x not in ['self', 'cls']]
+ func_defaults = argspec.defaults or {}
+
+ num_args = len(func_args)
+ num_defaults = len(func_defaults)
+
+ args = func_args[:num_args-num_defaults]
+ kwargs = dict(zip(func_args[num_args-num_defaults:], func_defaults))
+
+ info = func.__doc__
+
+ return ACTION_INFO(action, info, args, kwargs)
+
+ def RepairService(self, service, healthcheck, action, args, kwargs):
+ """Execute the repair action on the specified service.
+
+ Args:
+ service: The name of the service to be repaired.
+ healthcheck: The particular healthcheck we are repairing.
+ action: The name of the action to execute.
+ args: A list of positional arguments for the given repair action.
+ kwargs: A dictionary of keyword arguments for the given repair action.
+
+ Returns:
+ The same return value of GetStatus(service).
+ """
+ # No repair occurs if the service is not specified or is perfectly healthy.
+ status = self.service_states.get(service, None)
+ if status is None:
+ return SERVICE_STATUS(service, False, [])
+ elif isServiceHealthy(status):
+ return self.GetStatus(service)
+
+ # No repair occurs if the healthcheck is not specifed or perfectly healthy.
+ hc = [x for x in status.healthchecks if x.name == healthcheck]
+ if not hc or isHealthcheckHealthy(hc[0]):
+ return SERVICE_STATUS(healthcheck, False, [])
+ hc = hc[0]
+
+ # Get the repair action from the healthcheck.
+ repair_func = None
+ for a in hc.actions:
+ if a.__name__ == action:
+ repair_func = a
+ break
+
+ # TODO (msartori): Implement crbug.com/503373
+ if repair_func is not None:
+ try:
+ repair_func(*args, **kwargs)
+
+ # Update the service status and return.
+ # While actions are 'service-centric' from the perspective of the
+ # monitor, actions may have system-wide effect, so we must re-check
+ # all services.
+ self.Execute(force=True)
+ self.ConsolidateServiceStates()
+ except Exception, e:
+ LOGGER.error('Failed to execute the repair action "%s"'
+ ' for service "%s": %s', action, service, e,
+ exc_info=True)
+ else:
+ LOGGER.error('Failed to retrieve a suitable repair function for'
+ ' service="%s" and action="%s".', service, action,
+ exc_info=True)
+
+ return self.GetStatus(service)
diff --git a/src/mobmonitor/checkfile/manager_unittest.py b/src/mobmonitor/checkfile/manager_unittest.py
new file mode 100644
index 0000000..c3d41a5
--- /dev/null
+++ b/src/mobmonitor/checkfile/manager_unittest.py
@@ -0,0 +1,909 @@
+# -*- coding: utf-8 -*-
+# Copyright 2015 The Chromium OS Authors. All rights reserved.
+# Use of this source code is governed by a BSD-style license that can be
+# found in the LICENSE file.
+
+"""Unittests for Mob* Monitor checkfile manager."""
+
+from __future__ import print_function
+
+import imp
+import mock
+import os
+import subprocess
+import time
+import threading
+import unittest
+import mock
+
+from cherrypy.process import plugins
+import manager
+
+# Test health check and related attributes
+class TestHealthCheck(object):
+ """Test health check."""
+
+ def Check(self):
+ """Stub Check."""
+ return 0
+
+ def Diagnose(self, _errcode):
+ """Stub Diagnose."""
+ return ('Unknown Error.', [])
+
+
+class TestHealthCheckHasAttributes(object):
+ """Test health check with attributes."""
+
+ CHECK_INTERVAL_SEC = 10
+
+ def Check(self):
+ """Stub Check."""
+ return 0
+
+ def Diagnose(self, _errcode):
+ """Stub Diagnose."""
+ return ('Unknown Error.', [])
+
+
+class TestHealthCheckUnhealthy(object):
+ """Unhealthy test health check."""
+
+ def __init__(self):
+ self.x = -1
+
+ def Check(self):
+ """Stub Check."""
+ return self.x
+
+ def Diagnose(self, errcode):
+ """Stub Diagnose."""
+ if errcode == -1:
+ return ('Stub Error.', [self.Repair])
+ return ('Unknown Error.', [])
+
+ def Repair(self):
+ self.x = 0
+
+
+class TestHealthCheckMultipleActions(object):
+ """Unhealthy check with many actions that have different parameters."""
+
+ def __init__(self):
+ self.x = -1
+
+ def Check(self):
+ """Stub Check."""
+ return self.x
+
+ def Diagnose(self, errcode):
+ """Stub Diagnose."""
+ if errcode == -1:
+ return ('Stub Error.', [self.NoParams, self.PositionalParams,
+ self.DefaultParams, self.MixedParams])
+ return ('Unknown Error.', [])
+
+ def NoParams(self):
+ """NoParams Action."""
+ self.x = 0
+
+ # pylint: disable=unused-argument
+ def PositionalParams(self, x, y, z):
+ """PositionalParams Action."""
+ self.x = 0
+
+ def DefaultParams(self, x=1, y=2, z=3):
+ """DefaultParams Action."""
+ self.x = 0
+
+ def MixedParams(self, x, y, z=1):
+ """MixedParams Action."""
+ self.x = 0
+ # pylint: enable=unused-argument
+
+
+class TestHealthCheckQuasihealthy(object):
+ """Quasi-healthy test health check."""
+
+ def Check(self):
+ """Stub Check."""
+ return 1
+
+ def Diagnose(self, errcode):
+ """Stub Diagnose."""
+ if errcode == 1:
+ return ('Stub Error.', [self.RepairStub])
+ return ('Unknown Error.', [])
+
+ def RepairStub(self):
+ """Stub repair action."""
+
+
+class TestHealthCheckBroken(object):
+ """Broken test health check."""
+
+ def Check(self):
+ """Stub Check."""
+ raise ValueError()
+
+ def Diagnose(self, _errcode):
+ """A broken Diagnose function. A proper return should be a pair."""
+ raise ValueError()
+
+
+def TestAction():
+ return True
+
+
+TEST_SERVICE_NAME = 'test-service'
+TEST_MTIME = 100
+TEST_EXEC_TIME = 400
+CHECKDIR = '.'
+
+# Strings that are used to mock actual check modules.
+CHECKFILE_MANY_SIMPLE = '''
+SERVICE = 'test-service'
+
+class MyHealthCheck2(object):
+ def Check(self):
+ return 0
+
+ def Diagnose(self, errcode):
+ return ('Unknown error.', [])
+
+class MyHealthCheck3(object):
+ def Check(self):
+ return 0
+
+ def Diagnose(self, errcode):
+ return ('Unknown error.', [])
+
+class MyHealthCheck4(object):
+ def Check(self):
+ return 0
+
+ def Diagnose(self, errcode):
+ return ('Unknown error.', [])
+'''
+
+CHECKFILE_MANY_SIMPLE_ONE_BAD = '''
+SERVICE = 'test-service'
+
+class MyHealthCheck(object):
+ def Check(self):
+ return 0
+
+ def Diagnose(self, errcode):
+ return ('Unknown error.', [])
+
+class NotAHealthCheck(object):
+ def Diagnose(self, errcode):
+ return ('Unknown error.', [])
+
+class MyHealthCheck2(object):
+ def Check(self):
+ return 0
+
+ def Diagnose(self, errcode):
+ return ('Unknown error.', [])
+'''
+
+NOT_A_CHECKFILE = '''
+class NotAHealthCheck(object):
+ def NotCheckNorDiagnose(self):
+ return -1
+'''
+
+ANOTHER_NOT_A_CHECKFILE = '''
+class AnotherNotAHealthCheck(object):
+ def AnotherNotCheckNorDiagnose(self):
+ return -2
+'''
+
+ACTION_FILE = '''
+def TestAction():
+ return True
+
+def AnotherAction():
+ return False
+'''
+
+
+class RunCommand(threading.Thread):
+ """Helper class for executing the Mob* Monitor with a timeout."""
+
+ def __init__(self, cmd, timeout):
+ threading.Thread.__init__(self)
+ self.cmd = cmd
+ self.timeout = timeout
+ self.p = None
+
+ self.proc_stdout = None
+ self.proc_stderr = None
+
+ def run(self):
+ self.p = subprocess.Popen(self.cmd, stdout=subprocess.PIPE,
+ stderr=subprocess.STDOUT)
+ self.proc_stdout, self.proc_stderr = self.p.communicate()
+
+ def Stop(self):
+ self.join(self.timeout)
+
+ if self.is_alive():
+ self.p.terminate()
+ self.join(self.timeout)
+
+ if self.is_alive():
+ self.p.kill()
+ self.join(self.timeout)
+
+ return self.proc_stdout
+
+
+class CheckFileManagerHelperTest(unittest.TestCase):
+ """Unittests for CheckFileManager helper functions."""
+
+ def testMapHealthcheckStatusToDict(self):
+ """Test mapping a manager.HEALTHCHECK_STATUS to a dict."""
+ def _func():
+ pass
+
+ status = manager.HEALTHCHECK_STATUS('test', False, 'desc', [_func])
+ expect = {'name': 'test', 'health': False, 'description': 'desc',
+ 'actions': ['_func']}
+ self.assertEquals(expect, manager.MapHealthcheckStatusToDict(status))
+
+ def testMapServiceStatusToDict(self):
+ """Test mapping a manager.SERVICE_STATUS to a dict."""
+ def _func():
+ pass
+
+ hcstatus = manager.HEALTHCHECK_STATUS('test', False, 'desc', [_func])
+ hcexpect = {'name': 'test', 'health': False, 'description': 'desc',
+ 'actions': ['_func']}
+ status = manager.SERVICE_STATUS('test-service', False, [hcstatus])
+ expect = {'service': 'test-service', 'health': False,
+ 'healthchecks': [hcexpect]}
+ self.assertEquals(expect, manager.MapServiceStatusToDict(status))
+
+ def testMapActionInfoToDict(self):
+ """Test mapping a manager.ACTION_INFO to a dict."""
+ actioninfo = manager.ACTION_INFO('test', 'test', [1], {'a': 1})
+ expect = {'action': 'test', 'info': 'test', 'args': [1],
+ 'kwargs': {'a': 1}}
+ self.assertEquals(expect, manager.MapActionInfoToDict(actioninfo))
+
+ def testIsHealthcheckHealthy(self):
+ """Test checking whether health check statuses are healthy."""
+ # Test a healthy health check.
+ hch = manager.HEALTHCHECK_STATUS('healthy', True, manager.NULL_DESCRIPTION,
+ manager.EMPTY_ACTIONS)
+ self.assertTrue(manager.isHealthcheckHealthy(hch))
+
+ # Test a quasi-healthy health check.
+ hcq = manager.HEALTHCHECK_STATUS('quasi-healthy', True, 'Quasi-Healthy',
+ ['QuasiAction'])
+ self.assertFalse(manager.isHealthcheckHealthy(hcq))
+
+ # Test an unhealthy health check.
+ hcu = manager.HEALTHCHECK_STATUS('unhealthy', False, 'Unhealthy',
+ ['UnhealthyAction'])
+ self.assertFalse(manager.isHealthcheckHealthy(hcu))
+
+ # Test an object that is not a health check status.
+ s = manager.SERVICE_STATUS('service_status', True, [])
+ self.assertFalse(manager.isHealthcheckHealthy(s))
+
+ def testIsServiceHealthy(self):
+ """Test checking whether service statuses are healthy."""
+ # Define some health check statuses.
+ hch = manager.HEALTHCHECK_STATUS('healthy', True, manager.NULL_DESCRIPTION,
+ manager.EMPTY_ACTIONS)
+ hcq = manager.HEALTHCHECK_STATUS('quasi-healthy', True, 'Quasi-Healthy',
+ ['QuasiAction'])
+ hcu = manager.HEALTHCHECK_STATUS('unhealthy', False, 'Unhealthy',
+ ['UnhealthyAction'])
+
+ # Test a healthy service.
+ s = manager.SERVICE_STATUS('healthy', True, [])
+ self.assertTrue(manager.isServiceHealthy(s))
+
+ # Test a quasi-healthy service.
+ s = manager.SERVICE_STATUS('quasi-healthy', True, [hch, hcq])
+ self.assertFalse(manager.isServiceHealthy(s))
+
+ # Test an unhealthy service.
+ s = manager.SERVICE_STATUS('unhealthy', False, [hcu])
+ self.assertFalse(manager.isServiceHealthy(s))
+
+ # Test an object that is not a service status.
+ self.assertFalse(manager.isServiceHealthy(hch))
+
+ def testDetermineHealthcheckStatusHealthy(self):
+ """Test DetermineHealthCheckStatus on a healthy check."""
+ hcname = TestHealthCheck.__name__
+ testhc = TestHealthCheck()
+ expected = manager.HEALTHCHECK_STATUS(hcname, True,
+ manager.NULL_DESCRIPTION,
+ manager.EMPTY_ACTIONS)
+ self.assertEquals(expected,
+ manager.DetermineHealthcheckStatus(hcname, testhc))
+
+ def testDeterminHealthcheckStatusUnhealthy(self):
+ """Test DetermineHealthcheckStatus on an unhealthy check."""
+ hcname = TestHealthCheckUnhealthy.__name__
+ testhc = TestHealthCheckUnhealthy()
+ desc, actions = testhc.Diagnose(testhc.Check())
+ expected = manager.HEALTHCHECK_STATUS(hcname, False, desc, actions)
+ self.assertEquals(expected,
+ manager.DetermineHealthcheckStatus(hcname, testhc))
+
+ def testDetermineHealthcheckStatusQuasihealth(self):
+ """Test DetermineHealthcheckStatus on a quasi-healthy check."""
+ hcname = TestHealthCheckQuasihealthy.__name__
+ testhc = TestHealthCheckQuasihealthy()
+ desc, actions = testhc.Diagnose(testhc.Check())
+ expected = manager.HEALTHCHECK_STATUS(hcname, True, desc, actions)
+ self.assertEquals(expected,
+ manager.DetermineHealthcheckStatus(hcname, testhc))
+
+ def testDetermineHealthcheckStatusBrokenCheck(self):
+ """Test DetermineHealthcheckStatus raises on a broken health check."""
+ hcname = TestHealthCheckBroken.__name__
+ testhc = TestHealthCheckBroken()
+ result = manager.DetermineHealthcheckStatus(hcname, testhc)
+
+ self.assertEquals(hcname, result.name)
+ self.assertFalse(result.health)
+ self.assertFalse(result.actions)
+
+ def testIsHealthCheck(self):
+ """Test that IsHealthCheck properly asserts the health check interface."""
+
+ class NoAttrs(object):
+ """Test health check missing 'check' and 'diagnose' methods."""
+
+ class NoCheckAttr(object):
+ """Test health check missing 'check' method."""
+ def Diagnose(self, errcode):
+ pass
+
+ class NoDiagnoseAttr(object):
+ """Test health check missing 'diagnose' method."""
+ def Check(self):
+ pass
+
+ class GoodHealthCheck(object):
+ """Test health check that implements 'check' and 'diagnose' methods."""
+ def Check(self):
+ pass
+
+ def Diagnose(self, errcode):
+ pass
+
+ self.assertFalse(manager.IsHealthCheck(NoAttrs()))
+ self.assertFalse(manager.IsHealthCheck(NoCheckAttr()))
+ self.assertFalse(manager.IsHealthCheck(NoDiagnoseAttr()))
+ self.assertTrue(manager.IsHealthCheck(GoodHealthCheck()))
+
+ def testApplyHealthCheckAttributesNoAttrs(self):
+ """Test that we can apply attributes to a health check."""
+ testhc = TestHealthCheck()
+ result = manager.ApplyHealthCheckAttributes(testhc)
+ self.assertEquals(result.CHECK_INTERVAL_SEC,
+ manager.CHECK_INTERVAL_DEFAULT_SEC)
+
+ def testApplyHealthCheckAttributesHasAttrs(self):
+ """Test that we do not override an acceptable attribute."""
+ testhc = TestHealthCheckHasAttributes()
+ check_interval = testhc.CHECK_INTERVAL_SEC
+ result = manager.ApplyHealthCheckAttributes(testhc)
+ self.assertEquals(result.CHECK_INTERVAL_SEC, check_interval)
+
+ @mock.patch('manager.os')
+ @mock.patch('manager.imp')
+ def testImportFileAllHealthChecks(self, MockOs, MockImp):
+ """Test that health checks and service name are collected."""
+ manager.os.path.splitext.return_value = '/path/to/test_check.py'
+
+ manager.os.path.getmtime.return_value = TEST_MTIME
+
+ checkmodule = imp.new_module('test_check')
+ exec CHECKFILE_MANY_SIMPLE in checkmodule.__dict__
+ manager.imp.load_source.return_value = checkmodule
+
+
+ healthchecks, mtime = manager.ImportFile(TEST_SERVICE_NAME, '/')
+
+ self.assertEquals(len(healthchecks), 3)
+ self.assertEquals(mtime, TEST_MTIME)
+
+ @mock.patch('manager.os')
+ @mock.patch('manager.imp')
+ def testImportFileSomeHealthChecks(self, MockOs, MockImp):
+ """Test importing when not all classes are actually health checks."""
+ manager.os.path.splitext.return_value = '/path/to/test_check.py'
+
+ manager.os.path.getmtime.return_value = TEST_MTIME
+
+ checkmodule = imp.new_module('test_check')
+ exec CHECKFILE_MANY_SIMPLE_ONE_BAD in checkmodule.__dict__
+ manager.imp.load_source.return_value = checkmodule
+
+ healthchecks, mtime = manager.ImportFile(TEST_SERVICE_NAME, '/')
+
+ self.assertEquals(len(healthchecks), 2)
+ self.assertEquals(mtime, TEST_MTIME)
+
+
+class CheckFileManagerTest(unittest.TestCase):
+ """Unittests for CheckFileManager."""
+
+ @mock.patch('manager.os')
+ @mock.patch('manager.imp')
+ def testCollectionExecutionCallbackCheckfiles(self, MockOs, MockImp):
+ """Test the CollectionExecutionCallback on collecting checkfiles."""
+ manager.os.walk.return_value = iter([[CHECKDIR, [TEST_SERVICE_NAME], []]])
+ manager.os.listdir.return_value = ['test_check.py']
+ manager.os.path.isfile.return_value = True
+ manager.imp.find_module.return_value = (None, None, None)
+
+ manager.os.path.join.return_value = './%s/test_check.py' % TEST_SERVICE_NAME
+
+ myobj = TestHealthCheck()
+ manager.ImportFile = mock.Mock(return_value=[[myobj], TEST_MTIME])
+ cfm = manager.CheckFileManager(checkdir=CHECKDIR)
+ cfm.CollectionExecutionCallback()
+
+ manager.ImportFile.assert_called_once_with(
+ TEST_SERVICE_NAME, './%s/test_check.py' % TEST_SERVICE_NAME)
+
+ self.assertTrue(TEST_SERVICE_NAME in cfm.service_checks)
+ self.assertEquals(cfm.service_checks[TEST_SERVICE_NAME],
+ {myobj.__class__.__name__: (TEST_MTIME, myobj)})
+
+ @mock.patch('manager.os')
+ def testCollectionExecutionCallbackNoChecks(self, MockOs):
+ """Test the CollectionExecutionCallback with no valid check files."""
+ manager.os.walk.return_value = iter([['/checkdir/', [], ['test.py']]])
+
+ manager.ImportFile = mock.Mock(return_value=None)
+ cfm = manager.CheckFileManager(checkdir=CHECKDIR)
+ cfm.CollectionExecutionCallback()
+
+ self.assertFalse(manager.ImportFile.called)
+
+ self.assertFalse(TEST_SERVICE_NAME in cfm.service_checks)
+
+ def testStartCollectionExecution(self):
+ """Test the StartCollectionExecution method."""
+ plugins.Monitor = mock.Mock()
+
+ cfm = manager.CheckFileManager(checkdir=CHECKDIR)
+ cfm.StartCollectionExecution()
+
+ self.assertTrue(plugins.Monitor.called)
+
+ def testUpdateExistingHealthCheck(self):
+ """Test update when a health check exists and is not stale."""
+ cfm = manager.CheckFileManager(checkdir=CHECKDIR)
+
+ myobj = TestHealthCheck()
+
+ cfm.service_checks[TEST_SERVICE_NAME] = {myobj.__class__.__name__:
+ (TEST_MTIME, myobj)}
+
+ myobj2 = TestHealthCheck()
+ cfm.Update(TEST_SERVICE_NAME, [myobj2], TEST_MTIME)
+ self.assertTrue(TEST_SERVICE_NAME in cfm.service_checks)
+ self.assertEquals(cfm.service_checks[TEST_SERVICE_NAME],
+ {myobj.__class__.__name__: (TEST_MTIME, myobj)})
+
+ def testUpdateNonExistingHealthCheck(self):
+ """Test adding a new health check to the manager."""
+ cfm = manager.CheckFileManager(checkdir=CHECKDIR)
+ cfm.service_checks = {}
+
+ myobj = TestHealthCheck()
+ cfm.Update(TEST_SERVICE_NAME, [myobj], TEST_MTIME)
+
+ self.assertTrue(TEST_SERVICE_NAME in cfm.service_checks)
+ self.assertEquals(cfm.service_checks[TEST_SERVICE_NAME],
+ {myobj.__class__.__name__: (TEST_MTIME, myobj)})
+
+ @mock.patch('manager.time')
+ def testExecuteFresh(self, MockTime):
+ """Test executing a health check when the result is still fresh."""
+ exec_time_offset = TestHealthCheckHasAttributes.CHECK_INTERVAL_SEC / 2
+ manager.time.time.return_value = TEST_EXEC_TIME + exec_time_offset
+
+ cfm = manager.CheckFileManager(checkdir=CHECKDIR)
+ cfm.service_checks = {TEST_SERVICE_NAME:
+ {TestHealthCheckHasAttributes.__name__:
+ (TEST_MTIME, TestHealthCheckHasAttributes())}}
+ cfm.service_check_results = {
+ TEST_SERVICE_NAME: {TestHealthCheckHasAttributes.__name__:
+ (manager.HCEXECUTION_COMPLETED, TEST_EXEC_TIME,
+ None)}}
+
+ cfm.Execute()
+
+ _, exec_time, _ = cfm.service_check_results[TEST_SERVICE_NAME][
+ TestHealthCheckHasAttributes.__name__]
+
+ self.assertEquals(exec_time, TEST_EXEC_TIME)
+
+ @mock.patch('manager.time')
+ def testExecuteStale(self, MockTime):
+ """Test executing a health check when the result is stale."""
+ exec_time_offset = TestHealthCheckHasAttributes.CHECK_INTERVAL_SEC * 2
+ manager.time.time.return_value = TEST_EXEC_TIME + exec_time_offset
+
+ cfm = manager.CheckFileManager(checkdir=CHECKDIR)
+ cfm.service_checks = {TEST_SERVICE_NAME:
+ {TestHealthCheckHasAttributes.__name__:
+ (TEST_MTIME, TestHealthCheckHasAttributes())}}
+ cfm.service_check_results = {
+ TEST_SERVICE_NAME: {TestHealthCheckHasAttributes.__name__:
+ (manager.HCEXECUTION_COMPLETED, TEST_EXEC_TIME,
+ None)}}
+
+ cfm.Execute()
+
+ _, exec_time, _ = cfm.service_check_results[TEST_SERVICE_NAME][
+ TestHealthCheckHasAttributes.__name__]
+
+ self.assertEquals(exec_time, TEST_EXEC_TIME + exec_time_offset)
+
+ @mock.patch('manager.time')
+ def testExecuteNonExistent(self, MockTime):
+ """Test executing a health check when the result is nonexistent."""
+ manager.time.time.return_value = TEST_EXEC_TIME
+
+ cfm = manager.CheckFileManager(checkdir=CHECKDIR)
+ cfm.service_checks = {TEST_SERVICE_NAME:
+ {TestHealthCheck.__name__:
+ (TEST_MTIME, TestHealthCheck())}}
+
+ cfm.Execute()
+
+ resultsdict = cfm.service_check_results.get(TEST_SERVICE_NAME)
+ self.assertTrue(resultsdict is not None)
+
+ exec_status, exec_time, _ = resultsdict.get(TestHealthCheck.__name__,
+ (None, None, None))
+ self.assertTrue(exec_status is not None)
+ self.assertTrue(exec_time is not None)
+
+ self.assertEquals(exec_status, manager.HCEXECUTION_COMPLETED)
+ self.assertEquals(exec_time, TEST_EXEC_TIME)
+
+ @mock.patch('manager.time')
+ def testExecuteForce(self, MockTime):
+ """Test executing a health check by ignoring the check interval."""
+ exec_time_offset = TestHealthCheckHasAttributes.CHECK_INTERVAL_SEC / 2
+ manager.time.time.return_value = TEST_EXEC_TIME + exec_time_offset
+
+ cfm = manager.CheckFileManager(checkdir=CHECKDIR)
+ cfm.service_checks = {TEST_SERVICE_NAME:
+ {TestHealthCheckHasAttributes.__name__:
+ (TEST_MTIME, TestHealthCheckHasAttributes())}}
+ cfm.service_check_results = {
+ TEST_SERVICE_NAME: {TestHealthCheckHasAttributes.__name__:
+ (manager.HCEXECUTION_COMPLETED, TEST_EXEC_TIME,
+ None)}}
+
+ cfm.Execute(force=True)
+
+ _, exec_time, _ = cfm.service_check_results[TEST_SERVICE_NAME][
+ TestHealthCheckHasAttributes.__name__]
+
+ self.assertEquals(exec_time, TEST_EXEC_TIME + exec_time_offset)
+
+ def testConsolidateServiceStatesUnhealthy(self):
+ """Test consolidating state for a service with unhealthy checks."""
+ cfm = manager.CheckFileManager(checkdir=CHECKDIR)
+
+ # Setup some test check results
+ hcname = TestHealthCheck.__name__
+ statuses = [
+ manager.HEALTHCHECK_STATUS(hcname, False, 'Failed', ['Repair']),
+ manager.HEALTHCHECK_STATUS(hcname, True, 'Quasi', ['RepairQuasi']),
+ manager.HEALTHCHECK_STATUS(hcname, True, '', [])]
+
+ cfm.service_check_results.setdefault(TEST_SERVICE_NAME, {})
+ for i, status in enumerate(statuses):
+ name = '%s_%s' % (hcname, i)
+ cfm.service_check_results[TEST_SERVICE_NAME][name] = (
+ manager.HCEXECUTION_COMPLETED, TEST_EXEC_TIME,
+ status)
+
+ # Run and check the results.
+ cfm.ConsolidateServiceStates()
+ self.assertTrue(TEST_SERVICE_NAME in cfm.service_states)
+
+ _, health, healthchecks = cfm.service_states[TEST_SERVICE_NAME]
+ self.assertFalse(health)
+ self.assertEquals(2, len(healthchecks))
+ self.assertTrue(all([x in healthchecks for x in statuses[:2]]))
+
+ def testConsolidateServiceStatesQuasiHealthy(self):
+ """Test consolidating state for a service with quasi-healthy checks."""
+ cfm = manager.CheckFileManager(checkdir=CHECKDIR)
+
+ # Setup some test check results
+ hcname = TestHealthCheck.__name__
+ statuses = [
+ manager.HEALTHCHECK_STATUS(hcname, True, 'Quasi', ['RepairQuasi']),
+ manager.HEALTHCHECK_STATUS(hcname, True, '', [])]
+
+ cfm.service_check_results.setdefault(TEST_SERVICE_NAME, {})
+ for i, status in enumerate(statuses):
+ name = '%s_%s' % (hcname, i)
+ cfm.service_check_results[TEST_SERVICE_NAME][name] = (
+ manager.HCEXECUTION_COMPLETED, TEST_EXEC_TIME,
+ status)
+
+ # Run and check the results.
+ cfm.ConsolidateServiceStates()
+ self.assertTrue(TEST_SERVICE_NAME in cfm.service_states)
+
+ _, health, healthchecks = cfm.service_states[TEST_SERVICE_NAME]
+ self.assertTrue(health)
+ self.assertEquals(1, len(healthchecks))
+ self.assertTrue(statuses[0] in healthchecks)
+
+ def testConsolidateServiceStatesHealthy(self):
+ """Test consolidating state for a healthy service."""
+ cfm = manager.CheckFileManager(checkdir=CHECKDIR)
+
+ # Setup some test check results
+ hcname = TestHealthCheck.__name__
+ hcname2 = '%s_2' % hcname
+ statuses = [
+ manager.HEALTHCHECK_STATUS(hcname, True, '', []),
+ manager.HEALTHCHECK_STATUS(hcname2, True, '', [])]
+
+ cfm.service_check_results.setdefault(TEST_SERVICE_NAME, {})
+ cfm.service_check_results[TEST_SERVICE_NAME][hcname] = (
+ manager.HCEXECUTION_COMPLETED, TEST_EXEC_TIME, statuses[0])
+ cfm.service_check_results[TEST_SERVICE_NAME][hcname2] = (
+ manager.HCEXECUTION_IN_PROGRESS, TEST_EXEC_TIME, statuses[1])
+
+ # Run and check.
+ cfm.ConsolidateServiceStates()
+
+ self.assertTrue(TEST_SERVICE_NAME in cfm.service_states)
+
+ _, health, healthchecks = cfm.service_states.get(TEST_SERVICE_NAME)
+ self.assertTrue(health)
+ self.assertEquals(0, len(healthchecks))
+
+ def testGetServiceList(self):
+ """Test the GetServiceList RPC response."""
+ cfm = manager.CheckFileManager(checkdir=CHECKDIR)
+
+ self.assertEquals([], cfm.GetServiceList())
+
+ status = manager.SERVICE_STATUS(TEST_SERVICE_NAME, True, [])
+ cfm.service_states[TEST_SERVICE_NAME] = status
+
+ self.assertEquals([TEST_SERVICE_NAME], cfm.GetServiceList())
+
+ def testGetStatusNonExistent(self):
+ """Test the GetStatus RPC response when the service does not exist."""
+ cfm = manager.CheckFileManager(checkdir=CHECKDIR)
+
+ self.assertFalse(TEST_SERVICE_NAME in cfm.service_states)
+
+ status = manager.SERVICE_STATUS(TEST_SERVICE_NAME, False, [])
+ self.assertEquals(status, cfm.GetStatus(TEST_SERVICE_NAME))
+
+ def testGetStatusSingleService(self):
+ """Test the GetStatus RPC response for a single service."""
+ cfm = manager.CheckFileManager(checkdir=CHECKDIR)
+
+ s1name = TEST_SERVICE_NAME
+ s2name = '%s_2' % s1name
+ status1 = manager.SERVICE_STATUS(s1name, True, [])
+ status2 = manager.SERVICE_STATUS(s2name, True, [])
+ cfm.service_states[s1name] = status1
+ cfm.service_states[s2name] = status2
+
+ self.assertEquals(status1, cfm.GetStatus(s1name))
+ self.assertEquals(status2, cfm.GetStatus(s2name))
+
+ def testGetStatusAllServices(self):
+ """Test the GetStatus RPC response when no service is specified."""
+ cfm = manager.CheckFileManager(checkdir=CHECKDIR)
+
+ s1name = TEST_SERVICE_NAME
+ s2name = '%s_2' % s1name
+ status1 = manager.SERVICE_STATUS(s1name, True, [])
+ status2 = manager.SERVICE_STATUS(s2name, True, [])
+ cfm.service_states[s1name] = status1
+ cfm.service_states[s2name] = status2
+
+ result = cfm.GetStatus('')
+ self.assertEquals(2, len(result))
+ self.assertTrue(all([x in result for x in [status1, status2]]))
+
+ def testRepairServiceHealthy(self):
+ """Test the RepairService RPC when the service is healthy."""
+ cfm = manager.CheckFileManager(checkdir=CHECKDIR)
+
+ healthy_status = manager.SERVICE_STATUS(TEST_SERVICE_NAME, True, [])
+ cfm.service_states[TEST_SERVICE_NAME] = healthy_status
+
+ self.assertEquals(healthy_status, cfm.RepairService(TEST_SERVICE_NAME,
+ 'HealthcheckName',
+ 'RepairFuncName',
+ [], {}))
+
+ def testRepairServiceNonExistent(self):
+ """Test the RepairService RPC when the service does not exist."""
+ cfm = manager.CheckFileManager(checkdir=CHECKDIR)
+
+ self.assertFalse(TEST_SERVICE_NAME in cfm.service_states)
+
+ expected = manager.SERVICE_STATUS(TEST_SERVICE_NAME, False, [])
+ result = cfm.RepairService(TEST_SERVICE_NAME, 'DummyHealthcheck',
+ 'DummyAction', [], {})
+ self.assertEquals(expected, result)
+
+ def testRepairServiceInvalidAction(self):
+ """Test the RepairService RPC when the action is not recognized."""
+ cfm = manager.CheckFileManager(checkdir=CHECKDIR)
+
+ hcobj = TestHealthCheckUnhealthy()
+ cfm.service_checks[TEST_SERVICE_NAME] = {
+ hcobj.__class__.__name__: (TEST_MTIME, hcobj)}
+
+ unhealthy_status = manager.SERVICE_STATUS(
+ TEST_SERVICE_NAME, False,
+ [manager.HEALTHCHECK_STATUS(hcobj.__class__.__name__,
+ False, 'Always fails', [hcobj.Repair])])
+ cfm.service_states[TEST_SERVICE_NAME] = unhealthy_status
+
+ status = cfm.GetStatus(TEST_SERVICE_NAME)
+ self.assertFalse(status.health)
+ self.assertEquals(1, len(status.healthchecks))
+
+ status = cfm.RepairService(TEST_SERVICE_NAME, hcobj.__class__.__name__,
+ 'Blah', [], {})
+ self.assertFalse(status.health)
+ self.assertEquals(1, len(status.healthchecks))
+
+ def testRepairServiceInvalidActionArguments(self):
+ """Test the RepairService RPC when the action arguments are invalid."""
+ cfm = manager.CheckFileManager(checkdir=CHECKDIR)
+
+ hcobj = TestHealthCheckUnhealthy()
+ cfm.service_checks[TEST_SERVICE_NAME] = {
+ hcobj.__class__.__name__: (TEST_MTIME, hcobj)}
+
+ unhealthy_status = manager.SERVICE_STATUS(
+ TEST_SERVICE_NAME, False,
+ [manager.HEALTHCHECK_STATUS(hcobj.__class__.__name__,
+ False, 'Always fails', [hcobj.Repair])])
+ cfm.service_states[TEST_SERVICE_NAME] = unhealthy_status
+
+ status = cfm.GetStatus(TEST_SERVICE_NAME)
+ self.assertFalse(status.health)
+ self.assertEquals(1, len(status.healthchecks))
+
+ status = cfm.RepairService(TEST_SERVICE_NAME, hcobj.__class__.__name__,
+ 'Repair', [1, 2, 3], {})
+ self.assertFalse(status.health)
+ self.assertEquals(1, len(status.healthchecks))
+
+ def testRepairService(self):
+ """Test the RepairService RPC to repair an unhealthy service."""
+ cfm = manager.CheckFileManager(checkdir=CHECKDIR)
+
+ hcobj = TestHealthCheckUnhealthy()
+ cfm.service_checks[TEST_SERVICE_NAME] = {
+ hcobj.__class__.__name__: (TEST_MTIME, hcobj)}
+
+ unhealthy_status = manager.SERVICE_STATUS(
+ TEST_SERVICE_NAME, False,
+ [manager.HEALTHCHECK_STATUS(hcobj.__class__.__name__,
+ False, 'Always fails', [hcobj.Repair])])
+ cfm.service_states[TEST_SERVICE_NAME] = unhealthy_status
+
+ status = cfm.GetStatus(TEST_SERVICE_NAME)
+ self.assertFalse(status.health)
+ self.assertEquals(1, len(status.healthchecks))
+
+ status = cfm.RepairService(TEST_SERVICE_NAME,
+ hcobj.__class__.__name__,
+ hcobj.Repair.__name__,
+ [], {})
+ self.assertTrue(status.health)
+ self.assertEquals(0, len(status.healthchecks))
+
+ def testActionInfoServiceNonExistent(self):
+ """Test the ActionInfo RPC when the service does not exist."""
+ cfm = manager.CheckFileManager(checkdir=CHECKDIR)
+
+ self.assertFalse(TEST_SERVICE_NAME in cfm.service_states)
+
+ expect = manager.ACTION_INFO('test', 'Service not recognized.',
+ [], {})
+ result = cfm.ActionInfo(TEST_SERVICE_NAME, 'test', 'test')
+ self.assertEquals(expect, result)
+
+ def testActionInfoServiceHealthy(self):
+ """Test the ActionInfo RPC when the service is healthy."""
+ cfm = manager.CheckFileManager(checkdir=CHECKDIR)
+
+ healthy_status = manager.SERVICE_STATUS(TEST_SERVICE_NAME, True, [])
+ cfm.service_states[TEST_SERVICE_NAME] = healthy_status
+
+ expect = manager.ACTION_INFO('test', 'Service is healthy.',
+ [], {})
+ result = cfm.ActionInfo(TEST_SERVICE_NAME, 'test', 'test')
+ self.assertEquals(expect, result)
+
+ def testActionInfoActionNonExistent(self):
+ """Test the ActionInfo RPC when the action does not exist."""
+ cfm = manager.CheckFileManager(checkdir=CHECKDIR)
+
+ hcobj = TestHealthCheckUnhealthy()
+ cfm.service_checks[TEST_SERVICE_NAME] = {
+ hcobj.__class__.__name__: (TEST_MTIME, hcobj)}
+
+ unhealthy_status = manager.SERVICE_STATUS(
+ TEST_SERVICE_NAME, False,
+ [manager.HEALTHCHECK_STATUS(hcobj.__class__.__name__,
+ False, 'Always fails', [hcobj.Repair])])
+ cfm.service_states[TEST_SERVICE_NAME] = unhealthy_status
+
+ expect = manager.ACTION_INFO('test', 'Action not recognized.', [], {})
+ result = cfm.ActionInfo(TEST_SERVICE_NAME, hcobj.__class__.__name__,
+ 'test')
+ self.assertEquals(expect, result)
+
+ def testActionInfo(self):
+ """Test the ActionInfo RPC to collect information on a repair action."""
+ cfm = manager.CheckFileManager(checkdir=CHECKDIR)
+
+ hcobj = TestHealthCheckMultipleActions()
+ hcname = hcobj.__class__.__name__
+ actions = [hcobj.NoParams, hcobj.PositionalParams, hcobj.DefaultParams,
+ hcobj.MixedParams]
+
+ cfm.service_checks[TEST_SERVICE_NAME] = {hcname: (TEST_MTIME, hcobj)}
+
+ unhealthy_status = manager.SERVICE_STATUS(
+ TEST_SERVICE_NAME, False,
+ [manager.HEALTHCHECK_STATUS(hcname, False, 'Always fails', actions)])
+ cfm.service_states[TEST_SERVICE_NAME] = unhealthy_status
+
+ # Test ActionInfo when the action has no parameters.
+ expect = manager.ACTION_INFO('NoParams', 'NoParams Action.', [], {})
+ self.assertEquals(expect,
+ cfm.ActionInfo(TEST_SERVICE_NAME, hcname, 'NoParams'))
+
+ # Test ActionInfo when the action has only positional parameters.
+ expect = manager.ACTION_INFO('PositionalParams', 'PositionalParams Action.',
+ ['x', 'y', 'z'], {})
+ self.assertEquals(expect,
+ cfm.ActionInfo(TEST_SERVICE_NAME,
+ hcname, 'PositionalParams'))
+
+ # Test ActionInfo when the action has only default parameters.
+ expect = manager.ACTION_INFO('DefaultParams', 'DefaultParams Action.',
+ [], {'x': 1, 'y': 2, 'z': 3})
+ self.assertEquals(expect,
+ cfm.ActionInfo(TEST_SERVICE_NAME,
+ hcname, 'DefaultParams'))
+
+ # Test ActionInfo when the action has positional and default parameters.
+ expect = manager.ACTION_INFO('MixedParams', 'MixedParams Action.',
+ ['x', 'y'], {'z': 1})
+ self.assertEquals(expect, cfm.ActionInfo(TEST_SERVICE_NAME,
+ hcname, 'MixedParams'))
+
+if __name__ == '__main__':
+ unittest.main()
diff --git a/src/mobmonitor/checkfiles/devserver/__init__.py b/src/mobmonitor/checkfiles/devserver/__init__.py
new file mode 100644
index 0000000..e69de29
--- /dev/null
+++ b/src/mobmonitor/checkfiles/devserver/__init__.py
diff --git a/src/mobmonitor/checkfiles/devserver/gs_check.py b/src/mobmonitor/checkfiles/devserver/gs_check.py
new file mode 100644
index 0000000..495e1a5
--- /dev/null
+++ b/src/mobmonitor/checkfiles/devserver/gs_check.py
@@ -0,0 +1,129 @@
+# Copyright 2015 The Chromium OS Authors. All rights reserved.
+# Use of this source code is governed by a BSD-style license that can be
+# found in the LICENSE file.
+
+"""Google Storage health checks for devserver."""
+
+from __future__ import print_function
+
+import netifaces
+import os
+import ConfigParser
+
+from util import osutils
+
+
+GLOBAL_CONFIG = '/usr/local/autotest/global_config.ini'
+MOBLAB_CONFIG = '/usr/local/autotest/moblab_config.ini'
+SHADOW_CONFIG = '/usr/local/autotest/shadow_config.ini'
+
+GSUTIL_TIMEOUT_SEC = 5
+GSUTIL_USER = 'moblab'
+GSUTIL_INTERNAL_BUCKETS = ['gs://chromeos-image-archive']
+MOBLAB_SUBNET_ADDR = '192.168.231.1'
+
+
+def GetIp():
+ for iface in netifaces.interfaces():
+ if 'eth' not in iface:
+ continue
+ addrs = netifaces.ifaddresses(iface).get(netifaces.AF_INET)
+ if not addrs:
+ continue
+ for addr in addrs:
+ if MOBLAB_SUBNET_ADDR != addr.get('addr'):
+ return addr['addr']
+
+ return 'localhost'
+
+
+class GsBucket(object):
+ """Verify that we have access to the correct Google Storage bucket."""
+
+ def __init__(self):
+ self.bucket = None
+
+ def BucketReachable(self, gs_url):
+ """Check if we can reach the image server.
+
+ Args:
+ gs_url: The url of the Google Storage image server.
+
+ Returns:
+ True if |gs_url| can be reached.
+ False otherwise.
+ """
+ # If our boto key is non-existent or is not configured correctly
+ # for access to the right bucket, this check may take ~45 seconds.
+ # This really ruins the monitor's performance as we do not yet
+ # intelligently handle long running checks.
+ #
+ # Additionally, we cannot use chromite's timeout_util as it is
+ # based on Python's signal module, which cannot be used outside
+ # of the main thread. These checks are executed in a separate
+ # thread handled by the monitor.
+ #
+ # To handle this, we rely on the linux utility 'timeout' and
+ # forcefully kill our gsutil invocation if it is taking too long.
+
+ cmd = ['timeout', '-s', '9', str(GSUTIL_TIMEOUT_SEC),
+ 'gsutil', 'ls', '-b', gs_url]
+ try:
+ osutils.sudo_run_command(cmd, user=GSUTIL_USER)
+ except osutils.RunCommandError:
+ return False
+
+ return True
+
+ def Check(self):
+ """Verifies Google storage bucket access.
+
+ Returns:
+ 0 if we can access the correct bucket for our given Boto key.
+ -1 if a configuration file could not be found.
+ -2 if the configuration files did not contain the appropriate
+ information.
+ """
+ config_files = [GLOBAL_CONFIG, MOBLAB_CONFIG, SHADOW_CONFIG]
+ for f in config_files:
+ if not os.path.exists(f):
+ return -1
+
+ config = ConfigParser.ConfigParser()
+ config.read(config_files)
+
+ gs_url = None
+
+ for section in config.sections():
+ for option, value in config.items(section):
+ if 'image_storage_server' == option:
+ gs_url = value
+ break
+
+ if not (gs_url and self.BucketReachable(gs_url)):
+ self.bucket = gs_url
+ return -2
+
+ if gs_url in GSUTIL_INTERNAL_BUCKETS:
+ self.bucket = gs_url
+ return 1
+
+ return 0
+
+ def Diagnose(self, errcode):
+ if 1 == errcode:
+ return ('Using an internal Google Storage bucket %s' % self.bucket, [])
+
+ elif -1 == errcode:
+ return ('An autotest configuration file is missing.', [])
+
+ elif -2 == errcode:
+ return ('Moblab is not configured to access Google Storage.'
+ ' The current bucket name is set to %s. Please'
+ ' navigate to http://%s/moblab_setup/ and update'
+ ' the image_storage_server variable. For more information,'
+ ' please see https://www.chromium.org/chromium-os/testing/'
+ 'moblab/setup#TOC-Setting-up-the-boto-key-for-partners' % (
+ self.bucket, GetIp()), [])
+
+ return ('Unknown error reached with error code: %s' % errcode, [])
diff --git a/src/mobmonitor/checkfiles/moblab/__init__.py b/src/mobmonitor/checkfiles/moblab/__init__.py
new file mode 100644
index 0000000..e69de29
--- /dev/null
+++ b/src/mobmonitor/checkfiles/moblab/__init__.py
diff --git a/src/mobmonitor/checkfiles/moblab/autotest_common.py b/src/mobmonitor/checkfiles/moblab/autotest_common.py
new file mode 100644
index 0000000..ea539d0
--- /dev/null
+++ b/src/mobmonitor/checkfiles/moblab/autotest_common.py
@@ -0,0 +1,50 @@
+#!/usr/bin/python
+#
+# Copyright (c) 2014 The Chromium OS Authors. All rights reserved.
+# Use of this source code is governed by a BSD-style license that can be
+# found in the LICENSE file.
+
+"""A helper module to set up sys.path so that autotest_lib.* can be located."""
+
+# pylint: disable=F0401
+
+import os
+import re
+import sys
+
+import common # pylint:disable=W0611
+
+
+def InCrOSDevice():
+ """Returns True if running on a Chrome OS device."""
+ if not os.path.exists('/etc/lsb-release'):
+ return False
+ with open('/etc/lsb-release') as f:
+ lsb_release = f.read()
+ return re.match(r'^CHROMEOS_RELEASE', lsb_release, re.MULTILINE) is not None
+
+
+def import_autotest():
+ if InCrOSDevice():
+ autotest_dir = '/usr/local/autotest'
+ else:
+ CROS_WORKON_SRCROOT = os.environ['CROS_WORKON_SRCROOT']
+ autotest_dir = os.path.join(
+ CROS_WORKON_SRCROOT, 'src', 'third_party', 'autotest', 'files')
+
+ if os.path.isdir(autotest_dir):
+ oldpath = sys.path[:]
+ sys.path.insert(0, os.path.join(autotest_dir, 'client'))
+ import setup_modules
+ sys.path.pop(0)
+ setup_modules.setup(base_path=autotest_dir, root_module_name='autotest_lib')
+ try:
+ from autotest_lib.site_utils import cloud_console_client
+ from autotest_lib.client.common_lib import global_config
+ return True
+ except ImportError:
+ #The autotest is not ready yet
+ sys.path = oldpath
+ return False
+
+import_autotest()
diff --git a/src/mobmonitor/checkfiles/moblab/boto_check.py b/src/mobmonitor/checkfiles/moblab/boto_check.py
new file mode 100644
index 0000000..e6fc853
--- /dev/null
+++ b/src/mobmonitor/checkfiles/moblab/boto_check.py
@@ -0,0 +1,61 @@
+# Copyright 2015 The Chromium OS Authors. All rights reserved.
+# Use of this source code is governed by a BSD-style license that can be
+# found in the LICENSE file.
+
+"""Boto file health checks for devserver."""
+
+from __future__ import print_function
+
+import netifaces
+import os
+
+import common
+
+
+def GetIp():
+ for iface in netifaces.interfaces():
+ if 'eth' not in iface:
+ continue
+ addrs = netifaces.ifaddresses(iface).get(netifaces.AF_INET)
+ if not addrs:
+ continue
+ for addr in addrs:
+ if common.MOBLAB_SUBNET_ADDR != addr.get('addr'):
+ return addr['addr']
+
+ return 'localhost'
+
+
+class BotoFile(object):
+ """Verifies that a Boto key exists."""
+
+ def Check(self):
+ """Verifies that moblab has a Boto for proper gs_offloader functioning.
+
+ Returns:
+ 0 if a Boto key exists.
+ -1 if a Boto key does not exist.
+ -2 if the Boto key is empty.
+ """
+ if not os.path.exists(common.BOTO_PATH):
+ return -1
+
+ if 0 == os.stat(common.BOTO_PATH).st_size:
+ return -2
+
+ return 0
+
+ def Diagnose(self, errcode):
+ msg = ('Boto key is %s. Please upload a valid key'
+ ' at the following address:'
+ ' http://%s/moblab_setup/. For more information'
+ ' please see https://www.chromium.org/chromium-os/testing/'
+ 'moblab/setup#TOC-Setting-up-the-boto-key-for-partners')
+
+ if -1 == errcode:
+ return (msg % ('missing', GetIp()), [])
+
+ elif -2 == errcode:
+ return (msg % ('empty', GetIp()), [])
+
+ return ('Unknown error reached with error code: %s' % errcode, [])
diff --git a/src/mobmonitor/checkfiles/moblab/common.py b/src/mobmonitor/checkfiles/moblab/common.py
new file mode 100644
index 0000000..66b6478
--- /dev/null
+++ b/src/mobmonitor/checkfiles/moblab/common.py
@@ -0,0 +1,31 @@
+# Copyright 2015 The Chromium OS Authors. All rights reserved.
+# Use of this source code is governed by a BSD-style license that can be
+# found in the LICENSE file.
+
+"""Common lib for moblab checkfiles."""
+
+from __future__ import print_function
+
+
+#
+# Boto file constants.
+#
+BOTO_PATH = '/home/moblab/.boto'
+
+#
+# disk_check constants.
+#
+DISK_SIZE_BYTES = 32000000000
+
+MOBLAB_LABEL = 'MOBLAB-STORAGE'
+
+STATEFUL_MOUNTPOINT = '/mnt/stateful_partition'
+MOBLAB_MOUNTPOINT = '/mnt/moblab'
+MOBLAB_FILESYSTEM = 'ext4'
+
+DISK_FREE_THRESHOLD_PERCENT = 0.1
+
+#
+# Networking constants.
+#
+MOBLAB_SUBNET_ADDR = '192.168.231.1'
diff --git a/src/mobmonitor/checkfiles/moblab/disk_check.py b/src/mobmonitor/checkfiles/moblab/disk_check.py
new file mode 100644
index 0000000..7aba8af
--- /dev/null
+++ b/src/mobmonitor/checkfiles/moblab/disk_check.py
@@ -0,0 +1,206 @@
+# Copyright 2015 The Chromium OS Authors. All rights reserved.
+# Use of this source code is governed by a BSD-style license that can be
+# found in the LICENSE file.
+
+"""Disk monitoring health checks for moblab."""
+
+from __future__ import print_function
+
+import copy
+
+import moblab_actions
+import common
+
+from util import osutils
+from system import systeminfo
+
+
+class DiskDevices(object):
+ """Verifies that disks on moblab are set up correctly."""
+
+ DISK_INFO_VALID_SEC = 5
+
+ def __init__(self):
+ self.diskinfo = systeminfo.GetDisk(self.DISK_INFO_VALID_SEC)
+ self.useable_usbs = []
+
+ def CheckStateful(self):
+ """Helper check function for verifying the stateful partition.
+
+ Returns:
+ 0 if a sufficient stateful partition is exists.
+ -1 if no devices are found to be mounted at STATEFUL_MOUNTPOINT.
+ -2 if the stateful partition is too small.
+ """
+ diskpart = None
+ for part in self.diskinfo.DiskPartitions():
+ if part.mountpoint == common.STATEFUL_MOUNTPOINT:
+ diskpart = part
+
+ if not diskpart:
+ return -1
+
+ diskusage = self.diskinfo.DiskUsage(diskpart.mountpoint)
+ if diskusage.total < common.DISK_SIZE_BYTES:
+ return -2
+
+ return 0
+
+ def CheckUsb(self):
+ """Helper check function for verifying an adequate USB drive exists.
+
+ Returns:
+ 0 if a sufficient USB drive is exists.
+ -3 if no USB devices are found on the system.
+ -4 if no USB device is found to be of the appropriate size.
+ -5 if no USB device is found to be appropriately labelled.
+ -6 if no USB device is found to be formatted correctly.
+ -7 if no USB device is found to be mounted correctly.
+ """
+ # Get a list of all usb block devices.
+ usbs = [db for db in self.diskinfo.BlockDevices()
+ if any(x.startswith('usb') for x in db.ids)]
+
+ if not usbs:
+ return -3
+
+ # Get all well-sized usb block devices.
+ usbs = [usb for usb in usbs if usb.size >= common.DISK_SIZE_BYTES]
+
+ if not usbs:
+ return -4
+
+ # Get all well-labelled usb block devices.
+ self.useable_usbs = copy.copy(usbs)
+ usbs = [usb for usb in usbs
+ if any(x == common.MOBLAB_LABEL for x in usb.labels)]
+
+ if not usbs:
+ return -5
+
+ # Get all usbs formatted with the correct filesystem.
+ self.useable_usbs = copy.copy(usbs)
+ usbs = []
+ for usb in self.useable_usbs:
+ cmd = ['file', '-L', '-s', usb.device]
+ output = osutils.sudo_run_command(
+ cmd, error_code_ok=True).strip()
+ if common.MOBLAB_FILESYSTEM in output:
+ usbs.append(usb)
+
+ if not usbs:
+ return -6
+
+ # Get a list of mounts on this system and check if we have a usb
+ # mounted in the correctly.
+ self.useable_usbs = copy.copy(usbs)
+ mounts = self.diskinfo.DiskPartitions()
+ usbs = [usb for usb in usbs
+ if any(x.devicename == usb.device
+ and x.filesystem == common.MOBLAB_FILESYSTEM
+ for x in mounts)]
+
+ if not usbs:
+ return -7
+
+ # Looks like an adequate usb device exists!
+ return 0
+
+ def Check(self):
+ """Verifies that disk devices are setup correctly on this moblab.
+
+ Appropriate storage space may be satisfied by one of the following
+ conditions:
+
+ (1) A partition exists that is:
+ (i) mounted at /mnt/stateful_partition
+ (ii) >32GB in size
+
+ (2) A USB drive exists that is:
+ (i) mounted at /mnt/moblab
+ (ii) labelled MOBLAB-STORAGE
+ (iii) >32GB in size
+
+ Returns:
+ 0 if this moblab has appropriate disk space.
+ """
+ stateful_check = self.CheckStateful()
+ usb_check = self.CheckUsb()
+
+ if any(not x for x in [stateful_check, usb_check]):
+ return 0
+
+ if -3 == usb_check:
+ return stateful_check
+
+ return usb_check
+
+ def Diagnose(self, errcode):
+ if -1 == errcode:
+ return ('Critical error. No stateful partition was detected.', [])
+
+ elif -2 == errcode or -3 == errcode:
+ return ('Moblab internal storage is too small. Please insert'
+ ' a USB drive that is at least 32GB in size.', [])
+
+ elif -4 == errcode:
+ return ('All detected USB drives are too small. Please insert'
+ ' a USB drive that is at least 32GB in size.', [])
+
+ elif -5 == errcode:
+ return ('USB drive is not correctly labelled.'
+ ' Please review the setup instructions and label'
+ ' your USB drive. For more information please see'
+ ' https://www.chromium.org/chromium-os/testing/moblab/'
+ 'setup#TOC-Formatting-External-Storage-for-MobLab',
+ [])
+
+ elif -6 == errcode:
+ return ('USB drive is not formatted correctly.'
+ ' Please review the setup instructions and format'
+ ' your USB drive. For more information please see'
+ ' https://www.chromium.org/chromium-os/testing/moblab/'
+ 'setup#TOC-Formatting-External-Storage-for-MobLab',
+ [])
+
+ elif -7 == errcode:
+ usb = self.useable_usbs[0]
+ return ('USB drive is not mounted correctly.',
+ [moblab_actions.MountUsbWrapper(usb.device)])
+
+ return ('Unknown error reached with error code: %s' % errcode, [])
+
+
+class DiskSpace(object):
+ """Verifies that this moblab has enough remaining disk space."""
+
+ DISK_INFO_VALID_SEC = 5
+
+ def __init__(self):
+ self.diskinfo = systeminfo.GetDisk(self.DISK_INFO_VALID_SEC)
+
+ def Check(self):
+ """Verifies that moblab has enough free disk space.
+
+ Returns:
+ 0 if moblab has a sufficient amount of remaining disk space.
+ -1 if the moblab has <DISK_FREE_THRESHOLD_PERCENT free disk space
+ on either the moblab or stateful partition.
+ """
+ partitions = [common.MOBLAB_MOUNTPOINT, common.STATEFUL_MOUNTPOINT]
+ for part in partitions:
+ diskusage = self.diskinfo.DiskUsage(part)
+ free = 1 - diskusage.percent_used / 100
+ if free < common.DISK_FREE_THRESHOLD_PERCENT:
+ return -1
+
+ return 0
+
+ def Diagnose(self, errcode):
+ if -1 == errcode:
+ return ('Moblab has less than %s percent free disk space. Please'
+ ' free space to ensure proper functioning of your'
+ ' moblab.' % int(common.DISK_FREE_THRESHOLD_PERCENT * 100),
+ [moblab_actions.ClearOldDbRecords, moblab_actions.ClearLogs])
+
+ return ('Unknown error reached with error code: %s' % errcode, [])
diff --git a/src/mobmonitor/checkfiles/moblab/dut_check.py b/src/mobmonitor/checkfiles/moblab/dut_check.py
new file mode 100644
index 0000000..a295e49
--- /dev/null
+++ b/src/mobmonitor/checkfiles/moblab/dut_check.py
@@ -0,0 +1,65 @@
+# Copyright 2015 The Chromium OS Authors. All rights reserved.
+# Use of this source code is governed by a BSD-style license that can be
+# found in the LICENSE file.
+
+"""DUT health checks for moblab."""
+
+from __future__ import print_function
+
+import netifaces
+
+import common
+
+from util import osutils
+
+
+class DutExists(object):
+ """Verifies that a DUT is connected to moblab."""
+
+ def Check(self):
+ """Verify that at least one DUT is connected.
+
+ Returns:
+ 0 if at least one DUT exists on moblab's test subnet.
+ -1 if the moblab test subnet cannot be found.
+ -2 if no DUTs can be found.
+ """
+ # First, find the network interface the test subnet is connected to.
+ def FindInterface():
+ for iface in netifaces.interfaces():
+ addrs = netifaces.ifaddresses(iface).get(netifaces.AF_INET)
+ if not addrs:
+ continue
+ for addr in addrs:
+ if common.MOBLAB_SUBNET_ADDR == addr.get('addr'):
+ return addr['addr']
+
+ subnet_iface = FindInterface()
+
+ if not subnet_iface:
+ return -1
+
+ # Send ICMP ECHO_REQUESTs to DUTs.
+ cmd = ['fping', '-c', '1', '-g', '192.168.231.100', '192.168.231.120']
+ osutils.sudo_run_command(cmd, error_code_ok=True)
+
+ # Check ARP cache to determine which, if any, DUTs exist.
+ duts = []
+ arpcache = osutils.sudo_run_command(['arp']).strip()
+ for dut in arpcache.split('\n'):
+ if subnet_iface in dut and 'incomplete' not in dut:
+ duts.append(dut)
+
+ if not duts:
+ return -2
+
+ return 0
+
+ def Diagnose(self, errcode):
+ if -1 == errcode:
+ return ('Moblab test subnet interface could not be found.', [])
+
+ elif -2 == errcode:
+ return ('No DUTs were detected. Please plug in a device for test.', [])
+
+ return ('Unknown error reached with error code: %s' % errcode, [])
diff --git a/src/mobmonitor/checkfiles/moblab/heartbeat_check.py b/src/mobmonitor/checkfiles/moblab/heartbeat_check.py
new file mode 100644
index 0000000..99fda2d
--- /dev/null
+++ b/src/mobmonitor/checkfiles/moblab/heartbeat_check.py
@@ -0,0 +1,161 @@
+# Copyright 2015 The Chromium OS Authors. All rights reserved.
+# Use of this source code is governed by a BSD-style license that can be
+# found in the LICENSE file.
+
+"""Heartbeat health checks for moblab."""
+
+from __future__ import print_function
+
+import datetime
+import os
+import logging
+
+LOGGER = logging.getLogger(__name__)
+
+_MIN_HEARTBEAT_RATE=10
+
+import autotest_common
+
+# Needs the following import for unittests.
+try:
+ from autotest_lib.site_utils import cloud_console_client
+ from autotest_lib.client.common_lib import global_config
+except ImportError:
+ LOGGER.info('Autotest is not ready at loading.')
+
+# Abstract how the cache file is configured.
+def _GetHeartbeatCacheFile():
+ return global_config.global_config.get_config_value(
+ 'CROS', 'heartbeat_cache_file', default='/mnt/moblab/.HEARTBEAT')
+
+# Abstract how the rate is configured.
+def _GetHeartbeatRate():
+ return global_config.global_config.get_config_value(
+ 'CROS', 'heartbeat_rate_seconds', type=int, default=-1)
+
+def _GetLastHeartbeatTimeFromFile():
+ """Reads from the last heartbeat timestamp from cache file.
+
+ @returns A datetime object.
+ """
+ cache_file = _GetHeartbeatCacheFile()
+ if os.path.isfile(cache_file):
+ try:
+ f = open(cache_file, 'r')
+ try:
+ time_stamp = f.read()
+ if time_stamp:
+ return datetime.datetime.fromtimestamp(float(time_stamp))
+ except ValueError:
+ LOGGER.warning('The cache file contain bad timestamp')
+ finally:
+ f.close()
+ except (IOError, OSError) as e:
+ LOGGER.warning('Failed to open cache file to read.')
+
+ return None
+
+
+def _UpdateLastHeartbeatTime(time_stamp):
+ """Update the cache file with new timestamp.
+
+ @param time_stamp: A new timestamp as datetime.
+ """
+ cache_file = _GetHeartbeatCacheFile()
+ try:
+ f = open(cache_file, 'w')
+ try:
+ f.write(time_stamp.strftime('%s'))
+ finally:
+ f.close()
+ except (IOError, OSError) as e:
+ LOGGER.warning('Failed to open cache file to write.')
+
+
+def _NeedHeartbeat(now, last, rate):
+ """Rule for sending out hearbeat.
+
+ @param now: The now datetime object.
+ @param last: The datetime when last heartbeat is sent.
+ @param rate: The rate in seconds the heartbeat should be sent.
+
+ @returns True if heartbeat should be sent; otherwise, False.
+ """
+ if rate <= 0:
+ return False
+ return not last or (now - last).total_seconds() > rate
+
+
+class Heartbeat(object):
+ """Ensures that a heartbeat is sent out on time."""
+
+ def Check(self):
+ """Ensures that moblab sends out heartbeat on tmie.
+
+ Returns:
+ 0 if heartbeat is out on time.
+ -1 if cloud pubsub notificaiton is not enabled.
+ -2 if a heartbeat could not be sent out on time.
+ -3 if there is internal server error.
+ """
+ LOGGER.info('Start to check heartbeat')
+ # Mobmonitor could run before the autotest setup is finished.
+ # The /usr/local/autotest on moblab might not be created yet.
+ try:
+ from autotest_lib.site_utils import cloud_console_client
+ from autotest_lib.client.common_lib import global_config
+ except ImportError:
+ LOGGER.info('Try to import autotest.')
+ # The autotest is not ready yet
+ if not autotest_common.import_autotest():
+ LOGGER.warning('Autotest is not ready.')
+ else:
+ LOGGER.info('Autotest is ready now.')
+ return -3
+
+ heartbeat_rate = _GetHeartbeatRate()
+ if heartbeat_rate <= _MIN_HEARTBEAT_RATE:
+ LOGGER.info(
+ 'Skip heartbeat because rate is not set or set too small.')
+ return 0
+
+ if not cloud_console_client.is_cloud_notification_enabled():
+ LOGGER.warning('Could not send heartbeat because pubub is not enabled.')
+ return -1
+
+ now = datetime.datetime.now()
+ last_heartbeat = _GetLastHeartbeatTimeFromFile()
+ if _NeedHeartbeat(now, last_heartbeat, heartbeat_rate):
+ console_client = cloud_console_client.PubSubBasedClient()
+ if console_client.send_heartbeat():
+ _UpdateLastHeartbeatTime(now)
+ LOGGER.info('Heartbeat is sent and recorded the timestamp.')
+ else:
+ LOGGER.warning('Failed to send Heartbeat.')
+ return -2
+ else:
+ LOGGER.info('Skip heartbeat because previous one does not expire.')
+
+ return 0
+
+ def Diagnose(self, errcode):
+ msg_1 = ('Heartbeat could not be sent because pubsub is not enabled. Make'
+ 'sure your moblab is set up appropriately. For more information '
+ 'please see https://www.chromium.org/chromium-os/testing/'
+ 'moblab/setup#TOC-Setting-up-the-boto-key-for-partners')
+
+ msg_2 = ('Heartbeat could not be sent out on times. Make sure your moblab'
+ ' is set up appropriately. For more information'
+ ' please see https://www.chromium.org/chromium-os/testing/'
+ 'moblab/setup#TOC-Setting-up-the-boto-key-for-partners')
+
+ msg_3 = ('Internal software installation error.')
+
+ if -1 == errcode:
+ return msg_1
+ elif -2 == errcode:
+ return msg_2
+ elif -3 == errcode:
+ return msg_3
+
+ return ('Unknown error reached with error code: %s' % errcode, [])
diff --git a/src/mobmonitor/checkfiles/moblab/heartbeat_check_unittest.py b/src/mobmonitor/checkfiles/moblab/heartbeat_check_unittest.py
new file mode 100644
index 0000000..9aa145c
--- /dev/null
+++ b/src/mobmonitor/checkfiles/moblab/heartbeat_check_unittest.py
@@ -0,0 +1,155 @@
+#!/usr/bin/env python2
+# Copyright 2017 The Chromium OS Authors. All rights reserved.
+# Use of this source code is governed by a BSD-style license that can be
+# found in the LICENSE file.
+
+"""Unit test for heartbeat_check.py"""
+
+from __future__ import print_function
+
+import datetime
+import mox
+import os
+import tempfile
+import unittest
+
+import heartbeat_check
+from autotest_lib.site_utils import cloud_console_client
+
+class HeartbeatModuleTest(mox.MoxTestBase):
+ """Unit test for heartbeat module."""
+
+ def testGetLastHeartbeatTimeFromFile_NoFile(self):
+ """Tests read timestamp when the cahce file does not exists."""
+ self.mox.StubOutWithMock(os.path, 'isfile')
+ self.mox.StubOutWithMock(heartbeat_check, '_GetHeartbeatCacheFile')
+ heartbeat_check._GetHeartbeatCacheFile().AndReturn('/tmp/.HEARTBEAT')
+ os.path.isfile('/tmp/.HEARTBEAT').AndReturn(False)
+ self.mox.ReplayAll()
+ self.assertIsNone(heartbeat_check._GetLastHeartbeatTimeFromFile())
+ self.mox.VerifyAll()
+
+ def testHeartbeatCacheReadWrite(self):
+ """Tests normal read/write from and to the cache file."""
+ cache_file = tempfile.NamedTemporaryFile()
+ self.mox.StubOutWithMock(heartbeat_check, '_GetHeartbeatCacheFile')
+ heartbeat_check._GetHeartbeatCacheFile().MultipleTimes().AndReturn(
+ cache_file.name)
+ self.mox.ReplayAll()
+ now = datetime.datetime.now()
+ heartbeat_check._UpdateLastHeartbeatTime(now)
+ dt = heartbeat_check._GetLastHeartbeatTimeFromFile()
+ self.assertTrue(abs((now -dt).total_seconds()) < 1)
+ self.mox.VerifyAll()
+ cache_file.close()
+
+ def testNeedHeartbeatRule(self):
+ """Tests the logic if a new heartbeat is needed."""
+ now = datetime.datetime.now()
+ dt1 = now + datetime.timedelta(seconds=-50)
+ dt2 = now + datetime.timedelta(seconds=-70)
+
+ self.assertFalse(heartbeat_check._NeedHeartbeat(now, None, 0))
+ self.assertTrue(heartbeat_check._NeedHeartbeat(now, None, 60))
+ self.assertFalse(heartbeat_check._NeedHeartbeat(now, dt1, 60))
+ self.assertTrue(heartbeat_check._NeedHeartbeat(now, dt2, 60))
+
+
+class HeartbeatTest(mox.MoxTestBase):
+ """Unit test for Heartbeat class."""
+
+ def testCheckWhenHeartbeatNotEnabled(self):
+ """Check should returns 0 if heartbeat rate is set to 0."""
+ heartbeat = heartbeat_check.Heartbeat()
+ self.mox.StubOutWithMock(heartbeat_check, '_GetHeartbeatRate')
+ heartbeat_check._GetHeartbeatRate().AndReturn(0)
+ self.mox.ReplayAll()
+ self.assertEquals(0, heartbeat.Check())
+ self.mox.VerifyAll()
+
+ def testCheckWhenHeartbeatPubsubNotEnabled(self):
+ """Check should returns -1 if pusub is not enabled."""
+ heartbeat = heartbeat_check.Heartbeat()
+ self.mox.StubOutWithMock(heartbeat_check, '_GetHeartbeatRate')
+ heartbeat_check._GetHeartbeatRate().AndReturn(60)
+ self.mox.StubOutWithMock(cloud_console_client,
+ 'is_cloud_notification_enabled')
+ cloud_console_client.is_cloud_notification_enabled().AndReturn(False)
+ self.mox.ReplayAll()
+ self.assertEquals(-1, heartbeat.Check())
+ self.mox.VerifyAll()
+
+ def testCheckWhenHeartbeatSkip(self):
+ """Check should returns 0 if previous heartbeat not expire yet."""
+ heartbeat = heartbeat_check.Heartbeat()
+ self.mox.StubOutWithMock(heartbeat_check, '_GetHeartbeatRate')
+ heartbeat_check._GetHeartbeatRate().AndReturn(60)
+ self.mox.StubOutWithMock(cloud_console_client,
+ 'is_cloud_notification_enabled')
+ cloud_console_client.is_cloud_notification_enabled().AndReturn(True)
+ self.mox.StubOutWithMock(heartbeat_check,
+ '_GetLastHeartbeatTimeFromFile')
+ sometime = datetime.datetime.now()
+ heartbeat_check._GetLastHeartbeatTimeFromFile().AndReturn(sometime)
+ self.mox.StubOutWithMock(heartbeat_check, '_NeedHeartbeat')
+ heartbeat_check._NeedHeartbeat(
+ mox.IsA(datetime.datetime), sometime, 60).AndReturn(False)
+ self.mox.ReplayAll()
+ self.assertEquals(0, heartbeat.Check())
+ self.mox.VerifyAll()
+
+
+ def testCheckWhenHeartbeatFailedToSend(self):
+ """Check should returns 0 if previous heartbeat not expire yet."""
+ heartbeat = heartbeat_check.Heartbeat()
+ self.mox.StubOutWithMock(heartbeat_check, '_GetHeartbeatRate')
+ heartbeat_check._GetHeartbeatRate().AndReturn(60)
+ self.mox.StubOutWithMock(cloud_console_client,
+ 'is_cloud_notification_enabled')
+ cloud_console_client.is_cloud_notification_enabled().AndReturn(True)
+ self.mox.StubOutWithMock(heartbeat_check,
+ '_GetLastHeartbeatTimeFromFile')
+ sometime = datetime.datetime.now()
+ heartbeat_check._GetLastHeartbeatTimeFromFile().AndReturn(sometime)
+ self.mox.StubOutWithMock(heartbeat_check, '_NeedHeartbeat')
+ heartbeat_check._NeedHeartbeat(
+ mox.IsA(datetime.datetime), sometime, 60).AndReturn(True)
+ console_client = self.mox.CreateMock(
+ cloud_console_client.PubSubBasedClient)
+ self.mox.StubOutWithMock(cloud_console_client, 'PubSubBasedClient')
+ cloud_console_client.PubSubBasedClient().AndReturn(console_client)
+ console_client.send_heartbeat().AndReturn(False)
+ self.mox.ReplayAll()
+ self.assertEquals(-2, heartbeat.Check())
+ self.mox.VerifyAll()
+
+ def testCheckWhenHeartbeatIsSent(self):
+ """Check should returns 0 if previous heartbeat not expire yet."""
+ heartbeat = heartbeat_check.Heartbeat()
+ self.mox.StubOutWithMock(heartbeat_check, '_GetHeartbeatRate')
+ heartbeat_check._GetHeartbeatRate().AndReturn(60)
+ self.mox.StubOutWithMock(cloud_console_client,
+ 'is_cloud_notification_enabled')
+ cloud_console_client.is_cloud_notification_enabled().AndReturn(True)
+ self.mox.StubOutWithMock(heartbeat_check,
+ '_GetLastHeartbeatTimeFromFile')
+ sometime = datetime.datetime.now()
+ heartbeat_check._GetLastHeartbeatTimeFromFile().AndReturn(sometime)
+ self.mox.StubOutWithMock(heartbeat_check, '_NeedHeartbeat')
+ heartbeat_check._NeedHeartbeat(
+ mox.IsA(datetime.datetime), sometime, 60).AndReturn(True)
+ console_client = self.mox.CreateMock(
+ cloud_console_client.PubSubBasedClient)
+ self.mox.StubOutWithMock(cloud_console_client, 'PubSubBasedClient')
+ cloud_console_client.PubSubBasedClient().AndReturn(console_client)
+ console_client.send_heartbeat().AndReturn(True)
+ self.mox.StubOutWithMock(heartbeat_check,
+ '_UpdateLastHeartbeatTime')
+ heartbeat_check._UpdateLastHeartbeatTime(mox.IsA(datetime.datetime))
+ self.mox.ReplayAll()
+ self.assertEquals(0, heartbeat.Check())
+ self.mox.VerifyAll()
+
+
+if __name__ == '__main__':
+ unittest.main()
diff --git a/src/mobmonitor/checkfiles/moblab/moblab_actions.py b/src/mobmonitor/checkfiles/moblab/moblab_actions.py
new file mode 100644
index 0000000..fed3ce2
--- /dev/null
+++ b/src/mobmonitor/checkfiles/moblab/moblab_actions.py
@@ -0,0 +1,80 @@
+# Copyright 2015 The Chromium OS Authors. All rights reserved.
+# Use of this source code is governed by a BSD-style license that can be
+# found in the LICENSE file.
+
+"""Repair actions for moblab health checks."""
+
+from __future__ import print_function
+
+import common
+
+import os
+import re
+
+from util import osutils
+
+
+#
+# Disk actions.
+#
+def MountUsbWrapper(device):
+ """Wrapper for mounting USB devices.
+
+ Args:
+ device: The name of USB device we wish to mount.
+ """
+ def MountUsb():
+ """Mount the USB device."""
+ # Unmount |device| as it may already be mounted in the wrong location.
+ osutils.sudo_run_command(['umount', device], error_code_ok=True)
+
+ # Mount |device| in the correct location.
+ cmd = ['mount', device, common.MOBLAB_MOUNTPOINT, '-o', 'exec,dev,nosuid']
+ osutils.sudo_run_command(cmd)
+
+ return MountUsb
+
+
+def ClearOldDbRecords(date):
+ """Delete older database records.
+
+ Args:
+ date: Must be in the format 'yyyy-mm-dd'. We keep records that are
+ newer than the given date.
+ """
+ cmd = ['/usr/local/autotest/contrib/db_cleanup.py', date]
+ osutils.sudo_run_command(cmd)
+
+
+def ClearLogs():
+ """Deletes old log files."""
+ logdir = '/var/log'
+ oldlog = r'^.*\.[0-9]+$'
+
+ files_to_remove = []
+
+ for curdir, _, files in os.walk(logdir):
+ for f in files:
+ if re.match(oldlog, f):
+ files_to_remove.append(os.path.join(curdir, f))
+
+ if files_to_remove:
+ cmd = ['rm'] + files_to_remove
+ osutils.sudo_run_command(cmd)
+
+
+#
+# Upstart service actions.
+#
+def LaunchUpstartServiceWrapper(service):
+ """Wrapper for bringing up an upstart job.
+
+ Args:
+ service: The name of service we are trying to start.
+ """
+ def LaunchUpstartService():
+ """Launch the upstart service."""
+ cmd = ['start', service]
+ osutils.sudo_run_command(cmd)
+
+ return LaunchUpstartService
diff --git a/src/mobmonitor/checkfiles/moblab/network_check.py b/src/mobmonitor/checkfiles/moblab/network_check.py
new file mode 100644
index 0000000..c0fc104
--- /dev/null
+++ b/src/mobmonitor/checkfiles/moblab/network_check.py
@@ -0,0 +1,80 @@
+# Copyright 2015 The Chromium OS Authors. All rights reserved.
+# Use of this source code is governed by a BSD-style license that can be
+# found in the LICENSE file.
+
+"""Network monitoring health checks for moblab."""
+
+from __future__ import print_function
+
+import netifaces
+
+from util import osutils
+
+
+class InternetAccess(object):
+ """Verifies that moblab can connect to the internet."""
+
+ DNS = ['4.2.2.2', '4.2.2.3', '4.2.2.4', '8.8.4.4', '8.8.8.8']
+ PING_WAIT_SEC = '3'
+ PING_COUNT = '3'
+
+ def Check(self):
+ """Verify that this moblab device has internet access.
+
+ We test internet access by trying to connect to a handful
+ of public DNS addresses that would be unlikely be down,
+ or even to be down all at the same time.
+
+ Returns:
+ 0 if this moblab device successfully connects to at least
+ one of DNS.
+ -1 if moblab cannot connect to any of DNS.
+ """
+ for dns in self.DNS:
+ cmd = ['ping', '-W', self.PING_WAIT_SEC, '-c', self.PING_COUNT, dns]
+ try:
+ osutils.run_command(cmd)
+ return 0
+ except osutils.RunCommandError as e:
+ pass
+
+ return -1
+
+ def Diagnose(self, errcode):
+ if -1 == errcode:
+ return ('Moblab is having trouble connecting to the internet.'
+ ' Please double-check your physical connections.',
+ [])
+
+ return ('Unknown error reached with error code: %s' % errcode, [])
+
+
+class UsbEthernet(object):
+ """Verifies that moblab has a USB ethernet device connected."""
+
+ def Check(self):
+ """Verify that a USB ethernet device is present.
+
+ Returns:
+ 0 if a USB ethernet device exists.
+ -1 if no USB ethernet device is detected.
+ """
+ # External ethernet will be either eth0 or eth1. If a usb ethernet
+ # device is detected, we should have more than one eth* interface.
+ eth_ifaces = []
+ for iface in netifaces.interfaces():
+ if 'eth' in iface:
+ eth_ifaces.append(iface)
+
+ if len(eth_ifaces) > 1:
+ return 0
+
+ return -1
+
+ def Diagnose(self, errcode):
+ if -1 == errcode:
+ return ('No USB ethernet device detected. This may indicate that'
+ ' no DUT (or some other important device) is not connected.',
+ [])
+
+ return ('Unknown error reached with error code: %s' % errcode, [])
diff --git a/src/mobmonitor/checkfiles/moblab/servo_check.py b/src/mobmonitor/checkfiles/moblab/servo_check.py
new file mode 100644
index 0000000..ed16782
--- /dev/null
+++ b/src/mobmonitor/checkfiles/moblab/servo_check.py
@@ -0,0 +1,37 @@
+# Copyright 2015 The Chromium OS Authors. All rights reserved.
+# Use of this source code is governed by a BSD-style license that can be
+# found in the LICENSE file.
+
+"""Servo health checks for moblab."""
+
+from __future__ import print_function
+
+from util import osutils
+
+SERVO_VID = '18d1'
+SERVO_PID = '5002'
+
+
+class ServoExists(object):
+ """Verfies that a Servo is connected to moblab."""
+
+ def Check(self):
+ """Verify that a Servo is connected.
+
+ Returns:
+ 0 if a Servo exists.
+ 1 if a Servo device could not be found.
+ """
+ cmd = ['lsusb']
+ servo_id = '%s:%s' % (SERVO_VID, SERVO_PID)
+ usbs = osutils.sudo_run_command(cmd).strip()
+ for usb in usbs.split('\n'):
+ if servo_id in usb:
+ return 0
+ return 1
+
+ def Diagnose(self, errcode):
+ if 1 == errcode:
+ return ('No servo devices are connected.', [])
+
+ return ('Unknown error reached with error code: %s' % errcode, [])
diff --git a/src/mobmonitor/checkfiles/moblab/upstart_services_running_check.py b/src/mobmonitor/checkfiles/moblab/upstart_services_running_check.py
new file mode 100644
index 0000000..f21ce88
--- /dev/null
+++ b/src/mobmonitor/checkfiles/moblab/upstart_services_running_check.py
@@ -0,0 +1,166 @@
+# Copyright 2015 The Chromium OS Authors. All rights reserved.
+# Use of this source code is governed by a BSD-style license that can be
+# found in the LICENSE file.
+
+"""Ensure essential moblab services are up and running."""
+
+from __future__ import print_function
+
+import moblab_actions
+
+import os
+
+from util import osutils
+import logging
+
+
+LOGGER = logging.getLogger(__name__)
+
+
+def UpstartExists(service):
+ """Test if an upstart script for |service| exists on moblab.
+
+ Returns:
+ True if the upstart script for |service| exists.
+ False if it does not.
+ """
+ _, ext = os.path.splitext(service)
+ if not ext:
+ service = os.extsep.join([service, 'conf'])
+ return os.path.exists(os.path.join('/etc/init', service))
+
+
+def UpstartRunning(service):
+ """Test if |service| is running on moblab.
+
+ Returns:
+ True if the upstart job |service| is in 'start/running' state.
+ False otherwise.
+ """
+ cmd = ['status', service]
+ output = osutils.sudo_run_command(cmd, error_code_ok=True)
+
+ if output is not None and 'start/running' in output:
+ return True
+
+ return False
+
+
+def ServiceRunning(service):
+ """Test if |service| is running on moblab.
+
+ Returns:
+ True if |service| is running.
+ False otherwise.
+ """
+ cmd = ['ps', 'aux']
+ output = osutils.run_command(cmd, error_code_ok=True).strip()
+
+ for line in output.splitlines():
+ if service in line:
+ return True
+
+ return False
+
+
+class UpstartServiceRunning(object):
+ """Helper class for verifying that moblab services are up and running."""
+
+ SERVICES = []
+
+ def Check(self):
+ """Verfies this service is running.
+
+ Returns:
+ 0 if all SERVICES is running.
+ -(i+1) if SERVICE with index i is not running.
+ """
+ LOGGER.info('Checking services: %s', self.SERVICES)
+ for i, x in enumerate(self.SERVICES):
+ if not UpstartRunning(x):
+ return -(i+1)
+
+ return 0
+
+ def Diagnose(self, errcode):
+ if errcode and self.SERVICES:
+ index = -1*errcode - 1
+ service = self.SERVICES[index]
+ return ('Essential moblab service "%s" is not running.'
+ ' Please launch the service.' % service,
+ [moblab_actions.LaunchUpstartServiceWrapper(service)])
+
+ return ('Unknown error reached with error code: %s' % errcode, [])
+
+
+class GsOffloaderRunning(UpstartServiceRunning):
+ """Verifies that gs_offloader is running."""
+
+ SERVICES = ['moblab-gsoffloader-init', 'moblab-gsoffloader_s-init']
+
+
+class SchedulerRunning(UpstartServiceRunning):
+ """Verifies that scheduler is running."""
+
+ SERVICES = ['moblab-scheduler-init']
+
+
+class DevserverRunning(UpstartServiceRunning):
+ """Verifies that devserver is running."""
+
+ SERVICES = ['moblab-devserver-init', 'moblab-devserver-cleanup-init']
+
+
+class DatabaseRunning(UpstartServiceRunning):
+ """Verifies that database is up."""
+
+ SERVICES = ['moblab-database-init']
+
+
+class JobAborterRunning(UpstartServiceRunning):
+ """Verifies that job_aborter is up."""
+
+ SERVICES = ['job_aborter']
+
+
+class ApacheUpstartRunning(UpstartServiceRunning):
+ """Verifies that apache is running."""
+
+ SERVICES = ['moblab-apache-init']
+ SERVICE_SEARCH = 'apache'
+
+ def Check(self):
+ """Verifies that apache is running on moblab.
+
+ Returns:
+ 0 if apache is running.
+ -1 if apache is not running.
+ """
+ if not ServiceRunning(self.SERVICE_SEARCH):
+ return -1
+
+ return 0
+
+
+class DhcpUpstartRunning(UpstartServiceRunning):
+ """Verifies that dhcp is running."""
+
+ SERVICES = ['moblab-network-init', 'moblab-network-bridge-init']
+ SERVICE_SEARCH = 'dhcpcd'
+
+ def Check(self):
+ """Verfies that dhcp is running on moblab.
+
+ Returns:
+ 0 if dhcp is up.
+ -1 if dhcp is down and moblab-network-init is installed on moblab.
+ -2 if dhcp is down and moblab-network-bridge-init is installed on moblab.
+ """
+ if not ServiceRunning(self.SERVICE_SEARCH):
+ # DHCP is down. We need to figure which upstart script will
+ # correct the issue for this moblab.
+ for i, x in enumerate(self.SERVICES):
+ if UpstartExists(x):
+ return -(i+1)
+
+ return 0
diff --git a/src/mobmonitor/mobmonitor.py b/src/mobmonitor/mobmonitor.py
new file mode 100644
index 0000000..0dd5682
--- /dev/null
+++ b/src/mobmonitor/mobmonitor.py
@@ -0,0 +1,199 @@
+#!/usr/bin/env python2
+# -*- coding: utf-8 -*-
+# Copyright 2015 The Chromium OS Authors. All rights reserved.
+# Use of this source code is governed by a BSD-style license that can be
+# found in the LICENSE file.
+
+"""The Mob* Monitor web interface."""
+
+from __future__ import print_function
+
+import cherrypy
+import json
+import os
+import sys
+import logging
+import logging.handlers
+import argparse
+
+from cherrypy.lib.static import serve_file
+from checkfile import manager
+from util import collect_logs
+
+
+STATICDIR = '/usr/local/moblab/mobmonitor/static/'
+
+LOGDIR = '/var/log/mobmonitor/'
+LOGFILE = 'mobmonitor.log'
+LOGFILE_SIZE_BYTES = 1024 * 1024
+LOGFILE_COUNT = 10
+
+
+class MobMonitorRoot(object):
+ """The central object supporting the Mob* Monitor web interface."""
+
+ def __init__(self, checkfile_manager, staticdir=STATICDIR):
+ if not os.path.exists(staticdir):
+ raise IOError('Static directory does not exist: %s' % staticdir)
+
+ self.staticdir = staticdir
+ self.checkfile_manager = checkfile_manager
+
+ @cherrypy.expose
+ def index(self):
+ """Presents a welcome message."""
+ return open(os.path.join(self.staticdir, 'templates', 'index.html'))
+
+ @cherrypy.expose
+ def GetServiceList(self):
+ """Return a list of the monitored services.
+
+ Returns:
+ A list of the monitored services.
+ """
+ return json.dumps(self.checkfile_manager.GetServiceList())
+
+ @cherrypy.expose
+ def GetStatus(self, service=None):
+ """Return the health status of the specified service.
+
+ Args:
+ service: The service whose health status is being queried. If service
+ is None, return the health status of all monitored services.
+
+ Returns:
+ A list of dictionaries. Each dictionary contains the keys:
+ service: The name of the service.
+ health: A boolean describing the overall service health.
+ healthchecks: A list of unhealthy or quasi-healthy health checks.
+ """
+ service_statuses = self.checkfile_manager.GetStatus(service)
+ if not isinstance(service_statuses, list):
+ service_statuses = [service_statuses]
+
+ result = [
+ manager.MapServiceStatusToDict(status) for status in service_statuses]
+ return json.dumps(result)
+
+ @cherrypy.expose
+ def ActionInfo(self, service, healthcheck, action):
+ """Return usage and argument information for |action|.
+
+ Args:
+ service: A string. The name of a service being monitored.
+ healthcheck: A string. The name of the healthcheck the action belongs to.
+ action: A string. The name of an action specified by some healthcheck's
+ Diagnose method.
+
+ Returns:
+ TBD
+ """
+ result = self.checkfile_manager.ActionInfo(service, healthcheck, action)
+ return json.dumps(manager.MapActionInfoToDict(result))
+
+ @cherrypy.expose
+ def RepairService(self, service, healthcheck, action, args, kwargs):
+ """Execute the repair action on the specified service.
+
+ Args:
+ service: The service that the specified action will be applied to.
+ healthcheck: The particular healthcheck we are repairing.
+ action: The action to be applied.
+ args: A list of the positional arguments for the given repair action.
+ kwargs: A dictionary of keyword arguments for the given repair action.
+ """
+ # The mobmonitor's RPC library encodes arguments as strings when
+ # making a remote call to the monitor. The checkfile manager expects
+ # lists and dicts for the arugments, so we convert them here.
+ args = json.loads(args.replace('\'', '"'))
+ kwargs = json.loads(kwargs.replace('\'', '"'))
+
+ status = self.checkfile_manager.RepairService(service, healthcheck, action,
+ args, kwargs)
+ return json.dumps(manager.MapServiceStatusToDict(status))
+
+ @cherrypy.expose
+ def CollectLogs(self):
+ tarfile = collect_logs.collect_logs()
+ return serve_file(tarfile, 'application/x-download',
+ 'attachment', os.path.basename(tarfile))
+
+
+def SetupLogging(logdir):
+ logging.basicConfig(
+ level=logging.DEBUG,
+ format='%(asctime)s:%(name)s:%(levelname)-8s %(message)s',
+ datefmt='%Y-%m-%d %H:%M',
+ filename=os.path.join(logdir, LOGFILE),
+ filemode='w'
+ )
+ rotate = logging.handlers.RotatingFileHandler(
+ os.path.join(logdir, LOGFILE), maxBytes=LOGFILE_SIZE_BYTES,
+ backupCount=LOGFILE_COUNT)
+ logging.getLogger().addHandler(rotate)
+
+
+def ParseArguments(argv):
+ """Creates the argument parser."""
+ parser = argparse.ArgumentParser(description=__doc__)
+
+ parser.add_argument('-d', '--checkdir',
+ default='/usr/local/moblab/mobmonitor/checkfiles/',
+ help='The Mob* Monitor checkfile directory.')
+ parser.add_argument('-p', '--port', type=int, default=9991,
+ help='The Mob* Monitor port.')
+ parser.add_argument('-s', '--staticdir', default=STATICDIR,
+ help='Mob* Monitor web ui static content directory')
+ parser.add_argument('--logdir', dest='logdir', default=LOGDIR,
+ help='Mob* Monitor log file directory.')
+
+ return parser.parse_args(argv)
+
+def _validate_port(check_port):
+ port = int(check_port)
+ if port <= 0 or port >= 65536:
+ raise ValueError('%d is not a valid port' % port)
+ return port
+
+
+def main(argv):
+ options = ParseArguments(argv)
+
+ # Configure logger.
+ SetupLogging(options.logdir)
+
+ # Configure global cherrypy parameters.
+ cherrypy.config.update(
+ {'server.socket_host': '0.0.0.0',
+ 'server.socket_port': _validate_port(options.port)
+ })
+
+ mobmon_appconfig = {
+ '/':
+ {'tools.staticdir.root': options.staticdir
+ },
+ '/static':
+ {'tools.staticdir.on': True,
+ 'tools.staticdir.dir': ''
+ },
+ '/static/css':
+ {'tools.staticdir.dir': 'css'
+ },
+ '/static/js':
+ {'tools.staticdir.dir': 'js'
+ }
+ }
+
+ # Setup the mobmonitor
+ checkfile_manager = manager.CheckFileManager(checkdir=options.checkdir)
+ mobmonitor = MobMonitorRoot(checkfile_manager, staticdir=options.staticdir)
+
+ # Start the checkfile collection and execution background task.
+ checkfile_manager.StartCollectionExecution()
+
+ # Start the Mob* Monitor.
+ cherrypy.quickstart(mobmonitor, config=mobmon_appconfig)
+
+
+if __name__ == '__main__':
+ main(sys.argv[1:])
diff --git a/src/mobmonitor/mobmonitor_unittest.py b/src/mobmonitor/mobmonitor_unittest.py
new file mode 100644
index 0000000..dbb71ca
--- /dev/null
+++ b/src/mobmonitor/mobmonitor_unittest.py
@@ -0,0 +1,155 @@
+# -*- coding: utf-8 -*-
+# Copyright 2015 The Chromium OS Authors. All rights reserved.
+# Use of this source code is governed by a BSD-style license that can be
+# found in the LICENSE file.
+
+"""Unittests for the main Mob* Monitor script."""
+
+from __future__ import print_function
+
+import json
+import os
+import unittest
+import mock
+
+import mobmonitor
+
+from checkfile import manager
+from util import osutils
+
+
+class MockCheckFileManager(object):
+ """Mock CheckFileManager object that returns 'real' responses for testing."""
+
+ def __init__(self):
+ failed_check = manager.HEALTHCHECK_STATUS('hc1', False, 'Failed', [])
+
+ self.service_statuses = [
+ manager.SERVICE_STATUS('service1', True, []),
+ manager.SERVICE_STATUS('service2', False, [failed_check])]
+
+ self.action_info = manager.ACTION_INFO('DummyAction', '', ['x'], {})
+
+ def GetServiceList(self):
+ """Mock GetServiceList response."""
+ return ['test_service_1', 'test_service_2']
+
+ def GetStatus(self, service=None):
+ """Mock GetStatus response."""
+ if service is None:
+ return self.service_statuses
+
+ return self.service_statuses[0]
+
+ def ActionInfo(self, _service, _healthcheck, _action):
+ """Mock ActionInfo response."""
+ return self.action_info
+
+ def RepairService(self, _service, _healthcheck, _action, _args, _kwargs):
+ """Mock RepairService response."""
+ return self.service_statuses[0]
+
+
+class MobMonitorRootTest(unittest.TestCase):
+ """Unittests for the MobMonitorRoot."""
+
+ STATICDIR = 'static'
+
+ def _mock_checkfile_manager(self):
+ mock_checkfile_manager = mock.MagicMock()
+
+ failed_check = manager.HEALTHCHECK_STATUS('hc1', False, 'Failed', [])
+ service_statuses = [
+ manager.SERVICE_STATUS('service1', True, []),
+ manager.SERVICE_STATUS('service2', False, [failed_check])]
+ mock_checkfile_manager.service_statuses = service_statuses
+
+ action_info = manager.ACTION_INFO('DummyAction', '', ['x'], {})
+ mock_checkfile_manager.action_info = action_info
+
+ mock_checkfile_manager.GetServiceList.return_value = [
+ 'test_service_1', 'test_service_2']
+
+ def status_side_effect(service=None):
+ if service is None:
+ return service_statuses
+ else:
+ return service_statuses[0]
+
+ mock_checkfile_manager.GetStatus.side_effect = status_side_effect
+ mock_checkfile_manager.ActionInfo.return_value = action_info
+ mock_checkfile_manager.RepairService.return_value = service_statuses[0]
+
+ return mock_checkfile_manager
+
+
+ def setUp(self):
+ """Setup directories expected by the Mob* Monitor."""
+ self.staticdir = self.STATICDIR
+ self.mock_checkfile_manager = self._mock_checkfile_manager()
+
+ patch_os = mock.patch('mobmonitor.os')
+ self.addCleanup(patch_os.stop)
+ self.mock_os = patch_os.start()
+
+ self.mock_os.path.exists.return_value = True
+
+
+ def testGetServiceList(self):
+ """Test the GetServiceList RPC."""
+ cfm = self.mock_checkfile_manager
+ root = mobmonitor.MobMonitorRoot(cfm, staticdir=self.staticdir)
+ self.assertEqual(cfm.GetServiceList(), json.loads(root.GetServiceList()))
+
+ def testGetStatus(self):
+ """Test the GetStatus RPC."""
+ cfm = self.mock_checkfile_manager
+ root = mobmonitor.MobMonitorRoot(cfm, staticdir=self.staticdir)
+
+ # Test the result for a single service.
+ status = cfm.service_statuses[0]
+ expect = {'service': status.service, 'health': status.health,
+ 'healthchecks': []}
+ self.assertEquals([expect], json.loads(root.GetStatus(status.service)))
+
+ # Test the result for multiple services.
+ status1, status2 = cfm.service_statuses
+ check = status2.healthchecks[0]
+ expect = [{'service': status1.service, 'health': status1.health,
+ 'healthchecks': []},
+ {'service': status2.service, 'health': status2.health,
+ 'healthchecks': [{'name': check.name, 'health': check.health,
+ 'description': check.description,
+ 'actions': []}]}]
+ self.assertEquals(expect, json.loads(root.GetStatus()))
+
+ def testActionInfo(self):
+ """Test the ActionInfo RPC."""
+ cfm = self.mock_checkfile_manager
+ root = mobmonitor.MobMonitorRoot(cfm, staticdir=self.staticdir)
+
+ expect = {'action': 'DummyAction', 'info': '', 'args': ['x'], 'kwargs': {}}
+ self.assertEquals(expect,
+ json.loads(root.ActionInfo('service2',
+ 'dummy_healthcheck',
+ 'DummyAction')))
+
+ def testRepairService(self):
+ """Test the RepairService RPC."""
+ cfm = self.mock_checkfile_manager
+ root = mobmonitor.MobMonitorRoot(cfm, staticdir=self.staticdir)
+
+ status = cfm.service_statuses[0]
+ expect = {'service': status.service, 'health': status.health,
+ 'healthchecks': []}
+ string_args = '[1, 2]'
+ string_kwargs = '{"a": 1}'
+ self.assertEquals(expect,
+ json.loads(root.RepairService('dummy_service',
+ 'dummy_healthcheck',
+ 'dummy_action',
+ string_args,
+ string_kwargs)))
+
+if __name__ == '__main__':
+ unittest.main()
diff --git a/src/mobmonitor/static/css/style.css b/src/mobmonitor/static/css/style.css
new file mode 100644
index 0000000..1df1118
--- /dev/null
+++ b/src/mobmonitor/static/css/style.css
@@ -0,0 +1,158 @@
+/* General Style Settings */
+body {
+ background-color: #788999;
+ margin: 0;
+ padding: 0;
+}
+
+table {
+ border-collapse: collapse;
+}
+
+td {
+ border: 1px solid #999;
+ padding: 5px;
+ text-align: left;
+}
+
+.bold {
+ font-weight: bold;
+}
+
+.circle-healthy {
+ width: 20px;
+ height: 20px;
+ border-radius: 10px;
+ background: green;
+}
+
+.circle-quasi-healthy {
+ width: 20px;
+ height: 20px;
+ border-radius: 10px;
+ background: #CC7A00;
+}
+
+.circle-unhealthy {
+ width: 20px;
+ height: 20px;
+ border-radius: 10px;
+ background: red;
+}
+
+/* Site Header Settings */
+.site-header {
+ background-color: #20262c;
+ padding-top: 50px;
+ padding-right: 0px;
+ padding-left: 0px;
+ padding-bottom: 50px;
+ margin-bottom: 0px;
+}
+
+.site-header-title {
+ max-width: 1000px;
+ width: 90%;
+ margin: 0px auto;
+ margin-top: 0px;
+ margin-right: auto;
+ margin-bottom: 0px;
+ margin-left: auto;
+ text-align: center;
+ font-size: 32pt;
+ font-weight: bold;
+ color: #ACD6FF;
+}
+
+.collect-logs {
+ float: right;
+}
+
+/* Health Display Style Settings */
+.health-display {
+ max-width: 1000px;
+ width: 90%;
+ margin: 0px auto;
+ margin-top: 0px;
+ margin-right: auto;
+ margin-bottom: 0px;
+ margin-left: auto;
+}
+
+.health-container {
+ border: 1px solid black;
+ background-color: #FFF;
+}
+
+.health-container-header {
+ background-color: #FFFAAA;
+
+ width: 100%;
+}
+
+.health-container-header-element {
+ float: left;
+ margin: 10px;
+}
+
+.health-container-content {
+ background-color: #FFF;
+ padding: 10px;
+}
+
+.healthcheck-table {
+ margin: auto;
+ border: 1px solid black;
+ width: 100%;
+}
+
+.healthcheck-table-column1 {
+ width: 65%;
+}
+
+.healthcheck-table-column2 {
+ width: 35%;
+}
+
+.healthcheck-info {
+}
+
+.run-repair-action {
+ margin-right: 5px;
+}
+
+.color-healthy {
+ color: green;
+}
+
+.color-unhealthy {
+ color: red;
+}
+
+.color-quasi-healthy {
+ color: #CC7A00;
+}
+
+/* Action Repair Style Settings */
+.actionlist-dropdown {
+ width:95%;
+}
+
+.label-help {
+ width: 95%;
+ display: inline-block;
+ padding: 5px;
+ font-size: 12;
+ white-space: pre-wrap;
+}
+
+.input-text {
+ width: 95%;
+ margin-bottom: 10px;
+}
+
+.actionDisplay {
+ width: 95%;
+ height: 150px;
+ resize: vertical;
+}
diff --git a/src/mobmonitor/static/js/actionrepairdialog.js b/src/mobmonitor/static/js/actionrepairdialog.js
new file mode 100644
index 0000000..8b177ad
--- /dev/null
+++ b/src/mobmonitor/static/js/actionrepairdialog.js
@@ -0,0 +1,157 @@
+// Copyright 2015 The Chromium OS Authors. All rights reserved.
+// Use of this source code is governed by a BSD-style license that can be
+// found in the LICENSE file.
+
+'use strict';
+
+
+var ARGS_HELP_TEXT = 'Enter arguments as a comma separated list of the form: ' +
+ 'arg1,arg2,...,argN.';
+
+var ARGS_ACTION_HELP_TEXT = '\n\nYou must enter the arguments: ';
+
+var KWARGS_HELP_TEXT = 'Enter keyword arguments as a comma separated list of ' +
+ 'equal sign separated values of the form: ' +
+ 'kwarg1=value1,kwarg2=value2,...,kwargN=valueN.';
+
+var KWARGS_ACTION_HELP_TEXT = '\n\nYou may enter zero or more of the' +
+ ' following arguments: ';
+
+var NO_ARGS_TEXT = '\n\nNo arguments for you to add.';
+
+
+function ActionRepairDialog(service, actionInfo) {
+ // The actionInfo parameter is an object with the following fields:
+ // action: A string. The name of the repair action.
+ // info: A string. A description of the repair action.
+ // args: An array. The positional arguments taken by the action.
+ // kwargs: An object. The keyword arguments taken by the action.
+
+ var actionRepairDialog = this;
+
+ var templateData = {
+ action: actionInfo.action,
+ info: actionInfo.info
+ };
+
+ this.service = service;
+ this.actionInfo = actionInfo;
+
+ this.dialogElement_ = $(
+ renderTemplate('actionrepairdialog', templateData)).dialog({
+ autoOpen: false,
+ width: 575,
+ modal: true,
+
+ close: function(event, ui) {
+ $(this).dialog('destroy').remove();
+ },
+
+ buttons: {
+ 'Reset': function() {
+ actionRepairDialog.reset();
+ },
+ 'Submit': function() {
+ actionRepairDialog.submit();
+ }
+ }
+ });
+
+ // Commonly used elements of the dialog ui.
+ var d = this.dialogElement_;
+ this.dialogArgs = $(d).find('#args')[0];
+ this.dialogArgsHelp = $(d).find('#argsHelp')[0];
+ this.dialogKwargs = $(d).find('#kwargs')[0];
+ this.dialogKwargsHelp = $(d).find('#kwargsHelp')[0];
+
+ // Set default action info.
+ this.reset();
+}
+
+ActionRepairDialog.prototype.open = function() {
+ this.dialogElement_.dialog('open');
+};
+
+ActionRepairDialog.prototype.close = function() {
+ this.dialogElement_.dialog('close');
+};
+
+ActionRepairDialog.prototype.reset = function() {
+ var actionInfo = this.actionInfo;
+
+ // Clear old input.
+ this.dialogArgs.value = '';
+ this.dialogKwargs.value = '';
+
+ // Set the argument information.
+ if (!isEmpty(actionInfo.args)) {
+ $(this.dialogArgsHelp).text(ARGS_HELP_TEXT + ARGS_ACTION_HELP_TEXT +
+ actionInfo.args.join(','));
+ this.dialogArgs.disabled = false;
+ }
+ else {
+ $(this.dialogArgsHelp).text(ARGS_HELP_TEXT + NO_ARGS_TEXT);
+ this.dialogArgs.disabled = true;
+ }
+
+ // Set the kwarg information.
+ if (!isEmpty(actionInfo.kwargs)) {
+ var kwargs = [];
+ Object.keys(this.actionInfo.kwargs).forEach(function(key) {
+ kwargs.push(key + '=' + actionInfo.kwargs[key]);
+ });
+
+ this.dialogKwargs.value = kwargs.join(',');
+ this.dialogKwargs.disabled = false;
+ $(this.dialogKwargsHelp).text(
+ KWARGS_HELP_TEXT + KWARGS_ACTION_HELP_TEXT +
+ Object.keys(actionInfo.kwargs).join(','));
+ }
+ else {
+ $(this.dialogKwargsHelp).text(KWARGS_HELP_TEXT + NO_ARGS_TEXT);
+ this.dialogKwargs.disabled = true;
+ }
+};
+
+ActionRepairDialog.prototype.submit = function() {
+ // Caller must define the function 'submitHandler' on the created dialog.
+ // The submitHandler will be passed the following arguments:
+ // service: A string.
+ // action: A string.
+ // args: An array.
+ // kwargs: An object.
+
+ if (!this.submitHandler) {
+ alert('Caller must define submitHandler for ActionRepairDialog.');
+ return;
+ }
+
+ // Validate the argument input.
+ var args = this.dialogArgs.value;
+ var kwargs = this.dialogKwargs.value;
+
+ if (args && !/^([^,]+,)*[^,]+$/g.test(args)) {
+ alert('Arguments are not well-formed.\n' +
+ 'Expected form: a1,a2,...,aN');
+ return;
+ }
+
+ if (kwargs && !/^([^,=]+=[^,=]+,)*[^,]+=[^,=]+$/g.test(kwargs)) {
+ alert('Keyword argumetns are not well-formed.\n' +
+ 'Expected form: kw1=foo,...,kwN=bar');
+ return;
+ }
+
+ // Submit the action.
+ var submitArgs = args ? args.split(',') : [];
+ var submitKwargs = {};
+ kwargs.split(',').forEach(function(elem, index, array) {
+ var kv = elem.split('=');
+ submitKwargs[kv[0]] = kv[1];
+ });
+
+
+ this.submitHandler(this.service, this.actionInfo.action, submitArgs,
+ submitKwargs);
+ this.close();
+};
diff --git a/src/mobmonitor/static/js/healthdisplay.js b/src/mobmonitor/static/js/healthdisplay.js
new file mode 100644
index 0000000..06daa9e
--- /dev/null
+++ b/src/mobmonitor/static/js/healthdisplay.js
@@ -0,0 +1,150 @@
+// Copyright 2015 The Chromium OS Authors. All rights reserved.
+// Use of this source code is governed by a BSD-style license that can be
+// found in the LICENSE file.
+
+'use strict';
+
+var RECORD_TTL_MS = 10000;
+
+
+$.widget('mobmonitor.healthDisplay', {
+ options: {},
+
+ _create: function() {
+ this.element.addClass('health-display');
+
+ // Service status information. The variable serviceHealthStatusInfo
+ // is a mapping of the following form:
+ //
+ // {serviceName: {lastUpdatedTimestampInMs: lastUpdatedTimestampInMs,
+ // serviceStatus: serviceStatus}}
+ //
+ // Where serviceStatus objects are of the following form:
+ //
+ // {serviceName: serviceNameString,
+ // health: boolean,
+ // healthchecks: [
+ // {name: healthCheckName, health: boolean,
+ // description: descriptionString,
+ // actions: [actionNameString]}
+ // ]
+ // }
+ this.serviceHealthStatusInfo = {};
+ },
+
+ _destroy: function() {
+ this.element.removeClass('health-display');
+ this.element.empty();
+ },
+
+ _setOption: function(key, value) {
+ this._super(key, value);
+ },
+
+ // Private widget methods.
+ // TODO (msartori): Implement crbug.com/520746.
+ _updateHealthDisplayServices: function(services) {
+ var self = this;
+
+ function _removeService(service) {
+ // Remove the UI elements.
+ var id = 'div[id=' + SERVICE_CONTAINER_PREFIX + service + ']';
+ $(id).empty();
+ $(id).remove();
+
+ // Remove raw service status info that we are holding.
+ delete self.serviceHealthStatusInfo[service];
+ }
+
+ function _addService(serviceStatus) {
+ // This function is used as a callback to the rpcGetStatus.
+ // rpcGetStatus returns a list of service health statuses.
+ // In this widget, we add services one at a time, so take
+ // the first element.
+ serviceStatus = serviceStatus[0];
+
+ // Create the new content for the healthDisplay widget.
+ var templateData = jQuery.extend({}, serviceStatus);
+ templateData.serviceId = SERVICE_CONTAINER_PREFIX + serviceStatus.service;
+ templateData.errors = serviceStatus.healthchecks.filter(function(v) {
+ return !v.health;
+ });
+ templateData.warnings = serviceStatus.healthchecks.filter(function(v) {
+ return v.health;
+ });
+
+ var healthContainer = renderTemplate('healthstatuscontainer',
+ templateData);
+
+ // Insert the new container into the display widget.
+ $(healthContainer).appendTo(self.element);
+
+ // Maintain alphabetical order in our display.
+ self.element.children().sort(function(a, b) {
+ return $(a).attr('id') < $(b).attr('id') ? -1 : 1;
+ }).appendTo(self.element);
+
+ // Save information to do with this service.
+ var curtime = $.now();
+ var service = serviceStatus.service;
+
+ self.serviceHealthStatusInfo[service] = {
+ lastUpdatedTimestampInMs: curtime,
+ serviceStatus: serviceStatus
+ };
+ }
+
+ // Remove services that are no longer monitored or are stale.
+ var now = $.now();
+
+ Object.keys(this.serviceHealthStatusInfo).forEach(
+ function(elem, index, array) {
+ if ($.inArray(elem, services) < 0 ||
+ now > self.serviceHealthStatusInfo[elem].lastUpdatedTimestampInMs +
+ RECORD_TTL_MS) {
+ _removeService(elem);
+ }
+ });
+
+ // Get sublist of services to update.
+ var updateList =
+ $(services).not(Object.keys(this.serviceHealthStatusInfo)).get();
+
+ // Update the services.
+ updateList.forEach(function(elem, index, array) {
+ rpcGetStatus(elem, _addService);
+ });
+ },
+
+ // Public widget methods.
+ refreshHealthDisplay: function() {
+ var self = this;
+ rpcGetServiceList(function(services) {
+ self._updateHealthDisplayServices(services);
+ });
+ },
+
+ needsRepair: function(service) {
+ var serviceStatus = this.serviceHealthStatusInfo[service].serviceStatus;
+ return serviceStatus.health == 'false' ||
+ serviceStatus.healthchecks.length > 0;
+ },
+
+ markStale: function(service) {
+ this.serviceHealthStatusInfo[service].lastUpdatedTimestampInMs = 0;
+ },
+
+ getServiceActions: function(service) {
+ var actionSet = {};
+ var healthchecks =
+ this.serviceHealthStatusInfo[service].serviceStatus.healthchecks;
+
+ for (var i = 0; i < healthchecks.length; i++) {
+ for (var j = 0; j < healthchecks[i].actions.length; j++) {
+ actionSet[healthchecks[i].actions[j]] = true;
+ }
+ }
+
+ return Object.keys(actionSet);
+ }
+});
diff --git a/src/mobmonitor/static/js/main.js b/src/mobmonitor/static/js/main.js
new file mode 100644
index 0000000..f09d36c
--- /dev/null
+++ b/src/mobmonitor/static/js/main.js
@@ -0,0 +1,59 @@
+// Copyright 2015 The Chromium OS Authors. All rights reserved.
+// Use of this source code is governed by a BSD-style license that can be
+// found in the LICENSE file.
+
+'use strict';
+
+
+var HEALTH_DISPLAY_REFRESH_MS = 1000;
+
+
+$(document).ready(function() {
+ // Setup the health status widget.
+ $('#healthStatusDisplay').healthDisplay();
+ $('#healthStatusDisplay').healthDisplay('refreshHealthDisplay');
+
+ setInterval(function() {
+ $('#healthStatusDisplay').healthDisplay('refreshHealthDisplay');
+ }, HEALTH_DISPLAY_REFRESH_MS);
+
+ // Setup the log collection button.
+ $(document).on('click', '.collect-logs', function() {
+ window.open('/CollectLogs', '_blank');
+ });
+
+ // Setup the repair action buttons
+ $(document).on('click', '.run-repair-action', function() {
+ // Retrieve the service and action for this repair button.
+ var action = $(this).attr('action');
+ var healthcheck = $(this).closest('.healthcheck-info').attr('hcname');
+ var service = $(this).closest('.health-container').attr('id');
+ if (service.indexOf(SERVICE_CONTAINER_PREFIX) === 0) {
+ service = service.replace(SERVICE_CONTAINER_PREFIX, '');
+ }
+
+ // Do not launch dialog if this service does not need repair.
+ if (!$('#healthStatusDisplay').healthDisplay('needsRepair', service)) {
+ return;
+ }
+
+ function repairServiceCallback(response) {
+ $('#healthStatusDisplay').healthDisplay('markStale', response.service);
+ }
+
+ rpcActionInfo(service, healthcheck, action, function(response) {
+ if (isEmpty(response.args) && isEmpty(response.kwargs)) {
+ rpcRepairService(service, healthcheck, action,
+ [], {}, repairServiceCallback);
+ return;
+ }
+
+ var dialog = new ActionRepairDialog(service, response);
+ dialog.submitHandler = function(service, action, args, kwargs) {
+ rpcRepairService(service, healthcheck, action,
+ args, kwargs, repairServiceCallback);
+ };
+ dialog.open();
+ });
+ });
+});
diff --git a/src/mobmonitor/static/js/rpc.js b/src/mobmonitor/static/js/rpc.js
new file mode 100644
index 0000000..7b27e60
--- /dev/null
+++ b/src/mobmonitor/static/js/rpc.js
@@ -0,0 +1,50 @@
+// Copyright 2015 The Chromium OS Authors. All rights reserved.
+// Use of this source code is governed by a BSD-style license that can be
+// found in the LICENSE file.
+
+'use strict';
+
+
+function rpcGetServiceList(callback) {
+ $.getJSON('/GetServiceList', callback);
+}
+
+function rpcGetStatus(service, callback) {
+ $.getJSON('/GetStatus', {service: service}, callback);
+}
+
+function rpcActionInfo(service, healthcheck, action, callback) {
+ var data = {
+ service: service,
+ healthcheck: healthcheck,
+ action: action
+ };
+
+ $.getJSON('/ActionInfo', data, callback);
+}
+
+function rpcRepairService(service, healthcheck, action,
+ args, kwargs, callback) {
+
+ if (isEmpty(service))
+ throw new InvalidRpcArgumentError(
+ 'Must specify service in RepairService RPC');
+
+ if (isEmpty(healthcheck))
+ throw new InvalidRpcArgumentError(
+ 'Must specify healthcheck in RepairService RPC');
+
+ if (isEmpty(action))
+ throw new InvalidRpcArgumentError(
+ 'Must specify action in RepairService RPC');
+
+ var data = {
+ service: service,
+ healthcheck: healthcheck,
+ action: action,
+ args: JSON.stringify(args),
+ kwargs: JSON.stringify(kwargs)
+ };
+
+ $.post('/RepairService', data, callback, 'json');
+}
diff --git a/src/mobmonitor/static/js/template.js b/src/mobmonitor/static/js/template.js
new file mode 100644
index 0000000..7950f0c
--- /dev/null
+++ b/src/mobmonitor/static/js/template.js
@@ -0,0 +1,31 @@
+// Copyright 2015 The Chromium OS Authors. All rights reserved.
+// Use of this source code is governed by a BSD-style license that can be
+// found in the LICENSE file.
+
+'use strict';
+
+
+function renderTemplate(templateName, templateData) {
+ if (!renderTemplate.cache) {
+ renderTemplate.cache = {};
+ }
+
+ if (!renderTemplate.cache[templateName]) {
+ var dir = '/static/templates';
+ var url = dir + '/' + templateName + '.html';
+
+ var templateString;
+ $.ajax({
+ url: url,
+ method: 'GET',
+ async: false,
+ success: function(data) {
+ templateString = data;
+ }
+ });
+
+ renderTemplate.cache[templateName] = Handlebars.compile(templateString);
+ }
+
+ return renderTemplate.cache[templateName](templateData);
+}
diff --git a/src/mobmonitor/static/js/util.js b/src/mobmonitor/static/js/util.js
new file mode 100644
index 0000000..1609c9b
--- /dev/null
+++ b/src/mobmonitor/static/js/util.js
@@ -0,0 +1,20 @@
+// Copyright 2015 The Chromium OS Authors. All rights reserved.
+// Use of this source code is governed by a BSD-style license that can be
+// found in the LICENSE file.
+
+'use strict';
+
+var SERVICE_CONTAINER_PREFIX = 'serviceHealthContainer';
+
+
+function InvalidRpcArgumentError(message) {
+ this.message = message;
+}
+InvalidRpcArgumentError.prototype = new Error;
+
+function isEmpty(x) {
+ if (typeof(x) === 'undefined' || x === null)
+ return true;
+
+ return $.isEmptyObject(x);
+}
diff --git a/src/mobmonitor/static/templates/actionrepairdialog.html b/src/mobmonitor/static/templates/actionrepairdialog.html
new file mode 100644
index 0000000..3a4fb5b
--- /dev/null
+++ b/src/mobmonitor/static/templates/actionrepairdialog.html
@@ -0,0 +1,13 @@
+<div id="actionRepairDialog" title="Setup {{action}}">
+ <label for="actionDescription">Description:</label>
+ <textarea style="font-size:10pt" readonly name="actionDescription" id="actionDescription" class="actionDisplay">{{info}}
+ </textarea><br><br>
+
+ <label for="args">Arguments:</label><br>
+ <label for="args" id="argsHelp" class="label-help">Help text for args</label>
+ <input style="font-size:10pt" type="text" name="args" id="args" value="" class="input-text">
+
+ <label for="kwargs">Keyword Arguments:</label>
+ <label for="kwargs" id="kwargsHelp" class="label-help">Help text for kwargs</label>
+ <input style="font-size:10pt" type="text" name="kwargs" id="kwargs" value="" class="input-text">
+</div>
diff --git a/src/mobmonitor/static/templates/healthstatuscontainer.html b/src/mobmonitor/static/templates/healthstatuscontainer.html
new file mode 100644
index 0000000..2840094
--- /dev/null
+++ b/src/mobmonitor/static/templates/healthstatuscontainer.html
@@ -0,0 +1,80 @@
+<div id={{serviceId}} class="health-container">
+ <div class="health-container-header">
+ {{#if healthchecks.length}}
+ {{#if health}}
+ <div class="circle-quasi-healthy health-container-header-element"></div>
+ <div class="health-container-header-element"><b>{{service}}</b> is <font class="color-quasi-healthy">healthy</font>.</div>
+ {{else}}
+ <div class="circle-unhealthy health-container-header-element"></div>
+ <div class="health-container-header-element"><b>{{service}}</b> is <font class="color-unhealthy">unhealthy</font>.</div>
+ {{/if}}
+ {{else}}
+ <div class="circle-healthy health-container-header-element"></div>
+ <div class="health-container-header-element"><b>{{service}}</b> is <font class="color-healthy">healthy</font>.</div>
+ {{/if}}
+
+ <!--
+ Clear must be used or else floating causes the height of other
+ divs to warp.
+ -->
+ <div style="clear:both;"</div>
+ </div>
+
+ {{#if healthchecks.length}}
+ <div class="health-container-content">
+ {{#if errors.length}}
+ <b>Errors:</b>
+ <table class="healthcheck-table">
+ <tr>
+ <td class="bold healthcheck-table-column1">Issue Description</td>
+ <td class="bold healthcheck-table-column2">Recommended Action(s)</td>
+ </tr>
+ {{#each errors}}
+ <tr class="healthcheck-info" hcname="{{name}}">
+ <td>{{description}}</td>
+ <td>
+ <ul>
+ {{#each actions}}
+ <li>
+ <button class="run-repair-action" action="{{this}}">Run</button>
+ {{this}}
+ </li>
+ {{/each}}
+ </ul>
+ </td>
+ </tr>
+ {{/each}}
+ </table>
+ {{/if}}
+
+ {{#if warnings.length}}
+ {{#if errors.length}}
+ <br>
+ {{/if}}
+ <b>Warnings:</b>
+ <table class="healthcheck-table">
+ <tr>
+ <td class="bold healthcheck-table-column1">Issue Description</td>
+ <td class="bold healthcheck-table-column2">Recommended Action(s)</td>
+ </tr>
+ {{#each warnings}}
+ <tr class="healthcheck-info" hcname="{{name}}">
+ <td>{{description}}</td>
+ <td>
+ <ul>
+ {{#each actions}}
+ <li>
+ <button class="run-repair-action" action="{{this}}">Run</button>
+ {{this}}
+ </li>
+ {{/each}}
+ </ul>
+ </td>
+ </tr>
+ {{/each}}
+ </table>
+ {{/if}}
+ </div>
+ {{/if}}
+
+</div>
diff --git a/src/mobmonitor/static/templates/index.html b/src/mobmonitor/static/templates/index.html
new file mode 100644
index 0000000..7365389
--- /dev/null
+++ b/src/mobmonitor/static/templates/index.html
@@ -0,0 +1,29 @@
+<html>
+ <head>
+ <title>Mob* Monitor</title>
+ <link href="/static/css/style.css" rel="stylesheet">
+ <link href="https://ajax.googleapis.com/ajax/libs/jqueryui/1.11.4/themes/smoothness/jquery-ui.css" rel="stylesheet">
+ <script src="https://ajax.googleapis.com/ajax/libs/jquery/2.1.3/jquery.min.js"></script>
+ <script src="https://ajax.googleapis.com/ajax/libs/jqueryui/1.11.4/jquery-ui.min.js"></script>
+ <script src="https://cdnjs.cloudflare.com/ajax/libs/handlebars.js/3.0.3/handlebars.js"></script>
+ <script src="/static/js/util.js"></script>
+ <script src="/static/js/template.js"></script>
+ <script src="/static/js/rpc.js"></script>
+ <script src="/static/js/actionrepairdialog.js"></script>
+ <script src="/static/js/healthdisplay.js"></script>
+ <script src="/static/js/main.js"></script>
+ </head>
+
+ <body>
+ <header class="site-header">
+ <div class="site-header-title">Mob* Monitor</div>
+ </header>
+
+ <button class="collect-logs">Collect Logs</button>
+ <br><br>
+
+ <div id="healthStatusDisplay">
+ </div>
+
+ </body>
+</html>
diff --git a/src/mobmonitor/system/__init__.py b/src/mobmonitor/system/__init__.py
new file mode 100644
index 0000000..e69de29
--- /dev/null
+++ b/src/mobmonitor/system/__init__.py
diff --git a/src/mobmonitor/system/systeminfo.py b/src/mobmonitor/system/systeminfo.py
new file mode 100644
index 0000000..cedac04
--- /dev/null
+++ b/src/mobmonitor/system/systeminfo.py
@@ -0,0 +1,494 @@
+# -*- coding: utf-8 -*-
+# Copyright 2015 The Chromium OS Authors. All rights reserved.
+# Use of this source code is governed by a BSD-style license that can be
+# found in the LICENSE file.
+
+"""This module allows for easy access to common system information.
+
+In order to reduce stress due to excessive checking of system information,
+the records collected by the classes in this file are stored with a period
+of validity. If a record is requested, and there is an existing record
+of the same type, and it is still valid, it is returned. If there is no
+existing record, or it has gone stale, then the system information is
+actually collected.
+
+Each class uses a few attributes to regulate when they should do
+a collection, they are:
+ update_sec: An integer that is the period of validity for a record.
+ update_times: A dictionary that maps a resource name to an epoch time.
+ resources: A dictionary that maps a resource name to the actual record.
+
+On retrieving a new record, the time at which it is collected is stored
+in update_times with the record name as key, and the record itself is
+stored in resources. Every subsequent collection returns what is stored
+in the resources dict until the record goes stale.
+
+Users should not directly access the system information classes, but
+should instead use the 'getters' (ie. GetCpu, GetDisk, GetMemory) defined
+at the bottom of this file.
+
+Each of these getters is decorated with the CacheInfoClass decorator.
+This decorator caches instances of each storage class by the specified
+update interval. With this, multiple checkfiles can access the same
+information class instance, which can help to reduce additional and
+redundant system checks being performed.
+"""
+
+from __future__ import print_function
+
+import collections
+import functools
+import itertools
+import os
+import time
+
+from util import osutils
+
+
+SYSTEMFILE_PROC_MOUNTS = '/proc/mounts'
+SYSTEMFILE_PROC_MEMINFO = '/proc/meminfo'
+SYSTEMFILE_PROC_FILESYSTEMS = '/proc/filesystems'
+SYSTEMFILE_PROC_STAT = '/proc/stat'
+SYSTEMFILE_DEV_DISKBY = {
+ 'ids': '/dev/disk/by-id',
+ 'labels': '/dev/disk/by-label',
+}
+
+
+UPDATE_DEFAULT_SEC = 30
+UPDATE_MEMORY_SEC = UPDATE_DEFAULT_SEC
+UPDATE_DISK_SEC = UPDATE_DEFAULT_SEC
+UPDATE_CPU_SEC = 2
+
+RESOURCENAME_MEMORY = 'memory'
+RESOURCENAME_DISKPARTITIONS = 'diskpartitions'
+RESOURCENAME_DISKUSAGE = 'diskusage'
+RESOURCENAME_BLOCKDEVICE = 'blockdevice'
+RESOURCENAME_CPUPREVTIMES = 'cpuprevtimes'
+RESOURCENAME_CPUTIMES = 'cputimes'
+RESOURCENAME_CPULOADS = 'cpuloads'
+
+RESOURCE_MEMORY = collections.namedtuple('memory', ['total', 'available',
+ 'percent_used'])
+RESOURCE_DISKPARTITION = collections.namedtuple('diskpartition',
+ ['devicename', 'mountpoint',
+ 'filesystem'])
+RESOURCE_DISKUSAGE = collections.namedtuple('diskusage', ['total', 'used',
+ 'free',
+ 'percent_used'])
+RESOURCE_BLOCKDEVICE = collections.namedtuple('blockdevice',
+ ['device', 'size', 'ids',
+ 'labels'])
+RESOURCE_CPUTIME = collections.namedtuple('cputime', ['cpu', 'total', 'idle',
+ 'nonidle'])
+RESOURCE_CPULOAD = collections.namedtuple('cpuload', ['cpu', 'load'])
+
+
+CPU_ATTRS = ('cpu', 'user', 'nice', 'system', 'idle', 'iowait', 'irq',
+ 'softirq', 'steal', 'guest', 'guest_nice')
+CPU_IDLE_ATTRS = frozenset(['idle', 'iowait'])
+CPU_NONIDLE_ATTRS = frozenset(['user', 'nice', 'system', 'irq', 'softirq'])
+
+
+def CheckStorage(resource_basename):
+ """Decorate information functions to retrieve stored records if valid.
+
+ Args:
+ resource_basename: The data value basename we are checking our
+ local storage for.
+
+ Returns:
+ The real function decorator.
+ """
+
+ def func_deco(func):
+ """Return stored record if valid, else run function and update storage.
+
+ Args:
+ func: The collection function we are executing.
+
+ Returns:
+ The function wrapper.
+ """
+
+ @functools.wraps(func)
+ def wrapper(self, *args, **kwargs):
+ """Function wrapper.
+
+ Args:
+ args: Positional arguments that will be appended to dataname when
+ searching the local storage.
+
+ Returns:
+ The stored record or the new record as a result of running the
+ collection function.
+ """
+ dataname = resource_basename
+ if args:
+ dataname = '%s:%s' % (dataname, args[0])
+
+ if not self.NeedToUpdate(dataname):
+ return self.resources.get(dataname)
+
+ datavalue = func(self, *args, **kwargs)
+ self.Update(dataname, datavalue)
+
+ return datavalue
+
+ return wrapper
+
+ return func_deco
+
+
+def CacheInfoClass(class_name, update_default_sec):
+ """Cache system information class instances by update_sec interval time.
+
+ Args:
+ class_name: The name of the system information class.
+ update_default_sec: The default update interval for this class.
+
+ Returns:
+ The real function decorator.
+ """
+ def func_deco(func):
+ """Return the cached class instance.
+
+ Args:
+ func: The system information class 'getter'.
+
+ Returns:
+ The function wrapper.
+ """
+ cache = {}
+
+ @functools.wraps(func)
+ def wrapper(update_sec=update_default_sec):
+ """Function wrapper for caching system information class objects.
+
+ Args:
+ update_sec: The update interval for the class instance.
+
+ Returns:
+ The cached class instance that has this update interval.
+ """
+ key = '%s:%s' % (class_name, update_sec)
+
+ if key not in cache:
+ cache[key] = func(update_sec=update_sec)
+
+ return cache[key]
+
+ return wrapper
+
+ return func_deco
+
+
+class SystemInfoStorage(object):
+ """Store and access system information."""
+
+ def __init__(self, update_sec=UPDATE_DEFAULT_SEC):
+ self.update_sec = update_sec
+ self.update_times = {}
+ self.resources = {}
+
+ def Update(self, resource_name, data):
+ """Update local storage and collection times of the data.
+
+ Args:
+ resource_name: The key used for local storage and update times.
+ data: The data to store that is keyed by resource_name.
+ """
+ self.update_times[resource_name] = time.time()
+ self.resources[resource_name] = data
+
+ def NeedToUpdate(self, resource_name):
+ """Check if the record keyed by resource_name needs to be (re-)collected.
+
+ Args:
+ resource_name: A string representing some system value.
+
+ Returns:
+ A boolean. If True, the data must be collected. If False, the data
+ be retrieved from the self.resources dict with key resource_name.
+ """
+ if resource_name not in self.resources:
+ return True
+
+ if resource_name not in self.update_times:
+ return True
+
+ return time.time() > self.update_sec + self.update_times.get(resource_name)
+
+
+class Memory(SystemInfoStorage):
+ """Access memory information."""
+
+ def __init__(self, update_sec=UPDATE_MEMORY_SEC):
+ super(Memory, self).__init__(update_sec=update_sec)
+
+ @CheckStorage(RESOURCENAME_MEMORY)
+ def MemoryUsage(self):
+ """Collect memory information from /proc/meminfo.
+
+ Returns:
+ A named tuple with the following fields:
+ total: Corresponds to MemTotal of /proc/meminfo.
+ available: Corresponds to (MemFree+Buffers+Cached) of /proc/meminfo.
+ percent_used: The percentage of memory that is used based on
+ total and available.
+ """
+ # See MOCK_PROC_MEMINFO in the unittest file for this module
+ # to see an example of the file this function is reading from.
+ memtotal, memfree, buffers, cached = (0, 0, 0, 0)
+ with open(SYSTEMFILE_PROC_MEMINFO, 'rb') as f:
+ for line in f:
+ if line.startswith('MemTotal'):
+ memtotal = int(line.split()[1]) * 1024
+ if line.startswith('MemFree'):
+ memfree = int(line.split()[1]) * 1024
+ if line.startswith('Buffers'):
+ buffers = int(line.split()[1]) * 1024
+ if line.startswith('Cached'):
+ cached = int(line.split()[1]) * 1024
+
+ available = memfree + buffers + cached
+ percent_used = float(memtotal - available) / memtotal * 100
+
+ memory = RESOURCE_MEMORY(memtotal, available, percent_used)
+
+ return memory
+
+
+class Disk(SystemInfoStorage):
+ """Access disk information."""
+
+ def __init__(self, update_sec=UPDATE_DISK_SEC):
+ super(Disk, self).__init__(update_sec=update_sec)
+
+ @CheckStorage(RESOURCENAME_DISKPARTITIONS)
+ def DiskPartitions(self):
+ """Collect basic information about disk partitions.
+
+ Returns:
+ A list of named tuples. Each named tuple has the following fields:
+ devicename: The name of the partition.
+ mountpoint: The mount point of the partition.
+ filesystem: The file system in use on the partition.
+ """
+ # Read /proc/mounts for mounted filesystems.
+ # See MOCK_PROC_MOUNTS in the unittest file for this module
+ # to see an example of the file this function is reading from.
+ mounts = []
+ with open(SYSTEMFILE_PROC_MOUNTS, 'rb') as f:
+ for line in f:
+ iterline = iter(line.split())
+ try:
+ mounts.append([next(iterline), next(iterline), next(iterline)])
+ except StopIteration:
+ pass
+
+ # Read /proc/filesystems for a list of physical filesystems
+ # See MOCK_PROC_FILESYSTEMS in the unittest file for this module
+ # to see an example of the file this function is reading from.
+ physmounts = []
+ with open(SYSTEMFILE_PROC_FILESYSTEMS, 'rb') as f:
+ for line in f:
+ if not line.startswith('nodev'):
+ physmounts.append(line.strip())
+
+ # From these two sources, create a list of partitions
+ diskpartitions = []
+ for mountname, mountpoint, filesystem in mounts:
+ if filesystem not in physmounts:
+ continue
+ diskpartition = RESOURCE_DISKPARTITION(mountname, mountpoint, filesystem)
+ diskpartitions.append(diskpartition)
+
+ return diskpartitions
+
+ @CheckStorage(RESOURCENAME_DISKUSAGE)
+ def DiskUsage(self, partition):
+ """Collects usage information for the specified partition.
+
+ Args:
+ partition: The partition for which to check usage. This is the
+ same as the 'devicename' attribute given in the return value
+ of DiskPartitions.
+
+ Returns:
+ A named tuple with the following fields:
+ total: The total space on the partition.
+ used: The total amount of used space on the parition.
+ free: The total amount of unused space on the partition.
+ percent_used: The percentage of the partition that is used
+ based on total and used.
+ """
+ # Collect the partition information
+ vfsdata = os.statvfs(partition)
+ total = vfsdata.f_frsize * vfsdata.f_blocks
+ free = vfsdata.f_bsize * vfsdata.f_bfree
+ used = total - free
+ percent_used = float(used) / total * 100
+
+ diskusage = RESOURCE_DISKUSAGE(total, used, free, percent_used)
+
+ return diskusage
+
+ @CheckStorage(RESOURCENAME_BLOCKDEVICE)
+ def BlockDevices(self, device=''):
+ """Collects information about block devices.
+
+ This method combines information from:
+ (1) Reading through the SYSTEMFILE_DEV_DISKBY directories.
+ (2) Executing the 'lsblk' command provided by osutils.ListBlockDevices.
+
+ Returns:
+ A list of named tuples. Each tuple has the following fields:
+ device: The name of the block device.
+ size: The size of the block device in bytes.
+ ids: A list of ids assigned to this device.
+ labels: A list of labels assigned to this device.
+ """
+ devicefilter = os.path.basename(device)
+
+ # Data collected from the SYSTEMFILE_DEV_DISKBY directories.
+ ids = {}
+ labels = {}
+
+ # Data collected from 'lsblk'.
+ sizes = {}
+
+ # Collect diskby information.
+ for prop, diskdir in SYSTEMFILE_DEV_DISKBY.iteritems():
+ cmd = ['find', diskdir, '-lname', '*%s' % devicefilter]
+
+ output = osutils.run_command(cmd)
+
+ if not output:
+ continue
+
+ results = output.split()
+ for result in results:
+ devicename = os.path.abspath(osutils.resolve_symlink(result))
+ result = os.path.basename(result)
+
+ # Ensure that each of our data dicts have the same keys.
+ ids.setdefault(devicename, [])
+ labels.setdefault(devicename, [])
+ sizes.setdefault(devicename, 0)
+
+ if 'ids' == prop:
+ ids[devicename].append(result)
+ elif 'labels' == prop:
+ labels[devicename].append(result)
+
+ # Collect lsblk information.
+ for device in osutils.list_block_devices():
+ devicename = os.path.join('/dev', device.NAME)
+ if devicename in ids:
+ sizes[devicename] = int(device.SIZE)
+
+ return [RESOURCE_BLOCKDEVICE(device, sizes[device], ids[device],
+ labels[device])
+ for device in ids.iterkeys()]
+
+
+class Cpu(SystemInfoStorage):
+ """Access CPU information."""
+
+ def __init__(self, update_sec=UPDATE_CPU_SEC):
+ super(Cpu, self).__init__(update_sec=update_sec)
+
+ # CpuLoad depends on having two CpuTime collections at different
+ # points in time. One issue, is that the first call to CpuLoad,
+ # without prior calls to CpuTime will return a trivial value, that
+ # is, all cpus will be reported to have zero load. We solve this
+ # by doing an initial CpuTime collection here.
+ self.CpuTime()
+ self.update_times.pop(RESOURCENAME_CPUTIMES)
+ self.Update(RESOURCENAME_CPUPREVTIMES,
+ self.resources.pop(RESOURCENAME_CPUTIMES))
+
+ @CheckStorage(RESOURCENAME_CPUTIMES)
+ def CpuTime(self):
+ """Collect information on CPU time.
+
+ Returns:
+ A list of named tuples. Each named tuple has the following fields:
+ cpu: An identifier for the CPU.
+ total: The total CPU time in the measurement.
+ idle: The total time spent in an idle state.
+ nonidle: The total time spent not in an idle state.
+ """
+ # Collect CPU time information from /proc/stat
+ cputimes = []
+
+ # See MOCK_PROC_STAT in the unittest file for this module
+ # to see an example of the file this function is reading from.
+ with open(SYSTEMFILE_PROC_STAT, 'rb') as f:
+ for line in f:
+ if not line.startswith('cpu'):
+ continue
+ cpudesc = dict(itertools.izip(CPU_ATTRS, line.split()))
+ idle, nonidle = (0, 0)
+ for attr, value in cpudesc.iteritems():
+ if attr in CPU_IDLE_ATTRS:
+ idle += int(value)
+ if attr in CPU_NONIDLE_ATTRS:
+ nonidle += int(value)
+ total = idle + nonidle
+ cputimes.append(RESOURCE_CPUTIME(cpudesc.get('cpu'),
+ total, idle, nonidle))
+
+ # Store the previous cpu times if we have a 'current' measurement
+ # that is about to be replaced. This is very helpful for calculating
+ # load estimates over the update interval.
+ if RESOURCENAME_CPUTIMES in self.resources:
+ self.Update(RESOURCENAME_CPUPREVTIMES,
+ self.resources.get(RESOURCENAME_CPUTIMES))
+
+ return cputimes
+
+ @CheckStorage(RESOURCENAME_CPULOADS)
+ def CpuLoad(self):
+ """Estimate the CPU load.
+
+ Returns:
+ A list of named tuples. Each name tuple has the following fields:
+ cpu: An identifier for the CPU.
+ load: A number representing the load/usage ranging between 0 and 1.
+ """
+ prevcputimes = self.resources.get(RESOURCENAME_CPUPREVTIMES)
+ cputimes = self.CpuTime()
+ cpuloads = []
+ for prevtime, curtime in itertools.izip(prevcputimes, cputimes):
+ ct = curtime.total
+ ci = curtime.idle
+ pt = prevtime.total
+ pi = prevtime.idle
+
+ # Cpu load is estimated using a difference between cpu timing collections
+ # taken at different points in time. To estimate how much time in that
+ # interval was spent in a non-idle state, we calculate the percent change
+ # of the non-idle time using the relative differences in total and idle
+ # time between the two collections.
+ cpu = curtime.cpu
+ load = float((ct-pt)-(ci-pi))/(ct-pt) if (ct-pt) != 0 else 0
+ cpuloads.append(RESOURCE_CPULOAD(cpu, load))
+
+ return cpuloads
+
+
+@CacheInfoClass(Cpu.__name__, UPDATE_CPU_SEC)
+def GetCpu(update_sec=UPDATE_CPU_SEC):
+ return Cpu(update_sec=update_sec)
+
+
+@CacheInfoClass(Memory.__name__, UPDATE_MEMORY_SEC)
+def GetMemory(update_sec=UPDATE_MEMORY_SEC):
+ return Memory(update_sec=update_sec)
+
+
+@CacheInfoClass(Disk.__name__, UPDATE_DISK_SEC)
+def GetDisk(update_sec=UPDATE_DISK_SEC):
+ return Disk(update_sec=update_sec)
diff --git a/src/mobmonitor/system/systeminfo_unittest.py b/src/mobmonitor/system/systeminfo_unittest.py
new file mode 100644
index 0000000..25fee4a
--- /dev/null
+++ b/src/mobmonitor/system/systeminfo_unittest.py
@@ -0,0 +1,516 @@
+# -*- coding: utf-8 -*-
+# Copyright 2015 The Chromium OS Authors. All rights reserved.
+# Use of this source code is governed by a BSD-style license that can be
+# found in the LICENSE file.
+
+"""Unittests for systeminfo."""
+
+from __future__ import print_function
+
+import collections
+import io
+import mock
+import os
+import time
+import unittest
+
+# necessary in order to get systeminfo.py to import osutils correctly
+import sys
+sys.path.append('../') # we can run the test from mobmonitor/system/
+sys.path.append('.') # or from mobmonitor/
+import systeminfo
+
+# Strings that are used to mock particular system files for testing.
+MOCK_PROC_MEMINFO = u'''
+MemTotal: 65897588 kB
+MemFree: 24802380 kB
+Buffers: 1867288 kB
+Cached: 29458948 kB
+SwapCached: 0 kB
+Active: 34858204 kB
+Inactive: 3662692 kB
+Active(anon): 7223176 kB
+Inactive(anon): 136892 kB
+Active(file): 27635028 kB
+Inactive(file): 3525800 kB
+Unevictable: 27268 kB
+Mlocked: 27268 kB
+SwapTotal: 67035132 kB
+SwapFree: 67035132 kB
+Dirty: 304 kB
+Writeback: 0 kB
+AnonPages: 7221488 kB
+Mapped: 593360 kB
+Shmem: 139136 kB
+Slab: 1740892 kB
+SReclaimable: 1539144 kB
+SUnreclaim: 201748 kB
+KernelStack: 21384 kB
+PageTables: 103680 kB
+NFS_Unstable: 0 kB
+Bounce: 0 kB
+WritebackTmp: 0 kB
+CommitLimit: 99983924 kB
+Committed_AS: 19670964 kB
+VmallocTotal: 34359738367 kB
+VmallocUsed: 359600 kB
+VmallocChunk: 34325703384 kB
+HardwareCorrupted: 0 kB
+AnonHugePages: 2924544 kB
+HugePages_Total: 0
+HugePages_Free: 0
+HugePages_Rsvd: 0
+HugePages_Surp: 0
+Hugepagesize: 2048 kB
+DirectMap4k: 1619840 kB
+DirectMap2M: 46536704 kB
+DirectMap1G: 18874368 kB
+'''
+
+MOCK_PROC_MOUNTS = u'''
+rootfs / rootfs rw 0 0
+sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0
+proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0
+udev /dev devtmpfs rw,relatime,size=32935004k,nr_inodes=8233751,mode=755 0 0
+devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0
+tmpfs /run tmpfs rw,nosuid,noexec,relatime,size=6589760k,mode=755 0 0
+securityfs /sys/kernel/security securityfs rw,relatime 0 0
+/dev/mapper/dhcp--100--106--128--134--vg-root / ext4 rw,relatime,errors=remount-ro,i_version,data=ordered 0 0
+none /sys/fs/cgroup tmpfs rw,relatime,size=4k,mode=755 0 0
+none /sys/fs/fuse/connections fusectl rw,relatime 0 0
+none /sys/kernel/debug debugfs rw,relatime 0 0
+none /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0
+none /run/shm tmpfs rw,nosuid,nodev,relatime 0 0
+none /run/user tmpfs rw,nosuid,nodev,noexec,relatime,size=102400k,mode=755 0 0
+none /sys/fs/pstore pstore rw,relatime 0 0
+/dev/sdb1 /work ext4 rw,relatime,data=ordered 0 0
+/dev/sda1 /boot ext2 rw,relatime,i_version,stripe=4 0 0
+none /dev/cgroup/cpu cgroup rw,relatime,cpuacct,cpu 0 0
+none /dev/cgroup/devices cgroup rw,relatime,devices 0 0
+/dev/sdb1 /usr/local/autotest ext4 rw,relatime,data=ordered 0 0
+rpc_pipefs /run/rpc_pipefs rpc_pipefs rw,relatime 0 0
+/dev/mapper/dhcp--100--106--128--134--vg-usr+local+google /usr/local/google ext4 rw,relatime,i_version,data=writeback 0 0
+binfmt_misc /proc/sys/fs/binfmt_misc binfmt_misc rw,nosuid,nodev,noexec,relatime 0 0
+systemd /sys/fs/cgroup/systemd cgroup rw,nosuid,nodev,noexec,relatime,name=systemd 0 0
+/etc/auto.auto /auto autofs rw,relatime,fd=7,pgrp=2440,timeout=600,minproto=5,maxproto=5,indirect 0 0
+/etc/auto.home /home autofs rw,relatime,fd=13,pgrp=2440,timeout=600,minproto=5,maxproto=5,indirect 0 0
+srcfsd /google/src fuse.srcfsd rw,nosuid,nodev,relatime,user_id=906,group_id=65534,default_permissions,allow_other 0 0
+/dev/mapper/dhcp--100--106--128--134--vg-root /home/build ext4 rw,relatime,errors=remount-ro,i_version,data=ordered 0 0
+/dev/mapper/dhcp--100--106--128--134--vg-root /auto/buildstatic ext4 rw,relatime,errors=remount-ro,i_version,data=ordered 0 0
+cros /sys/fs/cgroup/cros cgroup rw,relatime,cpuset 0 0
+gvfsd-fuse /run/user/307266/gvfs fuse.gvfsd-fuse rw,nosuid,nodev,relatime,user_id=307266,group_id=5000 0 0
+objfsd /google/obj fuse.objfsd rw,nosuid,nodev,relatime,user_id=903,group_id=65534,default_permissions,allow_other 0 0
+x20fsd /google/data fuse.x20fsd rw,nosuid,nodev,relatime,user_id=904,group_id=65534,allow_other 0 0
+'''
+
+MOCK_PROC_FILESYSTEMS = u'''
+nodev sysfs
+nodev rootfs
+nodev ramfs
+nodev bdev
+nodev proc
+nodev cgroup
+nodev cpuset
+nodev tmpfs
+nodev devtmpfs
+nodev debugfs
+nodev securityfs
+nodev sockfs
+nodev pipefs
+nodev anon_inodefs
+nodev devpts
+ ext3
+ ext2
+ ext4
+nodev hugetlbfs
+ vfat
+nodev ecryptfs
+ fuseblk
+nodev fuse
+nodev fusectl
+nodev pstore
+nodev mqueue
+nodev rpc_pipefs
+nodev nfs
+nodev nfs4
+nodev nfsd
+nodev binfmt_misc
+nodev autofs
+ xfs
+ jfs
+ msdos
+ ntfs
+ minix
+ hfs
+ hfsplus
+ qnx4
+ ufs
+ btrfs
+'''
+
+MOCK_PROC_STAT = u'''
+cpu 36790888 1263256 11740848 7297360559 3637601 444 116964 0 0 0
+intr 3386552889 27 3 0 0 0 0 0 0 1 0 0 0 4 0 0 0 33 26 0 0 0 0 0 3508578 0 0 0
+ctxt 9855640605
+btime 1431536959
+processes 3593799
+procs_running 2
+procs_blocked 0
+softirq 652938841 3508230 208834832 49297758 52442647 16919039 0 47196001 147658748 3188900 123892686
+'''
+
+
+class SystemInfoStorageTest(unittest.TestCase):
+ """Unittests for SystemInfoStorage."""
+
+ def _CreateSystemInfoStorage(self, update_sec):
+ """Setup a SystemInfoStorage object."""
+ return systeminfo.SystemInfoStorage(update_sec=update_sec)
+
+ def testUpdate(self):
+ """Test SystemInfoStorage Update."""
+ si = self._CreateSystemInfoStorage(1)
+ si.Update('test', 'testvalue')
+ self.assertEquals(si.resources.get('test'), 'testvalue')
+
+ def testNeedToUpdateNoData(self):
+ """Test NeedToUpdate with no data."""
+ si = self._CreateSystemInfoStorage(1)
+ self.assertEquals(si.NeedToUpdate('test'), True)
+
+ def testNeedToUpdateNoUpdateTime(self):
+ """Test NeedToUpdate with data that has no update time recorded."""
+ si = self._CreateSystemInfoStorage(1)
+ dataname, data = ('testname', 'testvalue')
+ si.resources[dataname] = data
+ self.assertEquals(si.NeedToUpdate(dataname), True)
+
+ @mock.patch('systeminfo.time')
+ def testNeedToUpdateStaleData(self, MockTime):
+ """Test NeedToUpdate with data that is out of date."""
+ si = self._CreateSystemInfoStorage(1)
+
+ dataname, data = ('testname', 'testvalue')
+ curtime = time.time()
+
+ si.resources[dataname] = data
+ si.update_times[dataname] = curtime
+
+ MockTime.time.return_value = curtime + 2
+
+ self.assertEquals(si.NeedToUpdate(dataname), True)
+
+ @mock.patch('systeminfo.time')
+ def testNeedToUpdateFreshData(self, MockTime):
+ """Test NeedToUpdate with data that is not out of date."""
+ si = self._CreateSystemInfoStorage(1)
+
+ dataname, data = ('testname', 'testvalue')
+ curtime = time.time()
+
+ si.resources[dataname] = data
+ si.update_times[dataname] = curtime
+
+ MockTime.time.return_value = curtime + 0.5
+
+ self.assertEquals(si.NeedToUpdate(dataname), False)
+
+
+class MemoryTest(unittest.TestCase):
+ """Unittests for Memory."""
+
+ def _CreateMemory(self, update_sec):
+ """Setup a Memory object."""
+ return systeminfo.Memory(update_sec=update_sec)
+
+ def testMemoryExisting(self):
+ """Test memory information collection when a record exists."""
+ mem = self._CreateMemory(1)
+
+ dataname, data = (systeminfo.RESOURCENAME_MEMORY, 'testvalue')
+ mem.Update(dataname, data)
+
+ self.assertEquals(mem.MemoryUsage(), 'testvalue')
+
+ def testMemory(self):
+ """Test memory info when there is no record, or record is stale."""
+ mem = self._CreateMemory(1)
+
+ mock_file = io.StringIO(MOCK_PROC_MEMINFO)
+ with mock.patch('__builtin__.open', return_value=mock_file, create=True):
+ mem.MemoryUsage()
+
+ # The correct values that mem.MemoryUsage() should produce based on the
+ # mocked /proc/meminfo file.
+ mock_memtotal = 67479130112
+ mock_available = 57475702784
+ mock_percent_used = 14.824475821482267
+
+ mock_memory = systeminfo.RESOURCE_MEMORY(mock_memtotal, mock_available,
+ mock_percent_used)
+
+ self.assertEquals(mem.resources.get(systeminfo.RESOURCENAME_MEMORY),
+ mock_memory)
+
+
+class DiskTest(unittest.TestCase):
+ """Unittests for Disk."""
+
+ def _CreateDisk(self, update_sec):
+ """Setup a Disk object."""
+ return systeminfo.Disk(update_sec=update_sec)
+
+ def testDiskPartitionsExisting(self):
+ """Test disk partition information collection when a record exists."""
+ disk = self._CreateDisk(1)
+
+ dataname, data = (systeminfo.RESOURCENAME_DISKPARTITIONS, 'testvalue')
+ disk.Update(dataname, data)
+
+ self.assertEquals(disk.DiskPartitions(), 'testvalue')
+
+ def testDiskPartitions(self):
+ """Test disk partition info when there is no record, or record is stale."""
+ disk = self._CreateDisk(1)
+
+ mock_mounts_file = io.StringIO(MOCK_PROC_MOUNTS)
+ mock_filesystems_file = io.StringIO(MOCK_PROC_FILESYSTEMS)
+
+ def _file_returner(fname, _mode):
+ if systeminfo.SYSTEMFILE_PROC_MOUNTS == fname:
+ return mock_mounts_file
+ elif systeminfo.SYSTEMFILE_PROC_FILESYSTEMS == fname:
+ return mock_filesystems_file
+
+ with mock.patch('__builtin__.open', side_effect=_file_returner,
+ create=True):
+ disk.DiskPartitions()
+
+ # The expected return value.
+ mock_disk_partitions = [
+ systeminfo.RESOURCE_DISKPARTITION(
+ '/dev/mapper/dhcp--100--106--128--134--vg-root', '/', 'ext4'),
+ systeminfo.RESOURCE_DISKPARTITION('/dev/sdb1', '/work', 'ext4'),
+ systeminfo.RESOURCE_DISKPARTITION('/dev/sda1', '/boot', 'ext2'),
+ systeminfo.RESOURCE_DISKPARTITION(
+ '/dev/sdb1', '/usr/local/autotest', 'ext4'),
+ systeminfo.RESOURCE_DISKPARTITION(
+ '/dev/mapper/dhcp--100--106--128--134--vg-usr+local+google',
+ '/usr/local/google', 'ext4'),
+ systeminfo.RESOURCE_DISKPARTITION(
+ '/dev/mapper/dhcp--100--106--128--134--vg-root',
+ '/home/build', 'ext4'),
+ systeminfo.RESOURCE_DISKPARTITION(
+ '/dev/mapper/dhcp--100--106--128--134--vg-root',
+ '/auto/buildstatic', 'ext4')]
+
+ self.assertEquals(
+ disk.resources.get(systeminfo.RESOURCENAME_DISKPARTITIONS),
+ mock_disk_partitions)
+
+ def testDiskUsageExisting(self):
+ """Test disk usage information collection when a record exists."""
+ disk = self._CreateDisk(1)
+
+ partition = 'fakepartition'
+ dataname = '%s:%s' % (systeminfo.RESOURCENAME_DISKUSAGE, partition)
+ data = 'testvalue'
+
+ disk.Update(dataname, data)
+
+ self.assertEquals(disk.DiskUsage(partition), 'testvalue')
+
+ @mock.patch('systeminfo.os')
+ def testDiskUsage(self, MockOs):
+ """Test disk usage info when there is no record or record is stale."""
+ disk = self._CreateDisk(1)
+
+ partition = 'fakepartition'
+
+ # Mock value for test. These values came from statvfs'ing root.
+ mock_statvfs_return = collections.namedtuple(
+ 'mock_disk', ['f_bsize', 'f_frsize', 'f_blocks', 'f_bfree', 'f_bavail',
+ 'f_files', 'f_ffree', 'f_favail', 'f_flag', 'f_namemax'])
+ mock_value = mock_statvfs_return(4096, 4096, 9578876, 5332865, 4840526,
+ 2441216, 2079830, 2079830, 4096, 255)
+
+ MockOs.statvfs.return_value = mock_value
+
+ # Expected results of the test.
+ mock_total = 39235076096
+ mock_free = 21843415040
+ mock_used = 17391661056
+ mock_percent_used = 44.326818720693325
+ mock_diskusage = systeminfo.RESOURCE_DISKUSAGE(mock_total, mock_used,
+ mock_free,
+ mock_percent_used)
+
+ self.assertEquals(disk.DiskUsage(partition), mock_diskusage)
+
+ def testBlockDevicesExisting(self):
+ """Test block device information collection when a record exists."""
+ disk = self._CreateDisk(1)
+
+ device = '/dev/sda1'
+ dataname = '%s:%s' % (systeminfo.RESOURCENAME_BLOCKDEVICE, device)
+ data = 'testvalue'
+
+ disk.Update(dataname, data)
+
+ self.assertEquals(disk.BlockDevices(device), data)
+
+ # TODO mattmallett crbug.com/815014
+ # re-write this test so that it doesn't rely on chromite
+ # @mock.patch('systeminfo.osutils')
+ # def testBlockDevice(self, MockOsUtils):
+ # """Test block device info when there is no record or a record is stale."""
+ # disk = self._CreateDisk(1)
+ #
+ # mock_device = '/dev/sda1'
+ # mock_size = 12345678987654321
+ # mock_ids = ['ata-ST1000DM003-1ER162_Z4Y3WQDB-part1']
+ # mock_labels = ['BOOT-PARTITION']
+ # mock_lsblk = 'NAME="sda1" RM="0" TYPE="part" SIZE="%s"' % mock_size
+ #
+ # # self.StartPatcher(mock.patch('chromite.lib.osutils.ResolveSymlink'))
+ # systeminfo.osutils.ResolveSymlink.return_value = '/dev/sda1'
+ #
+ # with cros_build_lib_unittest.RunCommandMock() as rc_mock:
+ # rc_mock.AddCmdResult(
+ # partial_mock.In(systeminfo.SYSTEMFILE_DEV_DISKBY['ids']),
+ # output='\n'.join(mock_ids))
+ # rc_mock.AddCmdResult(
+ # partial_mock.In(systeminfo.SYSTEMFILE_DEV_DISKBY['labels']),
+ # output='\n'.join(mock_labels))
+ # rc_mock.AddCmdResult(partial_mock.In('lsblk'), output=mock_lsblk)
+ #
+ # mock_blockdevice = [systeminfo.RESOURCE_BLOCKDEVICE(mock_device,
+ # mock_size,
+ # mock_ids,
+ # mock_labels)]
+ #
+ # self.assertEquals(disk.BlockDevices(mock_device), mock_blockdevice)
+
+
+class Cpu(unittest.TestCase):
+ """Unittests for Cpu."""
+
+ def _CreateCpu(self, update_sec):
+ """Setup a Cpu object."""
+ return systeminfo.Cpu(update_sec=update_sec)
+
+ def testCpuTimeExisting(self):
+ """Test cpu timing information collection when a record exists."""
+ cpu = self._CreateCpu(1)
+
+ dataname, data = (systeminfo.RESOURCENAME_CPUTIMES, 'testvalue')
+ cpu.Update(dataname, data)
+
+ self.assertEquals(cpu.CpuTime(), 'testvalue')
+
+ def testCpuTime(self):
+ """Test cpu timing info when there is no record or record is stale."""
+ cpu = self._CreateCpu(1)
+
+ mock_file = io.StringIO(MOCK_PROC_STAT)
+ with mock.patch('__builtin__.open', return_value=mock_file, create=True):
+ cpu.CpuTime()
+
+ # The expected return value.
+ mock_cpu_name = 'cpu'
+ mock_cpu_total = 7350910560
+ mock_cpu_idle = 7300998160
+ mock_cpu_nonidle = 49912400
+
+ mock_cputimes = [systeminfo.RESOURCE_CPUTIME(mock_cpu_name, mock_cpu_total,
+ mock_cpu_idle,
+ mock_cpu_nonidle)]
+
+ self.assertEquals(cpu.resources.get(systeminfo.RESOURCENAME_CPUTIMES),
+ mock_cputimes)
+
+ def testCpuLoadExisting(self):
+ """Test cpu load information collection when a record exists."""
+ cpu = self._CreateCpu(1)
+
+ dataname, data = (systeminfo.RESOURCENAME_CPULOADS, 'testvalue')
+ cpu.Update(dataname, data)
+
+ self.assertEquals(cpu.CpuLoad(), 'testvalue')
+
+ def testCpuLoad(self):
+ """Test cpu load collection info when we have previous timing data."""
+ cpu = self._CreateCpu(1)
+
+ # Create a cpu timing record.
+ mock_cpu_prev_name = 'cpu'
+ mock_cpu_prev_total = 7340409560
+ mock_cpu_prev_idle = 7300497160
+ mock_cpu_prev_nonidle = 39912400
+
+ mock_cputimes_prev = [systeminfo.RESOURCE_CPUTIME(mock_cpu_prev_name,
+ mock_cpu_prev_total,
+ mock_cpu_prev_idle,
+ mock_cpu_prev_nonidle)]
+
+ cpu.Update(systeminfo.RESOURCENAME_CPUPREVTIMES, mock_cputimes_prev)
+
+ # Mock and execute.
+ mock_file = io.StringIO(MOCK_PROC_STAT)
+ with mock.patch('__builtin__.open', return_value=mock_file, create=True):
+ cpu.CpuLoad()
+
+ # The expected return value.
+ mock_cpu_name = 'cpu'
+ mock_cpu_load = 0.9522902580706599
+
+ mock_cpuloads = [systeminfo.RESOURCE_CPULOAD(mock_cpu_name, mock_cpu_load)]
+
+ self.assertEquals(cpu.resources.get(systeminfo.RESOURCENAME_CPULOADS),
+ mock_cpuloads)
+
+
+class InfoClassCacheTest(unittest.TestCase):
+ """Unittests for checking that information class caching."""
+
+ def testGetCpu(self):
+ """Test caching explicitly for Cpu information objects."""
+ cpus1 = [systeminfo.GetCpu(), systeminfo.GetCpu(),
+ systeminfo.GetCpu(systeminfo.UPDATE_CPU_SEC),
+ systeminfo.GetCpu(update_sec=systeminfo.UPDATE_CPU_SEC)]
+
+ cpus2 = [systeminfo.GetCpu(10), systeminfo.GetCpu(10),
+ systeminfo.GetCpu(update_sec=10)]
+
+ self.assertTrue(all(id(x) == id(cpus1[0]) for x in cpus1))
+ self.assertTrue(all(id(x) == id(cpus2[0]) for x in cpus2))
+ self.assertNotEqual(id(cpus1[0]), id(cpus2[0]))
+
+ def testGetMemory(self):
+ """Test caching explicitly for Memory information objects."""
+ mems1 = [systeminfo.GetMemory(), systeminfo.GetMemory(),
+ systeminfo.GetMemory(systeminfo.UPDATE_MEMORY_SEC),
+ systeminfo.GetMemory(update_sec=systeminfo.UPDATE_MEMORY_SEC)]
+
+ mems2 = [systeminfo.GetMemory(10), systeminfo.GetMemory(10),
+ systeminfo.GetMemory(update_sec=10)]
+
+ self.assertTrue(all(id(x) == id(mems1[0]) for x in mems1))
+ self.assertTrue(all(id(x) == id(mems2[0]) for x in mems2))
+ self.assertNotEqual(id(mems1[0]), id(mems2[0]))
+
+ def testGetDisk(self):
+ """Test caching explicitly for Disk information objects."""
+ disks1 = [systeminfo.GetDisk(), systeminfo.GetDisk(),
+ systeminfo.GetDisk(systeminfo.UPDATE_MEMORY_SEC),
+ systeminfo.GetDisk(update_sec=systeminfo.UPDATE_MEMORY_SEC)]
+
+ disks2 = [systeminfo.GetDisk(10), systeminfo.GetDisk(10),
+ systeminfo.GetDisk(update_sec=10)]
+
+ self.assertTrue(all(id(x) == id(disks1[0]) for x in disks1))
+ self.assertTrue(all(id(x) == id(disks2[0]) for x in disks2))
+ self.assertNotEqual(id(disks1[0]), id(disks2[0]))
+
+if __name__ == '__main__':
+ unittest.main()
diff --git a/src/mobmonitor/util/__init__.py b/src/mobmonitor/util/__init__.py
new file mode 100644
index 0000000..e69de29
--- /dev/null
+++ b/src/mobmonitor/util/__init__.py
diff --git a/src/mobmonitor/util/collect_logs.py b/src/mobmonitor/util/collect_logs.py
new file mode 100644
index 0000000..ba990dd
--- /dev/null
+++ b/src/mobmonitor/util/collect_logs.py
@@ -0,0 +1,60 @@
+# -*- coding: utf-8 -*-
+# Copyright 2015 The Chromium OS Authors. All rights reserved.
+# Use of this source code is governed by a BSD-style license that can be
+# found in the LICENSE file.
+
+"""Simple log collection script for Mob* Monitor"""
+
+from __future__ import print_function
+
+import glob
+import os
+import tempfile
+import shutil
+
+import osutils
+
+TMPDIR = '/mnt/moblab/tmp'
+TMPDIR_PREFIX = 'moblab_logs_'
+LOG_DIRS = {
+ 'apache_errors': '/var/log/apache2/error_log',
+ 'devserver_logs': '/var/log/devserver',
+ 'dhcp_leases': '/var/lib/dhcp',
+ 'messages': '/var/log/messages',
+ 'mysql': '/var/log/mysql',
+ 'servod': '/var/log/servod.log',
+ 'scheduler': '/usr/local/autotest/logs/scheduler.latest'
+}
+
+def remove_old_tarballs():
+ paths = glob.iglob(os.path.join(TMPDIR, '%s*.tgz' % TMPDIR_PREFIX))
+ for path in paths:
+ os.remove(path)
+
+
+def collect_logs():
+ remove_old_tarballs()
+ osutils.safe_mkdir(TMPDIR)
+ tempdir = tempfile.mkdtemp(prefix=TMPDIR_PREFIX, dir=TMPDIR)
+ os.chmod(tempdir, 0o777)
+
+ try:
+ for name, path in LOG_DIRS.iteritems():
+ if not os.path.exists(path):
+ continue
+ if os.path.isdir(path):
+ shutil.copytree(path, os.path.join(tempdir, name))
+ else:
+ shutil.copyfile(path, os.path.join(tempdir, name))
+
+ #cmd = ['mobmoncli', 'GetStatus']
+ #cros_build_lib.RunCommand(
+ # cmd,
+ # log_stdout_to_file=os.path.join(tempdir, 'mobmonitor_getstatus')
+ #)
+ #TODO mattmallett crbug.com/815013
+ finally:
+ tarball = '%s.tgz' % tempdir
+ osutils.create_tarball(tarball, tempdir)
+ osutils.rm_dir(tempdir, ignore_missing=True)
+ return tarball
diff --git a/src/mobmonitor/util/osutils.py b/src/mobmonitor/util/osutils.py
new file mode 100644
index 0000000..9678e6d
--- /dev/null
+++ b/src/mobmonitor/util/osutils.py
@@ -0,0 +1,159 @@
+# -*- coding: utf-8 -*-
+# Copyright 2018 The Chromium OS Authors. All rights reserved.
+# Use of this source code is governed by a BSD-style license that can be
+# found in the LICENSE file.
+
+"""Command line util library"""
+
+import subprocess
+import os
+import shutil
+import logging
+import errno
+import collections
+import re
+
+LOGGER = logging.getLogger(__name__)
+
+class RunCommandError(subprocess.CalledProcessError):
+ def __init__(self, returncode, cmd):
+ super(RunCommandError, self).__init__(returncode, cmd)
+
+def run_command(cmd, error_code_ok=False, log=False):
+ """Run a command.
+
+ Args:
+ cmd: string array of the command, and the command line arguments to
+ pass into the command, ex:
+ ['echo', 'hello']
+ error_code_ok: do not throw RunCommandError on non-zero exit code
+ log: log the command and the output
+
+ Return:
+ string containing the stdout of the output, None if no output
+ """
+ try:
+ output = subprocess.check_output(cmd)
+ if log:
+ LOGGER.info(' '.join(cmd))
+ LOGGER.info(output)
+ return output
+ except subprocess.CalledProcessError as e:
+ if log:
+ LOGGER.info(' '.join(cmd))
+ LOGGER.info(str(e))
+ if not error_code_ok:
+ raise RunCommandError(e.returncode, e.cmd)
+
+def sudo_run_command(cmd, user='root', error_code_ok=False):
+ """Run a command via sudo. All calls are logged.
+
+ Args:
+ cmd: string array of the command and args, see run_command for details
+ user: what user to run the command as, default root
+ error_code_ok: do not throw RunCommandError on non-zero exit code
+
+ Return:
+ string containing stdout of the output, None if no output
+ """
+ if user == 'root':
+ sudo_cmd = ['sudo'] + cmd
+ else:
+ sudo_cmd = ['sudo', '-u', user] + cmd
+ return run_command(sudo_cmd, log=True, error_code_ok=error_code_ok)
+
+def create_tarball(tarball_name, directory):
+ """Creates a gzipped tar archive
+
+ Args:
+ tarball_name: string name for the tarball
+ directory: directory to archive the contents of
+ """
+ run_command(['tar', '-czf', tarball_name, directory])
+
+def safe_mkdir(path):
+ """Create a directory and any parent directories if they don't exist.
+
+ Args:
+ path: string path of the new directory
+ """
+ run_command(['mkdir', '-p', path])
+
+def rm_dir(path, ignore_missing=False, sudo=False):
+ """Recursively remove a directory.
+
+ Args:
+ path: Path of directory to remove.
+ ignore_missing: Do not error when path does not exist.
+ sudo: Remove directories as root.
+ """
+ if sudo:
+ try:
+ sudo_run_command(
+ ['rm', '-r%s' % ('f' if ignore_missing else '',), '--', path])
+ except RunCommandError as e:
+ if not ignore_missing or os.path.exists(path):
+ # If we're not ignoring the rm ENOENT equivalent, throw it;
+ # if the pathway still exists, something failed, thus throw it.
+ raise
+ else:
+ try:
+ shutil.rmtree(path)
+ except EnvironmentError as e:
+ if not ignore_missing or e.errno != errno.ENOENT:
+ raise
+
+def resolve_symlink(file_name, root='/'):
+ """Resolve a symlink |file_name| relative to |root|.
+
+ For example:
+
+ ROOT-A/absolute_symlink --> /an/abs/path
+ ROOT-A/relative_symlink --> a/relative/path
+
+ absolute_symlink will be resolved to ROOT-A/an/abs/path
+ relative_symlink will be resolved to ROOT-A/a/relative/path
+
+ Args:
+ file_name: A path to the file.
+ root: A path to the root directory.
+
+ Returns:
+ file_name if file_name is not a symlink. Otherwise, the ultimate path
+ that file_name points to, with links resolved relative to |root|.
+ """
+ count = 0
+ while os.path.islink(file_name):
+ count += 1
+ if count > 128:
+ raise ValueError('Too many link levels for %s.' % file_name)
+ link = os.readlink(file_name)
+ if link.startswith('/'):
+ file_name = os.path.join(root, link[1:])
+ else:
+ file_name = os.path.join(os.path.dirname(file_name), link)
+ return file_name
+
+def list_block_devices():
+ """Lists all block devices.
+
+ Returns:
+ A list of BlockDevice items with attributes 'NAME', 'RM', 'TYPE',
+ 'SIZE' (RM stands for removable).
+ """
+ keys = ['NAME', 'RM', 'TYPE', 'SIZE']
+ BlockDevice = collections.namedtuple('BlockDevice', keys)
+
+ cmd = ['lsblk', '--pairs', '--bytes']
+ cmd += ['--output', ','.join(keys)]
+
+ output = run_command(cmd).strip()
+ devices = []
+ for line in output.splitlines():
+ d = {}
+ for k, v in re.findall(r'(\S+?)=\"(.+?)\"', line):
+ d[k] = v
+
+ devices.append(BlockDevice(**d))
+
+ return devices
diff --git a/src/mobmonitor/util/osutils_unittest.py b/src/mobmonitor/util/osutils_unittest.py
new file mode 100644
index 0000000..0a84f6e
--- /dev/null
+++ b/src/mobmonitor/util/osutils_unittest.py
@@ -0,0 +1,159 @@
+# -*- coding: utf-8 -*-
+# Copyright 2015 The Chromium OS Authors. All rights reserved.
+# Use of this source code is governed by a BSD-style license that can be
+# found in the LICENSE file.
+
+import unittest
+import mock
+
+import osutils
+
+class TestRunCommand(unittest.TestCase):
+ def test_run_command(self):
+ output = osutils.run_command(['echo', 'hello'])
+ self.assertEqual(output, 'hello\n')
+
+ def test_run_command_error(self):
+ with self.assertRaises(osutils.RunCommandError):
+ osutils.run_command(['false'])
+
+ def test_run_command_error_code_ok(self):
+ output = osutils.run_command(['false'], error_code_ok=True)
+
+ @mock.patch('osutils.LOGGER')
+ def test_run_command_log(self, MockLogger):
+ osutils.run_command(['echo', 'hello'], log=True)
+ MockLogger.info.assert_any_call('echo hello')
+ MockLogger.info.assert_any_call('hello\n')
+
+class TestSudoRunCommand(unittest.TestCase):
+ @mock.patch('osutils.LOGGER')
+ @mock.patch('osutils.subprocess')
+ def test_sudo_as_root(self, MockLogger, MockSubprocess):
+ osutils.subprocess.check_output.return_value = 'test'
+ output = osutils.sudo_run_command(['echo', 'hello'])
+
+ self.assertEqual(output, 'test')
+ osutils.subprocess.check_output.assert_called_with(
+ ['sudo', 'echo', 'hello'])
+ osutils.LOGGER.info.assert_any_call('sudo echo hello')
+
+ @mock.patch('osutils.LOGGER')
+ @mock.patch('osutils.subprocess')
+ def test_sudo_as_other_user(self, MockLogger, MockSubprocess):
+ osutils.subprocess.check_output.return_value = 'test'
+ output = osutils.sudo_run_command(['echo', 'hello'], user='moblab')
+
+ self.assertEqual(output, 'test')
+ osutils.subprocess.check_output.assert_called_with(
+ ['sudo', '-u', 'moblab', 'echo', 'hello'])
+ osutils.LOGGER.info.assert_any_call('sudo -u moblab echo hello')
+
+class TestCreateTarball(unittest.TestCase):
+ @mock.patch('osutils.run_command')
+ def test_create_tarball(self, MockRunCommand):
+ osutils.create_tarball('mytarball.tgz', '/my/dir')
+ osutils.run_command.assert_called_with(
+ ['tar', '-czf', 'mytarball.tgz', '/my/dir'])
+
+class TestSafeMkdir(unittest.TestCase):
+ @mock.patch('osutils.run_command')
+ def test_safe_mkdir(self, MockRunCommand):
+ osutils.safe_mkdir('/path/to/my/dir')
+ osutils.run_command.assert_called_with(
+ ['mkdir', '-p', '/path/to/my/dir'])
+
+class TestRmDir(unittest.TestCase):
+ @mock.patch('osutils.shutil')
+ @mock.patch('osutils.run_command')
+ def test_rm_dir(self, MockShutil, MockRunCommand):
+ osutils.rm_dir('rm/this/old/dir')
+
+ @mock.patch('osutils.shutil')
+ @mock.patch('osutils.run_command')
+ def test_rm_dir_error(self, MockShutil, MockRunCommand):
+ osutils.shutil.rmtree.side_effect = EnvironmentError()
+ with self.assertRaises(EnvironmentError):
+ osutils.rm_dir('rm/this/old/dir')
+
+ @mock.patch('osutils.shutil')
+ @mock.patch('osutils.run_command')
+ def test_ignore_missing(self, MockShutil, MockRunCommand):
+ def side_effect(arg):
+ error = EnvironmentError()
+ error.errno = osutils.errno.ENOENT
+ osutils.shutil.rmtree.side_effect = side_effect
+ osutils.rm_dir('rm/this/old/dir', ignore_missing=True)
+
+ @mock.patch('osutils.shutil')
+ @mock.patch('osutils.run_command')
+ def test_rm_dir_root(self, MockShutil, MockRunCommand):
+ osutils.rm_dir('rm/this/old/dir', sudo=True)
+ osutils.run_command.assert_called_with(
+ ['sudo', 'rm', '-r', '--', 'rm/this/old/dir'],
+ log=True, error_code_ok=False)
+
+ @mock.patch('osutils.shutil')
+ @mock.patch('osutils.run_command')
+ @mock.patch('osutils.os')
+ def test_rm_dir_root_ignore_missing(
+ self, MockShutil, MockRunCommand, MockOs):
+ osutils.rm_dir('rm/this/old/dir', sudo=True, ignore_missing=True)
+ osutils.run_command.side_effect = osutils.RunCommandError('', 1)
+ osutils.os.path.exists.return_value = False
+ osutils.run_command.assert_called_with(
+ ['sudo', 'rm', '-rf', '--', 'rm/this/old/dir'],
+ log=True, error_code_ok=False)
+
+ @mock.patch('osutils.shutil')
+ @mock.patch('osutils.run_command')
+ @mock.patch('osutils.os')
+ def test_rm_dir_root_error(
+ self, MockShutil, MockRunCommand, MockOs):
+ osutils.run_command.side_effect = osutils.RunCommandError('', 1)
+ with self.assertRaises(osutils.RunCommandError):
+ osutils.rm_dir('rm/this/old/dir', sudo=True)
+
+class TestResolveSymlink(unittest.TestCase):
+ @mock.patch('osutils.os')
+ def test_resolve_symlink(self, MockOs):
+ osutils.os.path.islink.side_effect = [True, False]
+ osutils.os.readlink.return_value = '/my/file'
+ osutils.os.path.join.return_value = '/path/to/my/file'
+ file_name = osutils.resolve_symlink('file')
+ self.assertEqual(file_name, '/path/to/my/file')
+
+ @mock.patch('osutils.os')
+ def test_symlink_error(self, MockOs):
+ osutils.os.path.islink.return_value = True
+ osutils.os.readlink.return_value = '/my/file'
+ osutils.os.path.join.return_value = '/path/to/my/file'
+ with self.assertRaises(ValueError):
+ osutils.resolve_symlink('file')
+
+class TestListBlockDevices(unittest.TestCase):
+ LSBLK_OUT = ('NAME="sda" RM="0" TYPE="disk" SIZE="128035676160"\n'
+ 'NAME="sda1" RM="0" TYPE="part" SIZE="123578875904"\n'
+ 'NAME="sda2" RM="0" TYPE="part" SIZE="16777216"')
+
+ @mock.patch('osutils.run_command')
+ def test_list_block_devices(self, MockRunCommand):
+ osutils.run_command.return_value = self.LSBLK_OUT
+ devices = osutils.list_block_devices()
+
+ osutils.run_command.assert_called_with([
+ 'lsblk', '--pairs', '--bytes', '--output',
+ 'NAME,RM,TYPE,SIZE'])
+
+ self.assertEqual(devices[0].NAME, 'sda')
+ self.assertEqual(devices[0].RM, '0')
+ self.assertEqual(devices[0].TYPE, 'disk')
+ self.assertEqual(devices[0].SIZE, '128035676160')
+
+ self.assertEqual(devices[1].NAME, 'sda1')
+ self.assertEqual(devices[2].NAME, 'sda2')
+
+ self.assertEqual(len(devices), 3)
+
+if __name__ == '__main__':
+ unittest.main()
