Roll recipe dependencies (trivial).

This is an automated CL created by the recipe roller. This CL rolls
recipe changes from upstream projects (depot_tools) into this repository.

depot_tools:
https://chromium.googlesource.com/chromium/tools/depot_tools.git/+/1cabb17575917b73ec2e270d4187656c20b1ab0c
  1cabb17 (sokcevic@google.com)
      Set branchref on Changelist if set by user

More info is at https://goo.gl/zkKdpD. Use https://goo.gl/noib3a to file a bug.

R=bjoyce@chromium.org

Recipe-Tryjob-Bypass-Reason: Autoroller
Bugdroid-Send-Email: False
Change-Id: Ib90c1c768f13891f880edf0841fc0bd6ce9264b3
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/tools/franky/+/2815940
Commit-Queue: Recipe Roller <recipe-mega-autoroller@chops-service-accounts.iam.gserviceaccount.com>
Bot-Commit: Recipe Roller <recipe-mega-autoroller@chops-service-accounts.iam.gserviceaccount.com>
2 files changed
tree: 5cd690f55f66e7e37bb4eddd38d5e38daf1d66cc
  1. franky/
  2. infra/
  3. recipes/
  4. scripts/
  5. .gitignore
  6. codereview.settings
  7. CONTRIBUTING.md
  8. LICENSE
  9. Makefile
  10. OWNERS
  11. PRESUBMIT.py
  12. README.md
  13. setup.py
  14. test.py
  15. test.py.vpython
README.md

Franky: Android and iOS Test Automation Tool

This application is not an official Google product.

Franky is a data driven mobile test automation tool designed for driving applications on Android and iOS. Franky is best suited to projects that need to test stand alone applications by interacting with the applications Graphical User Interface. Because of the nature of GUI testing, Franky is not a good choice for to implement in the presubmit queues of a project build workflow.

Franky can interact with your application and mobile system just as a user can. Consequently anything that can be selected or configured by a user is accessible to a user, including objects that are not part of the application (system settings, user accounts, etc).

Franky is written in Python, and depends on Appium and WebDriverAgent (for iOS device testing).

Installation

First, install Appium:

npm install -g appium

Follow their installation instructions for your platform. In particular, pay close attention to iOS requirements. You may need to find the WebDriverAgent copy in the Appium installation and update its WebDriverAgent.xcodeproj with your provisioning profile and signing credentials.

It is highly recommended to install Franky in a separate virtualenv environment:

mkdir ~/virtualenv
cd ~/virtualenv
virtualenv franky
source franky/bin/activate
cd ~
git clone https://chromium.googlesource.com/chromium/src/tools/franky
cd franky
./setup.py install

This will install a new command franky in ~/virtualenv/franky/bin. You can test it by running:

franky --help

Installation using VPython

VPython is a drop-in Python replacement used by Chromium operations to create hermetic Python environments on per-script basis. It relies on the CIPD service.

Make sure cipd and vpython executables are in your PATH. The simplest way to get them is to clone depot_tools and add it to you PATH:

cd ~
git clone https://chromium.googlesource.com/chromium/tools/depot_tools.git

and add the following line to your ~/.bashrc file:

PATH="$PATH":"$HOME/depot_tools"
export PATH

Alternatively, you can compile both yourself from CIPD and VPython sources and install as you like.

Make sure you have an up-to-date and clean checkout:

git pull
git status  # There should be no modified or untracked files

Upload a new Franky Python wheel to CIPD (you need to have enough permissions in the CIPD package ACLs; view ACLs by running cipd acl-list infra/python/wheels/franky-py2):

cipd auth-login  # Do this once per machine
make release

Create an executable shim script franky:

#!/usr/bin/env vpython

import franky.runner

if __name__ == '__main__':
  franky.runner.main()

and accompany it with the VPython spec file franky.vpython containing all the Python dependencies, and in particular, a reference to the Franky package:

wheel {
  name: "infra/python/wheels/franky-py2"
  version: "git_revision:deadbeefe66bf36e0d50eaaba80ff3d538ba5716"
}

See test.py and test.py.vpython for an example.

Components

Franky consists of 4 main components:

Appium: The Test automation framework on top of which Franky is built. Appium is an open source test automation framework developed and supported by Sauce Labs to automate native and hybrid mobile apps. It uses JSON wire protocol internally to interact with iOS and Android native apps using the Selenium WebDriver which is a testing framework for web apps. Appium is designed to encourage a 2-tier architecture: a machine runs the test written in one language (python in this case) and another one (the test server) actually executes it. Furthermore the WebDriver protocol targets scalability (because based on HTTP), which makes Appium very scalable as well.

Franky Core: The core component of Franky, it is a client to the appium server.

Dashboard: The web interface that displays test results output.

Data Source: The UI interface used to construct tests to be automated.

Franky is architectured in such a way that the test data or test plan is completely separated from the script that executes it. This gives the ability to any non technical person to use the system at ease to automate tests.

FRANKY MODULES

runner.py

This is the entry point and the main orchestra of Franky. It is the mediator between different components of the system. After getting all global constants including the path to the app to automate and where to locate appium server, runner.py module reads test suites from test_data_reader.py, gets needed app under test info from app_info.py and device info from device_info.py. Runner.py also instantiates an object driver_provider.py which will be passed along to each step action method providing here a way to access the Appium driver. The _RunTestSuite method in runner.py is responsible for creating unittest test suites where each test is going to be executed as a unittest test with a SetUp and tearDown methods. The test method testFrankySteps maps each action in the test step to its corresponding type and executes the test step. After all the actions are done executing in a suite, runner.py sends the result to generate_report.py as a json file.

app_info.py

This script provides the needed information about the application under test. It will gather the information and store it, which includes the name of the applications, the platform, and some version information.

device_info.py

Gathers the needed device information to run tests against it.

driver_provider.py

This class provides a property called driver by returning a lazy initialized instance of DriverProvider class. This driver is used to run all tests and therefore executeall action methods.

step.py

This module provides all of the needed framework to support a test step. Each test step is composed of a description, action_type, action, by (name of the first element locator), path (name of the 2nd element locator), value, duration, start_coordinate, end_coordinate, device_type, and device_versions.

test_data_reader.py

This module automatically reads test suites from csv files in Google Sheet. It also validates each test step according to its device type. A valid test step is that which does not have a device type attribute or if given, matches the device type of the connected device. A non valid test step is that whose device type is different from the device type of the connected device. If a non valid test step action is StartTest, then the entire set of test steps below that invalid test step will also be ignored until the next test step with action StartTest is found, marking here the beginning of a new test. If the first step in a test case is not StartTest then all succeeding test steps are going to be ignored until a step with StartTest is found.

reports folder

generate_report.py

Generates a JSON test report from all of the test results.

html_report.py

Will take all of the test results and write a HTML test result report.

html_report_template.py

The template used by html_report to generate the html report.

test_util folder

The test_util folder contains all of the modules that contains logic of implementation Franky test cases and test suites.

franky_testcase.py

It is a wrapper over unittest.TestCase that allows to execute test steps. In case of failure franky_testcase tries to recover test infrastructure with Android/iOS specific. Also franky_testcase allows to add additional steps before/after test.

franky_testsuite.py

unittest.TestSuite is a skeleton of Franky that parses test cases from csv files(s) and execute test cases. Test suite supports.

appium_util Folder

The appium_util folder contains all of the modules that directly pertain to the Appium product.

appium_driver_util.py

This is the module that starts Appium Server and also create Appium driver. The only client to this module is driver_provider.py and the only way to get or rest the driver is therefore through driver_provider.py. All the Appium desired capabilities all well defined in appium_driver_utils.py among which the app path, launchTimeout, newCommandTimeout, waitForAppScript, backendRetries etc.

input_formatter.py

Provides methods to format objects before they are input.

path_constants.py

Provides path constants that are used across different modules.

timeout_util.py

Provides ways to manipulate the driver timeouts.

actions Folder

In the actions folder, resides all type of actions files which each implements a specific user action. An action is the process of doing something passive or active by the user on the screen such as tapping a button, installing an app, or verifying that a button is present or not, typically to achieve an aim which is to make sure there is success. Every action corresponds to a method with arguments driver_provider and a Step tuple.

Here are the different types of actions:

alert.py

This module is responsible for handling alerts or pop-ups during test execution. This includes in app alerts, local notification, battery warning and privacy access permission alerts such as location, contacts and photos. System update alerts are not handled here. By default Appium does not handle any alert on the screen and Franky dismisses all alerts unless told not to do so. AutoDismissAlerts and AutoAcceptAlerts are methods in this module.

element.py

Implements all non interactive actions only used to verify or compare: VerifyElementIsPresent, VerifyElementIsAbsent, VerifyElementIsDisplayed, VerifyElementIsHidden, VerifyElementIsEnabled, VerifyElementIsDisabled, VerifyElementIsSelected, VerifyElementNameContains, VerifyElementNameEquals, VerifyElementValueContains, VerifyElementValueEquals, VerifyElementLabelContains, VerifyElementLabelEquals.

franky.py

This module provides a common interface for creating action objects.

menu.py

This provides actions to open and close Chrome tabs and menus.

omnibox.py

For all omnibox related actions: SearchFromOmnibox, SearchFromOmniboxIncognito, VerifyOmniboxURLContains, VerifyOmniboxURLEquals, VerifyIncognitoOmniboxURLContains, VerifyIncognitoOmniboxURLEquals, LoadCurrentURLFromOmnibox, LoadCurrentURLFromOmniboxIncognito.

onboarding.py

Provides all onboarding related actions.

scroll.py

For all scrolling, swiping and dragging related actions like: Overscroll, DragInsideWithOptions, DragFromToForDuration, ScrollToVisible, ScrollDownToElement, ScrollUpToElement. Swipe: Swipes the screen from one direction to the opposite. Direction is passed as a value of the step. So if Swipe has value Left, it will swipe from right to left and vice versa. In some instances, the user could want to swipe starting or ending in a specific place of the screen like swiping left but starting from the omnibox, the x_coordinate or y_coordinate should therefore be given. For example if step.y_coordinate is 0.5 and step.value is Left, this method will swipe from right to left and starts from the middle of the screen. Coordinates are optional for Swipe and if they are omitted then SwipeLeft will start from the right edge, SwipeUp will start from the bottom edge, etc.. An example of a use case here is the action to SwipeLeft from the omnibox to go to another tab. This action will specify the omnibox location to SwipeLeft with a start_coordinate of (1, 0.5).

settings.py

For all settings related actions: RemoveAccounts, AddAccount, and ClearBrowsingData.

system.py

For all system or app related actions like: InstallApp, RemoveApp, LaunchApp, CloseApp, ResetAppiumSession, ReinstallApp, ReinstallAndLaunchApp, DesactivateAppForDuration, SetDeviceOrientation, VerifyDeviceOrientation, EnableWiFi, LaunchSplitViewApp, ReinstallAndLaunchAndByPassFirstRun.

tap.py

For all tapping related actions like: Tap, DoubleTap, TapIfPresent, TapHiddenElement, TouchAndHold, TapWithOptions: Performs the specified gesture on the specified element using a dictionary to specify gesture attributes. You can use offsets to achieve finer precision in specifying the hitpoint within the rect for the specified element. The offset comprises a pair of x and y values, each ranging from 0.0 to 1.0. These values represent, respectively, relative horizontal and vertical positions within the rect, with {x:0.0, y:0.0} as the top left and {x:1.0, y:1.0} as the bottom right. Thus, {x:0.3, y:0.6} specifies a position just below and to the left of center, and {x:1.0, y:0.5} specifies a position centered vertically at the far right.

timeout.py

This module provides ways to manipulates the driver timeouts. It is very important to note that as well as supporting Selenium webdriver elements, Appium also supports UIAlements and there are some differences in the way these 2 types of elements are handled. Handling timeout for UIAlement is done right after the app is launched and PushTimeout() method will push a timeout to wait for a UIAlement to be present before failing. This timeout will only affect all UIAelements during the test and will not affect Selenium webdriver elements. Selenium webdriver timeout to wait for an element to be present before failing is only set when finding an element and is done individually for each Selenium webdriver element using the selenium.webdriver.support.expect_conditions module.

user_input.py

Provides all user input related actions like inputting characters.

utilities Folder

utilities contains common helpers that allows to download/upload reports and test suites, store and delete temporary files, run external process, filter and collect logcat data from Android device.

ERROR HANDLING

Appium throws Selenium errors (NoSuchElementException, TimeoutException, WebDriverException, etc…) when an element is not found on the screen or is non interactive, like a hidden element. Franky catches these exceptions in order to recover from the test failure (make sure wi-fi is ON and by-pass first run), leaving the app in the appropriate state for the next test to run.

ActionError.py is Franky’s base class for handling Selenium errors thrown by Appium such that an ActionError is thrown instead of Selenium error. VerificationError, TapError, NoElementFoundError are subclasses of ActionError and each error is thrown when an expected condition is unmet.

DATA PROVIDER

Test cases data provider has ten attributes:

  1. Description: This is used to give test case as well as a test step a description. It describes in a very brief and succinct way what the test step does in a human friendly language. An example will be: “Tap on the Back button to return to the home screen”. A test step description could be null but it is very recommended to always. This description is later used in the html report after the steps have been executed.

  2. Action type: Type of action to use for the test step. Any action belongs to a category based on its function and the way it interacts with the app. tap, scroll, sendkeys, verify, etc... are examples of types of actions.

  3. Action: Test step action is the actual Appium command to be sent to the Appium server. An action is encapsulated in a method that calls Appium webdriver for its execution. DoubleTap, Type, ScrollUpToElement, VerifyPresenceOfElement are example of actions.The list of all actions from Actions.py is mapped to this field and serves as a data validator. Any action used to build a test should be part of the list of these actions.

  4. By: This field is the first element of the locator tuple. This field may or may not be required depending on the type of action used. In Appium, elements can be found on the screen by id, xpath, uiautomation, classname, etc...

  5. Path: This field is the second element of the locator tuple. A path could be the accessibility label or accessibility identifier of an element. This field is only required if by field is provided.

  6. Value: Test step value. The value is one of the attributes of an element and could be the text that populates a text field. A test step value depends on the type of actions used and can just serve as an argument to an action method. This attribute depends may or may not be compulsory depending of the type of action used.

  7. Duration: This field is the time in seconds a test step action needs to accomplish a specific task. For example DeactivateAppForDuration action uses duration field as the time for the app to remain inactive. A test step duration field may or may not be required depending on the type of action used.

  8. start_coordinate: This field is the horizontal value in a pair of coordinates used to find a point in an element on the screen. A test step start_coordinate field may or may not be required depending on the type of action used. Coordinate range is [0.0;1.0], where 0.0 is left and 1.0 is right.

  9. end_coordinate: This field is the vertical value in a pair of coordinates used to find a point in an element on the screen. A test step end_coordinate field may or may not be required depending on the type of action used. Coordinate range is [0.0;1.0], where 0.0 is top and 1.0 is bottom.

  10. Device Type: A string used to disable an entire test case or a set of test steps. If given at the beginning of a test case, the test case will not be executed on a device configuration different from the one specified. If used on a test step, that test step will not be executed. A test case that has a blank device type attribute will run on all devices.

  11. Device Versions: A list of strings delimited by a comma for all the device versions a test can be executed on. If specified at the beginning of a test case, the test will not be executed on a device whose version is different from the one(s) listed. A test case that has a blank device version attribute will run on all device versions.

Each line in the data provider Spreadsheet represents a test step (Step tuple made up of all the 10 attributes above). The beginning of a new test case is tagged with StartTest. A test case is composed of one or more test steps and any lines below StartTest is the set of test steps that is part of that test until the next line containing StartTest is encountered. A test Suite is composed of many test cases and is also represented by an entire sheet in the Data Provider Spreadsheet.

REPORT

Generate_report.py module is responsible for generating a report in a json or html format out of the test results data at the end of each test suite run. If HTML, the report will be generated by using html_report.py which will in turn use the structure of the HTML template which resides in html_report_template.py. Tests results are generated after each run in a form of a list that is passed to the report module which processes the data. An HTML report will be generated by default.