The Layout Test Framework that Blink uses is a regression testing tool that is multi-platform and it has a large amount of tools that help test varying types of regression, such as pixel diffs, text diffs, etc. The framework is mainly used by Blink, however it was made to be extensible so that other projects can use it test different parts of chrome (such as Print Preview). This is a guide to help people who want to actually the framework to test whatever they want.
Before you can start actually extending the framework, you should be familiar with how to use it. See the layout tests documentation.
There are two parts to actually extending framework to test a piece of software. The first part is extending certain files in: /third_party/blink/tools/blinkpy/web_tests/ The code in
blinkpy/web_tests is the layout test framework itself
The second part is creating a driver (program) to actually communicate the layout test framework. This part is significantly more tricky and dependent on what exactly exactly is being tested.
This part isn’t too difficult. There are basically two classes that need to be extended (ideally, just inherited from). These classes are:
Driver. Located in
layout_tests/port/driver.py. Each instance of this is the class that will actually an instance of the program that produces the test data (program in Part 2).
Port. Located in
layout_tests/port/base.py. This class is responsible creating drivers with the correct settings, giving access to certain OS functionality to access expected files, etc.
As said, Driver launches the program from Part 2. Said program will communicate with the driver class to receive instructions and send back data. All of the work for driver gets done in
Driver.run_test. Everything else is a helper or initialization function.
run_test() most likely doesn’t need to be overridden unless extra blocks of data need to be sent to/read from the test program. However, you do need to know how it works because it will influence what functions you need to override. Here are the ones you’re probably going to need to override
This function creates a set of command line arguments to run the test program, so the function will almost certainly need to be overridden.
It creates the command line to run the program.
subprocess.popen to create the process, which takes the name of the test program and any options it might need.
The first item in the list of arguments should be the path to test program using this function:
This is an absolute path to the test program. This is the bare minimum you need to get the driver to launch the test program, however if you have options you need to append, just append them to the list.
If your program has any special startup needs, then this will be the place to put it.
That’s mostly it. The Driver class has almost all the functionality you could want, so there isn’t much to override here. If extra data needs to be read or sent, extra data members should be added to
This class is responsible for providing functionality such as where to look for tests, where to store test results, what driver to run, what timeout to use, what kind of files can be run, etc. It provides a lot of functionality, however it isn’t really sufficient because it doesn’t account of platform specific problems, therefore port itself shouldn’t be extend. Instead LinuxPort, WinPort, and MacPort (and maybe the android port class) should be extended as they provide platform specific overrides/extensions that implement most of the important functionality. While there are many functions in Port, overriding one function will affect most of the other ones to get the desired behavior. For example, if
layout_tests_dir() is overridden, not only will the code look for tests in that directory, but it will find the correct TestExpectations file, the platform specific expected files, etc.
Here are some of the functions that most likely need to be overridden.
The rest of the functions can definitely be overridden for your projects specific needs, however these are the bare minimum needed to get it running. There are also functions you can override to make certain actions that aren’t on by default always take place. For example, the layout test framework always checks for system dependencies unless you pass in a switch. If you want them disabled for your project, just override
check_sys_deps to always return OK. This way you don’t need to pass in so many switches.
As said earlier, you should override LinuxPort, MacPort, and/or WinPort. You should create a class that implements the platform independent overrides (such as
driver_class) and then create a separate class for each platform specific port of your program that inherits from the class with the independent overrides and the platform port you want. For example, you might want to have a different timeout for your project, but on Windows the timeout needs to be vastly different than the others. In this case you can just create a default override that every class uses except your Windows port. In that port you can just override the function again to provide the specific timeout you need. This way you don’t need to maintain the same function on each platform if they all do the same thing.
Port that’s basically it unless you need to make many odd modifications. Lots of functionality is already there so you shouldn’t really need to do much.
This is the part where you create the program that your driver class launches. This part is very application dependent, so it will not be a guide on how implement certain features, just what should be implemented and the order in which events should occur and some guidelines about what to do/not do. For a good example of how to implement your test program, look at MockDRT in
mock_drt.pyin the same directory as
driver.py. It goes through all the steps described below and is very clear and concise. It is written in python, but your driver can be anything that can be run by
subprocess.popen and has stdout, stdin, stderr.
Your goal for this part of the project is to create a program (or extend a program) to interface with the layout test framework. The layout test framework will communicate with this program to tell it what to do and it will accept data from this program to perform the regression testing or create new base line files.
This is how your code should be laid out.
run_test()in the driver.
content_shellneeds to wait for an html file to navigate to, render it, then convert that rendering to a PNG. It does this constantly, until a signal/message is sent to indicate that no more tests should be processed
driver.py. This information will be passed through stdin and is just one large string, with each part of the command being split with apostrophes (ex: “/path’foo” is path to the test file, then foo is some setting that your program might need).
content_shell, this would be the part where it navigates to the test file, then renders it. After the program acts on the input, it needs to send some text to the driver code to indicate that it has acted on the input. This text will indicate something that you want to test. For example, if you want to make sure you program always prints “foo” you should send it to the driver. If the program every prints “bar” (or anything else), that would indicate a failure and the test will fail.
That’s basically what the skeleton of your program should be.
This is information about how to do some specific things, such as sending data to the layout test framework.
read_blockfunctions in Driver.py