annotation-protocol/README.md

Annotation-Protocol: Tests for the Web Annotation Protocol
==========================================================

The [Web Annotation Protocol](https://www.w3.org/TR/annotation-protocol)
specification presents set of messages to allow Annotation clients and servers
to interact seamlessly.

The purpose of these tests is to help validate that clients send and are
capable of receiving correctly formatted messages, and that servers are
able to receive and respond to correctly structured requests.

The general approach for this testing is to enable both manual and
automated testing. However, since the specification has no actual user
interface requirements, there is no general automation mechanism that
can be presented for testing clients.  Also the server tests need to be
pointed at a server implementation to exercise.  However, once provided
the basic information, testing is automated.

Implementors could take advantage of the plumbing we provide here to
help their implementations talk to the endpoint we provide or exercise
their endpoint with the provided server tests.  This assumes knowledge
of the requirements of each test / collection of tests so that the input
data is relevant.  Each test or test collection contains information
sufficient for the task.

With regard to server tests, the browser tests we provide can be
pointed at an endpoint and will exercise that endpoint using well
defined messages.  This is done semi-automatically, although some set-up
is required.

Running Tests
-------------

In the case of this test collection, we will be initially creating manual
tests.  These will automatically determine pass or fail and generate output for
the main WPT window.  The plan is to minimize the number of such tests to
ease the burden on the testers while still exercising all the features.

The workflow for running these tests is something like:

1. Start up the test driver window and select the annotation-protocol tests -
   either client or server - then click "Start".
2. A window pops up that shows a test - the description of which tells the
   tester what is required.  The window will contain fields into which some
   information is provided.
3. In the case of client testing the tester (presumably in another window) brings up their
   annotation client and points it at the supplied endpoint.  They they perform the
   action specified (annotating content in the test window, requesting an annotation from the server, etc.).
4. The server receives the information from the client, evaluates it, and reports the result of testing.
   In the event of multi-step messages, the cycle repeats until complete.
5. Repeat steps 2-4 until done.
6. Download the JSON format report of test results, which can then be visually
   inspected, reported on using various tools, or passed on to W3C for
   evaluation and collection in the Implementation Report via github.

**Remember that while these tests are written to help exercise implementations,
their other (important) purpose is to increase confidence that there are
interoperable implementations.** So, implementers are our audience, but these
tests are not meant to be a comprehensive collection of tests for an implementor.
The bulk of the tests are manual because there are no UI requirements in the
Recommendation that would make it possible to effectively stimulate every client portably.

Having said that, because the structure of these "manual" tests is very rigid,
it is possible for an implementer who understands test automation to use an
open source tool such as [Selenium](http://www.seleniumhq.org/) to run these
"manual" tests against their implementation - exercising their implementation
against content they provide to create annotations and feed the data into our
test input field and run the test.

Capturing and Reporting Results
-------------------------------

As tests are run against implementations, if the results of testing are
submitted to [test-results](https://github.com/w3c/test-results/) then they will
be automatically included in documents generated by
[wptreport](https://www.github.com/w3c/wptreport). The same tool can be used
locally to view reports about recorded results.

Automating Test Execution
-------------------------

Writing Tests
-------------

If you are interested in writing tests for this environment, see the
associated [CONTRIBUTING](CONTRIBUTING.md) document.