blob: d15654e1c1df7c2a379ea79899e2090d32e6ae81 [file] [log] [blame]
Coverage TODO
Key:
* Heading
- Not done yet.
+ Done.
x Not going to do.
* 4.0
- What defaults should change?
- --source = . ?
- --branch = True ?
- Remove 2.3, 2.4, 2.5 limitations
+ set, sorted, reversed, rpartition
+ generator expressions
+ decorators
+ collections.defaultdict
+ .startswith((,))
+ "with" statements
- .format() ?
+ try/except/finally
+ with assertRaises
- addCleaup instead of tearDown (only in 2.7!)
+ exec statement can look like a function in py2 (since when?)
- runpy ?
- we can use "except ExcClass as e:"
+ Remove code only run on <2.6
- Change data file to json
- Create data api
- New ast-based branch coverage?
+ gevent, etc.
+ Remove the old command-line syntax
+ A pain, b/c of the structure of the tests.
+ BTW: make an easier way to write those tests.
- Documentation
- Plugins!
Once per process
Once per file
- create a file tracer
- call its has_dynamic_source_file()
Once per call
Once per line
* --source stuff:
+ warn if a package is never found.
+ warn if no data was collected
- tie --source into reporting
* Soon
+ Better omit handling that ignores files during measurement.
- Deal with ~ in specified paths correctly.
+ while TRUE claims to be partial.
+ A way to mark lines as partial branches, with a regex?
+ Default to "while True:", "while 1:"
+ HTML keyboard short cuts
* 3.2
+ Some kind of indication in the HTML where yellow lines aren't going.
- Profile the reporting code: it's REALLY slow.
- parser is doing some redundant work.
+ Analysis class should do rolling up of stats also (actually Numbers)
+ Update docs for --branch.
- self.coverage.data.has_arcs is ugly.
+ Branches that never jump to nocover lines shouldn't be marked as partial.
(see top of test_cogapp for examples)
+ Maybe turning off yellow lines should make those lines green?
+ A missing branch to leave the function shows an annotation of -1. Now "exit".
+ XML report needs to get branch information.
+ Add branch info to "coverage debug data"
+ Polish up the help, and double-check the docs.
* Speed
+ C extension collector
- bitvector in trace extension.
- Ignore certain modules
+ Record linenos rather than (file,lineno) pairs in tracer.
x Tricky swapping of collector like figleaf, pycov, et al. (Don't need to do
this with C collector).
- Seems like there should be a faster way to manage all the line number sets in
CodeParser.raw_parse.
- If tracing, canonical_filename_cache overlaps with should_trace_cache. Skip
canonical_filename_cache. Maybe it isn't even worth it...
- Would pre-allocating line number integers make the C tracer faster? It would
use less memory.
* Accuracy
- Record magic number of module to ensure code hasn't changed
- Record version of coverage data file, so we can update what's stored there.
- Record options in coverage data file, so multiple runs are certain to make
sense together.
- Do I still need the lines in annotate_file that deal specially with "else"?
* Power
+ Branch coverage
Titus' idea:
1: if a:
2: b = 2
3: c = 3
if the coverage data shows 1,2,3, it was if-then. if it's 1,3, then the
missing else was executed.
- API for getting coverage data.
- Instruction tracing instead of line tracing.
- Path tracing (how does this even work?)
- Count execution of lines
- Track callers of functions (ala std module trace)
- Method/Class/Module coverage reporting.
- .coverage files that can be kept separate, rather than accumulated.
- test/coverage map: http://rbtcollins.wordpress.com/2009/09/16/back-from-hiatus/
- Similar to figleaf's sections.
* Convenience
- Command line modules should also be directories, meaning all the modules in that
directory.
- Why can't a morf also be a string, the name of a module?
- ignore by module as well as file?
+ Use a .coveragerc file to control Coverage without the programmatic API.
- Add a --data switch to explicitly control the data file on the command line.
x Why can't you specify execute (-x) and report (-r) in the same invocation?
Maybe just because -x needs the rest of the command line?
+ Support 2.3 - 3.1!
http://pythonology.blogspot.com/2009/02/making-code-run-on-python-20-through-30.html
http://www.rfk.id.au/blog/entry/preparing-pyenchant-for-python-3
http://pydev.blogspot.com/2008/11/making-code-work-in-python-2-and-3.html
+ Explicitly set pickle protocol to 2.
- An inference mode that marks lines as executed if they "must have been" executed:
class definitions, etc, when coverage is started after the class is defined.
- Different categories of exclude pragma? So you can enable and disable them
from the command line, to reconsider exclusions.
+ Reporting on files never touched by coverage (package completeness)
- A setup.py command? http://jeetworks.org/node/50
- Deltas: indicate the change in coverage percentage from the last run.
+ Show lines missing rather than lines run in the reporting, since that's what
you need to focus on.
* Beauty
+ HTML report
- Colored bars indicating coverage per file.
- Package navigation.
- Rolled-up statistics.
- Some way to focus in on red and yellow
- Show only lines near highlights?
+ Jump to next highlight?
+ Keyboard navigation: j and k.
- Cookie for changes to pyfile.html state.
+ Clickable column headers on the index page.
+ Syntax coloring in HTML report.
+ Dynamic effects in HTML report.
+ Footer in reports pointing to coverage home page.
+ Baseline grid for linenumber font.
+ Separate out css and HTML.
+ Does it work right with utf-8 source files? http://www.python.org/dev/peps/pep-0263/
- Use vim modeline to determine tab width: http://vimdoc.sourceforge.net/htmldoc/options.html#modeline
* Community
+ New docs, rather than pointing to Gareth's
+ Min python version is 2.3.
- Three phases of work:
- Collection
- Analysis
- Reporting
- Distinction between:
- ignore (files not to collect)
- exclude (lines not to report as missed)
- omit (files not to report)
- Changes from coverage 2.x:
- Bare "except:" lines now count as executable code.
- Double function decorators: all decorator lines count as executable code.
- Document the .coverage file format.
+ HTML reporting.
- Much more detail about what's in the report.
- References between pages are off:
- They have <em> tags around them.
- They use #anchors that don't survive the px->html conversion.
+ Be sure --help text is complete (-i is missing).
+ Host the project somewhere with a real bug tracker: bitbucket.org
+ Point discussion to TIP
- PEP 8 compliance?
* Programmability
+ Don't use sys.exit in CoverageScript.
+ Remove singleton
+ Initialization of instance variables in the class.
* Installation
x How will Coverage package install over coverage.py module?
x pip can't install it: it reads the coverage html page, and finds the kit link,
but then can't handle the root-relative link.
* Modernization
+ Decide on minimum supported version
+ 2.3
+ Get rid of the basestring protection
+ Use enumerate
+ Use sets instead of dicts
+ Switch from getopt to optparse.
+ Get rid of the recursive nonsense.
+ Docstrings.
+ Remove huge document-style comments.
- Better names:
+ self.cache -> self.cache_filename -> CoverageData.filename
+ self.usecache -> CoverageData.use_file
- More classes:
- Module munging
+ Coverage data files
+ Why are some imports at the top of the file, and some in functions?
+ Get rid of sys.exitfunc use.
+ True and False (with no backward adaptation: the constants are new in 2.2.1)
+ Get rid of compiler module
+ In analyzing code
+ In test_coverage.py
+ Style:
+ lineno
+ filename
* Correctness
- What does -p (parallel mode) mean with -e (erase data)?
* Tests
+ Switch to a real test runner, like nose.
+ Test both the C trace function and the Python trace function.
+ parser.py has no direct tests.
- Tests about the .coverage file.
+ Tests about the --long-form of arguments.
+ Tests about overriding the .coverage filename.
- Tests about parallel mode.
+ Tests about assigning a multi-line string.
- Tests about tricky docstrings.
+ Coverage test Coverage!
- Tests that tracing stops after calling stop()
- More intensive thread testing.
x Tests about the "import __main__" in cmdline.py
+ What happens if the -x script raises an exception?
- Test that the kit has all the proper contents.