sml:
- Assume different tools for different language but methods of packaging and distributing need to be similar.
- Full desktops apps -- for even command line tools with complex local configuration -- could be a support problem.
- A service-model could be useful to managers and architects providing oversight.
- Interesting in integrating compliance checking into Bamboo build process - implies scriptable command-line tools.
raeburn:
Bug detection is an important part, but the more general goal (for Ken's project) is code quality improvement. So while finding defects is important, even false positives that require careful inspection to identify as such may indicate areas where the code could be made clearer or more maintainable. Refactoring tools, especially if they automatically detect things like code duplication, may also be of use.
Tools or services to consider (list obviously needs winnowing to make manageable):
- various gcc warnings
- various Sun compiler warnings
- Windows compiler warnings?
- ...other compilers?
- Solaris lint: supposedly significantly enhanced compared to traditional lint
- pc-lint/flexelint
- other flavors of lint?
- splint (already tried on parts of krb5 code, found some bugs, has issues).
- Web site: http://www.splint.org/
- Languages supported: C (C90, most C99 extensions, some GNU C extensions).
- License: GNU GPL
- Platforms: UNIX
- Problems detected include (list is excerpt from manual):
- Dereferencing a possibly null pointer
- Using possibly undefined storage or returning storage that is not properly defined
- Type mismatches, with greater precision and flexibility than provided by C compilers
- Violations of information hiding
- Memory management errors including uses of dangling references and memory leaks
- Dangerous aliasing
- Modifications and global variable uses that are inconsistent with specified interfaces
- Problematic control flow such as likely infinite loops, fall through cases or incomplete switches, and suspicious statements
- Buffer overflow vulnerabilities
- Dangerous macro implementations or invocations
- Violations of customized naming conventions.
- Observations: Splint maintenance seems minimal these days, and development pretty much stopped. The memory management tracking is interesting, based on the idea that one pointer handle "owns" an object at any one time, ownership can be transferred, functions shouldn't remember pointers they don't own, etc. However, it doesn't understand functions with different behavior in success and failure cases – like C's very critical realloc. Inline comments in code can disable warnings, alert splint to memory ownership handling that's different from its default assumptions, etc.
- polyspace (www.mathworks.com; supports C/C++, Ada for embedded systems)
- coverity (current status as of early February: Kerberos team evaluating)
- klocwork insight, klocwork developer (www.klocwork.com; works on c, c++, java)
- pmd (java only)
- fortify findbugs (java only)
- codesonar (www.grammatech.com; commercial, free trial available; supports c/c++, runs on Windows, Linux and Solaris; does interprocedural, whole-program analysis)
- fortify sca
- flawfinder
- simian (similarity analyser; www.redhillconsulting.com.au/products/simian/overview.html; identifies duplication in c, c++, c#, java, html, ml, vb, text, etc; runs in .net 1.1 or java 1.4 or later; free for non-commercial or open source use)
- checkstyle (checkstyle.sourceforge.net; runs many checks on java code including coding conventions, code duplication)
- xrefactory (www.xref-tech.com; c and java refactoring tool and source browser; includes emacs support)
- ccfinder, ccfinderx (www.ccfinder.net; code clone finder; supports Java, C/C++, VB, C#; runs on Windows XP)
- unpaste (finds parallel syntactic constructs that are sometimes duplicated or nearly identical code)
- cadvise (hpux only)
- oink (based on cqual) www.cubewano.org/oink
- its4 (www.cigital.com/its4; not supported; just matches on token sequences in un-preprocessed code)
- rats (Rough Auditing Tool for Security; rough analysis intended as a starting point for manual analysis; http://www.fortifysoftware.com/security-resources/rats.jsp)
- Veracode SecurityReview (binary code analysis service?)
- SmartRisk Analyzer (gone? originally @stake, which was acquired by Symantec)
- PScan (format string problems mainly; flawfinder, RATS, and gcc can do similar things; server not responding 1/24)
- calysto (work in progress by Domagoj Babic; already tried on krb5 code, found some problems; currently a service only, send email to developer)
- sparse (http://www.kernel.org/pub/software/devel/sparse/)
- pychecker (Python only)
- Pixy (http://pixybox.seclab.tuwien.ac.at/pixy/) checks PHP for XSS and SQL injection vulnerabilities.
- Eclipse metrics tools:
See also lists at:
There are also various tools out there for building analysis tools, like bddbddb or LLVM, but unless they easily enable some specific sorts of useful checks that are difficult to get through other tools, let's not invest effort at this time.
Some possible areas of code analysis:
- memory management
- buffer overruns
- dangling, uninitialized or null pointer use
- bad free() calls
- memory leaks
- arithmetic overflow bugs
- portability issues
- coding standards compliance
- error code management
- avoiding operator precedence confusion in C
- avoiding = vs == confusion in C
- error conditions not checked
- even simple stuff like whitespace
- code duplication, other automatic refactoring opportunity detection
- code complexity: A good score is not really an indicator of good code, but excessive complexity would suggest that the code may be hard to understand and maintain, and easier to break with a seemingly simple change.
- cross-site scripting
- unsafe use of attacker-influenced data (e.g., "taint" tracking)
- SQL injection
- format strings
- command line or pathname generation
- ...
Languages we care about:
- c, c++
- objective c?
- java
- php
- perl?
- python?
- c#?
Evaluation criteria:
- tool status: prototype, fully functional, development ongoing, maintained, stagnant, dead
- license: Is it open source, or do we have to keep access restricted? What restrictions are there on how we can use it? Can we make minor fixes if necessary? Price? Can we make public a review or analysis of the tool?
- support: Is help available if we run into problems?
- languages: Which programming languages are supported, and how well?
- platforms: Windows? Mac? UNIX? What implementation language? Other packages that need to be installed to support it?
- ease of use: Invoke via command line or makefile? Plug in to Eclipse? Has its own GUI? Does it need to be fed all the source for a program at once? Can it analyze libraries we write, and applications using analyzed libraries, or only whole-program analysis?
- intrusiveness: Does it require stylized code, magic comments, additional input or generated files? Would the stylized code, if needed, trigger complaints from other tools?
- Types of analysis: What kind of problems or issues does it look for?
- hit rate: Does it miss a lot of problems? Does it report a lot of false positives?
- Can we suppress false positives we've analyzed and found to be okay?
- Is it being actively developed, or at least maintained?