sml:
- Assume different tools for different language but methods of packaging and distributing need to be similar.
- Full desktops apps -- for even command line tools with complex local configuration -- could be a support problem.
- A service-model could be useful to managers and architects providing oversight.
- Interesting in integrating compliance checking into Bamboo build process - implies scriptable command-line tools.
Bug detection is an important part, but the more general goal (for Ken's project) is code quality improvement. So while finding defects is important, even false positives that require careful inspection to identify as such may indicate areas where the code could be made clearer or more maintainable. Refactoring tools, especially if they automatically detect things like code duplication, may also be of use. (raeburn)
Tools:
Compiler warnings:
- gcc (various)
- Sun cc
- Windows?
- ...other compilers?
Lint variants:
- clint: designed to check C++ for common programmer errors and suggest improvements; seems unmaintained (http://sourceforge.net/projects/clint/)
- pc-lint/flexelint: commercial windows-only tool (http://www.gimpel.com/)
- Solaris lint: supposedly significantly enhanced compared to traditional lint
- other flavors of lint?
- splint (already tried on parts of krb5 code, found some bugs, has issues).
- Web site: http://www.splint.org/
- Languages supported: C (C90, most C99 extensions, some GNU C extensions).
- License: GNU GPL
- Platforms: UNIX
- Problems detected include (list is excerpt from manual): Dereferencing a possibly null pointer; Using possibly undefined storage or returning storage that is not properly defined; Type mismatches, with greater precision and flexibility than provided by C compilers; Violations of information hiding; Memory management errors including uses of dangling references and memory leaks; Dangerous aliasing; Modifications and global variable uses that are inconsistent with specified interfaces; Problematic control flow such as likely infinite loops, fall through cases or incomplete switches, and suspicious statements; Buffer overflow vulnerabilities; Dangerous macro implementations or invocations; Violations of customized naming conventions.
- Observations: Splint maintenance seems minimal these days, and development pretty much stopped. The memory management tracking is interesting, based on the idea that one pointer handle "owns" an object at any one time, ownership can be transferred, functions shouldn't remember pointers they don't own, etc. However, it doesn't understand functions with different behavior in success and failure cases - like C's very critical realloc. Inline comments in code can disable warnings, alert splint to memory ownership handling that's different from its default assumptions, etc.
Other:
- BLAST: BLAST is a software model checker for C programs (http://mtc.epfl.ch/software-tools/blast/)
- BOON: BOON is a tool for automatically finding buffer overrun vulnerabilities in C source code (http://www.cs.berkeley.edu/~daw/boon/)
- cadvise (hpux only)
- calysto (work in progress by Domagoj Babic; already tried on krb5 code, found some problems; currently a service only, send email to developer)
- ccfinder, ccfinderx (www.ccfinder.net; code clone finder; supports Java, C/C++, VB, C#; runs on Windows XP)
- checkstyle (checkstyle.sourceforge.net; runs many checks on java code including coding conventions, code duplication)
- codesonar (www.grammatech.com; commercial, free trial available; supports c/c++, runs on Windows, Linux and Solaris; does interprocedural, whole-program analysis)
- coverity (current status as of early February: Kerberos team evaluating)
- crap4j: java Change Risk Analysis and Predictions tool: http://www.crap4j.org/
- Eclipse metrics tools:
- flawfinder: basic scanning, easy to set up, GPL -amb (http://www.dwheeler.com/flawfinder/, http://sourceforge.net/projects/flawfinder/)
- fortify findbugs (java only)
- fortify sca
- its4 (www.cigital.com/its4; not supported; just matches on token sequences in un-preprocessed code)
- klocwork insight, klocwork developer (www.klocwork.com; works on c, c++, java)
- MOPS: a tool for finding security bugs in C programs and for verifying conformance to rules of defensive programming http://www.cs.berkeley.edu/~daw/mops/; requires user-supplied properties to check; not currently maintained?
- oink (based on cqual) www.cubewano.org/oink
- Ounce Labs' patented Contextual Analysis technology allows source code to be automatically analyzed in a depth and level of detail never before possible: http://www.ouncelabs.com/solutions/solutions-software-portfolio-security.asp
- Pixy (http://pixybox.seclab.tuwien.ac.at/pixy/) checks PHP for XSS and SQL injection vulnerabilities.
- pmd (java only)
- polyspace (www.mathworks.com; supports C/C++, Ada for embedded systems)
- PScan (format string problems mainly; flawfinder, RATS, and gcc can do similar things; server not responding 1/24)
- pychecker (Python only)
- rats (Rough Auditing Tool for Security; rough analysis intended as a starting point for manual analysis; http://www.fortifysoftware.com/security-resources/rats.jsp)
- simian (similarity analyser; www.redhillconsulting.com.au/products/simian/overview.html; identifies duplication in c, c++, c#, java, html, ml, vb, text, etc; runs in .net 1.1 or java 1.4 or later; free for non-commercial or open source use)
- skavenger: mostly for php (fancy grep replacement, really? not interesting. -amb) (http://code.google.com/p/skavenger/)
- SmartRisk Analyzer (gone? originally @stake, which was acquired by Symantec)
- SMATCH: Smatch is C source checker but mainly focused checking the Linux kernel code (http://smatch.sourceforge.net/)
- SourceAudit: C/C++; interesting on paper, at least; costs money? -amb (http://www.sourceaudit.com/products_sa.php)
- SPARROW (http://www.spa-arrow.com/) looks for memory leaks, use-after-free, buffer overruns. Supports Mac, Windows, Linux, Solaris, FreeBSD. On-site demo and trial copy available.
- sparse (http://www.kernel.org/pub/software/devel/sparse/)
- xrefactory (www.xref-tech.com; c and java refactoring tool and source browser; includes emacs support)
- unpaste (finds parallel syntactic constructs that are sometimes duplicated or nearly identical code)
- Veracode SecurityReview (binary code analysis service; web site says results are generally returned in 24-72 hours, which might be useful when preparing for release or deployment but perhaps not as a regular, automatic part of the development process)
See also lists at:
There are also various tools out there for building analysis tools, like bddbddb or LLVM, but unless they easily enable some specific sorts of useful checks that are difficult to get through other tools, let's not invest effort at this time.
Some possible areas of code analysis:
- memory management
- buffer overruns
- dangling, uninitialized or null pointer use
- bad free() calls
- memory leaks
- arithmetic overflow bugs
- portability issues
- coding standards compliance
- error code management
- avoiding operator precedence confusion in C
- avoiding = vs == confusion in C
- error conditions not checked
- even simple stuff like whitespace
- code duplication, other automatic refactoring opportunity detection
- code complexity: A good score is not really an indicator of good code, but excessive complexity would suggest that the code may be hard to understand and maintain, and easier to break with a seemingly simple change.
- cross-site scripting
- unsafe use of attacker-influenced data (e.g., "taint" tracking)
- SQL injection
- format strings
- command line or pathname generation
- validation of externally-provided binary data structures before use
- conversion issues
- more overflow potential
- bad casting
- ...
Languages we care about:
- c, c++
- objective c?
- java
- php
- perl?
- python?
- c#?
Evaluation criteria:
- tool status: prototype, fully functional, development ongoing, maintained, stagnant, dead
- license: Is it open source, or do we have to keep access restricted? What restrictions are there on how we can use it? (Can we use it on a shared build engine?) Can we make minor fixes if necessary? Price? Can we make public a review or analysis of the tool? Do we have to restrict who can see the results?
- support: Is help available if we run into problems?
- languages: Which programming languages are supported, and how well?
- platforms: Windows? Mac? UNIX? What implementation language? Other packages that need to be installed to support it?
- ease of use: Invoke via command line or makefile? Plug in to Eclipse? Has its own GUI? Does it need to be fed all the source for a program at once? Can it analyze libraries we write, and applications using analyzed libraries, or only whole-program analysis?
- intrusiveness: Does it require stylized code, magic comments, additional input or generated files? Would the stylized code, if needed, trigger complaints from other tools?
- Types of analysis: What kind of problems or issues does it look for?
- hit rate: Does it miss a lot of problems? Does it report a lot of false positives?
- Can we suppress false positives we've analyzed and found to be okay?
- Is it being actively developed, or at least maintained?