1. Examine Proc Doc

2. Examine Sys Doc

3. Interview Developers

4. Interview Users

5. Examine Code I

6. Examine Code II

7. Determine Truth

8. Generate Test Cases

9. Run Test Cases

10. Review Test Output

11. Assessment


SDP Template

SQPP Template

SCMP Template

WBS Template


Introduction

What and Why of VV&A



Step 11 -- Assessment

Recommendation: does code meet requirements for its intended use? Possible answers are:
  • Yes, use for this application
  • For the following types of cases, yes; for others, no
  • If auxiliary code (pre-processors, post-processors) provided
  • If following documentation is provided
  • If specified user training provided
  • If following changes are made to documentation
  • If following changes are made to user training
  • If following changes are made to code
  • Do more V&V first
  • No, do not use for this application

    • Performance Assessment
  • How close does the output come to truth (validation)?
  • How close does the output need to come to truth (acceptability criteria)?
  • How well can truth be determined?
  • How easy is it to compare output to truth?
  • How thorough are the test cases/comparisons to truth?
  • Error bounds?
  • Crashes/freezes/failures?
  • Graceful failure?
  • Are there workarounds?

    • Code Assessment
  • How complex is the code (via McCabe, Halstead and other measures)?
  • How well is the code designed?
  • Is the code modular?
  • Object oriented (information hiding etc)?
  • How robust is the code?
  • Does it include examples/test cases/pre-processors/post-processors?

    • Documentation Assessment
  • In-line documentation
  • User documentation/manuals
  • How well does documentation help the user run the sim?
  • Are examples of inputs and outputs included?
  • How well does documentation describe limitations/problems/workarounds?
  • How well does documentation describe algorithms/assumptions?
  • As-designed or as-built?
  • How complete?

    • Assessment of User Needs/Training/Sophistication
  • Does sim calculate the things the user wants it to?
  • How easy is it to use the sim?
  • Input checking?
  • Error checking?
  • Warnings (upon input or during calculation)?
  • How much user training/knowledge required?
  • What kinds of errors/problems does the user encounter?
  • How serious are the errors/problems?
  • Are there workarounds?

    • Perceived CMM level of developers
  • A single number that says "this sim probably does (or doesn't) meet requirements"
  • Relates to verification, not validation
  • Quality of developer requirements management
  • Quality of developer process documentation
  • Quality of developer CM practices
  • Quality of developer QA practices
  • Quality of developer change (requirements/bug fix) practices
  • Quality of developer testing
  • Helps in evaluating future sims by that developer
  • Helps keep developer CMM assessment honest