Methodology of Benchmarks

• Free Availability but copyright and procedure protected.
• Time Measurement in clock wall time.
• Full disclosure agreement
  • all parameters, hardware and software configuration (optional price)
  • meta algorithm, used algorithm, or reference (if appropriate)
  • no implementation details ?
  • application submission form
• Reproducibility, reproduction procedure, competitors.
• Benchmark Database, design, features, availability. Scope of systems (What can be solved by which system ?)