Evaluation of Research

This chapter is based on the notes 2022-07-19_Evaluation_PaperWriting.pdf

Why provide an evaluation of proposed concepts?

  • For readers: To convince

  • For us: To find weaknesses in the approach

How to evaluate?

  • Competitions

  • Experiments

  • Data Analysis

  • Surveys

  • Proofs

Competitions

  • Comparative evaluation

  • Participant: Team + Tool

  • Team selects best possible tool configuration for task

  • Open call for participation + call for benchmark instances

  • Ideally: objective ranking (example: SV-COMP; counterexample: VerifyThis)

  • Biases: Scoring scheme and benchmark (community agreement)

Example competition types:

  • Performance + Precision

  • Usability / Expressivenes

  • Research projects (e.g., student research competition)

Experiments

  1. Hypothesis that should be falsified/supported

  2. Experiment setup and plan

  3. Execution

  4. Results

    • Interpretation

    • Effect on hypothesis

Experiments must be available and reproducible.

  • Reliability

  • Validity

  • Objectiveness

Further information (in German): A. Butz and Antonio Krüger: Section 13: Evaluation. In Mensch-Maschine-Interaktion, De Gruyter Oldenburg 2017.

Data Analysis

  • Data already available

  1. Data scraping / mining

  2. Analysis

Common tools for analysis:

Surveys

  • Interviews

  • Questionnairs

  • Monitoring/Supervision

  • Focus groups

Proofs

  • Theorem, Proposition, Lemma

  • Proof, proof draft

    • proof by induction

    • proof by deduction

    • proof by construction

    • proof by contradiction

  • Statements that are not provable are ‘conjectures’

Excourse: Numbers in LaTeX

For typesetting numbers in LaTeX (e.g., experimental data), \usepackage{siunitx}.

\SI{15}{\giga\byte}
\SI{15}{\minute}