# Evaluation of Research

*This chapter is based on the notes [2022-07-19_Evaluation_PaperWriting.pdf](../../course_notes/2022-07-19_Evaluation_PaperWriting.pdf)*

Why provide an evaluation of proposed concepts?

- For readers: To convince
- For us: To find weaknesses in the approach

How to evaluate?

- Competitions
- Experiments
- Data Analysis
- Surveys
- Proofs

## Competitions

- Comparative evaluation
- Participant: Team + Tool
- Team selects best possible tool configuration for task
- Open call for participation + call for benchmark instances
- Ideally: objective ranking (example: [SV-COMP](https://sv-comp.sosy-lab.org/2022/); counterexample: [VerifyThis](https://verifythis.github.io/))
- Biases: Scoring scheme and benchmark (community agreement)

Example competition types:

- Performance + Precision
- Usability / Expressivenes
- Research projects (e.g., student research competition)

## Experiments

1. Hypothesis that should be falsified/supported
2. Experiment setup and plan
3. Execution
4. Results
    - Interpretation
    - Effect on hypothesis

Experiments must be available and reproducible.

- Reliability
- Validity
- Objectiveness

Further information (in German):
[A. Butz and Antonio Krüger: Section 13: Evaluation. In Mensch-Maschine-Interaktion, De Gruyter Oldenburg 2017.](https://www-degruyter-com.emedien.ub.uni-muenchen.de/document/doi/10.1515/9783110476378/)

## Data Analysis

- Data already available

1. Data scraping / mining
2. Analysis

Common tools for analysis:

- [Jupyter Notebooks](https://jupyter.org/)
- [R](https://www.r-project.org/)
- [Python](https://www.python.org/)
- [Jasp](https://jasp-stats.org/)
- [gnuplot](http://www.gnuplot.info/)
- [PGF plots (tikz)](https://ctan.org/pkg/pgfplots?lang=en)
- Plotting in python with [seaborn](https://seaborn.pydata.org/)


## Surveys

- Interviews
- Questionnairs
- Monitoring/Supervision
- Focus groups

## Proofs

- Theorem, Proposition, Lemma
- Proof, proof draft
    - proof by induction
    - proof by deduction
    - proof by construction
    - proof by contradiction
- Statements that are not provable are 'conjectures'



## Excourse: Numbers in LaTeX

For typesetting numbers in LaTeX (e.g., experimental data),
[`\usepackage{siunitx}`](https://ctan.org/pkg/siunitx).

```
\SI{15}{\giga\byte}
\SI{15}{\minute}
```