Reproducibility Workshop

MV group

What’s reproducibility?

Why is it important?

Reproducibility:

To recreate an idea or process

Is there a reproducibility crisis in science?

The real issue behind the reproducibility crisis?

Only focus on production of results

Reproducibility problem:

Academia Edition

  • No time
  • Publication pressure
  • Poor organization
  • Result fishing
  • Lack of ideas
  • No will to learn new skills

Undermine scientific research credibility!

Reproducibility problem:

Industry Edition

  • No time
  • Task completion pressure
  • Lack of planning beforehand
  • Non-structured work
  • Competititve focus on growth
  • Lack of communication

Risk of:

  • Inefficiency
  • Innovation slow-down
  • Lack of accountability

How is it in real life?

Academia Edition

Industry Edition

Factors of the reproducibility problem

Failure to reproduce results

Small differences in execution

Selective reporting

Detractors of execution

Factors of the reproducibility problem

Failure to reproduce results

In your own research

Results in a paper you found

Results from a previous collaboration

Factors of the reproducibility problem

Failure to reproduce results

Small differences in execution

Selective reporting

Detractors of execution

Factors of the reproducibility problem

Small differences in execution

Steps in physical experiments

Randomization in computational experiments

Changes in software version

Factors of the reproducibility problem

Failure to reproduce results

Small differences in execution

Selective reporting

Detractors of execution

Factors of the reproducibility problem

Selective reporting

Difficult parameter tuning in modelling

Difficult data processing

Deep data selection

Factors of the reproducibility problem

Failure to reproduce results

Small differences in execution

Selective reporting

Detractors of execution

Factors of the reproducibility problem

Detractors of execution

Lack of time

Incomplete release of material

Publication pressure

Lack of organization

Lack of interest to learn new skills

Experiences and examples

Is it really a problem?

Dear researcher…

I read your publications, is it possible to have access to the data and methods you’ve used in this work?

Dear researcher…

I read your publications, is it possible to have access to the data and methods you’ve used in this work?

Yes of course!

What is the final product in research?

A discovery, a project, a publication, a prize … ?

What is the final product in research?

A paper

A project

.
├── case_studies
│   ├── case1
│   │   ├── experiments
│   │   └── images
│   └── case2
├── README.md
├── software 
│   └── bin
└── results 
    ├── figures
    └── measurements

Figures of interest

How often do you find code and data in published research?

Data availability

Analysis of sources

Analysis of sources per year

Hands-on part 1

Let’s reproduce!

Exercise 1

Can you reproduce a project …

  1. when it is a completely automatic process?

  2. when it has a few manual steps?

  3. when it is completely manual?

Let’s reproduce! (Click here)

Elements of reproducibility

3 pillars of reproducibility

Elements of reproducibility

Strategy

Structure

Technical aspects

Strategy

How do I proceed with my project?

%%{init: {'theme':'forest'}}%%
flowchart TB
  A(Familiarize\n yourself) --> B(Test!)
  B -.-> B
  B --> C{Design}
  C --> D[Manual?]
  C --> E[Automatic?]
  D --> F(Execute)
  E --> F
  F <--> G(Add documentation)

Elements of reproducibility

Strategy

Structure

Technical aspects

Structure

Who is the NEXT USER of my project?

.
├── case_studies # Input
│   ├── case1
│   │   ├── experiments
│   │   └── images
│   └── case2
├── README.md # Documentation
├── software # Process
│   └── bin
└── results # Output
    ├── figures
    └── measurements

%%{init: {'theme':'forest'}}%%
flowchart TB
  A(Input) -- Add documentation --> B[Process]
  B -- Add documentation --> C[Output]

Elements of reproducibility

Strategy

Structure

Technical aspects

Technical aspects

What tools do I need for my project?

%%{init: {'theme':'forest'}}%%
graph TB
S(Project) -.- A
S -.- E
subgraph Experimental
    A(In the lab) --> B(Instruments)
    A --> C(Materials)
    A --> D(Protocols)
end
subgraph Computational
    E(On PC) --> F(Operative system)
    E --> G(Software)
    E --> H(Programming <br> language)
end

Delivering a project

%%{init: {'theme':'forest'}}%%
graph TB
A[Strategy] --> D{Project}
B[Structure] --> D
C[Technical Aspects] --> D
D -- Outreach --> E(Publication)
D -- Reproduce --> F(Repository)

Hands-on part 2

Let’s get reproducible!

Exercise 2

How reproducible are you?

a) Pimp my project

  • Choose a project of your own
  • Duplicate it
  • Turn it into a reproducible project!

b) What a mess!

  • Copy the provided project
  • Try not to cry
  • Turn it into a reproducible project!

Let’s get reproducible! (Click here)

Thank you!

Contact us

mv-learn.netlify.app