Characterization of different user behaviors for demand response in data centers

This repository is the artifact associated with the publication "Characterization of different user behaviors for demand response in data centers" presented at Euro-Par 2022. It contains all the scripts and files needed to reproduce the experiments presented in the paper.

General description of the experiments

We use a simulation of the RJMS (Resource and Job Management System) of a data center to study the impact of the users' submission behavior on the energy consumed, the average waiting time and the average slowdown. The simulated system is depicted below:

We replay the trace of 105 days taken from the workload metacentrum. For each of these days, we replay the trace as in the original file exept during a window of 1 (resp. 4) hour where all the users are assumed to adopt one of the following submission behaviors:

Description of the repository

• prepare_workload.ipynb: Jupyter notebook downloading and preparing the workload trace used in the experiments
• campaign.py: Python script preparing and launching in parallel the 105 experiments. Each experiment corresponds to one instance of instance.py
• analyse_campaign.ipynb: Jupyter notebook analysing the results from the campaign and plotting the graphs shown in the article

We also made use of some handy scripts to install the environment and launch the experiments in the Grid'5000 testbed. They are not directly necessary to reproduce our results but we left them in the folder scripts for interested readers.

Steps to reproduce

You should be able to reproduce the experiments with a standard Linux machine (laptop, desktop, one cluster node) that has a graphical interface for the Jupyter notebooks. We personnaly used a 16-core Intel Xeon E5-2630 v3 machine from Grid'5000 on which the experiments ran in less than two hours.

You will need 5GB disk space on / for the dependencies and ~15GB for the input and output data.

1. Install

For the sake reproductibility, all the dependancies for these experiments and their version (release tag or commit number) are managed with the package manager Nix. If you don't have it on your machine, the following command should install it. Otherwise, please refer to their documentation.

curl -L https://nixos.org/nix/install | sh

The main software used (and configured in the file default.nix) are:

• Batsim and SimGrid for the infrastructure simulation
• Batmen: our set of schedulers for batsim and plugin to simulate users
• python3, pandas, jupyter, matplotlib etc. for the data analysis

Enter a shell with all dependencies managed. This will take some time (~5mn) to download and compile averything the first time you launch it, but then all the environment is cached for future use.

nix-shell -A exp_env --pure

2. Prepare input workload

Inside the nix shell, start a notebook and follow the steps presented in prepare_workload.ipynb (~5mn):

jupyter notebook 0_prepare_workload.ipynb

3. Launch the campaign

Still inside the nix shell, launch the python script campaign.py. It will prepare and launch in parallel the 105 experiments. Each experiment corresponds to one instance of instance.py.

python3 campaign.py

This step took 96 minutes on a 2 x Intel Xeon E5-2660 v2 10 cores/CPU.

4. Analyse the results

Finally and still in the nix shell (otherwise you just need to install with pip some python libraries like pandas, evalys, matplotlib...), run this Jupyter notebook to plot the graphs displayed in the article (~10mn):

jupyter notebook analyse_campaign.ipynb