Skip to content
Snippets Groups Projects
Commit 2066d4d3 authored by Maël Madon's avatar Maël Madon
Browse files

some cleaning and update readme

parent a4149fcf
No related branches found
No related tags found
No related merge requests found
...@@ -2,15 +2,22 @@ ...@@ -2,15 +2,22 @@
This repository contains the material for the practical session of course [RM4ES](https://moodle.univ-tlse3.fr/course/view.php?id=4996) given in the Spring semester of 2024. This repository contains the material for the practical session of course [RM4ES](https://moodle.univ-tlse3.fr/course/view.php?id=4996) given in the Spring semester of 2024.
## Install ## Session 1 : getting started
**Objective for this session:**
- setting up all the simulation environment
- understanding the workload representation in [Batsim]
- having a first scheduling algorithm running
### Install
Ensure that you are in your home directory (run `cd`). Ensure that you are in your home directory (run `cd`).
Clone this repository: Clone this repository:
``` ```
git clone git@gitlab.irit.fr:sepia-pub/mael/RM4ES-practicals.git git clone https://gitlab.irit.fr/sepia-pub/mael/RM4ES-practicals.git
``` ```
Install [Nix]: Install [Nix], that will manage for you all the dependencies you need for the course:
```bash ```bash
# download and install nix-user-chroot, a hack to install nix without root privilege # download and install nix-user-chroot, a hack to install nix without root privilege
...@@ -25,43 +32,20 @@ mkdir -p ~/.config/nix && cp ~/RM4ES-practicals/util/nix.conf ~/.config/nix ...@@ -25,43 +32,20 @@ mkdir -p ~/.config/nix && cp ~/RM4ES-practicals/util/nix.conf ~/.config/nix
echo 'source ~/.nix-profile/etc/profile.d/nix.sh' >> ~/.bashrc echo 'source ~/.nix-profile/etc/profile.d/nix.sh' >> ~/.bashrc
``` ```
Now, each time you want to test your schedulers, you need to run, from the repository `RM4ES-practicals`: **IMPORTANT:** Now, each time you want to test your schedulers, you need to run, from the repository `RM4ES-practicals`:
```bash ```bash
# enter the nix-user-chroot shell
~/nix-user-chroot ~/.nix bash ~/nix-user-chroot ~/.nix bash
# enter the nix shell defined in file default.nix, with all dependencies managed.
# can take a few minute the first time you run it, to download everything.
nix-shell -A expe nix-shell -A expe
``` ```
Or, as a one-liner:
```bash
~/nix-user-chroot ~/.nix bash -c "nix-shell -A expe"
```
## Status rn
We can start a small expe!
On one shell:
```
ns -A dev
pybatsim newSched
```
On the other shell:
```
ns -A expe
batsim -p expe/1machine.xml -w expe/2jobs.json -e out/
```
## Session 1 : getting started
**Objective for this session:**
- setting up all the simulation environment
- understanding the workload representation in [Batsim]
- having a first scheduling algorithm running
### Introduction ### Introduction to Batsim
The resource and job management simulator [Batsim], that we are going to use during these practicals, needs three inputs: The resource and job management simulator [Batsim], that we are going to use during these practicals, needs three inputs:
...@@ -92,13 +76,14 @@ The workload is represented by a JSON file (see [expe/2jobs.json](./expe/2jobs.j ...@@ -92,13 +76,14 @@ The workload is represented by a JSON file (see [expe/2jobs.json](./expe/2jobs.j
The meaning of each field in the JSON is explained in [Batsim workload documentation](https://batsim.readthedocs.io/en/latest/input-workload.html). The meaning of each field in the JSON is explained in [Batsim workload documentation](https://batsim.readthedocs.io/en/latest/input-workload.html).
The scheduling algorithm has to be developed separately, and communicates with [Batsim] through the messages defined in [Batsim protocol](https://batsim.readthedocs.io/en/latest/protocol.html). The scheduling algorithm has to be developed separately, and communicates with [Batsim] through the messages defined in [Batsim protocol](https://batsim.readthedocs.io/en/latest/protocol.html).
In this tutorial, we will develop our schedulers in Python thanks to [pybatsim](https://gitlab.inria.fr/batsim/pybatsim). In this tutorial, we will develop our schedulers in Python thanks to [pybatsim] (version 3.2.0).
The scheduler implementations will be located in the folder [sched/](./sched/). The scheduler implementations will be located in the folder [sched/](./sched/).
### Exercise 1 ### Exercise 1
Represent the workload used previously in this course (exercise 1 from the [exercise document](https://docs.google.com/document/d/15nfFvYgTwStwS0JGt5DMaQs6JDya8dKzFj69wMLTuTw/edit)) in Batsim JSON format. Represent the workload used previously in this course (exercise 1 from the [exercise document](https://docs.google.com/document/d/15nfFvYgTwStwS0JGt5DMaQs6JDya8dKzFj69wMLTuTw/edit)) in Batsim JSON format.
You will put it in a new file named `expe/ex1.json` You will put it in a new file named `expe/ex1.json`.
We will use the field "walltime" to store the "deadline".
### Exercise 2 ### Exercise 2
TODO run a simulation with pre-defined scheduler `rejector`. TODO run a simulation with pre-defined scheduler `rejector`.
...@@ -127,5 +112,21 @@ Visualize the output thanks to evalys. ...@@ -127,5 +112,21 @@ Visualize the output thanks to evalys.
- no working poetry in nix-shell - no working poetry in nix-shell
- will that work in the lab's machines? - will that work in the lab's machines?
## Status rn
We can start a small expe!
On one shell:
```
ns -A dev
pybatsim newSched
```
On the other shell:
```
ns -A expe
batsim -p expe/1machine.xml -w expe/2jobs.json -e out/
```
[Batsim]: https://batsim.org/ [Batsim]: https://batsim.org/
[Nix]: https://nixos.org/nix/ [Nix]: https://nixos.org/nix/
\ No newline at end of file [pybatsim]: https://gitlab.inria.fr/batsim/pybatsim
\ No newline at end of file
{
"description": "Test easy backfill.",
"nb_res": 2,
"jobs": [
{"id": "0", "profile": "100sec", "res": 1, "walltime":110, "subtime": 0},
{"id": "1", "profile": "100sec", "res": 2, "walltime":110, "subtime": 10},
{"id": "2", "profile": "10sec", "res": 1, "walltime":20, "subtime": 50},
{"id": "3", "profile": "100sec", "res": 1, "walltime":110, "subtime": 300},
{"id": "4", "profile": "100sec", "res": 2, "walltime":110, "subtime": 310},
{"id": "5", "profile": "10sec", "res": 1, "walltime":80, "subtime": 350}
],
"profiles": {
"100sec": {"delay": 100,"type": "delay"},
"10sec": {"delay": 10,"type": "delay"}
}
}
\ No newline at end of file
===============================
Pybatsim - Example
===============================
This is an example folder to show you how to setup and package your own scheduler in Pybatsim
Setup your scheduler
~~~~~~~~~~~~~~~~~~~~
A small scheduler is given as an example, which simply rejects all submitted jobs.
To use your own scheduler, put the source file in this current repository (along with `universal_rejection.py, then register it in the pybatsim entry point by adding a line in the `pyproject.toml` file under the `[tool.poetry.plugins."pybatsim.schedulers"]` section.
The line must follow the pattern::
`<scheduler_name> = <file_name>:<Scheduler_class_name>`.
Start your scheduler
~~~~~~~~~~~~~~~~~~~~
Once done, to use your scheduler you can use the `example-shell` nix recipee defined in the root folder of Pybatsim to enter a virtual environment with everything you need by running (from this directory)::
nix-shell ../default.nix -A example-shell
And then launch your scheduler with::
pybatsim <scheduler_name>
Or with a one liner::
nix-shell ../default.nix -A example-shell --command 'pybatsim <scheduler_name>'
Develop your scheduler
~~~~~~~~~~~~~~~~~~~~~~
To develop your scheduler, it is convenient to enter a poetry environment by running (from this directory)::
poetry install
poetry shell
Then, you can modify your scheduler and test it by directly running::
pybatsim <scheduler_name>
Note for nix users: a shell containing `poetry` is already defined in the `default.nix` file in the root forled of Pybatsim.
From this directory, before running the poetry commands, you can run::
nix-shell ../default.nix -A dev-shell
This diff is collapsed.
[build-system]
requires = ["poetry-core>=1.0.0"]
build-backend = "poetry.core.masonry.api"
[tool.poetry]
name = "RM4ES"
version = "0.0.0"
description = "Small schedulers for RM4ES course"
authors = []
packages = [
{include = "*.py"},
]
[tool.poetry.dependencies]
python = "^3.7.1"
pybatsim = "^4.0.0a0"
[tool.poetry.plugins."pybatsim.schedulers"]
rejector = "universal_rejection:UniversalRejectionScheduler"
fcfs2 = "fcfs:Fcfs2"
newSched = "new_sched:Newsched"
""" """
Trivial example scheduler that rejects any job. Trivial example scheduler that rejects any job.
No hard feelings!
""" """
from pybatsim.batsim.batsim import BatsimScheduler from batsim.batsim import BatsimScheduler, Batsim
class UniversalRejectionScheduler(BatsimScheduler): class Rejector(BatsimScheduler):
def onJobSubmission(self, job): def onJobSubmission(self, job):
self.bs.reject_jobs([job]) # nope! self.bs.reject_jobs([job]) # nope!
{
kapack ? import
(fetchTarball "https://github.com/oar-team/nur-kapack/archive/901a5b656f695f2c82d17c091a55db2318ed3f39.tar.gz")
{}
, pkgs ? kapack.pkgs
, python3 ? pkgs.python3
, python3Packages ? pkgs.python3Packages
}:
let
jobs = rec {
evalys = kapack.evalys.overrideAttrs (attr: rec {
name = "evalys";
version = "d4d47bd2f4d076730b05f1345b1bf9032cd28753";
src = pkgs.fetchgit rec {
url = "https://github.com/Mema5/evalys.git";
rev = version;
sha256 = "sha256-eHGRNj2xWBoEgG2wvAZUoH1Us5d1yu9emu1Zyyms1JU=";
};
});
expe = pkgs.mkShell rec {
buildInputs = [
kapack.batsim-410
kapack.pybatsim-320
python3Packages.matplotlib
evalys
];
QT_QPA_PLATFORM_PLUGIN_PATH= with pkgs.libsForQt5.qt5; "${qtbase.bin}/lib/qt-${qtbase.version}/plugins/platforms";
};
};
in
jobs
\ No newline at end of file
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Please register or to comment