slight change in the platform size to match the new workload

0_prepare_workload.ipynb

@@ -55,7 +55,9 @@
    "cell_type": "code",
    "execution_count": 1,
    "id": "ff40dcdd",
-   "metadata": {},
+   "metadata": {
+    "scrolled": true
+   },
    "outputs": [
     {
      "name": "stdout",
@@ -63,36 +65,7 @@
      "text": [
       "Unix Time Jun  1st 2014: 1401573600\n",
       "Unix Time Nov 30th 2014: 1417388399\n",
-      "We should keep all the jobs submitted between 44578794 and 60393593\n"
-     ]
-    }
-   ],
-   "source": [
-    "from time import *\n",
-    "begin_trace = 1356994806 # according to original SWF header\n",
-    "jun1_unix_time, nov30_unix_time = mktime(strptime('Sun Jun  1 00:00:00 2014')), mktime(strptime('Sun Nov 30 23:59:59 2014'))\n",
-    "jun1, nov30 = (int) (jun1_unix_time - begin_trace), (int) (nov30_unix_time - begin_trace)\n",
-    "print(\"Unix Time Jun  1st 2014: {:.0f}\".format( jun1_unix_time ))\n",
-    "print(\"Unix Time Nov 30th 2014: {:.0f}\".format( nov30_unix_time ))\n",
-    "print(\"We should keep all the jobs submitted between {:d} and {:d}\".format(jun1, nov30))\n",
-    "\n",
-    "# Create a swf with only the selected clusters and the 6 selected months \n",
-    "! scripts/swf_moulinette.py workload/METACENTRUM-2013-3.swf \\\n",
-    "    -o workload/METACENTRUM_6months.swf \\\n",
-    "    --keep_only=\"submit_time >= {jun1} and submit_time <= {nov30}\" \\\n",
-    "    --partitions_to_select 1 2 3 5 7 8 9 10 11 12 14 15 18 19 20 21 22 23 25 26 31"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 9,
-   "id": "6ec15ee8",
-   "metadata": {},
-   "outputs": [
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
+      "We should keep all the jobs submitted between 44578794 and 60393593\n",
       "Processing swf line 100000\n",
       "Processing swf line 200000\n",
       "Processing swf line 300000\n",
@@ -152,16 +125,71 @@
       "Processing swf line 5700000\n",
       "-------------------\n",
       "End parsing\n",
-      "Total 4836170 jobs and 838 users have been created.\n",
-      "Total number of core-hours: 16805967\n",
-      "887919 valid jobs were not selected (keep_only) for 77201656 core-hour\n",
-      "Jobs not selected: 15.5% in number, 82.1% in core-hour\n",
+      "Total 1649029 jobs and 556 users have been created.\n",
+      "Total number of core-hours: 18222722\n",
+      "4075060 valid jobs were not selected (keep_only) for 75784902 core-hour\n",
+      "Jobs not selected: 71.2% in number, 80.6% in core-hour\n",
       "7119 out of 5731209 lines in the file did not match the swf format\n",
       "30 jobs were not valid\n"
      ]
     }
    ],
    "source": [
+    "# First selection\n",
+    "# Create a swf with only the selected clusters and the 6 selected months \n",
+    "from time import *\n",
+    "begin_trace = 1356994806 # according to original SWF header\n",
+    "jun1_unix_time, nov30_unix_time = mktime(strptime('Sun Jun  1 00:00:00 2014')), mktime(strptime('Sun Nov 30 23:59:59 2014'))\n",
+    "jun1, nov30 = (int) (jun1_unix_time - begin_trace), (int) (nov30_unix_time - begin_trace)\n",
+    "print(\"Unix Time Jun  1st 2014: {:.0f}\".format( jun1_unix_time ))\n",
+    "print(\"Unix Time Nov 30th 2014: {:.0f}\".format( nov30_unix_time ))\n",
+    "print(\"We should keep all the jobs submitted between {:d} and {:d}\".format(jun1, nov30))\n",
+    "\n",
+    "! ./scripts/swf_moulinette.py workload/METACENTRUM-2013-3.swf \\\n",
+    "    -o workload/METACENTRUM_6months.swf \\\n",
+    "    --keep_only=\"submit_time >= {jun1} and submit_time <= {nov30}\" \\\n",
+    "    --partitions_to_select 1 2 3 5 7 8 9 10 11 12 14 15 18 19 20 21 22 23 25 26 31"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "id": "6ec15ee8",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Processing swf line 100000\n",
+      "Processing swf line 200000\n",
+      "Processing swf line 300000\n",
+      "Processing swf line 400000\n",
+      "Processing swf line 500000\n",
+      "Processing swf line 600000\n",
+      "Processing swf line 700000\n",
+      "Processing swf line 800000\n",
+      "Processing swf line 900000\n",
+      "Processing swf line 1000000\n",
+      "Processing swf line 1100000\n",
+      "Processing swf line 1200000\n",
+      "Processing swf line 1300000\n",
+      "Processing swf line 1400000\n",
+      "Processing swf line 1500000\n",
+      "Processing swf line 1600000\n",
+      "-------------------\n",
+      "End parsing\n",
+      "Total 1604201 jobs and 546 users have been created.\n",
+      "Total number of core-hours: 4785357\n",
+      "44828 valid jobs were not selected (keep_only) for 13437365 core-hour\n",
+      "Jobs not selected: 2.7% in number, 73.7% in core-hour\n",
+      "0 out of 1649030 lines in the file did not match the swf format\n",
+      "1 jobs were not valid\n"
+     ]
+    }
+   ],
+   "source": [
+    "# Second selection\n",
     "# Keep only the selected jobs\n",
     "! ./scripts/swf_moulinette.py workload/METACENTRUM_6months.swf \\\n",
     "    -o workload/MC_selection_article.swf \\\n",
@@ -174,21 +202,29 @@
    "metadata": {},
    "source": [
     "## Platform\n",
-    "According to the system specifications given in the [corresponding page in Parallel Workload Archive](https://www.cs.huji.ac.il/labs/parallel/workload/l_metacentrum2/index.html), if we exclude nodes with >16 cores, there are $\\#cores_{total} = 6416$ cores on May 1st 2014.(1)\n",
+    "According to the system specifications given in the [corresponding page in Parallel Workload Archive](https://www.cs.huji.ac.il/labs/parallel/workload/l_metacentrum2/index.html): from June 1st 2014 to Nov 30th 2014 there is no change in the platform for the clusters considered in our study (<16 cores). There is a total of **6304 cores**.(1)\n",
     "\n",
-    "We build a platform file adapted to the remaining workload. We choose to make it homogeneous with 16-core nodes. To have a coherent number of nodes, we count:\n",
+    "We build a platform file adapted to the remaining workload. We see above that the second selection cut 73.7\\% of core-hours from the original workload. We choose to make an homogeneous with 16-core nodes. To have a coherent number of nodes, we count:\n",
     "\n",
-    "$\\#nodes = \\frac{\\#cores_{total} * \\%kept_{core.hour}}{\\#corePerNode} = 6416 * .294 / 16 = 118$\n",
+    "$\\#nodes = \\frac{\\#cores_{total} * \\%kept_{core.hour}}{\\#corePerNode} = 6304 * .263 / 16 = 104$\n",
     "\n",
     "In SimGrid platform language, this corresponds to such a cluster:\n",
     "```xml\n",
-    "<cluster id=\"cluster_MC\" prefix=\"MC_\" suffix=\"\" radical=\"0-117\" core=\"16\">\n",
+    "<cluster id=\"cluster_MC\" prefix=\"MC_\" suffix=\"\" radical=\"0-103\" core=\"16\">\n",
     "```\n",
     "\n",
     "The corresponding SimGrid platform file can be found in `platform/average_metacentrum.xml`.\n",
     "\n",
-    "(1) clusters decomissionned before or comissionned after May 1st 2014 have also been removed: $8+480+160+1792+256+576+88+416+108+168+752+112+588+152+160+160+192+24+224 = 6416$"
+    "(1) clusters decomissionned before or comissionned after the 6-month period have been removed: $8+480+160+1792+256+576+88+416+108+168+752+112+588+48+152+160+192+24+224 = 6304$"
    ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "north-meeting",
+   "metadata": {},
+   "outputs": [],
+   "source": []
   }
  ],
  "metadata": {
@@ -207,7 +243,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.8.10"
+   "version": "3.8.9"
   }
  },
  "nbformat": 4,

platform/average_metacentrum.xml

@@ -4,7 +4,7 @@
 
     <zone id="AS0" routing="Full">
 
-        <cluster id="cluster_MC" prefix="MC_" suffix="" radical="0-117"
+        <cluster id="cluster_MC" prefix="MC_" suffix="" radical="0-103"
         core="16" bw="10Gbps" lat="0.0"
         speed="11.77Gf, 1e-9Mf, 0.166666666666667f, 0.006666666666667f">
             <prop id="wattage_per_state" value="100:100:217, 9.75:9.75:9.75, 100:100:100, 125:125:125" />