Cleans the jupyter file

1cbb23b1 · Georges Da Costa · a0a36145 · 1cbb23b1
Commit 1cbb23b1 authored 3 years ago by Georges Da Costa
--- a/0_prepare_workload.ipynb
+++ b/0_prepare_workload.ipynb
@@ -19,242 +19,9 @@
   "source": [
    "# Download the workload (548.3 MB unzipped)\n",
    "!wget https://www.cs.huji.ac.il/labs/parallel/workload/l_metacentrum2/METACENTRUM-2013-3.swf.gz \\\n",
-    "    --output-file workload/METACENTRUM-2013-3.swf.gz "
+    "      --no-check-certificate -nc -P workload workload/METACENTRUM-2013-3.swf.gz"
   ]
  },
-  {
-   "cell_type": "code",
-   "execution_count": 5,
-   "id": "a8982775",
-   "metadata": {},
-   "outputs": [
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "GNU Wget 1.21.2, a non-interactive network retriever.\r\n",
-      "Usage: wget [OPTION]... [URL]...\r\n",
-      "\r\n",
-      "Mandatory arguments to long options are mandatory for short options too.\r\n",
-      "\r\n",
-      "Startup:\r\n",
-      "  -V,  --version                   display the version of Wget and exit\r\n",
-      "  -h,  --help                      print this help\r\n",
-      "  -b,  --background                go to background after startup\r\n",
-      "  -e,  --execute=COMMAND           execute a `.wgetrc'-style command\r\n",
-      "\r\n",
-      "Logging and input file:\r\n",
-      "  -o,  --output-file=FILE          log messages to FILE\r\n",
-      "  -a,  --append-output=FILE        append messages to FILE\r\n",
-      "  -d,  --debug                     print lots of debugging information\r\n",
-      "  -q,  --quiet                     quiet (no output)\r\n",
-      "  -v,  --verbose                   be verbose (this is the default)\r\n",
-      "  -nv, --no-verbose                turn off verboseness, without being quiet\r\n",
-      "       --report-speed=TYPE         output bandwidth as TYPE.  TYPE can be bits\r\n",
-      "  -i,  --input-file=FILE           download URLs found in local or external FILE\r\n",
-      "  -F,  --force-html                treat input file as HTML\r\n",
-      "  -B,  --base=URL                  resolves HTML input-file links (-i -F)\r\n",
-      "                                     relative to URL\r\n",
-      "       --config=FILE               specify config file to use\r\n",
-      "       --no-config                 do not read any config file\r\n",
-      "       --rejected-log=FILE         log reasons for URL rejection to FILE\r\n",
-      "\r\n",
-      "Download:\r\n",
-      "  -t,  --tries=NUMBER              set number of retries to NUMBER (0 unlimits)\r\n",
-      "       --retry-connrefused         retry even if connection is refused\r\n",
-      "       --retry-on-http-error=ERRORS    comma-separated list of HTTP errors to retry\r\n",
-      "  -O,  --output-document=FILE      write documents to FILE\r\n",
-      "  -nc, --no-clobber                skip downloads that would download to\r\n",
-      "                                     existing files (overwriting them)\r\n",
-      "       --no-netrc                  don't try to obtain credentials from .netrc\r\n",
-      "  -c,  --continue                  resume getting a partially-downloaded file\r\n",
-      "       --start-pos=OFFSET          start downloading from zero-based position OFFSET\r\n",
-      "       --progress=TYPE             select progress gauge type\r\n",
-      "       --show-progress             display the progress bar in any verbosity mode\r\n",
-      "  -N,  --timestamping              don't re-retrieve files unless newer than\r\n",
-      "                                     local\r\n",
-      "       --no-if-modified-since      don't use conditional if-modified-since get\r\n",
-      "                                     requests in timestamping mode\r\n",
-      "       --no-use-server-timestamps  don't set the local file's timestamp by\r\n",
-      "                                     the one on the server\r\n",
-      "  -S,  --server-response           print server response\r\n",
-      "       --spider                    don't download anything\r\n",
-      "  -T,  --timeout=SECONDS           set all timeout values to SECONDS\r\n",
-      "       --dns-timeout=SECS          set the DNS lookup timeout to SECS\r\n",
-      "       --connect-timeout=SECS      set the connect timeout to SECS\r\n",
-      "       --read-timeout=SECS         set the read timeout to SECS\r\n",
-      "  -w,  --wait=SECONDS              wait SECONDS between retrievals\r\n",
-      "                                     (applies if more then 1 URL is to be retrieved)\r\n",
-      "       --waitretry=SECONDS         wait 1..SECONDS between retries of a retrieval\r\n",
-      "                                     (applies if more then 1 URL is to be retrieved)\r\n",
-      "       --random-wait               wait from 0.5*WAIT...1.5*WAIT secs between retrievals\r\n",
-      "                                     (applies if more then 1 URL is to be retrieved)\r\n",
-      "       --no-proxy                  explicitly turn off proxy\r\n",
-      "  -Q,  --quota=NUMBER              set retrieval quota to NUMBER\r\n",
-      "       --bind-address=ADDRESS      bind to ADDRESS (hostname or IP) on local host\r\n",
-      "       --limit-rate=RATE           limit download rate to RATE\r\n",
-      "       --no-dns-cache              disable caching DNS lookups\r\n",
-      "       --restrict-file-names=OS    restrict chars in file names to ones OS allows\r\n",
-      "       --ignore-case               ignore case when matching files/directories\r\n",
-      "  -4,  --inet4-only                connect only to IPv4 addresses\r\n",
-      "  -6,  --inet6-only                connect only to IPv6 addresses\r\n",
-      "       --prefer-family=FAMILY      connect first to addresses of specified family,\r\n",
-      "                                     one of IPv6, IPv4, or none\r\n",
-      "       --user=USER                 set both ftp and http user to USER\r\n",
-      "       --password=PASS             set both ftp and http password to PASS\r\n",
-      "       --ask-password              prompt for passwords\r\n",
-      "       --use-askpass=COMMAND       specify credential handler for requesting \r\n",
-      "                                     username and password.  If no COMMAND is \r\n",
-      "                                     specified the WGET_ASKPASS or the SSH_ASKPASS \r\n",
-      "                                     environment variable is used.\r\n",
-      "       --no-iri                    turn off IRI support\r\n",
-      "       --local-encoding=ENC        use ENC as the local encoding for IRIs\r\n",
-      "       --remote-encoding=ENC       use ENC as the default remote encoding\r\n",
-      "       --unlink                    remove file before clobber\r\n",
-      "       --xattr                     turn on storage of metadata in extended file attributes\r\n",
-      "\r\n",
-      "Directories:\r\n",
-      "  -nd, --no-directories            don't create directories\r\n",
-      "  -x,  --force-directories         force creation of directories\r\n",
-      "  -nH, --no-host-directories       don't create host directories\r\n",
-      "       --protocol-directories      use protocol name in directories\r\n",
-      "  -P,  --directory-prefix=PREFIX   save files to PREFIX/..\r\n",
-      "       --cut-dirs=NUMBER           ignore NUMBER remote directory components\r\n",
-      "\r\n",
-      "HTTP options:\r\n",
-      "       --http-user=USER            set http user to USER\r\n",
-      "       --http-password=PASS        set http password to PASS\r\n",
-      "       --no-cache                  disallow server-cached data\r\n",
-      "       --default-page=NAME         change the default page name (normally\r\n",
-      "                                     this is 'index.html'.)\r\n",
-      "  -E,  --adjust-extension          save HTML/CSS documents with proper extensions\r\n",
-      "       --ignore-length             ignore 'Content-Length' header field\r\n",
-      "       --header=STRING             insert STRING among the headers\r\n",
-      "       --compression=TYPE          choose compression, one of auto, gzip and none. (default: none)\r\n",
-      "       --max-redirect              maximum redirections allowed per page\r\n",
-      "       --proxy-user=USER           set USER as proxy username\r\n",
-      "       --proxy-password=PASS       set PASS as proxy password\r\n",
-      "       --referer=URL               include 'Referer: URL' header in HTTP request\r\n",
-      "       --save-headers              save the HTTP headers to file\r\n",
-      "  -U,  --user-agent=AGENT          identify as AGENT instead of Wget/VERSION\r\n",
-      "       --no-http-keep-alive        disable HTTP keep-alive (persistent connections)\r\n",
-      "       --no-cookies                don't use cookies\r\n",
-      "       --load-cookies=FILE         load cookies from FILE before session\r\n",
-      "       --save-cookies=FILE         save cookies to FILE after session\r\n",
-      "       --keep-session-cookies      load and save session (non-permanent) cookies\r\n",
-      "       --post-data=STRING          use the POST method; send STRING as the data\r\n",
-      "       --post-file=FILE            use the POST method; send contents of FILE\r\n",
-      "       --method=HTTPMethod         use method \"HTTPMethod\" in the request\r\n",
-      "       --body-data=STRING          send STRING as data. --method MUST be set\r\n",
-      "       --body-file=FILE            send contents of FILE. --method MUST be set\r\n",
-      "       --content-disposition       honor the Content-Disposition header when\r\n",
-      "                                     choosing local file names (EXPERIMENTAL)\r\n",
-      "       --content-on-error          output the received content on server errors\r\n",
-      "       --auth-no-challenge         send Basic HTTP authentication information\r\n",
-      "                                     without first waiting for the server's\r\n",
-      "                                     challenge\r\n",
-      "\r\n",
-      "HTTPS (SSL/TLS) options:\r\n",
-      "       --secure-protocol=PR        choose secure protocol, one of auto, SSLv2,\r\n",
-      "                                     SSLv3, TLSv1, TLSv1_1, TLSv1_2 and PFS\r\n",
-      "       --https-only                only follow secure HTTPS links\r\n",
-      "       --no-check-certificate      don't validate the server's certificate\r\n",
-      "       --certificate=FILE          client certificate file\r\n",
-      "       --certificate-type=TYPE     client certificate type, PEM or DER\r\n",
-      "       --private-key=FILE          private key file\r\n",
-      "       --private-key-type=TYPE     private key type, PEM or DER\r\n",
-      "       --ca-certificate=FILE       file with the bundle of CAs\r\n",
-      "       --ca-directory=DIR          directory where hash list of CAs is stored\r\n",
-      "       --crl-file=FILE             file with bundle of CRLs\r\n",
-      "       --pinnedpubkey=FILE/HASHES  Public key (PEM/DER) file, or any number\r\n",
-      "                                   of base64 encoded sha256 hashes preceded by\r\n",
-      "                                   'sha256//' and separated by ';', to verify\r\n",
-      "                                   peer against\r\n",
-      "       --random-file=FILE          file with random data for seeding the SSL PRNG\r\n",
-      "\r\n",
-      "       --ciphers=STR           Set the priority string (GnuTLS) or cipher list string (OpenSSL) directly.\r\n",
-      "                                   Use with care. This option overrides --secure-protocol.\r\n",
-      "                                   The format and syntax of this string depend on the specific SSL/TLS engine.\r\n",
-      "HSTS options:\r\n",
-      "       --no-hsts                   disable HSTS\r\n",
-      "       --hsts-file                 path of HSTS database (will override default)\r\n",
-      "\r\n",
-      "FTP options:\r\n",
-      "       --ftp-user=USER             set ftp user to USER\r\n",
-      "       --ftp-password=PASS         set ftp password to PASS\r\n",
-      "       --no-remove-listing         don't remove '.listing' files\r\n",
-      "       --no-glob                   turn off FTP file name globbing\r\n",
-      "       --no-passive-ftp            disable the \"passive\" transfer mode\r\n",
-      "       --preserve-permissions      preserve remote file permissions\r\n",
-      "       --retr-symlinks             when recursing, get linked-to files (not dir)\r\n",
-      "\r\n",
-      "FTPS options:\r\n",
-      "       --ftps-implicit                 use implicit FTPS (default port is 990)\r\n",
-      "       --ftps-resume-ssl               resume the SSL/TLS session started in the control connection when\r\n",
-      "                                         opening a data connection\r\n",
-      "       --ftps-clear-data-connection    cipher the control channel only; all the data will be in plaintext\r\n",
-      "       --ftps-fallback-to-ftp          fall back to FTP if FTPS is not supported in the target server\r\n",
-      "WARC options:\r\n",
-      "       --warc-file=FILENAME        save request/response data to a .warc.gz file\r\n",
-      "       --warc-header=STRING        insert STRING into the warcinfo record\r\n",
-      "       --warc-max-size=NUMBER      set maximum size of WARC files to NUMBER\r\n",
-      "       --warc-cdx                  write CDX index files\r\n",
-      "       --warc-dedup=FILENAME       do not store records listed in this CDX file\r\n",
-      "       --no-warc-compression       do not compress WARC files with GZIP\r\n",
-      "       --no-warc-digests           do not calculate SHA1 digests\r\n",
-      "       --no-warc-keep-log          do not store the log file in a WARC record\r\n",
-      "       --warc-tempdir=DIRECTORY    location for temporary files created by the\r\n",
-      "                                     WARC writer\r\n",
-      "\r\n",
-      "Recursive download:\r\n",
-      "  -r,  --recursive                 specify recursive download\r\n",
-      "  -l,  --level=NUMBER              maximum recursion depth (inf or 0 for infinite)\r\n",
-      "       --delete-after              delete files locally after downloading them\r\n",
-      "  -k,  --convert-links             make links in downloaded HTML or CSS point to\r\n",
-      "                                     local files\r\n",
-      "       --convert-file-only         convert the file part of the URLs only (usually known as the basename)\r\n",
-      "       --backups=N                 before writing file X, rotate up to N backup files\r\n",
-      "  -K,  --backup-converted          before converting file X, back up as X.orig\r\n",
-      "  -m,  --mirror                    shortcut for -N -r -l inf --no-remove-listing\r\n",
-      "  -p,  --page-requisites           get all images, etc. needed to display HTML page\r\n",
-      "       --strict-comments           turn on strict (SGML) handling of HTML comments\r\n",
-      "\r\n",
-      "Recursive accept/reject:\r\n",
-      "  -A,  --accept=LIST               comma-separated list of accepted extensions\r\n",
-      "  -R,  --reject=LIST               comma-separated list of rejected extensions\r\n",
-      "       --accept-regex=REGEX        regex matching accepted URLs\r\n",
-      "       --reject-regex=REGEX        regex matching rejected URLs\r\n",
-      "       --regex-type=TYPE           regex type (posix|pcre)\r\n",
-      "  -D,  --domains=LIST              comma-separated list of accepted domains\r\n",
-      "       --exclude-domains=LIST      comma-separated list of rejected domains\r\n",
-      "       --follow-ftp                follow FTP links from HTML documents\r\n",
-      "       --follow-tags=LIST          comma-separated list of followed HTML tags\r\n",
-      "       --ignore-tags=LIST          comma-separated list of ignored HTML tags\r\n",
-      "  -H,  --span-hosts                go to foreign hosts when recursive\r\n",
-      "  -L,  --relative                  follow relative links only\r\n",
-      "  -I,  --include-directories=LIST  list of allowed directories\r\n",
-      "       --trust-server-names        use the name specified by the redirection\r\n",
-      "                                     URL's last component\r\n",
-      "  -X,  --exclude-directories=LIST  list of excluded directories\r\n",
-      "  -np, --no-parent                 don't ascend to the parent directory\r\n",
-      "\r\n",
-      "Email bug reports, questions, discussions to <bug-wget@gnu.org>\r\n",
-      "and/or open issues at https://savannah.gnu.org/bugs/?func=additem&group=wget.\r\n"
-     ]
-    }
-   ],
-   "source": [
-    "!wget --help"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "id": "e75c1fdf",
-   "metadata": {},
-   "outputs": [],
-   "source": []
-  },
  {
   "cell_type": "code",
   "execution_count": 6,
@@ -274,231 +41,6 @@
    "!gunzip workload/METACENTRUM-2013-3.swf.gz"
   ]
  },
-  {
-   "cell_type": "code",
-   "execution_count": 1,
-   "id": "d4cd4f2c",
-   "metadata": {},
-   "outputs": [
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "GNU Wget 1.21.2, a non-interactive network retriever.\r\n",
-      "Usage: wget [OPTION]... [URL]...\r\n",
-      "\r\n",
-      "Mandatory arguments to long options are mandatory for short options too.\r\n",
-      "\r\n",
-      "Startup:\r\n",
-      "  -V,  --version                   display the version of Wget and exit\r\n",
-      "  -h,  --help                      print this help\r\n",
-      "  -b,  --background                go to background after startup\r\n",
-      "  -e,  --execute=COMMAND           execute a `.wgetrc'-style command\r\n",
-      "\r\n",
-      "Logging and input file:\r\n",
-      "  -o,  --output-file=FILE          log messages to FILE\r\n",
-      "  -a,  --append-output=FILE        append messages to FILE\r\n",
-      "  -d,  --debug                     print lots of debugging information\r\n",
-      "  -q,  --quiet                     quiet (no output)\r\n",
-      "  -v,  --verbose                   be verbose (this is the default)\r\n",
-      "  -nv, --no-verbose                turn off verboseness, without being quiet\r\n",
-      "       --report-speed=TYPE         output bandwidth as TYPE.  TYPE can be bits\r\n",
-      "  -i,  --input-file=FILE           download URLs found in local or external FILE\r\n",
-      "  -F,  --force-html                treat input file as HTML\r\n",
-      "  -B,  --base=URL                  resolves HTML input-file links (-i -F)\r\n",
-      "                                     relative to URL\r\n",
-      "       --config=FILE               specify config file to use\r\n",
-      "       --no-config                 do not read any config file\r\n",
-      "       --rejected-log=FILE         log reasons for URL rejection to FILE\r\n",
-      "\r\n",
-      "Download:\r\n",
-      "  -t,  --tries=NUMBER              set number of retries to NUMBER (0 unlimits)\r\n",
-      "       --retry-connrefused         retry even if connection is refused\r\n",
-      "       --retry-on-http-error=ERRORS    comma-separated list of HTTP errors to retry\r\n",
-      "  -O,  --output-document=FILE      write documents to FILE\r\n",
-      "  -nc, --no-clobber                skip downloads that would download to\r\n",
-      "                                     existing files (overwriting them)\r\n",
-      "       --no-netrc                  don't try to obtain credentials from .netrc\r\n",
-      "  -c,  --continue                  resume getting a partially-downloaded file\r\n",
-      "       --start-pos=OFFSET          start downloading from zero-based position OFFSET\r\n",
-      "       --progress=TYPE             select progress gauge type\r\n",
-      "       --show-progress             display the progress bar in any verbosity mode\r\n",
-      "  -N,  --timestamping              don't re-retrieve files unless newer than\r\n",
-      "                                     local\r\n",
-      "       --no-if-modified-since      don't use conditional if-modified-since get\r\n",
-      "                                     requests in timestamping mode\r\n",
-      "       --no-use-server-timestamps  don't set the local file's timestamp by\r\n",
-      "                                     the one on the server\r\n",
-      "  -S,  --server-response           print server response\r\n",
-      "       --spider                    don't download anything\r\n",
-      "  -T,  --timeout=SECONDS           set all timeout values to SECONDS\r\n",
-      "       --dns-timeout=SECS          set the DNS lookup timeout to SECS\r\n",
-      "       --connect-timeout=SECS      set the connect timeout to SECS\r\n",
-      "       --read-timeout=SECS         set the read timeout to SECS\r\n",
-      "  -w,  --wait=SECONDS              wait SECONDS between retrievals\r\n",
-      "                                     (applies if more then 1 URL is to be retrieved)\r\n",
-      "       --waitretry=SECONDS         wait 1..SECONDS between retries of a retrieval\r\n",
-      "                                     (applies if more then 1 URL is to be retrieved)\r\n",
-      "       --random-wait               wait from 0.5*WAIT...1.5*WAIT secs between retrievals\r\n",
-      "                                     (applies if more then 1 URL is to be retrieved)\r\n",
-      "       --no-proxy                  explicitly turn off proxy\r\n",
-      "  -Q,  --quota=NUMBER              set retrieval quota to NUMBER\r\n",
-      "       --bind-address=ADDRESS      bind to ADDRESS (hostname or IP) on local host\r\n",
-      "       --limit-rate=RATE           limit download rate to RATE\r\n",
-      "       --no-dns-cache              disable caching DNS lookups\r\n",
-      "       --restrict-file-names=OS    restrict chars in file names to ones OS allows\r\n",
-      "       --ignore-case               ignore case when matching files/directories\r\n",
-      "  -4,  --inet4-only                connect only to IPv4 addresses\r\n",
-      "  -6,  --inet6-only                connect only to IPv6 addresses\r\n",
-      "       --prefer-family=FAMILY      connect first to addresses of specified family,\r\n",
-      "                                     one of IPv6, IPv4, or none\r\n",
-      "       --user=USER                 set both ftp and http user to USER\r\n",
-      "       --password=PASS             set both ftp and http password to PASS\r\n",
-      "       --ask-password              prompt for passwords\r\n",
-      "       --use-askpass=COMMAND       specify credential handler for requesting \r\n",
-      "                                     username and password.  If no COMMAND is \r\n",
-      "                                     specified the WGET_ASKPASS or the SSH_ASKPASS \r\n",
-      "                                     environment variable is used.\r\n",
-      "       --no-iri                    turn off IRI support\r\n",
-      "       --local-encoding=ENC        use ENC as the local encoding for IRIs\r\n",
-      "       --remote-encoding=ENC       use ENC as the default remote encoding\r\n",
-      "       --unlink                    remove file before clobber\r\n",
-      "       --xattr                     turn on storage of metadata in extended file attributes\r\n",
-      "\r\n",
-      "Directories:\r\n",
-      "  -nd, --no-directories            don't create directories\r\n",
-      "  -x,  --force-directories         force creation of directories\r\n",
-      "  -nH, --no-host-directories       don't create host directories\r\n",
-      "       --protocol-directories      use protocol name in directories\r\n",
-      "  -P,  --directory-prefix=PREFIX   save files to PREFIX/..\r\n",
-      "       --cut-dirs=NUMBER           ignore NUMBER remote directory components\r\n",
-      "\r\n",
-      "HTTP options:\r\n",
-      "       --http-user=USER            set http user to USER\r\n",
-      "       --http-password=PASS        set http password to PASS\r\n",
-      "       --no-cache                  disallow server-cached data\r\n",
-      "       --default-page=NAME         change the default page name (normally\r\n",
-      "                                     this is 'index.html'.)\r\n",
-      "  -E,  --adjust-extension          save HTML/CSS documents with proper extensions\r\n",
-      "       --ignore-length             ignore 'Content-Length' header field\r\n",
-      "       --header=STRING             insert STRING among the headers\r\n",
-      "       --compression=TYPE          choose compression, one of auto, gzip and none. (default: none)\r\n",
-      "       --max-redirect              maximum redirections allowed per page\r\n",
-      "       --proxy-user=USER           set USER as proxy username\r\n",
-      "       --proxy-password=PASS       set PASS as proxy password\r\n",
-      "       --referer=URL               include 'Referer: URL' header in HTTP request\r\n",
-      "       --save-headers              save the HTTP headers to file\r\n",
-      "  -U,  --user-agent=AGENT          identify as AGENT instead of Wget/VERSION\r\n",
-      "       --no-http-keep-alive        disable HTTP keep-alive (persistent connections)\r\n",
-      "       --no-cookies                don't use cookies\r\n",
-      "       --load-cookies=FILE         load cookies from FILE before session\r\n",
-      "       --save-cookies=FILE         save cookies to FILE after session\r\n",
-      "       --keep-session-cookies      load and save session (non-permanent) cookies\r\n",
-      "       --post-data=STRING          use the POST method; send STRING as the data\r\n",
-      "       --post-file=FILE            use the POST method; send contents of FILE\r\n",
-      "       --method=HTTPMethod         use method \"HTTPMethod\" in the request\r\n",
-      "       --body-data=STRING          send STRING as data. --method MUST be set\r\n",
-      "       --body-file=FILE            send contents of FILE. --method MUST be set\r\n",
-      "       --content-disposition       honor the Content-Disposition header when\r\n",
-      "                                     choosing local file names (EXPERIMENTAL)\r\n",
-      "       --content-on-error          output the received content on server errors\r\n",
-      "       --auth-no-challenge         send Basic HTTP authentication information\r\n",
-      "                                     without first waiting for the server's\r\n",
-      "                                     challenge\r\n",
-      "\r\n",
-      "HTTPS (SSL/TLS) options:\r\n",
-      "       --secure-protocol=PR        choose secure protocol, one of auto, SSLv2,\r\n",
-      "                                     SSLv3, TLSv1, TLSv1_1, TLSv1_2 and PFS\r\n",
-      "       --https-only                only follow secure HTTPS links\r\n",
-      "       --no-check-certificate      don't validate the server's certificate\r\n",
-      "       --certificate=FILE          client certificate file\r\n",
-      "       --certificate-type=TYPE     client certificate type, PEM or DER\r\n",
-      "       --private-key=FILE          private key file\r\n",
-      "       --private-key-type=TYPE     private key type, PEM or DER\r\n",
-      "       --ca-certificate=FILE       file with the bundle of CAs\r\n",
-      "       --ca-directory=DIR          directory where hash list of CAs is stored\r\n",
-      "       --crl-file=FILE             file with bundle of CRLs\r\n",
-      "       --pinnedpubkey=FILE/HASHES  Public key (PEM/DER) file, or any number\r\n",
-      "                                   of base64 encoded sha256 hashes preceded by\r\n",
-      "                                   'sha256//' and separated by ';', to verify\r\n",
-      "                                   peer against\r\n",
-      "       --random-file=FILE          file with random data for seeding the SSL PRNG\r\n",
-      "\r\n",
-      "       --ciphers=STR           Set the priority string (GnuTLS) or cipher list string (OpenSSL) directly.\r\n",
-      "                                   Use with care. This option overrides --secure-protocol.\r\n",
-      "                                   The format and syntax of this string depend on the specific SSL/TLS engine.\r\n",
-      "HSTS options:\r\n",
-      "       --no-hsts                   disable HSTS\r\n",
-      "       --hsts-file                 path of HSTS database (will override default)\r\n",
-      "\r\n",
-      "FTP options:\r\n",
-      "       --ftp-user=USER             set ftp user to USER\r\n",
-      "       --ftp-password=PASS         set ftp password to PASS\r\n",
-      "       --no-remove-listing         don't remove '.listing' files\r\n",
-      "       --no-glob                   turn off FTP file name globbing\r\n",
-      "       --no-passive-ftp            disable the \"passive\" transfer mode\r\n",
-      "       --preserve-permissions      preserve remote file permissions\r\n",
-      "       --retr-symlinks             when recursing, get linked-to files (not dir)\r\n",
-      "\r\n",
-      "FTPS options:\r\n",
-      "       --ftps-implicit                 use implicit FTPS (default port is 990)\r\n",
-      "       --ftps-resume-ssl               resume the SSL/TLS session started in the control connection when\r\n",
-      "                                         opening a data connection\r\n",
-      "       --ftps-clear-data-connection    cipher the control channel only; all the data will be in plaintext\r\n",
-      "       --ftps-fallback-to-ftp          fall back to FTP if FTPS is not supported in the target server\r\n",
-      "WARC options:\r\n",
-      "       --warc-file=FILENAME        save request/response data to a .warc.gz file\r\n",
-      "       --warc-header=STRING        insert STRING into the warcinfo record\r\n",
-      "       --warc-max-size=NUMBER      set maximum size of WARC files to NUMBER\r\n",
-      "       --warc-cdx                  write CDX index files\r\n",
-      "       --warc-dedup=FILENAME       do not store records listed in this CDX file\r\n",
-      "       --no-warc-compression       do not compress WARC files with GZIP\r\n",
-      "       --no-warc-digests           do not calculate SHA1 digests\r\n",
-      "       --no-warc-keep-log          do not store the log file in a WARC record\r\n",
-      "       --warc-tempdir=DIRECTORY    location for temporary files created by the\r\n",
-      "                                     WARC writer\r\n",
-      "\r\n",
-      "Recursive download:\r\n",
-      "  -r,  --recursive                 specify recursive download\r\n",
-      "  -l,  --level=NUMBER              maximum recursion depth (inf or 0 for infinite)\r\n",
-      "       --delete-after              delete files locally after downloading them\r\n",
-      "  -k,  --convert-links             make links in downloaded HTML or CSS point to\r\n",
-      "                                     local files\r\n",
-      "       --convert-file-only         convert the file part of the URLs only (usually known as the basename)\r\n",
-      "       --backups=N                 before writing file X, rotate up to N backup files\r\n",
-      "  -K,  --backup-converted          before converting file X, back up as X.orig\r\n",
-      "  -m,  --mirror                    shortcut for -N -r -l inf --no-remove-listing\r\n",
-      "  -p,  --page-requisites           get all images, etc. needed to display HTML page\r\n",
-      "       --strict-comments           turn on strict (SGML) handling of HTML comments\r\n",
-      "\r\n",
-      "Recursive accept/reject:\r\n",
-      "  -A,  --accept=LIST               comma-separated list of accepted extensions\r\n",
-      "  -R,  --reject=LIST               comma-separated list of rejected extensions\r\n",
-      "       --accept-regex=REGEX        regex matching accepted URLs\r\n",
-      "       --reject-regex=REGEX        regex matching rejected URLs\r\n",
-      "       --regex-type=TYPE           regex type (posix|pcre)\r\n",
-      "  -D,  --domains=LIST              comma-separated list of accepted domains\r\n",
-      "       --exclude-domains=LIST      comma-separated list of rejected domains\r\n",
-      "       --follow-ftp                follow FTP links from HTML documents\r\n",
-      "       --follow-tags=LIST          comma-separated list of followed HTML tags\r\n",
-      "       --ignore-tags=LIST          comma-separated list of ignored HTML tags\r\n",
-      "  -H,  --span-hosts                go to foreign hosts when recursive\r\n",
-      "  -L,  --relative                  follow relative links only\r\n",
-      "  -I,  --include-directories=LIST  list of allowed directories\r\n",
-      "       --trust-server-names        use the name specified by the redirection\r\n",
-      "                                     URL's last component\r\n",
-      "  -X,  --exclude-directories=LIST  list of excluded directories\r\n",
-      "  -np, --no-parent                 don't ascend to the parent directory\r\n",
-      "\r\n",
-      "Email bug reports, questions, discussions to <bug-wget@gnu.org>\r\n",
-      "and/or open issues at https://savannah.gnu.org/bugs/?func=additem&group=wget.\r\n"
-     ]
-    }
-   ],
-   "source": [
-    "!wget --help"
-   ]
-  },
  {
   "cell_type": "markdown",
   "id": "graphic-rabbit",
@@ -687,7 +229,7 @@
 ],
 "metadata": {
  "kernelspec": {
-   "display_name": "Python 3 (ipykernel)",
+   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
@@ -701,7 +243,7 @@
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
-   "version": "3.9.6"
+   "version": "3.10.4"
  }
 },
 "nbformat": 4,

 %% Cell type:markdown id:forced-resolution tags:
 # Downloading and preparing the workload and platform
 ## Workload
 We use the reconverted log `METACENTRUM-2013-3.swf` available on [Parallel Workload Archive](https://www.cs.huji.ac.il/labs/parallel/workload/l_metacentrum2/index.html).
 %% Cell type:code id:f66eb756 tags:
 ``` python
 # Download the workload (548.3 MB unzipped)
 !wget https://www.cs.huji.ac.il/labs/parallel/workload/l_metacentrum2/METACENTRUM-2013-3.swf.gz \
-    --output-file workload/METACENTRUM-2013-3.swf.gz
+      --no-check-certificate -nc -P workload workload/METACENTRUM-2013-3.swf.gz
-```
-%% Cell type:code id:a8982775 tags:
-``` python
-!wget --help
-```
-%% Output
-    GNU Wget 1.21.2, a non-interactive network retriever.
-    Usage: wget [OPTION]... [URL]...
-    Mandatory arguments to long options are mandatory for short options too.
-    Startup:
-      -V,  --version                   display the version of Wget and exit
-      -h,  --help                      print this help
-      -b,  --background                go to background after startup
-      -e,  --execute=COMMAND           execute a `.wgetrc'-style command
-    Logging and input file:
-      -o,  --output-file=FILE          log messages to FILE
-      -a,  --append-output=FILE        append messages to FILE
-      -d,  --debug                     print lots of debugging information
-      -q,  --quiet                     quiet (no output)
-      -v,  --verbose                   be verbose (this is the default)
-      -nv, --no-verbose                turn off verboseness, without being quiet
-           --report-speed=TYPE         output bandwidth as TYPE.  TYPE can be bits
-      -i,  --input-file=FILE           download URLs found in local or external FILE
-      -F,  --force-html                treat input file as HTML
-      -B,  --base=URL                  resolves HTML input-file links (-i -F)
-                                         relative to URL
-           --config=FILE               specify config file to use
-           --no-config                 do not read any config file
-           --rejected-log=FILE         log reasons for URL rejection to FILE
-    Download:
-      -t,  --tries=NUMBER              set number of retries to NUMBER (0 unlimits)
-           --retry-connrefused         retry even if connection is refused
-           --retry-on-http-error=ERRORS    comma-separated list of HTTP errors to retry
-      -O,  --output-document=FILE      write documents to FILE
-      -nc, --no-clobber                skip downloads that would download to
-                                         existing files (overwriting them)
-           --no-netrc                  don't try to obtain credentials from .netrc
-      -c,  --continue                  resume getting a partially-downloaded file
-           --start-pos=OFFSET          start downloading from zero-based position OFFSET
-           --progress=TYPE             select progress gauge type
-           --show-progress             display the progress bar in any verbosity mode
-      -N,  --timestamping              don't re-retrieve files unless newer than
-                                         local
-           --no-if-modified-since      don't use conditional if-modified-since get
-                                         requests in timestamping mode
-           --no-use-server-timestamps  don't set the local file's timestamp by
-                                         the one on the server
-      -S,  --server-response           print server response
-           --spider                    don't download anything
-      -T,  --timeout=SECONDS           set all timeout values to SECONDS
-           --dns-timeout=SECS          set the DNS lookup timeout to SECS
-           --connect-timeout=SECS      set the connect timeout to SECS
-           --read-timeout=SECS         set the read timeout to SECS
-      -w,  --wait=SECONDS              wait SECONDS between retrievals
-                                         (applies if more then 1 URL is to be retrieved)
-           --waitretry=SECONDS         wait 1..SECONDS between retries of a retrieval
-                                         (applies if more then 1 URL is to be retrieved)
-           --random-wait               wait from 0.5*WAIT...1.5*WAIT secs between retrievals
-                                         (applies if more then 1 URL is to be retrieved)
-           --no-proxy                  explicitly turn off proxy
-      -Q,  --quota=NUMBER              set retrieval quota to NUMBER
-           --bind-address=ADDRESS      bind to ADDRESS (hostname or IP) on local host
-           --limit-rate=RATE           limit download rate to RATE
-           --no-dns-cache              disable caching DNS lookups
-           --restrict-file-names=OS    restrict chars in file names to ones OS allows
-           --ignore-case               ignore case when matching files/directories
-      -4,  --inet4-only                connect only to IPv4 addresses
-      -6,  --inet6-only                connect only to IPv6 addresses
-           --prefer-family=FAMILY      connect first to addresses of specified family,
-                                         one of IPv6, IPv4, or none
-           --user=USER                 set both ftp and http user to USER
-           --password=PASS             set both ftp and http password to PASS
-           --ask-password              prompt for passwords
-           --use-askpass=COMMAND       specify credential handler for requesting
-                                         username and password.  If no COMMAND is
-                                         specified the WGET_ASKPASS or the SSH_ASKPASS
-                                         environment variable is used.
-           --no-iri                    turn off IRI support
-           --local-encoding=ENC        use ENC as the local encoding for IRIs
-           --remote-encoding=ENC       use ENC as the default remote encoding
-           --unlink                    remove file before clobber
-           --xattr                     turn on storage of metadata in extended file attributes
-    Directories:
-      -nd, --no-directories            don't create directories
-      -x,  --force-directories         force creation of directories
-      -nH, --no-host-directories       don't create host directories
-           --protocol-directories      use protocol name in directories
-      -P,  --directory-prefix=PREFIX   save files to PREFIX/..
-           --cut-dirs=NUMBER           ignore NUMBER remote directory components
-    HTTP options:
-           --http-user=USER            set http user to USER
-           --http-password=PASS        set http password to PASS
-           --no-cache                  disallow server-cached data
-           --default-page=NAME         change the default page name (normally
-                                         this is 'index.html'.)
-      -E,  --adjust-extension          save HTML/CSS documents with proper extensions
-           --ignore-length             ignore 'Content-Length' header field
-           --header=STRING             insert STRING among the headers
-           --compression=TYPE          choose compression, one of auto, gzip and none. (default: none)
-           --max-redirect              maximum redirections allowed per page
-           --proxy-user=USER           set USER as proxy username
-           --proxy-password=PASS       set PASS as proxy password
-           --referer=URL               include 'Referer: URL' header in HTTP request
-           --save-headers              save the HTTP headers to file
-      -U,  --user-agent=AGENT          identify as AGENT instead of Wget/VERSION
-           --no-http-keep-alive        disable HTTP keep-alive (persistent connections)
-           --no-cookies                don't use cookies
-           --load-cookies=FILE         load cookies from FILE before session
-           --save-cookies=FILE         save cookies to FILE after session
-           --keep-session-cookies      load and save session (non-permanent) cookies
-           --post-data=STRING          use the POST method; send STRING as the data
-           --post-file=FILE            use the POST method; send contents of FILE
-           --method=HTTPMethod         use method "HTTPMethod" in the request
-           --body-data=STRING          send STRING as data. --method MUST be set
-           --body-file=FILE            send contents of FILE. --method MUST be set
-           --content-disposition       honor the Content-Disposition header when
-                                         choosing local file names (EXPERIMENTAL)
-           --content-on-error          output the received content on server errors
-           --auth-no-challenge         send Basic HTTP authentication information
-                                         without first waiting for the server's
-                                         challenge
-    HTTPS (SSL/TLS) options:
-           --secure-protocol=PR        choose secure protocol, one of auto, SSLv2,
-                                         SSLv3, TLSv1, TLSv1_1, TLSv1_2 and PFS
-           --https-only                only follow secure HTTPS links
-           --no-check-certificate      don't validate the server's certificate
-           --certificate=FILE          client certificate file
-           --certificate-type=TYPE     client certificate type, PEM or DER
-           --private-key=FILE          private key file
-           --private-key-type=TYPE     private key type, PEM or DER
-           --ca-certificate=FILE       file with the bundle of CAs
-           --ca-directory=DIR          directory where hash list of CAs is stored
-           --crl-file=FILE             file with bundle of CRLs
-           --pinnedpubkey=FILE/HASHES  Public key (PEM/DER) file, or any number
-                                       of base64 encoded sha256 hashes preceded by
-                                       'sha256//' and separated by ';', to verify
-                                       peer against
-           --random-file=FILE          file with random data for seeding the SSL PRNG
-           --ciphers=STR           Set the priority string (GnuTLS) or cipher list string (OpenSSL) directly.
-                                       Use with care. This option overrides --secure-protocol.
-                                       The format and syntax of this string depend on the specific SSL/TLS engine.
-    HSTS options:
-           --no-hsts                   disable HSTS
-           --hsts-file                 path of HSTS database (will override default)
-    FTP options:
-           --ftp-user=USER             set ftp user to USER
-           --ftp-password=PASS         set ftp password to PASS
-           --no-remove-listing         don't remove '.listing' files
-           --no-glob                   turn off FTP file name globbing
-           --no-passive-ftp            disable the "passive" transfer mode
-           --preserve-permissions      preserve remote file permissions
-           --retr-symlinks             when recursing, get linked-to files (not dir)
-    FTPS options:
-           --ftps-implicit                 use implicit FTPS (default port is 990)
-           --ftps-resume-ssl               resume the SSL/TLS session started in the control connection when
-                                             opening a data connection
-           --ftps-clear-data-connection    cipher the control channel only; all the data will be in plaintext
-           --ftps-fallback-to-ftp          fall back to FTP if FTPS is not supported in the target server
-    WARC options:
-           --warc-file=FILENAME        save request/response data to a .warc.gz file
-           --warc-header=STRING        insert STRING into the warcinfo record
-           --warc-max-size=NUMBER      set maximum size of WARC files to NUMBER
-           --warc-cdx                  write CDX index files
-           --warc-dedup=FILENAME       do not store records listed in this CDX file
-           --no-warc-compression       do not compress WARC files with GZIP
-           --no-warc-digests           do not calculate SHA1 digests
-           --no-warc-keep-log          do not store the log file in a WARC record
-           --warc-tempdir=DIRECTORY    location for temporary files created by the
-                                         WARC writer
-    Recursive download:
-      -r,  --recursive                 specify recursive download
-      -l,  --level=NUMBER              maximum recursion depth (inf or 0 for infinite)
-           --delete-after              delete files locally after downloading them
-      -k,  --convert-links             make links in downloaded HTML or CSS point to
-                                         local files
-           --convert-file-only         convert the file part of the URLs only (usually known as the basename)
-           --backups=N                 before writing file X, rotate up to N backup files
-      -K,  --backup-converted          before converting file X, back up as X.orig
-      -m,  --mirror                    shortcut for -N -r -l inf --no-remove-listing
-      -p,  --page-requisites           get all images, etc. needed to display HTML page
-           --strict-comments           turn on strict (SGML) handling of HTML comments
-    Recursive accept/reject:
-      -A,  --accept=LIST               comma-separated list of accepted extensions
-      -R,  --reject=LIST               comma-separated list of rejected extensions
-           --accept-regex=REGEX        regex matching accepted URLs
-           --reject-regex=REGEX        regex matching rejected URLs
-           --regex-type=TYPE           regex type (posix|pcre)
-      -D,  --domains=LIST              comma-separated list of accepted domains
-           --exclude-domains=LIST      comma-separated list of rejected domains
-           --follow-ftp                follow FTP links from HTML documents
-           --follow-tags=LIST          comma-separated list of followed HTML tags
-           --ignore-tags=LIST          comma-separated list of ignored HTML tags
-      -H,  --span-hosts                go to foreign hosts when recursive
-      -L,  --relative                  follow relative links only
-      -I,  --include-directories=LIST  list of allowed directories
-           --trust-server-names        use the name specified by the redirection
-                                         URL's last component
-      -X,  --exclude-directories=LIST  list of excluded directories
-      -np, --no-parent                 don't ascend to the parent directory
-    Email bug reports, questions, discussions to <bug-wget@gnu.org>
-    and/or open issues at https://savannah.gnu.org/bugs/?func=additem&group=wget.
-%% Cell type:code id:e75c1fdf tags:
-``` python
 ```
 %% Cell type:code id:bound-harvey tags:
 ``` python
 # Unzip the workload
 !gunzip workload/METACENTRUM-2013-3.swf.gz
 ```
 %% Output
    gzip: workload/METACENTRUM-2013-3.swf already exists; do you wish to overwrite (y or n)? ^C
-%% Cell type:code id:d4cd4f2c tags:
-``` python
-!wget --help
-```
-%% Output
-    GNU Wget 1.21.2, a non-interactive network retriever.
-    Usage: wget [OPTION]... [URL]...
-    Mandatory arguments to long options are mandatory for short options too.
-    Startup:
-      -V,  --version                   display the version of Wget and exit
-      -h,  --help                      print this help
-      -b,  --background                go to background after startup
-      -e,  --execute=COMMAND           execute a `.wgetrc'-style command
-    Logging and input file:
-      -o,  --output-file=FILE          log messages to FILE
-      -a,  --append-output=FILE        append messages to FILE
-      -d,  --debug                     print lots of debugging information
-      -q,  --quiet                     quiet (no output)
-      -v,  --verbose                   be verbose (this is the default)
-      -nv, --no-verbose                turn off verboseness, without being quiet
-           --report-speed=TYPE         output bandwidth as TYPE.  TYPE can be bits
-      -i,  --input-file=FILE           download URLs found in local or external FILE
-      -F,  --force-html                treat input file as HTML
-      -B,  --base=URL                  resolves HTML input-file links (-i -F)
-                                         relative to URL
-           --config=FILE               specify config file to use
-           --no-config                 do not read any config file
-           --rejected-log=FILE         log reasons for URL rejection to FILE
-    Download:
-      -t,  --tries=NUMBER              set number of retries to NUMBER (0 unlimits)
-           --retry-connrefused         retry even if connection is refused
-           --retry-on-http-error=ERRORS    comma-separated list of HTTP errors to retry
-      -O,  --output-document=FILE      write documents to FILE
-      -nc, --no-clobber                skip downloads that would download to
-                                         existing files (overwriting them)
-           --no-netrc                  don't try to obtain credentials from .netrc
-      -c,  --continue                  resume getting a partially-downloaded file
-           --start-pos=OFFSET          start downloading from zero-based position OFFSET
-           --progress=TYPE             select progress gauge type
-           --show-progress             display the progress bar in any verbosity mode
-      -N,  --timestamping              don't re-retrieve files unless newer than
-                                         local
-           --no-if-modified-since      don't use conditional if-modified-since get
-                                         requests in timestamping mode
-           --no-use-server-timestamps  don't set the local file's timestamp by
-                                         the one on the server
-      -S,  --server-response           print server response
-           --spider                    don't download anything
-      -T,  --timeout=SECONDS           set all timeout values to SECONDS
-           --dns-timeout=SECS          set the DNS lookup timeout to SECS
-           --connect-timeout=SECS      set the connect timeout to SECS
-           --read-timeout=SECS         set the read timeout to SECS
-      -w,  --wait=SECONDS              wait SECONDS between retrievals
-                                         (applies if more then 1 URL is to be retrieved)
-           --waitretry=SECONDS         wait 1..SECONDS between retries of a retrieval
-                                         (applies if more then 1 URL is to be retrieved)
-           --random-wait               wait from 0.5*WAIT...1.5*WAIT secs between retrievals
-                                         (applies if more then 1 URL is to be retrieved)
-           --no-proxy                  explicitly turn off proxy
-      -Q,  --quota=NUMBER              set retrieval quota to NUMBER
-           --bind-address=ADDRESS      bind to ADDRESS (hostname or IP) on local host
-           --limit-rate=RATE           limit download rate to RATE
-           --no-dns-cache              disable caching DNS lookups
-           --restrict-file-names=OS    restrict chars in file names to ones OS allows
-           --ignore-case               ignore case when matching files/directories
-      -4,  --inet4-only                connect only to IPv4 addresses
-      -6,  --inet6-only                connect only to IPv6 addresses
-           --prefer-family=FAMILY      connect first to addresses of specified family,
-                                         one of IPv6, IPv4, or none
-           --user=USER                 set both ftp and http user to USER
-           --password=PASS             set both ftp and http password to PASS
-           --ask-password              prompt for passwords
-           --use-askpass=COMMAND       specify credential handler for requesting
-                                         username and password.  If no COMMAND is
-                                         specified the WGET_ASKPASS or the SSH_ASKPASS
-                                         environment variable is used.
-           --no-iri                    turn off IRI support
-           --local-encoding=ENC        use ENC as the local encoding for IRIs
-           --remote-encoding=ENC       use ENC as the default remote encoding
-           --unlink                    remove file before clobber
-           --xattr                     turn on storage of metadata in extended file attributes
-    Directories:
-      -nd, --no-directories            don't create directories
-      -x,  --force-directories         force creation of directories
-      -nH, --no-host-directories       don't create host directories
-           --protocol-directories      use protocol name in directories
-      -P,  --directory-prefix=PREFIX   save files to PREFIX/..
-           --cut-dirs=NUMBER           ignore NUMBER remote directory components
-    HTTP options:
-           --http-user=USER            set http user to USER
-           --http-password=PASS        set http password to PASS
-           --no-cache                  disallow server-cached data
-           --default-page=NAME         change the default page name (normally
-                                         this is 'index.html'.)
-      -E,  --adjust-extension          save HTML/CSS documents with proper extensions
-           --ignore-length             ignore 'Content-Length' header field
-           --header=STRING             insert STRING among the headers
-           --compression=TYPE          choose compression, one of auto, gzip and none. (default: none)
-           --max-redirect              maximum redirections allowed per page
-           --proxy-user=USER           set USER as proxy username
-           --proxy-password=PASS       set PASS as proxy password
-           --referer=URL               include 'Referer: URL' header in HTTP request
-           --save-headers              save the HTTP headers to file
-      -U,  --user-agent=AGENT          identify as AGENT instead of Wget/VERSION
-           --no-http-keep-alive        disable HTTP keep-alive (persistent connections)
-           --no-cookies                don't use cookies
-           --load-cookies=FILE         load cookies from FILE before session
-           --save-cookies=FILE         save cookies to FILE after session
-           --keep-session-cookies      load and save session (non-permanent) cookies
-           --post-data=STRING          use the POST method; send STRING as the data
-           --post-file=FILE            use the POST method; send contents of FILE
-           --method=HTTPMethod         use method "HTTPMethod" in the request
-           --body-data=STRING          send STRING as data. --method MUST be set
-           --body-file=FILE            send contents of FILE. --method MUST be set
-           --content-disposition       honor the Content-Disposition header when
-                                         choosing local file names (EXPERIMENTAL)
-           --content-on-error          output the received content on server errors
-           --auth-no-challenge         send Basic HTTP authentication information
-                                         without first waiting for the server's
-                                         challenge
-    HTTPS (SSL/TLS) options:
-           --secure-protocol=PR        choose secure protocol, one of auto, SSLv2,
-                                         SSLv3, TLSv1, TLSv1_1, TLSv1_2 and PFS
-           --https-only                only follow secure HTTPS links
-           --no-check-certificate      don't validate the server's certificate
-           --certificate=FILE          client certificate file
-           --certificate-type=TYPE     client certificate type, PEM or DER
-           --private-key=FILE          private key file
-           --private-key-type=TYPE     private key type, PEM or DER
-           --ca-certificate=FILE       file with the bundle of CAs
-           --ca-directory=DIR          directory where hash list of CAs is stored
-           --crl-file=FILE             file with bundle of CRLs
-           --pinnedpubkey=FILE/HASHES  Public key (PEM/DER) file, or any number
-                                       of base64 encoded sha256 hashes preceded by
-                                       'sha256//' and separated by ';', to verify
-                                       peer against
-           --random-file=FILE          file with random data for seeding the SSL PRNG
-           --ciphers=STR           Set the priority string (GnuTLS) or cipher list string (OpenSSL) directly.
-                                       Use with care. This option overrides --secure-protocol.
-                                       The format and syntax of this string depend on the specific SSL/TLS engine.
-    HSTS options:
-           --no-hsts                   disable HSTS
-           --hsts-file                 path of HSTS database (will override default)
-    FTP options:
-           --ftp-user=USER             set ftp user to USER
-           --ftp-password=PASS         set ftp password to PASS
-           --no-remove-listing         don't remove '.listing' files
-           --no-glob                   turn off FTP file name globbing
-           --no-passive-ftp            disable the "passive" transfer mode
-           --preserve-permissions      preserve remote file permissions
-           --retr-symlinks             when recursing, get linked-to files (not dir)
-    FTPS options:
-           --ftps-implicit                 use implicit FTPS (default port is 990)
-           --ftps-resume-ssl               resume the SSL/TLS session started in the control connection when
-                                             opening a data connection
-           --ftps-clear-data-connection    cipher the control channel only; all the data will be in plaintext
-           --ftps-fallback-to-ftp          fall back to FTP if FTPS is not supported in the target server
-    WARC options:
-           --warc-file=FILENAME        save request/response data to a .warc.gz file
-           --warc-header=STRING        insert STRING into the warcinfo record
-           --warc-max-size=NUMBER      set maximum size of WARC files to NUMBER
-           --warc-cdx                  write CDX index files
-           --warc-dedup=FILENAME       do not store records listed in this CDX file
-           --no-warc-compression       do not compress WARC files with GZIP
-           --no-warc-digests           do not calculate SHA1 digests
-           --no-warc-keep-log          do not store the log file in a WARC record
-           --warc-tempdir=DIRECTORY    location for temporary files created by the
-                                         WARC writer
-    Recursive download:
-      -r,  --recursive                 specify recursive download
-      -l,  --level=NUMBER              maximum recursion depth (inf or 0 for infinite)
-           --delete-after              delete files locally after downloading them
-      -k,  --convert-links             make links in downloaded HTML or CSS point to
-                                         local files
-           --convert-file-only         convert the file part of the URLs only (usually known as the basename)
-           --backups=N                 before writing file X, rotate up to N backup files
-      -K,  --backup-converted          before converting file X, back up as X.orig
-      -m,  --mirror                    shortcut for -N -r -l inf --no-remove-listing
-      -p,  --page-requisites           get all images, etc. needed to display HTML page
-           --strict-comments           turn on strict (SGML) handling of HTML comments
-    Recursive accept/reject:
-      -A,  --accept=LIST               comma-separated list of accepted extensions
-      -R,  --reject=LIST               comma-separated list of rejected extensions
-           --accept-regex=REGEX        regex matching accepted URLs
-           --reject-regex=REGEX        regex matching rejected URLs
-           --regex-type=TYPE           regex type (posix|pcre)
-      -D,  --domains=LIST              comma-separated list of accepted domains
-           --exclude-domains=LIST      comma-separated list of rejected domains
-           --follow-ftp                follow FTP links from HTML documents
-           --follow-tags=LIST          comma-separated list of followed HTML tags
-           --ignore-tags=LIST          comma-separated list of ignored HTML tags
-      -H,  --span-hosts                go to foreign hosts when recursive
-      -L,  --relative                  follow relative links only
-      -I,  --include-directories=LIST  list of allowed directories
-           --trust-server-names        use the name specified by the redirection
-                                         URL's last component
-      -X,  --exclude-directories=LIST  list of excluded directories
-      -np, --no-parent                 don't ascend to the parent directory
-    Email bug reports, questions, discussions to <bug-wget@gnu.org>
-    and/or open issues at https://savannah.gnu.org/bugs/?func=additem&group=wget.
 %% Cell type:markdown id:graphic-rabbit tags:
 It is a 2-year-long trace from MetaCentrum, the national grid of the Czech republic. As mentionned in the [original paper releasing the log](https://www.cs.huji.ac.il/~feit/parsched/jsspp15/p5-klusacek.pdf), the platform is **very heterogeneous** and underwent majors changes during the logging period. For the purpose of our study, we perform the following selection.
 First:
 - we remove from the workload all the clusters whose nodes have **more than 16 cores**
 - we truncate the workload to keep only 6 month (June to November 2014) where no major change was performed in the infrastructure (no cluster < 16 cores added nor removed, no reconfiguration in the scheduling system)
 Second:
 - we remove from the workload the jobs with an **execution time greater than one day**
 - we remove from the workload the jobs with a **number of requested cores greater than 16**
 To do so, we use a the home-made SWF parser `swf_moulinette.py`:
 %% Cell type:code id:ff40dcdd tags:
 ``` python
 # First selection
 # Create a swf with only the selected clusters and the 6 selected months
 from time import *
 begin_trace = 1356994806 # according to original SWF header
 jun1_unix_time, nov30_unix_time = mktime(strptime('Sun Jun  1 00:00:00 2014')), mktime(strptime('Sun Nov 30 23:59:59 2014'))
 jun1, nov30 = (int) (jun1_unix_time - begin_trace), (int) (nov30_unix_time - begin_trace)
 print("Unix Time Jun  1st 2014: {:.0f}".format( jun1_unix_time ))
 print("Unix Time Nov 30th 2014: {:.0f}".format( nov30_unix_time ))
 print("We should keep all the jobs submitted between {:d} and {:d}".format(jun1, nov30))
 ! ./scripts/swf_moulinette.py workload/METACENTRUM-2013-3.swf \
    -o workload/METACENTRUM_6months.swf \
    --keep_only="submit_time >= {jun1} and submit_time <= {nov30}" \
    --partitions_to_select 1 2 3 5 7 8 9 10 11 12 14 15 18 19 20 21 22 23 25 26 31
 ```
 %% Output
    Unix Time Jun  1st 2014: 1401573600
    Unix Time Nov 30th 2014: 1417388399
    We should keep all the jobs submitted between 44578794 and 60393593
    Processing swf line 100000
    Processing swf line 200000
    Processing swf line 300000
    Processing swf line 400000
    Processing swf line 500000
    Processing swf line 600000
    Processing swf line 700000
    Processing swf line 800000
    Processing swf line 900000
    Processing swf line 1000000
    Processing swf line 1100000
    Processing swf line 1200000
    Processing swf line 1300000
    Processing swf line 1400000
    Processing swf line 1500000
    Processing swf line 1600000
    Processing swf line 1700000
    Processing swf line 1800000
    Processing swf line 1900000
    Processing swf line 2000000
    Processing swf line 2100000
    Processing swf line 2200000
    Processing swf line 2300000
    Processing swf line 2400000
    Processing swf line 2500000
    Processing swf line 2600000
    Processing swf line 2700000
    Processing swf line 2800000
    Processing swf line 2900000
    Processing swf line 3000000
    Processing swf line 3100000
    Processing swf line 3200000
    Processing swf line 3300000
    Processing swf line 3400000
    Processing swf line 3500000
    Processing swf line 3600000
    Processing swf line 3700000
    Processing swf line 3800000
    Processing swf line 3900000
    Processing swf line 4000000
    Processing swf line 4100000
    Processing swf line 4200000
    Processing swf line 4300000
    Processing swf line 4400000
    Processing swf line 4500000
    Processing swf line 4600000
    Processing swf line 4700000
    Processing swf line 4800000
    Processing swf line 4900000
    Processing swf line 5000000
    Processing swf line 5100000
    Processing swf line 5200000
    Processing swf line 5300000
    Processing swf line 5400000
    Processing swf line 5500000
    Processing swf line 5600000
    Processing swf line 5700000
    -------------------
    End parsing
    Total 1649029 jobs and 556 users have been created.
    Total number of core-hours: 18222722
    4075060 valid jobs were not selected (keep_only) for 75784902 core-hour
    Jobs not selected: 71.2% in number, 80.6% in core-hour
    7119 out of 5731209 lines in the file did not match the swf format
    30 jobs were not valid
 %% Cell type:code id:6ec15ee8 tags:
 ``` python
 # Second selection
 # Keep only the selected jobs
 ! ./scripts/swf_moulinette.py workload/METACENTRUM_6months.swf \
    -o workload/MC_selection_article.swf \
    --keep_only="nb_res <= 16 and run_time <= 24*3600"
 ```
 %% Output
    Processing swf line 100000
    Processing swf line 200000
    Processing swf line 300000
    Processing swf line 400000
    Processing swf line 500000
    Processing swf line 600000
    Processing swf line 700000
    Processing swf line 800000
    Processing swf line 900000
    Processing swf line 1000000
    Processing swf line 1100000
    Processing swf line 1200000
    Processing swf line 1300000
    Processing swf line 1400000
    Processing swf line 1500000
    Processing swf line 1600000
    -------------------
    End parsing
    Total 1604201 jobs and 546 users have been created.
    Total number of core-hours: 4785357
    44828 valid jobs were not selected (keep_only) for 13437365 core-hour
    Jobs not selected: 2.7% in number, 73.7% in core-hour
    0 out of 1649030 lines in the file did not match the swf format
    1 jobs were not valid
 %% Cell type:markdown id:afde35e8 tags:
 ## Platform
 According to the system specifications given in the [corresponding page in Parallel Workload Archive](https://www.cs.huji.ac.il/labs/parallel/workload/l_metacentrum2/index.html): from June 1st 2014 to Nov 30th 2014 there is no change in the platform for the clusters considered in our study (<16 cores). There is a total of **6304 cores**.(1)
 We build a platform file adapted to the remaining workload. We see above that the second selection cuts 73.7\% of core-hours from the original workload. We choose to make an homogeneous cluster with 16-core nodes. To have a coherent number of nodes, we count:
 $\#nodes = \frac{\#cores_{total} * \%kept_{core.hour}}{\#corePerNode} = 6304 * .263 / 16 = 104$
 In SimGrid platform language, this corresponds to such a cluster:
 ```xml
 <cluster id="cluster_MC" prefix="MC_" suffix="" radical="0-103" core="16">
 ```
 The corresponding SimGrid platform file can be found in `platform/average_metacentrum.xml`.
 (1) clusters decomissionned before or comissionned after the 6-month period have been removed: $8+480+160+1792+256+576+88+416+108+168+752+112+588+48+152+160+192+24+224 = 6304$