From 100a95736d6e5ab1afb4b79013f82631969d28bc Mon Sep 17 00:00:00 2001 From: Michael Prokop Date: Fri, 27 Nov 2020 17:55:56 +0100 Subject: [PATCH 01/16] Bump Standards-Version to 4.5.1 --- debian/control | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/debian/control b/debian/control index 05d2c41..e5c58b0 100644 --- a/debian/control +++ b/debian/control @@ -15,7 +15,7 @@ Build-Depends: flake8, isort, xsltproc, -Standards-Version: 4.5.0 +Standards-Version: 4.5.1 Homepage: https://grml.org/grml2usb/ Vcs-git: git://git.grml.org/grml2usb.git Vcs-Browser: https://git.grml.org/?p=grml2usb.git -- 2.1.4 From a0f5a4e00741da298835127ad333744f0f7ee73a Mon Sep 17 00:00:00 2001 From: Michael Prokop Date: Fri, 27 Nov 2020 17:56:40 +0100 Subject: [PATCH 02/16] Release new version 0.18.4 --- debian/changelog | 20 ++++++++++++++++++++ 1 file changed, 20 insertions(+) diff --git a/debian/changelog b/debian/changelog index bde4e0a..d8db4af 100644 --- a/debian/changelog +++ b/debian/changelog @@ -1,3 +1,23 @@ +grml2usb (0.18.4) unstable; urgency=medium + + [ Michael Prokop ] + * [3664e4a] Code cleanups. Thanks to Chris Hofstaedtler for review + + feedback + * [1dfee7d] Use "GRML" as FAT label when creating the file system + * [272b66e] Fix race condition with blockdev/BLKRRPART due to lack of + fsync. Thanks to dann frazier for bug report and + patch (Closes: #975015) + * [100a957] Bump Standards-Version to 4.5.1 + + [ Darshaka Pathirana ] + * [7656c18] Use consistent case of 'USB' and 'Grml' + + [ Manuel Rom ] + * [6f3eb52] Add unit testing capabilities and basic tests for + check_for_usbdevice + + -- Michael Prokop Fri, 27 Nov 2020 17:56:02 +0100 + grml2usb (0.18.3) unstable; urgency=medium * [9e7dd96] codecheck: fix flake8 issues with versions >=3.8.3. Thanks -- 2.1.4 From 01ed11d39312404c265da1a53938c921ce03c4f9 Mon Sep 17 00:00:00 2001 From: Michael Prokop Date: Fri, 22 Jan 2021 10:20:41 +0100 Subject: [PATCH 03/16] Fix --grub and --syslinux handling With commit 3664e4ae we removed an argument from the grub_option() and syslinux_warning() callback functions, which was reported as false positives by vulture tool. Fixes regression from 3664e4ae, failing grml2usb execution with --grub (and deprecated --syslinux) command line option with: | TypeError: grub_option() takes 3 positional arguments but 4 were given Thanks: Ralf Moll for the bugreport --- grml2usb | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/grml2usb b/grml2usb index d8c5053..8f689a9 100755 --- a/grml2usb +++ b/grml2usb @@ -64,7 +64,7 @@ RE_P_PARTITION = re.compile(r"(.*?\d+)p(\d+)$") RE_LOOP_DEVICE = re.compile(r"/dev/loop\d+$") -def syslinux_warning(option, opt, opt_parser): +def syslinux_warning(option, opt, _value, opt_parser): """A helper function for printing a warning about deprecated option""" # pylint: disable-msg=W0613 sys.stderr.write( @@ -75,7 +75,7 @@ def syslinux_warning(option, opt, opt_parser): # if grub option is set, unset syslinux option -def grub_option(option, opt, opt_parser): +def grub_option(option, opt, _value, opt_parser): """A helper function adjusting other option values""" # pylint: disable-msg=W0613 setattr(opt_parser.values, option.dest, True) -- 2.1.4 From 7cda8fa07780825c0819ed81a836554a69349afa Mon Sep 17 00:00:00 2001 From: Michael Prokop Date: Fri, 22 Jan 2021 10:32:54 +0100 Subject: [PATCH 04/16] Release new version 0.18.5 --- debian/changelog | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/debian/changelog b/debian/changelog index d8db4af..2389236 100644 --- a/debian/changelog +++ b/debian/changelog @@ -1,3 +1,10 @@ +grml2usb (0.18.5) unstable; urgency=medium + + * [01ed11d] Fix --grub and --syslinux handling. Thanks to Ralf Moll for + the bugreport + + -- Michael Prokop Fri, 22 Jan 2021 10:32:43 +0100 + grml2usb (0.18.4) unstable; urgency=medium [ Michael Prokop ] -- 2.1.4 From 40221ebfc36eaccf1634153a0b57ac2db5020966 Mon Sep 17 00:00:00 2001 From: Manuel Rom Date: Mon, 22 Mar 2021 11:12:42 +0100 Subject: [PATCH 05/16] Add Github action workflows for CI/CD * Add .github/workflows/check-full.yml, enabling: * Code checks (flake8, black, isort, vulture) * Unit tests (pytest) * Adapt the Makefile * Add vulture to codecheck target * Add test target with pytest --- .github/workflows/check-full.yml | 57 ++++++++++++++++++++++++++++++++++++++++ Makefile | 4 +++ 2 files changed, 61 insertions(+) create mode 100644 .github/workflows/check-full.yml diff --git a/.github/workflows/check-full.yml b/.github/workflows/check-full.yml new file mode 100644 index 0000000..91d5ab6 --- /dev/null +++ b/.github/workflows/check-full.yml @@ -0,0 +1,57 @@ +name: Code Testing + +on: + push: + pull_request: + schedule: + - cron: '42 1 * * *' + +jobs: + codecheck: + runs-on: ubuntu-latest + name: Run codecheck + + steps: + - name: Checkout source + uses: actions/checkout@v2 + + - name: Install virtualenv + python3-setuptools + run: sudo apt-get install virtualenv python3-setuptools + + - name: Set up Python virtualenv environment + run: virtualenv -p /usr/bin/python3 venv3 + + - name: Activate Python virtualenv environment + run: . ./venv3/bin/activate + + - name: pip install wheel (to make install black work) + run: pip3 install wheel + + - name: pip install flake8, isort + black, vulture + run: pip3 install flake8 isort black vulture + + - name: Codecheck execution + run: make codecheck + + unittests: + runs-on: ubuntu-latest + name: Run unit tests + + steps: + - name: Checkout source + uses: actions/checkout@v2 + + - name: Install virtualenv + python3-setuptools + run: sudo apt-get install virtualenv python3-setuptools + + - name: Set up Python virtualenv environment + run: virtualenv -p /usr/bin/python3 venv3 + + - name: Activate Python virtualenv environment + run: . ./venv3/bin/activate + + - name: Install pytest + run: pip3 install pytest + + - name: Run Pytest + run: pytest diff --git a/Makefile b/Makefile index 7a3f516..6d506eb 100644 --- a/Makefile +++ b/Makefile @@ -42,6 +42,10 @@ codecheck: flake8 grml2usb isort --check-only grml2usb black --check grml2usb + vulture grml2iso grml2usb test/grml2usb_test.py + +test: + pytest # graph: # sudo pycallgraph grml2usb /grml/isos/grml-small_2008.11.iso /dev/sdb1 -- 2.1.4 From 3f776793aa188a538869f2276915b29c7d6e91d5 Mon Sep 17 00:00:00 2001 From: Michael Prokop Date: Sat, 24 Apr 2021 21:58:15 +0200 Subject: [PATCH 06/16] Fix vulture usage and add vulture to Build-Depends * grml2iso is a shell script, executing vulture doesn't work on it * the Debian package build process executes 'make codecheck:', so everything what's needed for its execution needs to be covered through the Build-Dependencies, accordingly add add vulture to Build-Depends --- Makefile | 2 +- debian/control | 1 + 2 files changed, 2 insertions(+), 1 deletion(-) diff --git a/Makefile b/Makefile index 6d506eb..dd12e74 100644 --- a/Makefile +++ b/Makefile @@ -42,7 +42,7 @@ codecheck: flake8 grml2usb isort --check-only grml2usb black --check grml2usb - vulture grml2iso grml2usb test/grml2usb_test.py + vulture grml2usb test/grml2usb_test.py test: pytest diff --git a/debian/control b/debian/control index e5c58b0..d6cac4f 100644 --- a/debian/control +++ b/debian/control @@ -14,6 +14,7 @@ Build-Depends: docbook-xsl, flake8, isort, + vulture, xsltproc, Standards-Version: 4.5.1 Homepage: https://grml.org/grml2usb/ -- 2.1.4 From a6cce2293f8dd8ca787e7617d640304b682d5dfe Mon Sep 17 00:00:00 2001 From: Michael Prokop Date: Sat, 24 Apr 2021 22:19:12 +0200 Subject: [PATCH 07/16] Github action: do not install virtualenv + python3-setuptools pip3 and python3-setuptools are available out of the box on ubuntu-latest. By not having to install python3-virtualenv we gain ~6 seconds in each of of the codecheck and unit tests runs. So instead let's rely on system wide pip3, given that we're running in a throw-away VM anyway. Thanks: Chris Hofstaedtler --- .github/workflows/check-full.yml | 18 ------------------ 1 file changed, 18 deletions(-) diff --git a/.github/workflows/check-full.yml b/.github/workflows/check-full.yml index 91d5ab6..4536d0f 100644 --- a/.github/workflows/check-full.yml +++ b/.github/workflows/check-full.yml @@ -15,15 +15,6 @@ jobs: - name: Checkout source uses: actions/checkout@v2 - - name: Install virtualenv + python3-setuptools - run: sudo apt-get install virtualenv python3-setuptools - - - name: Set up Python virtualenv environment - run: virtualenv -p /usr/bin/python3 venv3 - - - name: Activate Python virtualenv environment - run: . ./venv3/bin/activate - - name: pip install wheel (to make install black work) run: pip3 install wheel @@ -41,15 +32,6 @@ jobs: - name: Checkout source uses: actions/checkout@v2 - - name: Install virtualenv + python3-setuptools - run: sudo apt-get install virtualenv python3-setuptools - - - name: Set up Python virtualenv environment - run: virtualenv -p /usr/bin/python3 venv3 - - - name: Activate Python virtualenv environment - run: . ./venv3/bin/activate - - name: Install pytest run: pip3 install pytest -- 2.1.4 From 78ae858b306428eb130c8338d103ba2779c12857 Mon Sep 17 00:00:00 2001 From: Michael Prokop Date: Fri, 11 Mar 2022 18:20:01 +0100 Subject: [PATCH 08/16] grml2iso: support parallel execution Usage of a static working directory name doesn't make sense, and prevents usage running grml2iso multiple times in parallel. Instead use a temporary working directory. This work was funded by Grml-Forensic. --- grml2iso | 25 ++++++++++++++----------- 1 file changed, 14 insertions(+), 11 deletions(-) diff --git a/grml2iso b/grml2iso index 5f32804..b29c4df 100755 --- a/grml2iso +++ b/grml2iso @@ -11,10 +11,8 @@ PATH="${PATH}:/sbin:/usr/local/sbin:/usr/sbin" # adjust variables if necessary through environment {{{ # path to the grml2usb script you'd like to use - [ -n "$GRML2USB" ] || GRML2USB='grml2usb' -# work directory for creating the filesystem - [ -n "$TMPDIR" ] && WRKDIR="${TMPDIR}/grml2iso.tmp" - [ -n "$WRKDIR" ] || WRKDIR='/tmp/grml2iso.tmp' +[ -n "$GRML2USB" ] || GRML2USB='grml2usb' + # support mkisofs as well as genisoimage if which xorriso >/dev/null 2>&1 ; then MKISOFS='xorriso -as mkisofs' @@ -54,7 +52,8 @@ Options: restrictions in the bootprocess only IPs are allowed. Supported protocols are: http and ftp -t Directory Directory that should be used for temporary files - during build. Defaults to /tmp/grml2iso.tmp if unset. + during build, instead of using a temporary directory + created by mktemp(1). Examples: $0 -s http://192.168.23.42:8000/grml/ -o small.iso grml64-small_2018.12.iso @@ -95,10 +94,6 @@ Options: GRML2USB_OPTS+=(--bootoptions="fetch=$URI") fi - if [ -n "$WRKDIR" ] ; then - GRML2USB_OPTS+=(--tmpdir="$WRKDIR") - fi - # make sure -o is specified [ -n "$ISOFILE" ] || usage 1 @@ -143,8 +138,16 @@ Options: esac # }}} -# create necessary stuff under WRKDIR {{{ - [ -d "$WRKDIR" ] && WRKDIR_EXISTED='true' || WRKDIR_EXISTED='false' +# ensure to properly set up working directory {{{ + WRKDIR_EXISTED='false' + if [ -z "$WRKDIR" ] ; then + WRKDIR="$(mktemp -d)" + else + [ -d "$WRKDIR" ] && WRKDIR_EXISTED='true' + fi + + GRML2USB_OPTS+=(--tmpdir="$WRKDIR") + rm -rf "$WRKDIR/cddir" "$WRKDIR/grub_tmp" mkdir -p "$WRKDIR/cddir" # }}}} -- 2.1.4 From 8126bbf87f43e55098d09cd7592a60418ea19d2a Mon Sep 17 00:00:00 2001 From: Michael Prokop Date: Fri, 11 Mar 2022 18:25:31 +0100 Subject: [PATCH 09/16] grml2iso: execute under pipefail Quoting from bash(1): | The return status of a pipeline is the exit status of the last | command, unless the pipefail option is enabled. If pipefail is enabled, | the pipeline's return status is the value of the last (rightmost) | command to exit with a non-zero status, or zero if all commands exit | successfully. Furthermore, move the "set -e" to the beginning of the file, to ensure it's active as soon as possible. While at it, drop trailing whitespace and update ISO name to a more recent one. This work was funded by Grml-Forensic. --- grml2iso | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/grml2iso b/grml2iso index b29c4df..3e48688 100755 --- a/grml2iso +++ b/grml2iso @@ -6,6 +6,8 @@ # License: This file is licensed under the GPL v2 or any later version. ################################################################################ +set -e -o pipefail + # make sure we have the sbin directories in our PATH to find grml2usb ootb PATH="${PATH}:/sbin:/usr/local/sbin:/usr/sbin" @@ -32,8 +34,6 @@ fi # }}} # helper stuff {{{ - set -e - usage() { echo >&2 "Usage: $0 [OPTIONS] -o target.iso source1.iso [source2.iso ...]" echo >&2 " @@ -56,7 +56,7 @@ Options: created by mktemp(1). Examples: - $0 -s http://192.168.23.42:8000/grml/ -o small.iso grml64-small_2018.12.iso + $0 -s http://192.168.23.42:8000/grml/ -o small.iso grml64-small_2021.07.iso Will generate a file small.iso which tries to download the squashfs file from http://192.168.23.42:8000/grml/ - the squashfs file is placed in the same -- 2.1.4 From 6a409984ab0316d0a59c5e4ab38dac86a48282ae Mon Sep 17 00:00:00 2001 From: Michael Prokop Date: Mon, 21 Mar 2022 16:20:56 +0100 Subject: [PATCH 10/16] Release new version 0.19.0 --- debian/changelog | 14 ++++++++++++++ 1 file changed, 14 insertions(+) diff --git a/debian/changelog b/debian/changelog index 2389236..f712727 100644 --- a/debian/changelog +++ b/debian/changelog @@ -1,3 +1,17 @@ +grml2usb (0.19.0) unstable; urgency=medium + + [ Manuel Rom ] + * [40221eb] Add Github action workflows for CI/CD + + [ Michael Prokop ] + * [3f77679] Fix vulture usage and add vulture to Build-Depends + * [a6cce22] Github action: do not install virtualenv + + python3-setuptools. Thanks to Chris Hofstaedtler + * [78ae858] grml2iso: support parallel execution + * [8126bbf] grml2iso: execute under pipefail + + -- Michael Prokop Mon, 21 Mar 2022 16:19:54 +0100 + grml2usb (0.18.5) unstable; urgency=medium * [01ed11d] Fix --grub and --syslinux handling. Thanks to Ralf Moll for -- 2.1.4 From 27eba4ace1f4cde430f773c01a629c240b6bf207 Mon Sep 17 00:00:00 2001 From: Michael Prokop Date: Wed, 7 Sep 2022 13:57:23 +0200 Subject: [PATCH 11/16] Replace egrep usage with grep -E grep 3.8 deprecated support for egrep + fgrep, and now prints a warning on stderr: | egrep: warning: egrep is obsolescent; using grep -E | fgrep: warning: fgrep is obsolescent; using grep -F --- debian/rules | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/debian/rules b/debian/rules index f83d809..7d696b5 100755 --- a/debian/rules +++ b/debian/rules @@ -10,7 +10,7 @@ VERSION:=$(shell dpkg-parsechangelog | awk '/Version: / { print $$2 }') override_dh_auto_build: dh_testdir - egrep -q '^PROG_VERSION = "\*\*\*UNKNOWN\*\*\*"' grml2usb || (echo "PROG_VERSION in grml2usb wrong." && exit 2) + grep -qE '^PROG_VERSION = "\*\*\*UNKNOWN\*\*\*"' grml2usb || (echo "PROG_VERSION in grml2usb wrong." && exit 2) $(MAKE) -C mbr $(MAKE) -- 2.1.4 From ec28a5006697e567fbc5496b05da9a50ce42d333 Mon Sep 17 00:00:00 2001 From: Michael Prokop Date: Wed, 8 Feb 2023 08:27:47 +0100 Subject: [PATCH 12/16] Bump Standards-Version to 4.6.2 --- debian/control | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/debian/control b/debian/control index d6cac4f..c60f8fb 100644 --- a/debian/control +++ b/debian/control @@ -16,7 +16,7 @@ Build-Depends: isort, vulture, xsltproc, -Standards-Version: 4.5.1 +Standards-Version: 4.6.2 Homepage: https://grml.org/grml2usb/ Vcs-git: git://git.grml.org/grml2usb.git Vcs-Browser: https://git.grml.org/?p=grml2usb.git -- 2.1.4 From 00ce55f518f6abc61c2ede0320bffc90ff3eb09a Mon Sep 17 00:00:00 2001 From: Michael Prokop Date: Wed, 8 Feb 2023 08:28:02 +0100 Subject: [PATCH 13/16] Release new version 0.19.1 --- debian/changelog | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/debian/changelog b/debian/changelog index f712727..05f89ef 100644 --- a/debian/changelog +++ b/debian/changelog @@ -1,3 +1,10 @@ +grml2usb (0.19.1) unstable; urgency=medium + + * [27eba4a] Replace egrep usage with grep -E + * [ec28a50] Bump Standards-Version to 4.6.2 + + -- Michael Prokop Wed, 08 Feb 2023 08:27:50 +0100 + grml2usb (0.19.0) unstable; urgency=medium [ Manuel Rom ] -- 2.1.4 From 2955aaf78a680a9c1a96ef5d3a4e03599b2dd670 Mon Sep 17 00:00:00 2001 From: Michael Prokop Date: Mon, 20 Feb 2023 11:27:09 +0100 Subject: [PATCH 14/16] codecheck: reformat with black, version 23.1.0 Drop empty newline as expected by black >=23.1.0 See https://github.com/psf/black/blob/main/CHANGES.md Closes: #1031466 --- grml2usb | 1 - 1 file changed, 1 deletion(-) diff --git a/grml2usb b/grml2usb index 8f689a9..5ac6e76 100755 --- a/grml2usb +++ b/grml2usb @@ -1614,7 +1614,6 @@ def handle_syslinux_config(grml_flavour, target): new_grml_cfg = "%s/%s_grml.cfg" % (syslinux_target, flavour_filename) if os.path.isfile(defaults_file): - # remove default menu entry in menu remove_default_entry(new_default_with_path) -- 2.1.4 From 4bbd7a8a99e58c13ee09917fcb28caf720d85b86 Mon Sep 17 00:00:00 2001 From: Michael Prokop Date: Mon, 20 Feb 2023 11:35:14 +0100 Subject: [PATCH 15/16] Release new version 0.19.2 --- debian/changelog | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/debian/changelog b/debian/changelog index 05f89ef..f818fc7 100644 --- a/debian/changelog +++ b/debian/changelog @@ -1,3 +1,10 @@ +grml2usb (0.19.2) unstable; urgency=medium + + * [2955aaf] codecheck: reformat with black, version 23.1.0 + (Closes: #1031466) + + -- Michael Prokop Mon, 20 Feb 2023 11:33:34 +0100 + grml2usb (0.19.1) unstable; urgency=medium * [27eba4a] Replace egrep usage with grep -E -- 2.1.4 From 5f40f1c76e15c5de09a8a42fc4700284781f5be5 Mon Sep 17 00:00:00 2001 From: Michael Prokop Date: Wed, 13 Mar 2024 12:36:30 +0100 Subject: [PATCH 16/16] grml2usb docs: drop references to deprecated grml-medium grml-medium is gone since many years, so there's no point in further referring to it Thanks to Christoph Biedl for spotting --- grml2usb.8.txt | 11 +---------- 1 file changed, 1 insertion(+), 10 deletions(-) diff --git a/grml2usb.8.txt b/grml2usb.8.txt index 8e966ad..ec8a197 100644 --- a/grml2usb.8.txt +++ b/grml2usb.8.txt @@ -214,12 +214,6 @@ Directory layout on USB device | |-- grml64 | | |-- linux26 [Kernel] | | |-- initrd.gz [initramfs] - | |-- grml-medium - | | |-- linux26 [...] - | | |-- initrd.gz - | |-- grml64-medium - | | |-- linux26 - | | |-- initrd.gz | |-- grml-small | | |-- linux26 | | |-- initrd.gz @@ -256,11 +250,8 @@ Directory layout on USB device |-- grml/ | |-- filesystem.module [module specifying which squashfs should be used for grml] | `-- grml.squashfs [squashfs file for grml] - |-- grml-medium/ - | |-- filesystem.module [module specifying which squashfs should be used for grml-medium] - | `-- grml-medium.squashfs [squashfs file for grml-medium] |-- grml-small/ - | |-- filesystem.module [module specifying which squashfs should be used for grml-medium] + | |-- filesystem.module [module specifying which squashfs should be used for grml-small] | `-- grml-small.squashfs [squashfs file for grml-small] `-- ... -- 2.1.4