From: Michael Prokop Date: Wed, 13 Mar 2024 11:36:30 +0000 (+0100) Subject: grml2usb docs: drop references to deprecated grml-medium X-Git-Url: https://git.grml.org/?p=grml2usb.git;a=commitdiff_plain;h=HEAD;hp=6a4ec4bb15a2df99e382aadc2b53e395c58cebb0 grml2usb docs: drop references to deprecated grml-medium grml-medium is gone since many years, so there's no point in further referring to it Thanks to Christoph Biedl for spotting --- diff --git a/.github/workflows/check-full.yml b/.github/workflows/check-full.yml new file mode 100644 index 0000000..4536d0f --- /dev/null +++ b/.github/workflows/check-full.yml @@ -0,0 +1,39 @@ +name: Code Testing + +on: + push: + pull_request: + schedule: + - cron: '42 1 * * *' + +jobs: + codecheck: + runs-on: ubuntu-latest + name: Run codecheck + + steps: + - name: Checkout source + uses: actions/checkout@v2 + + - name: pip install wheel (to make install black work) + run: pip3 install wheel + + - name: pip install flake8, isort + black, vulture + run: pip3 install flake8 isort black vulture + + - name: Codecheck execution + run: make codecheck + + unittests: + runs-on: ubuntu-latest + name: Run unit tests + + steps: + - name: Checkout source + uses: actions/checkout@v2 + + - name: Install pytest + run: pip3 install pytest + + - name: Run Pytest + run: pytest diff --git a/Makefile b/Makefile index 7a3f516..dd12e74 100644 --- a/Makefile +++ b/Makefile @@ -42,6 +42,10 @@ codecheck: flake8 grml2usb isort --check-only grml2usb black --check grml2usb + vulture grml2usb test/grml2usb_test.py + +test: + pytest # graph: # sudo pycallgraph grml2usb /grml/isos/grml-small_2008.11.iso /dev/sdb1 diff --git a/debian/changelog b/debian/changelog index bde4e0a..f818fc7 100644 --- a/debian/changelog +++ b/debian/changelog @@ -1,3 +1,58 @@ +grml2usb (0.19.2) unstable; urgency=medium + + * [2955aaf] codecheck: reformat with black, version 23.1.0 + (Closes: #1031466) + + -- Michael Prokop Mon, 20 Feb 2023 11:33:34 +0100 + +grml2usb (0.19.1) unstable; urgency=medium + + * [27eba4a] Replace egrep usage with grep -E + * [ec28a50] Bump Standards-Version to 4.6.2 + + -- Michael Prokop Wed, 08 Feb 2023 08:27:50 +0100 + +grml2usb (0.19.0) unstable; urgency=medium + + [ Manuel Rom ] + * [40221eb] Add Github action workflows for CI/CD + + [ Michael Prokop ] + * [3f77679] Fix vulture usage and add vulture to Build-Depends + * [a6cce22] Github action: do not install virtualenv + + python3-setuptools. Thanks to Chris Hofstaedtler + * [78ae858] grml2iso: support parallel execution + * [8126bbf] grml2iso: execute under pipefail + + -- Michael Prokop Mon, 21 Mar 2022 16:19:54 +0100 + +grml2usb (0.18.5) unstable; urgency=medium + + * [01ed11d] Fix --grub and --syslinux handling. Thanks to Ralf Moll for + the bugreport + + -- Michael Prokop Fri, 22 Jan 2021 10:32:43 +0100 + +grml2usb (0.18.4) unstable; urgency=medium + + [ Michael Prokop ] + * [3664e4a] Code cleanups. Thanks to Chris Hofstaedtler for review + + feedback + * [1dfee7d] Use "GRML" as FAT label when creating the file system + * [272b66e] Fix race condition with blockdev/BLKRRPART due to lack of + fsync. Thanks to dann frazier for bug report and + patch (Closes: #975015) + * [100a957] Bump Standards-Version to 4.5.1 + + [ Darshaka Pathirana ] + * [7656c18] Use consistent case of 'USB' and 'Grml' + + [ Manuel Rom ] + * [6f3eb52] Add unit testing capabilities and basic tests for + check_for_usbdevice + + -- Michael Prokop Fri, 27 Nov 2020 17:56:02 +0100 + grml2usb (0.18.3) unstable; urgency=medium * [9e7dd96] codecheck: fix flake8 issues with versions >=3.8.3. Thanks diff --git a/debian/control b/debian/control index 05d2c41..c60f8fb 100644 --- a/debian/control +++ b/debian/control @@ -14,8 +14,9 @@ Build-Depends: docbook-xsl, flake8, isort, + vulture, xsltproc, -Standards-Version: 4.5.0 +Standards-Version: 4.6.2 Homepage: https://grml.org/grml2usb/ Vcs-git: git://git.grml.org/grml2usb.git Vcs-Browser: https://git.grml.org/?p=grml2usb.git diff --git a/debian/rules b/debian/rules index f83d809..7d696b5 100755 --- a/debian/rules +++ b/debian/rules @@ -10,7 +10,7 @@ VERSION:=$(shell dpkg-parsechangelog | awk '/Version: / { print $$2 }') override_dh_auto_build: dh_testdir - egrep -q '^PROG_VERSION = "\*\*\*UNKNOWN\*\*\*"' grml2usb || (echo "PROG_VERSION in grml2usb wrong." && exit 2) + grep -qE '^PROG_VERSION = "\*\*\*UNKNOWN\*\*\*"' grml2usb || (echo "PROG_VERSION in grml2usb wrong." && exit 2) $(MAKE) -C mbr $(MAKE) diff --git a/grml2iso b/grml2iso index 5f32804..3e48688 100755 --- a/grml2iso +++ b/grml2iso @@ -6,15 +6,15 @@ # License: This file is licensed under the GPL v2 or any later version. ################################################################################ +set -e -o pipefail + # make sure we have the sbin directories in our PATH to find grml2usb ootb PATH="${PATH}:/sbin:/usr/local/sbin:/usr/sbin" # adjust variables if necessary through environment {{{ # path to the grml2usb script you'd like to use - [ -n "$GRML2USB" ] || GRML2USB='grml2usb' -# work directory for creating the filesystem - [ -n "$TMPDIR" ] && WRKDIR="${TMPDIR}/grml2iso.tmp" - [ -n "$WRKDIR" ] || WRKDIR='/tmp/grml2iso.tmp' +[ -n "$GRML2USB" ] || GRML2USB='grml2usb' + # support mkisofs as well as genisoimage if which xorriso >/dev/null 2>&1 ; then MKISOFS='xorriso -as mkisofs' @@ -34,8 +34,6 @@ fi # }}} # helper stuff {{{ - set -e - usage() { echo >&2 "Usage: $0 [OPTIONS] -o target.iso source1.iso [source2.iso ...]" echo >&2 " @@ -54,10 +52,11 @@ Options: restrictions in the bootprocess only IPs are allowed. Supported protocols are: http and ftp -t Directory Directory that should be used for temporary files - during build. Defaults to /tmp/grml2iso.tmp if unset. + during build, instead of using a temporary directory + created by mktemp(1). Examples: - $0 -s http://192.168.23.42:8000/grml/ -o small.iso grml64-small_2018.12.iso + $0 -s http://192.168.23.42:8000/grml/ -o small.iso grml64-small_2021.07.iso Will generate a file small.iso which tries to download the squashfs file from http://192.168.23.42:8000/grml/ - the squashfs file is placed in the same @@ -95,10 +94,6 @@ Options: GRML2USB_OPTS+=(--bootoptions="fetch=$URI") fi - if [ -n "$WRKDIR" ] ; then - GRML2USB_OPTS+=(--tmpdir="$WRKDIR") - fi - # make sure -o is specified [ -n "$ISOFILE" ] || usage 1 @@ -143,8 +138,16 @@ Options: esac # }}} -# create necessary stuff under WRKDIR {{{ - [ -d "$WRKDIR" ] && WRKDIR_EXISTED='true' || WRKDIR_EXISTED='false' +# ensure to properly set up working directory {{{ + WRKDIR_EXISTED='false' + if [ -z "$WRKDIR" ] ; then + WRKDIR="$(mktemp -d)" + else + [ -d "$WRKDIR" ] && WRKDIR_EXISTED='true' + fi + + GRML2USB_OPTS+=(--tmpdir="$WRKDIR") + rm -rf "$WRKDIR/cddir" "$WRKDIR/grub_tmp" mkdir -p "$WRKDIR/cddir" # }}}} diff --git a/grml2usb b/grml2usb index d206f1d..5ac6e76 100755 --- a/grml2usb +++ b/grml2usb @@ -14,7 +14,6 @@ This script installs a Grml system (either a running system or ISO[s]) to a USB """ -import datetime import fileinput import glob import logging @@ -22,11 +21,9 @@ import os import os.path import re import shutil -import struct import subprocess import sys import tempfile -import time import uuid from inspect import isclass, isroutine from optparse import OptionParser @@ -53,9 +50,6 @@ except Exception: # global variables MOUNTED = set() # register mountpoints TMPFILES = set() # register tmpfiles -DATESTAMP = time.mktime( - datetime.datetime.now().timetuple() -) # unique identifier for syslinux.cfg GRML_FLAVOURS = set() # which flavours are being installed? GRML_DEFAULT = None UUID = None @@ -63,7 +57,6 @@ SYSLINUX_LIBS = [ "/usr/lib/syslinux/modules/bios/", # Debian "/usr/lib/syslinux/bios/", # Arch Linux ] -GPT_HEADER = b"\x55\xaa\x45\x46\x49\x20\x50\x41\x52\x54" # original GPT header GRUB_INSTALL = None RE_PARTITION = re.compile(r"([a-z/]*?)(\d+)$") @@ -71,9 +64,8 @@ RE_P_PARTITION = re.compile(r"(.*?\d+)p(\d+)$") RE_LOOP_DEVICE = re.compile(r"/dev/loop\d+$") -def syslinux_warning(option, opt, value, opt_parser): - """A helper function for printing a warning about deprecated option - """ +def syslinux_warning(option, opt, _value, opt_parser): + """A helper function for printing a warning about deprecated option""" # pylint: disable-msg=W0613 sys.stderr.write( "Note: the --syslinux option is deprecated as syslinux " @@ -83,9 +75,8 @@ def syslinux_warning(option, opt, value, opt_parser): # if grub option is set, unset syslinux option -def grub_option(option, opt, value, opt_parser): - """A helper function adjusting other option values - """ +def grub_option(option, opt, _value, opt_parser): + """A helper function adjusting other option values""" # pylint: disable-msg=W0613 setattr(opt_parser.values, option.dest, True) setattr(opt_parser.values, "syslinux", False) @@ -273,29 +264,11 @@ class VerifyException(Exception): @Exception: message""" -# The following two functions help to operate on strings as -# array (list) of bytes (octets). In Python 3000, the bytes -# datatype will need to be used. This is intended for using -# with manipulation of files on the octet level, like shell -# arrays, e.g. in MBR creation. - - -def array2string(*a): - """Convert a list of integers [0;255] to a string.""" - return struct.pack("%sB" % len(a), *a) - - -def string2array(s): - """Convert a (bytes) string into a list of integers.""" - return struct.unpack("%sB" % len(s), s) - - def cleanup(): - """Cleanup function to make sure there aren't any mounted devices left behind. - """ + """Cleanup function to make sure there aren't any mounted devices left behind.""" - def del_failed(fn, path, exc): - logging.warn("Deletion of %s failed in temporary folder" % path) + def del_failed(_fn, path, _exc): + logging.warning("Deletion of %s failed in temporary folder", path) logging.info("Cleaning up before exiting...") proc = subprocess.Popen(["sync"]) @@ -427,8 +400,7 @@ def which(program): def get_defaults_file(iso_mount, flavour, name): - """get the default file for syslinux - """ + """get the default file for syslinux""" bootloader_dirs = ["/boot/isolinux/", "/boot/syslinux/"] for directory in bootloader_dirs: for name in name, "%s_%s" % (get_flavour_filename(flavour), name): @@ -465,8 +437,7 @@ def search_file( retval.append(os.path.abspath(os.path.join(current_dir, filename))) if not lst_return: break - # pylint: disable-msg=W0612 - for current_dir, directories, files in os.walk(path): + for current_dir, _directories, _files in os.walk(path): if match_file(current_dir): retval.append(os.path.abspath(os.path.join(current_dir, filename))) if not lst_return: @@ -545,35 +516,14 @@ def mkfs_fat16(device): logging.info("Formating partition with fat16 filesystem") logging.debug("mkfs.vfat -F 16 %s", device) - proc = subprocess.Popen(["mkfs.vfat", "-F", "16", device]) + proc = subprocess.Popen(["mkfs.vfat", "-n", "GRML", "-F", "16", device]) proc.wait() if proc.returncode != 0: raise CriticalException("error executing mkfs.vfat") -def generate_isolinux_splash(grml_flavour): - """Generate bootsplash for isolinux/syslinux - - @grml_flavour: name of grml flavour the configuration should be generated for""" - - grml_name = grml_flavour - - return """\ -17 /boot/syslinux/logo.16 - -Some information and boot options available via keys F2 - F10. http://grml.org/ -%(grml_name)s -""" % { - "grml_name": grml_name - } - - -def generate_main_syslinux_config(*arg): - """Generate main configuration for use in syslinux.cfg - - @*arg: just for backward compatibility""" - # pylint: disable-msg=W0613 - # remove warning about unused arg +def generate_main_syslinux_config(): + """Generate main configuration for use in syslinux.cfg""" return """\ label - @@ -780,7 +730,7 @@ def install_mbr(mbrtemplate, device, partition, ismirbsdmbr=True): raise CriticalException("%s can not be read." % mbrtemplate) if partition is not None and ((partition < 0) or (partition > 3)): - logging.warn("Cannot activate partition %d", partition) + logging.warning("Cannot activate partition %d", partition) partition = None if ismirbsdmbr: @@ -856,7 +806,9 @@ def install_mbr(mbrtemplate, device, partition, ismirbsdmbr=True): set_rw(device) logging.debug( - "executing: dd if='%s' of='%s' bs=512 count=1 conv=notrunc", tmpf.name, device + "executing: dd if='%s' of='%s' bs=512 count=1 conv=notrunc,fsync", + tmpf.name, + device, ) proc = subprocess.Popen( [ @@ -865,7 +817,7 @@ def install_mbr(mbrtemplate, device, partition, ismirbsdmbr=True): "of=%s" % device, "bs=512", "count=1", - "conv=notrunc", + "conv=notrunc,fsync", ], stderr=open(os.devnull, "r+"), ) @@ -874,11 +826,6 @@ def install_mbr(mbrtemplate, device, partition, ismirbsdmbr=True): raise Exception("error executing dd (third run)") del tmpf - # make sure we sync filesystems before returning - logging.debug("executing: sync") - proc = subprocess.Popen(["sync"]) - proc.wait() - logging.debug("Probing device via 'blockdev --rereadpt %s'", device) proc = subprocess.Popen(["blockdev", "--rereadpt", device]) proc.wait() @@ -890,20 +837,6 @@ def install_mbr(mbrtemplate, device, partition, ismirbsdmbr=True): set_rw(device) -def is_writeable(device): - """Check if the device is writeable for the current user - - @device: partition where bootloader should be installed to""" - - if not device: - return False - - if not os.path.exists(device): - return False - - return os.access(device, os.W_OK) and os.access(device, os.R_OK) - - def mount(source, target, mount_options): """Mount specified source on given target @@ -966,13 +899,21 @@ def unmount(target, unmount_options): unregister_mountpoint(target) +def extract_device_name(device): + """Extract the device name of a given path + + @device: device name, like /dev/sda1 or /dev/sda + """ + return re.match(r"/dev/(.*?)\d*$", device).group(1) + + def check_for_usbdevice(device): """Check whether the specified device is a removable USB device @device: device name, like /dev/sda1 or /dev/sda """ - usbdevice = re.match(r"/dev/(.*?)\d*$", device).group(1) + usbdevice = extract_device_name(device) # newer systems: usbdev = os.path.realpath("/sys/class/block/" + usbdevice + "/removable") if not os.path.isfile(usbdev): @@ -1138,54 +1079,6 @@ def copy_system_files(grml_flavour, iso_mount, target): exec_rsync(source, dest) -def update_grml_versions(iso_mount, target): - """Update the grml version file on a cd - Returns true if version was updated successfully, - False if grml-version does not exist yet on the mountpoint - - @iso_mount: string of the iso mount point - @target: path of the target mount point - """ - grml_target = target + "/grml/" - target_grml_version_file = search_file("grml-version", grml_target) - if target_grml_version_file: - iso_grml_version_file = search_file("grml-version", iso_mount) - if not iso_grml_version_file: - logging.warn( - "Warning: %s could not be found - can not install it", - iso_grml_version_file, - ) - return False - try: - # read the flavours from the iso image - iso_versions = {} - iso_file = open(iso_grml_version_file, "r") - for line in iso_file: - iso_versions[get_flavour(line)] = line.strip() - - # update the existing flavours on the target - for line in fileinput.input([target_grml_version_file], inplace=1): - flavour = get_flavour(line) - if flavour in list(iso_versions.keys()): - print(iso_versions.pop(flavour)) - else: - print(line.strip()) - fileinput.close() - - target_file = open(target_grml_version_file, "a") - # add the new flavours from the current iso - for flavour in iso_versions: - target_file.write("%s\n" % iso_versions[flavour]) - except IOError: - logging.warn("Warning: Could not write file") - finally: - iso_file.close() - target_file.close() - return True - else: - return False - - def copy_grml_files(grml_flavour, iso_mount, target): """copy some minor grml files to a given target @@ -1203,7 +1096,9 @@ def copy_grml_files(grml_flavour, iso_mount, target): exec_rsync(filename, grml_target) break else: - logging.warn("Warning: could not find flavour directory for %s ", grml_flavour) + logging.warning( + "Warning: could not find flavour directory for %s ", grml_flavour + ) def copy_addons(iso_mount, target): @@ -1428,8 +1323,7 @@ def get_device_from_partition(partition): def get_flavour(flavour_str): - """Returns the flavour of a grml version string - """ + """Returns the flavour of a grml version string""" return re.match(r"[\w-]*", flavour_str).group() @@ -1720,7 +1614,6 @@ def handle_syslinux_config(grml_flavour, target): new_grml_cfg = "%s/%s_grml.cfg" % (syslinux_target, flavour_filename) if os.path.isfile(defaults_file): - # remove default menu entry in menu remove_default_entry(new_default_with_path) @@ -1905,8 +1798,7 @@ def install_grml(mountpoint, device): def remove_mountpoint(mountpoint): - """remove a registered mountpoint - """ + """remove a registered mountpoint""" try: unmount(mountpoint, "") @@ -1929,7 +1821,7 @@ def handle_mbr(device): mbr_device, partition_number = get_device_from_partition(device) if partition_number is None: - logging.warn("Could not detect partition number, not activating partition") + logging.warning("Could not detect partition number, not activating partition") # if we get e.g. /dev/loop1 as device we don't want to put the MBR # into /dev/loop of course, therefore use /dev/loop1 as mbr_device @@ -2164,7 +2056,7 @@ def main(): if not os.path.isdir(device): if device[-1:].isdigit(): if int(device[-1:]) > 4 or device[-2:].isdigit(): - logging.warn( + logging.warning( "Warning: installing on partition number >4, booting *might* fail depending on your system." ) diff --git a/grml2usb.8.txt b/grml2usb.8.txt index 65ae948..ec8a197 100644 --- a/grml2usb.8.txt +++ b/grml2usb.8.txt @@ -3,7 +3,7 @@ grml2usb(8) Name ---- -grml2usb - install Grml ISO(s) on usb device for booting +grml2usb - install Grml ISO(s) on USB device for booting Synopsis -------- @@ -17,7 +17,7 @@ Important! The Grml team does not take responsibility for loss of any data! Introduction ------------ -grml2usb installs Grml on a given partition of your usb device and makes +grml2usb installs Grml on a given partition of your USB device and makes it bootable. It provides multiboot ISO support, meaning you can specify several Grml ISOs on the command line at once and select the Grml flavour you would like to boot on the bootprompt then. Note that the @@ -197,7 +197,7 @@ Developers Corner ----------------- [[directory-layout]] -Directory layout on usb device +Directory layout on USB device ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ boot/ -> @@ -214,12 +214,6 @@ Directory layout on usb device | |-- grml64 | | |-- linux26 [Kernel] | | |-- initrd.gz [initramfs] - | |-- grml-medium - | | |-- linux26 [...] - | | |-- initrd.gz - | |-- grml64-medium - | | |-- linux26 - | | |-- initrd.gz | |-- grml-small | | |-- linux26 | | |-- initrd.gz @@ -256,11 +250,8 @@ Directory layout on usb device |-- grml/ | |-- filesystem.module [module specifying which squashfs should be used for grml] | `-- grml.squashfs [squashfs file for grml] - |-- grml-medium/ - | |-- filesystem.module [module specifying which squashfs should be used for grml-medium] - | `-- grml-medium.squashfs [squashfs file for grml-medium] |-- grml-small/ - | |-- filesystem.module [module specifying which squashfs should be used for grml-medium] + | |-- filesystem.module [module specifying which squashfs should be used for grml-small] | `-- grml-small.squashfs [squashfs file for grml-small] `-- ... @@ -315,9 +306,9 @@ See http://www.methods.co.nz/asciidoc/userguide.html [horizontal] *Error message*:: ran out of input data. System halted -*Reason*:: Everything OK, except for the filesystem used on your usb device. So +*Reason*:: Everything OK, except for the filesystem used on your USB device. So instead of fat16 you are using for example fat32. Fix: use the appropriate -filesystem (fat16 for usb pens usually). The Bootsplash might be displayed, the +filesystem (fat16 for USB flash drive usually). The Bootsplash might be displayed, the kernel loads but you very soon get the error message. *Error message*:: Invalid operating system @@ -427,7 +418,7 @@ What's grml2iso? ~~~~~~~~~~~~~~~~ grml2iso is a script which uses grml2usb to generate a multiboot ISO out of -several grml ISOs. See 'man grml2iso' for further details. +several Grml ISOs. See 'man grml2iso' for further details. [[menu-lst]] Why is there a menu.lst and a grub.cfg inside /boot/grub/? diff --git a/pytest.ini b/pytest.ini new file mode 100644 index 0000000..6bba0a2 --- /dev/null +++ b/pytest.ini @@ -0,0 +1,3 @@ +[pytest] +markers = + check_for_usbdevice diff --git a/tarball.sh b/tarball.sh index 8679a45..046ec20 100755 --- a/tarball.sh +++ b/tarball.sh @@ -16,7 +16,7 @@ cat > "${DIR}"/README << EOF README ------ -grml2usb installs grml ISO(s) on usb device for booting. +grml2usb installs Grml ISO(s) on USB device for booting. This tarball provides all the necessary files for running grml2usb. Execute the script install.sh with root permissions to install the diff --git a/test/grml2usb.py b/test/grml2usb.py new file mode 120000 index 0000000..07ca4fc --- /dev/null +++ b/test/grml2usb.py @@ -0,0 +1 @@ +../grml2usb \ No newline at end of file diff --git a/test/grml2usb_test.py b/test/grml2usb_test.py new file mode 100644 index 0000000..56b2085 --- /dev/null +++ b/test/grml2usb_test.py @@ -0,0 +1,40 @@ +""" +grml2usb basic pytests +~~~~~~~~~~~~~~~~~~~~~~ + +This script contains basic "unit" tests, implemented for and executed with pytest. + +Requirements: +pytest (pip install pytest) + +Runwith: +$ pytest [-m {basic}] + +:copyright: (c) 2020 by Manuel Rom +:license: GPL v2 or any later version +:bugreports: http://grml.org/bugs/ +""" + + +import importlib + +import pytest + +grml2usb = importlib.import_module("grml2usb", ".") + + +@pytest.mark.check_for_usbdevice +def test_extract_device_name(): + """Assert, that 'extract_device_name' returns a device name for a given path""" + assert grml2usb.extract_device_name("/dev/sda") == "sda" + assert grml2usb.extract_device_name("/dev/sdb") == "sdb" + assert grml2usb.extract_device_name("/dev/sdb4") == "sdb" + + +@pytest.mark.check_for_usbdevice +def test_extract_device_name_invalid(): + """Assert, that 'extract_device_name' raises an Error, when given an incorrect string""" + with pytest.raises(AttributeError): + assert grml2usb.extract_device_name("/dev") + with pytest.raises(AttributeError): + assert grml2usb.extract_device_name("foobar")