Merge branch 'RPi-Distro-dev' into dev

Author: Landrash

README.md

@@ -16,9 +16,18 @@ Build instructions:
 - Retrieve your freshly built Raspberry Pi image from the `rpi_gen\deploy` folder.
 
 
-### Dependencies
+## Dependencies
+
+On Debian-based systems:
+
+```bash
+apt-get install quilt parted realpath qemu-user-static debootstrap zerofree pxz zip \
+dosfstools bsdtar libcap2-bin grep rsync
+```
+
+The file `depends` contains a list of tools needed.  The format of this
+package is `<tool>[:<debian-package>]`.
 
-`quilt parted realpath qemu-user-static debootstrap zerofree pxz zip dosfstools bsdtar libcap2-bin grep rsync`
 
 ## Config
 
@@ -28,34 +37,73 @@ environment variables.
 
 The following environment variables are supported:
 
- * `IMG_NAME`, the name of the distribution to build (required)
  * `APT_PROXY`, proxy/cache URL to be included in the build
 
+ * `IMG_NAME`, the name of the distribution to build (required)
+   The name of the image to build with the current stage directories.  Setting
+   `IMG_NAME=Raspbian` is logical for an unmodified RPi-Distro/pi-gen build,
+   but you should use something else for a customized version.  Export files
+   in stages may add suffixes to `IMG_NAME`.
+
+ * `APT_PROXY` (Default: unset)
+
+   If you require the use of an apt proxy, set it here.  This proxy setting
+   will not be included in the image, making it safe to use an `apt-cacher` or
+   similar package for development.
+
+ * `BASE_DIR`  (Default: location of `build.sh`)
+
+   **CAUTION**: Currently, changing this value will probably break build.sh
+
+   Top-level directory for `pi-gen`.  Contains stage directories, build
+   scripts, and by default both work and deployment directories.
+
+ * `WORK_DIR`  (Default: `"$BASE_DIR/work"`)
+
+   Directory in which `pi-gen` builds the target system.  This value can be
+   changed if you have a suitably large, fast storage location for stages to
+   be built and cached.  Note, `WORK_DIR` stores a complete copy of the target
+   system for each build stage, amounting to tens of gigabytes in the case of
+   Raspbian.
+
+ * `DEPLOY_DIR`  (Default: `"$BASE_DIR/deploy"`)
+
+   Output directory for target system images and NOOBS bundles.
+
 A simple example for building Hassbian:
+=======
 
 ```bash
 IMG_NAME='Hassbian'
 ```
 
+
 ## Docker Build
 
 ```bash
 nano config         # Edit your config file. See above.
 ./build-docker.sh
 ```
+
 If everything goes well, your finished image will be in the `deploy/` folder.
-You can then remove the build container with `docker rm pigen_work`
+You can then remove the build container with `docker rm -v pigen_work`
 
 If something breaks along the line, you can edit the corresponding scripts, and
 continue:
 
-```
+```bash
 CONTINUE=1 ./build-docker.sh
 ```
 
-There is a possibility that even when running from a docker container, the installation of `qemu-user-static` will silently fail when building the image because `binfmt-support` _must be enabled on the underlying kernel_. An easy fix is to ensure `binfmt-support` is installed on the host machine before starting the `./build-docker.sh` script (or using your own docker build solution).
+There is a possibility that even when running from a docker container, the
+installation of `qemu-user-static` will silently fail when building the image
+because `binfmt-support` _must be enabled on the underlying kernel_. An easy
+fix is to ensure `binfmt-support` is installed on the host machine before
+starting the `./build-docker.sh` script (or using your own docker build
+solution).
+
 
-### Raspbian Stage Anatomy
+### Hasspbian Stage Anatomy
 
 The build of Hassbian is divided up into several stages for logical clarity
 and modularity.  This causes some initial complexity, but it simplifies
@@ -101,15 +149,26 @@ maintenance and allows for more easy customization.
    The original **Stage 4** and **Stage 5** are removed since they are not
    used on the HASSbian image.
 
-### Stage specification
-If you wish to build up to a specified stage (such as building up to stage 2 for a lite system), place an empty file named `SKIP` in each of the `./stage` directories you wish not to include.
 
-Then remove the `EXPORT*` files from `./stage3` (if building up to stage 2) and add them to `./stage2`.
-
-```
 ## Example for building a lite system without Home Assistant
 $ touch ./stage3/SKIP 
 $ rm stage3/EXPORT*
 $ touch stage3/EXPORT_IMAGE
 ```
 If you wish to build further configurations upon (for example) the lite system, you can also delete the contents of `./stage3` and replace with your own contents in the same format.
+
+=======
+If you wish to build up to a specified stage (such as building up to stage 2
+for a lite system), place an empty file named `SKIP` in each of the `./stage`
+directories you wish not to include.
+
+Then remove the `EXPORT*` files from `./stage4` (if building up to stage 2) or
+from `./stage2` (if building a minimal system).
+
+```bash
+# Example for building a lite system
+echo "IMG_NAME='Raspbian'" > config
+touch ./stage3/SKIP ./stage4/SKIP ./stage5/SKIP
+rm stage4/EXPORT*
+sudo ./build.sh  # or ./build-docker.sh
+```

build-docker.sh

@@ -12,9 +12,9 @@ if ! $DOCKER ps >/dev/null; then
 fi
 set -e
 
-config_mount=
+config_mount="/dev/null:/dev/null"
 if [ -f config ]; then
-	config_mount="-v $(pwd)/config:/pi-gen/config:ro"
+	config_mount="$(pwd)/config:/pi-gen/config:ro"
 	source config
 fi
 
@@ -45,7 +45,7 @@ fi
 if [ "$CONTAINER_EXISTS" != "" ] && [ "$CONTINUE" != "1" ]; then
 	echo "Container $CONTAINER_NAME already exists and you did not specify CONTINUE=1. Aborting."
 	echo "You can delete the existing container like this:"
-	echo "  docker rm $CONTAINER_NAME"
+	echo "  docker rm -v $CONTAINER_NAME"
 	exit 1
 fi
 
@@ -54,6 +54,7 @@ if [ "$CONTAINER_EXISTS" != "" ]; then
 	trap "echo 'got CTRL+C... please wait 5s';docker stop -t 5 ${CONTAINER_NAME}_cont" SIGINT SIGTERM
 	time $DOCKER run --rm --privileged \
 		--volumes-from="${CONTAINER_NAME}" --name "${CONTAINER_NAME}_cont" \
+		-e IMG_NAME=${IMG_NAME}\
 		pi-gen \
 		bash -e -o pipefail -c "dpkg-reconfigure qemu-user-static &&
 	cd /pi-gen; ./build.sh;
@@ -62,8 +63,9 @@ if [ "$CONTAINER_EXISTS" != "" ]; then
 else
 	trap "echo 'got CTRL+C... please wait 5s'; docker stop -t 5 ${CONTAINER_NAME}" SIGINT SIGTERM
 	$DOCKER run --name "${CONTAINER_NAME}" --privileged \
-		-v $(pwd)/deploy:/pi-gen/deploy \
-		${config_mount} \
+		-e IMG_NAME=${IMG_NAME}\
+		-v "$(pwd)/deploy:/pi-gen/deploy" \
+		-v "${config_mount}" \
 		pi-gen \
 		bash -e -o pipefail -c "dpkg-reconfigure qemu-user-static &&
 	cd /pi-gen; ./build.sh &&

build.sh

@@ -128,7 +128,7 @@ if [ -z "${IMG_NAME}" ]; then
 	exit 1
 fi
 
-export IMG_DATE=${IMG_DATE:-"$(date -u +%Y-%m-%d)"}
+export IMG_DATE=${IMG_DATE:-"$(date +%Y-%m-%d)"}
 
 export BASE_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
 export SCRIPT_DIR="${BASE_DIR}/scripts"

export-image/04-finalise/01-run.sh

@@ -35,6 +35,8 @@ rm -f ${ROOTFS_DIR}/var/lib/dbus/machine-id
 
 true > ${ROOTFS_DIR}/etc/machine-id
 
+ln -nsf /proc/mounts ${ROOTFS_DIR}/etc/mtab
+
 for _FILE in $(find ${ROOTFS_DIR}/var/log/ -type f); do
 	true > ${_FILE}
 done

export-image/prerun.sh

@@ -8,12 +8,12 @@ rm -f ${IMG_FILE}
 rm -rf ${ROOTFS_DIR}
 mkdir -p ${ROOTFS_DIR}
 
-BOOT_SIZE=$(du -s ${EXPORT_ROOTFS_DIR}/boot --block-size=1 | cut -f 1)
-TOTAL_SIZE=$(du -s ${EXPORT_ROOTFS_DIR} --exclude var/cache/apt/archives --block-size=1 | cut -f 1)
+BOOT_SIZE=$(du --apparent-size -s ${EXPORT_ROOTFS_DIR}/boot --block-size=1 | cut -f 1)
+TOTAL_SIZE=$(du --apparent-size -s ${EXPORT_ROOTFS_DIR} --exclude var/cache/apt/archives --block-size=1 | cut -f 1)
 
-IMG_SIZE=$((BOOT_SIZE + TOTAL_SIZE + (400 * 1024 * 1024)))
+IMG_SIZE=$((BOOT_SIZE + TOTAL_SIZE + (800 * 1024 * 1024)))
 
-fallocate -l ${IMG_SIZE} ${IMG_FILE}
+truncate -s ${IMG_SIZE} ${IMG_FILE}
 fdisk -H 255 -S 63 ${IMG_FILE} <<EOF
 o
 n
@@ -51,7 +51,7 @@ echo "/boot: offset $BOOT_OFFSET, length $BOOT_LENGTH"
 echo "/:     offset $ROOT_OFFSET, length $ROOT_LENGTH"
 
 mkdosfs -n boot -F 32 -v $BOOT_DEV > /dev/null
-mkfs.ext4 -O ^huge_file $ROOT_DEV > /dev/null
+mkfs.ext4 -O ^metadata_csum,^huge_file $ROOT_DEV > /dev/null
 
 mount -v $ROOT_DEV ${ROOTFS_DIR} -t ext4
 mkdir -p ${ROOTFS_DIR}/boot

export-noobs/00-release/files/os.json

@@ -1,7 +1,7 @@
 {
     "description": "NOOBS_DESCRIPTION",
     "feature_level": 35120124,
-    "kernel": "4.4",
+    "kernel": "4.9",
     "name": "NOOBS_NAME",
     "password": "raspberry",
     "release_date": "UNRELEASED",

export-noobs/00-release/files/release_notes.txt

@@ -1,4 +1,18 @@
 UNRELEASED:
+  * 
+2017-07-05:
+  * New kernel and firmware
+  * Filesystem created without the metadata_csum feature
+2017-06-21:
+  * Scratch 2 application included
+  * Thonny Python IDE included
+  * New icons with thinner outlines
+  * Volume control more linear in behaviour
+  * Updated Flash player
+  * Updated RealVNC server and viewer
+  * Various tweaks and bugfixes
+  * New kernel and firmware
+2017-04-10:
   * Wolfram Mathematica updated to version 11.0.1
   * Adobe Flash Player updated to version 25.0.0.127
   * Use PARTUUID to support USB boot

scripts/common

@@ -16,9 +16,9 @@ bootstrap(){
 	fi
 
 	capsh --drop=cap_setfcap -- -c "${BOOTSTRAP_CMD} --components=main,contrib,non-free \
-		--arch armhf\
-		--no-check-gpg \
-		$1 $2 $3"
+		--arch armhf \
+		--keyring "${STAGE_DIR}/files/raspberrypi.gpg" \
+		$1 $2 $3" || rmdir "$2/debootstrap"
 }
 export -f bootstrap