From d022b96ea2a4fef4e86ae9ef283b9551365c02f8 Mon Sep 17 00:00:00 2001 From: Tony Green Date: Wed, 8 Aug 2018 07:47:39 +1000 Subject: [PATCH 1/2] Documentation, changelog and metadata updates prior to release --- CHANGELOG.md | 9 ++++ README.md | 126 ++++++++++++++++++++++++++++++++++++---------- manifests/init.pp | 55 ++++++++++++++++++-- metadata.json | 2 +- 4 files changed, 161 insertions(+), 31 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 046fa081..5fd2ad96 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,14 @@ # Changelog +## Release 0.2.1 + +- Major rewrite on most areas of the module +- Patching task now written in ruby and has much better reporting and error handling +- Facter is now ruby based but still uses cache data from a cron job +- Blackout window functionality included +- Documentation updates +- SHOULD be backwards compatible + ## Release 0.1.19 - Bugfix on the debian task which caused incorrect reporting of patching status diff --git a/README.md b/README.md index 1f334dd8..bbe7731a 100644 --- a/README.md +++ b/README.md @@ -5,17 +5,7 @@ This module contains a set of tasks and custom facts to allow the automation of ## Table of contents -- [os_patching](#ospatching) - - [Table of contents](#table-of-contents) - - [Description](#description) - - [Setup](#setup) - - [What os_patching affects](#what-ospatching-affects) - - [Beginning with os_patching](#beginning-with-ospatching) - - [Usage](#usage) - - [Reference](#reference) - - [Limitations](#limitations) - - [Development](#development) - - [Contributors](#contributors) + ## Description @@ -25,11 +15,13 @@ If you're looking for a simple way to report on your OS patch levels, this modul It also uses security metadata (where available) to determine if there are security updates. On Redhat, this is provided from Redhat as additional metadata in YUM. On Debian, checks are done for which repo the updates are coming from. There is a parameter to the task to only apply security updates. +Blackout windows enable the support for time based change freezes where no patching can happen. There can be multiple windows defined and each which will automatically expire after reaching the defined end date. + ## Setup ### What os_patching affects -The module, when added to a node, installs a script to generate the patching facts and a cron job to run the script. +The module provides an additional fact (`os_patching`) and has a task to allow the patching of a server. When the `os_patching` manifest is added to a node it installs a script and cron job to generate cache data used by the `os_patching` fact. ### Beginning with os_patching @@ -37,27 +29,62 @@ Install the module using the Puppetfile, include it on your nodes and then use t ## Usage +### Manifest Include the module: ```puppet include os_patching ``` -Run a basic patching task from the command line: -```bash -# puppet task show os_patching::patch_server +More advanced usage: +```puppet +class { 'os_patching': + patch_window => 'Week3', + reboot_override => true, + blackout_windows => { 'End of year change freeze': + { + 'start': '2018-12-15T00:00:00+1000', + 'end': '2019-01-15T23:59:59+1000', + } + }, +} +``` -os_patching::patch_server - Carry out OS patching on the server, optionally including a reboot +In that example, the node is assigned to a "patch window", will be forced to reboot regardless of the setting specified in the task and has a blackout window defined for the period of 2018-12-15 - 2019-01-15, during which time no patching through the task can be carried out. + +### Task +Run a basic patching task from the command line: +``` +os_patching::patch_server - Carry out OS patching on the server, optionally including a reboot and/or only applying security related updates USAGE: -$ puppet task run os_patching::patch_server reboot= security_only= <[--nodes, -n ] | [--query, -q <'query'>]> +$ puppet task run os_patching::patch_server [dpkg_params=] [reboot=] [security_only=] [timeout=] [yum_params=] <[--nodes, -n ] | [--query, -q <'query'>]> PARAMETERS: -- reboot : Boolean - Should the server reboot after patching has been applied? -- security_only : Boolean - Limit patches to those tagged as security related? +- dpkg_params : Optional[String] + Any additional parameters to include in the dpkg command +- reboot : Optional[Boolean] + Should the server reboot after patching has been applied? (Defaults to false) +- security_only : Optional[Boolean] + Limit patches to those tagged as security related? (Defaults to false) +- timeout : Optional[Integer] + How many seconds should we wait until timing out the patch run? (Defaults to 3600 seconds) +- yum_params : Optional[String] + Any additional parameters to include in the yum upgrade command (such as including/excluding repos) ``` +Example: +```bash +$ puppet task run os_patching::patch_server reboot=true security_only=false --query="inventory[certname] { facts.os_patching.patch_window = 'Week3' and facts.os_patching.blocked = false and facts.os_patching.package_update_count > 0}" +``` + +This will run a patching task against all nodes which have facts matching: + +* `os_patching.patch_window` of 'Week3' +* `os_patching.blocked` equals `false` +* `os_patching.package_update_count` greater than 0 + +The task will apply all patches (`security_only=false`) and will reboot the node after patching (`reboot=true`). + ## Reference ### Facts @@ -65,16 +92,32 @@ PARAMETERS: Most of the reporting is driven off the custom fact `os_patching_data`, for example: ```yaml -# facter os_patching +# facter -p os_patching { - package_update_count => 0 + package_update_count => 0, + package_updates => [], + security_package_updates => [], + security_package_update_count => 0, + blocked => false, + blocked_reasons => [], + blackouts => {}, + patch_window = 'Week3', + pinned_packages => [], + last_run => { + date => "2018-08-07T21:55:20+10:00", + message => "Patching complete", + return_code => "Success", + post_reboot => "false", + security_only => "false", + job_id => "60" + } } ``` This shows there are no updates which can be applied to this server. When there are updates to add, you will see similar to this: ```yaml -# facter os_patching +# facter -p os_patching { package_update_count => 6, package_updates => [ @@ -85,10 +128,40 @@ This shows there are no updates which can be applied to this server. When there "procps-ng.x86_64", "python-perf.x86_64" ] + security_package_updates => [], + security_package_update_count => 0, + blocked => false, + blocked_reasons => [], + blackouts => { + Test change freeze 2 => { + start => "2018-08-01T09:17:10+1000", + end => "2018-08-01T11:15:50+1000" + } + }, + pinned_packages => [], + patch_window = 'Week3', + last_run => { + date => "2018-08-07T21:55:20+10:00", + message => "Patching complete", + return_code => "Success", + post_reboot => "false", + security_only => "false", + job_id => "60" + } } ``` -Where it shows 6 packages with available updates, along with an array of the package names. +Where it shows 6 packages with available updates, along with an array of the package names. None of the packges are tagged as security related (requires Debian or a subscription to RHEL). There are no blockers to patching and the blackout window defined is not in effect. + +The pinned packages entry lists any packages which have been specifically excluded from being patched, from [version lock](https://access.redhat.com/solutions/98873) on Red Hat or by [pinning](https://wiki.debian.org/AptPreferences) in Debian. + +Last run shows a summary of the information from the last `os_patching::patch_server` task. + +The fact `os_patching.patch_window` can be used to assign nodes to an arbitrary group. The fact can be used as part of the query fed into the task to determine which nodes to patch: + +```bash +$ puppet task run os_patching::patch_server --query="inventory[certname] {facts.os_patching.patch_window = 'Week3'}" +``` ### Task output @@ -97,7 +170,6 @@ If there is nothing to be done, the task will report: ```puppet { "date" : "Mon May 28 12:23:33 AEST 2018", - "fqdn" : "example.lab.vm", "reboot" : "true", "message" : "yum dry run shows no patching work to do", "return-code" : "success", @@ -110,7 +182,6 @@ If patching was executed, the task will report: ```puppet { "date": "Mon May 28 12:29:23 AEST 2018", - "fqdn": "example.lab.vm", "reboot": "true", "message": "Patching complete", "return-code": "Success", @@ -125,6 +196,7 @@ If patching was executed, the task will report: } ``` +A summary of the patch run is also written to `/etc/os_patching/run_history`, the last line of which is used by the `os_patching.last_run` fact. ## Limitations diff --git a/manifests/init.pp b/manifests/init.pp index 1a2f2333..4a414c70 100644 --- a/manifests/init.pp +++ b/manifests/init.pp @@ -1,6 +1,55 @@ -# @summary This manifest sets up the script and cron job to write -# custom structured facts that contain patching data. -# @example include os_patching +# @summary This manifest sets up a script and cron job to populate +# the `os_patching` fact. +# +# @param [String] patch_data_owner User name for the owner of the patch data +# @param [String] patch_data_group Group name for the owner of the patch data +# @param [String] patch_cron_user User name to run the cron job as (defaults to patch_data_owner) +# @param [Boolean] install_delta_rpm Should the deltarpm package be installed on RedHat family nodes +# @param [Boolean] reboot_override Controls on a node level if a reboot should/should not be done after patching. +# This overrides the setting in the task +# @param [Hash] blackout_windows A hash containing the patch blackout windows, which prevent patching +# @param [String] patch_window A freeform text entry used to allocate a node to a specific patch window (Optional) +# @param [String] patch_cron_hour The hour(s) for the cron job to run (defaults to absent, which means '*' in cron) +# @param [String] patch_cron_month The month(s) for the cron job to run (defaults to absent, which means '*' in cron) +# @param [String] patch_cron_monthday The monthday(s) for the cron job to run (defaults to absent, which means '*' in cron) +# @param [String] patch_cron_weekday The weekday(s) for the cron job to run (defaults to absent, which means '*' in cron) +# @param [String] patch_cron_min The min(s) for the cron job to run (defaults to a random number between 0 and 59) +# +# @example assign node to 'Week3' patching window, force a reboot and create a blackout window for the end of the year +# class { 'os_patching': +# patch_window => 'Week3', +# reboot_override => true, +# blackout_windows => { 'End of year change freeze': +# { +# 'start': '2018-12-15T00:00:00+1000', +# 'end': '2019-01-15T23:59:59+1000', +# } +# }, +# } +# +# @example An example profile to setup patching, sourcing blackout windows from hiera +# class profiles::soe::patching ( +# $patch_window = undef, +# $blackout_windows = undef, +# $reboot_override = undef, +# ){ +# # Pull any blackout windows out of hiera +# $hiera_blackout_windows = lookup('profiles::soe::patching::blackout_windows',Hash,hash,{}) +# +# # Merge the blackout windows from the parameter and hiera +# $full_blackout_windows = $hiera_blackout_windows + $blackout_windows +# +# # Call the os_patching class to set everything up +# class { 'os_patching': +# patch_window => $patch_window, +# reboot_override => $reboot_override, +# blackout_windows => $full_blackout_windows, +# } +# } +# +# @example JSON hash to specify a change freeze from 2018-12-15 to 2019-01-15 +# {"End of year change freeze": {"start": "2018-12-15T00:00:00+1000", "end": "2019-01-15T23:59:59+1000"}} +# class os_patching ( String $patch_data_owner = 'root', String $patch_data_group = 'root', diff --git a/metadata.json b/metadata.json index 77916d96..ad2b296c 100644 --- a/metadata.json +++ b/metadata.json @@ -1,6 +1,6 @@ { "name": "albatrossflavour-os_patching", - "version": "0.1.19", + "version": "0.2.1", "author": "Tony Green", "summary": "Perform OS level patching through tasks and report on patch levels through facts", "license": "Apache-2.0", From 91c03266224a3dc27e7e112580207c1ba51f6f8e Mon Sep 17 00:00:00 2001 From: Tony Green Date: Wed, 8 Aug 2018 08:11:55 +1000 Subject: [PATCH 2/2] Final doco updates --- README.md | 77 +++++++++++++++++++++++++++++++++-------------- manifests/init.pp | 3 +- 2 files changed, 56 insertions(+), 24 deletions(-) diff --git a/README.md b/README.md index bbe7731a..0636231e 100644 --- a/README.md +++ b/README.md @@ -3,9 +3,7 @@ This module contains a set of tasks and custom facts to allow the automation of and reporting on operating system patching, currently restricted to Redhat and Debian derivatives. -## Table of contents - - +Under the hood it uses the OS level tools to carry out the actual patching. ## Description @@ -167,41 +165,74 @@ $ puppet task run os_patching::patch_server --query="inventory[certname] {facts. If there is nothing to be done, the task will report: -```puppet +```json { - "date" : "Mon May 28 12:23:33 AEST 2018", - "reboot" : "true", - "message" : "yum dry run shows no patching work to do", - "return-code" : "success", - "securityonly" : "false" + "pinned_packages" : [ ], + "security" : false, + "return" : "Success", + "start_time" : "2018-08-08T07:52:28+10:00", + "debug" : "", + "end_time" : "2018-08-08T07:52:46+10:00", + "reboot" : false, + "packages_updated" : "", + "job_id" : "", + "message" : "No patches to apply" } ``` -If patching was executed, the task will report: +If patching was executed, the task will report similar to below: -```puppet +```json { - "date": "Mon May 28 12:29:23 AEST 2018", - "reboot": "true", - "message": "Patching complete", - "return-code": "Success", - "packagesupdated": [ - "kernel-tools-3.10.0-862.2.3.el7.x86_64", - "kernel-tools-libs-3.10.0-862.2.3.el7.x86_64", - "postfix-2:2.10.1-6.el7.x86_64", - "procps-ng-3.3.10-17.el7.x86_64", - "python-perf-3.10.0-862.2.3.el7.x86_64" - ], - "securityonly": "false" + "pinned_packages" : [ ], + "security" : false, + "return" : "Success", + "start_time" : "2018-08-07T21:55:20+10:00", + "debug" : "TRIMMED DUE TO LENGTH FOR THIS EXAMPLE, WOULD NORMALLY CONTAIN FULL COMMAND OUTPUT", + "end_time" : "2018-08-07T21:57:11+10:00", + "reboot" : false, + "packages_updated" : [ "NetworkManager-1:1.10.2-14.el7_5.x86_64", "NetworkManager-libnm-1:1.10.2-14.el7_5.x86_64", "NetworkManager-team-1:1.10.2-14.el7_5.x86_64", "NetworkManager-tui-1:1.10.2-14.el7_5.x86_64", "binutils-2.27-27.base.el7.x86_64", "centos-release-7-5.1804.el7.centos.2.x86_64", "git-1.8.3.1-13.el7.x86_64", "gnupg2-2.0.22-4.el7.x86_64", "kernel-tools-3.10.0-862.3.3.el7.x86_64", "kernel-tools-libs-3.10.0-862.3.3.el7.x86_64", "perl-Git-1.8.3.1-13.el7.noarch", "python-2.7.5-68.el7.x86_64", "python-libs-2.7.5-68.el7.x86_64", "python-perf-3.10.0-862.3.3.el7.centos.plus.x86_64", "selinux-policy-3.13.1-192.el7_5.3.noarch", "selinux-policy-targeted-3.13.1-192.el7_5.3.noarch", "sudo-1.8.19p2-13.el7.x86_64", "yum-plugin-fastestmirror-1.1.31-45.el7.noarch", "yum-utils-1.1.31-45.el7.noarch" ], + "job_id" : "60", + "message" : "Patching complete" } ``` +If patching was blocked, the task will report similar to below: + +```json +Error: Task exited : 100 +Patching blocked +``` A summary of the patch run is also written to `/etc/os_patching/run_history`, the last line of which is used by the `os_patching.last_run` fact. +``` +2018-08-07T14:47:24+10:00|No patches to apply|Success|false|false| +2018-08-07T14:56:56+10:00|Patching complete|Success|false|false|121 +2018-08-07T15:04:42+10:00|yum timeout after 2 seconds : Loaded plugins: versionlock|1||| +2018-08-07T15:05:51+10:00|yum timeout after 3 seconds : Loaded plugins: versionlock|1||| +2018-08-07T15:10:16+10:00|Patching complete|Success|false|false|127 +2018-08-07T21:31:47+10:00|Patching blocked |100||| +2018-08-08T07:53:59+10:00|Patching blocked |100||| +``` + +### `/etc/os_patching` directory + +This directory contains the various control files needed for the fact and task to work correctly. They are managed by the manifest. + +* `/etc/os_patching/blackout_windows` - contains name, start and end time for all blackout windows +* `/etc/os_patching/package_updates` - a list of all package updates available, populated by `/usr/local/bin/os_patching_fact_generation.sh`, triggered through cron +* `/etc/os_patching/security_package_updates` - a list of all security_package updates available, populated by `/usr/local/bin/os_patching_fact_generation.sh`, triggered through cron +* `/etc/os_patching/run_history` - a summary of each run of the `os_patching::patch_server` task, populated by the task +* `/etc/os_patching/reboot_override` - if present, overrides the `reboot=` parameter to the task +* `/etc/os_patching/patch_window` - if present, sets the value for the fact `os_patching.patch_window` + + ## Limitations This module is for PE2018+ with agents capable of running tasks. It is currently limited to the Red Hat and Debian operating system. +Debian nodes currently do not allow `security_only` patch tasks to be set to `true`, a fix for this is being worked on. + ## Development Fork, develop, submit a pull request diff --git a/manifests/init.pp b/manifests/init.pp index 4a414c70..3699c387 100644 --- a/manifests/init.pp +++ b/manifests/init.pp @@ -7,7 +7,8 @@ # @param [Boolean] install_delta_rpm Should the deltarpm package be installed on RedHat family nodes # @param [Boolean] reboot_override Controls on a node level if a reboot should/should not be done after patching. # This overrides the setting in the task -# @param [Hash] blackout_windows A hash containing the patch blackout windows, which prevent patching +# @param [Hash] blackout_windows A hash containing the patch blackout windows, which prevent patching. +# The dates are in full ISO8601 format. # @param [String] patch_window A freeform text entry used to allocate a node to a specific patch window (Optional) # @param [String] patch_cron_hour The hour(s) for the cron job to run (defaults to absent, which means '*' in cron) # @param [String] patch_cron_month The month(s) for the cron job to run (defaults to absent, which means '*' in cron)