Command-line test selection/rejection

.github/issue_template.md

@@ -1,6 +1,6 @@
 ## System
 
-> Feel free to delete this section if you're submitting a feature request
+<!-- Feel free to delete this section if you're submitting a feature request -->
 
 - OS: _(Travis/OSX/Linux/Windows)_
 - `ruby -v`:

.github/pull_request_template.md

@@ -2,7 +2,8 @@
 
 * See CHANGELOG.md for more
 
-
+<!--
 ## Issues Fixed
 
 * Fixes #3000
+-->

CHANGELOG.md

@@ -7,6 +7,10 @@ and this project adheres to [Semantic Versioning](http://semver.org/).
 ## [Unreleased]
 ### Added
 - Provide an `itoa` function. It is present in Arduino's runtime environment but not on most (all?) host systems because itoa is not a portable standard function.
+- `to_h` and `to_s` functions for `ci_config.rb`
+- `CIConfig::clone`
+- Ability to override `CIConfig` from a hash instead of just a file
+- `arduino_ci_remote.rb` now supports command line switches `--testfile-select=GLOB` and `--testfile-reject=GLOB` (which can both be repeated)
 
 ### Changed
 - Simplified the use of `Array.each` with a return statement; it's now simply `Array.find`
@@ -19,6 +23,7 @@ and this project adheres to [Semantic Versioning](http://semver.org/).
 
 ### Fixed
 - Determining a working OSX launch command no longer breaks on non-English installations
+- `arduino_ci_remote.rb` now honors selected and rejected test files
 
 ### Security
 

README.md

@@ -34,6 +34,25 @@ source 'https://rubygems.org'
 gem 'arduino_ci'
 ```
 
+### Testing Locally
+
+First, pull in the `arduino_ci` library as a dependency.
+
+```
+$ bundle install
+```
+
+
+With that installed, just the following shell command each time you want the tests to execute:
+
+```
+$ bundle exec arduino_ci_remote.rb
+```
+
+
+
+### Testing with remote CI
+
 > **Note:** `arduino_ci_remote.rb` expects to be run from the root directory of your Arduino project library.
 
 

REFERENCE.md

@@ -9,6 +9,28 @@ These defaults are specified in [misc/default.yml](misc/default.yml).  You are f
 
 ## Overriding default build behavior
 
+### From the command line
+
+The following options are currently available in the `arduino_ci_remote.rb` test runner.
+
+```
+Usage: arduino_ci_remote.rb [options]
+        --testfile-select=GLOB       Unit test file (or glob) to select
+        --testfile-reject=GLOB       Unit test file (or glob) to reject
+    -h, --help                       Prints this help
+```
+
+#### `--testfile-select` option
+
+This allows a file (or glob) pattern to be executed in your tests directory, creating a whitelist of files to test.  E.g. `--testfile-select=test_animal_*.cpp` would match `test_animal_cat.cpp` and `test_animal_dog.cpp` (testing only those) and not `test_plant_rose.cpp`.
+
+#### `--testfile-reject` option
+
+This allows a file (or glob) pattern to be executed in your tests directory, creating a blacklist of files to skip.  E.g. `--testfile-reject=test_animal_*.cpp` would match `test_animal_cat.cpp` and `test_animal_dog.cpp` (skipping those) and test only `test_plant_rose.cpp`, `test_plant_daisy.cpp`, etc.
+
+
+### From configuration
+
 `.arduino-ci.yml` files will override the default behavior.  There are 3 places you can put them:
 
 1. the root of your library

exe/arduino_ci_remote.rb

@@ -2,13 +2,49 @@
 require 'arduino_ci'
 require 'set'
 require 'pathname'
+require 'optparse'
 
 WIDTH = 80
 FIND_FILES_INDENT = 4
 
 @failure_count = 0
 @passfail = proc { |result| result ? "✓" : "✗" }
 
+# Use some basic parsing to allow command-line overrides of config
+class Parser
+  def self.parse(options)
+    parsed_config = {}
+    parsed_config["unittest"] = {}
+
+    opt_parser = OptionParser.new do |opts|
+      opts.banner = "Usage: #{File.basename(__FILE__)} [options]"
+
+      opts.on("--testfile-select=GLOB", "Unit test file (or glob) to select") do |p|
+        parsed_config["unittest"]["testfiles"] ||= {}
+        parsed_config["unittest"]["testfiles"]["select"] ||= []
+        parsed_config["unittest"]["testfiles"]["select"] << p
+      end
+
+      opts.on("--testfile-reject=GLOB", "Unit test file (or glob) to reject") do |p|
+        parsed_config["unittest"]["testfiles"] ||= {}
+        parsed_config["unittest"]["testfiles"]["reject"] ||= []
+        parsed_config["unittest"]["testfiles"]["reject"] << p
+      end
+
+      opts.on("-h", "--help", "Prints this help") do
+        puts opts
+        exit
+      end
+    end
+
+    opt_parser.parse!(options)
+    parsed_config
+  end
+end
+
+# Read in command line options and make them read-only
+@cli_options = (Parser.parse ARGV).freeze
+
 # terminate after printing any debug info.  TODO: capture debug info
 def terminate(final = nil)
   puts "Failures: #{@failure_count}"
@@ -28,6 +64,12 @@ def terminate(final = nil)
 #   without altering the signature because it only leaves space
 #   for the checkmark _after_ the multiline, it doesn't know how
 #   to make that conditionally the body
+# @param message String the text of the progress indicator
+# @param multiline boolean whether multiline output is expected
+# @param mark_fn block (string) -> string that says how to describe the result
+# @param on_fail_msg String custom message for failure
+# @param tally_on_fail boolean whether to increment @failure_count
+# @param abort_on_fail boolean whether to abort immediately on failure (i.e. if this is a fatal error)
 def perform_action(message, multiline, mark_fn, on_fail_msg, tally_on_fail, abort_on_fail)
   line = "#{message}... "
   endline = "...#{message} "
@@ -111,7 +153,10 @@ def display_files(pathname)
   non_hidden.each { |p| puts "#{margin}#{p}" }
 end
 
-def perform_unit_tests(config)
+def perform_unit_tests(file_config)
+  puts file_config.to_h[:unittest].to_s
+  config = file_config.with_override_config(@cli_options)
+  puts config.to_h[:unittest].to_s
   cpp_library = ArduinoCI::CppLibrary.new(Pathname.new("."), @arduino_cmd.lib_dir)
 
   # check GCC
@@ -149,7 +194,7 @@ def perform_unit_tests(config)
     inform("Skipping unit tests") { "no platforms were requested" }
   else
     config.platforms_to_unittest.each do |p|
-      cpp_library.test_files.each do |unittest_path|
+      config.allowable_unittest_files(cpp_library.test_files).each do |unittest_path|
         unittest_name = unittest_path.basename.to_s
         compilers.each do |gcc_binary|
           attempt_multiline("Unit testing #{unittest_name} with #{gcc_binary}") do

lib/arduino_ci/ci_config.rb

@@ -72,6 +72,21 @@ def initialize
       @unittest_info = {}
     end
 
+    # @return [Hash] config data as a hash
+    def to_h
+      {
+        packages: @package_info,
+        platforms: @platform_info,
+        compile: @compile_info,
+        unittest: @unittest_info
+      }
+    end
+
+    # @return [String] config data as a string
+    def to_s
+      to_h.to_s
+    end
+
     # Deep-clone a hash
     # @param hash [Hash] the source data
     # @return [Hash] a copy
@@ -112,6 +127,13 @@ def load_yaml(path)
       yml = YAML.load_file(path)
       raise ConfigurationError, "The YAML file at #{path} failed to load" unless yml
 
+      apply_configuration(yml)
+    end
+
+    # Load configuration from a hash
+    # @param yml [Hash] the source data
+    # @return [ArduinoCI::CIConfig] a reference to self
+    def apply_configuration(yml)
       if yml.include?("packages")
         yml["packages"].each do |k, v|
           valid_data = validate_data("packages", v, PACKAGE_SCHEMA)
@@ -139,19 +161,35 @@ def load_yaml(path)
       self
     end
 
+    # Create a clone of this configuration and return it
+    # @return [ArduinoCI::CIConfig] the new settings object
+    def clone
+      cloned_config = self.class.new
+      cloned_config.package_info  = deep_clone(@package_info)
+      cloned_config.platform_info = deep_clone(@platform_info)
+      cloned_config.compile_info  = deep_clone(@compile_info)
+      cloned_config.unittest_info = deep_clone(@unittest_info)
+      cloned_config
+    end
+
     # Override these settings with settings from another file
     # @param path [String] the path to the settings yaml file
     # @return [ArduinoCI::CIConfig] the new settings object
     def with_override(path)
-      overridden_config = self.class.new
-      overridden_config.package_info  = deep_clone(@package_info)
-      overridden_config.platform_info = deep_clone(@platform_info)
-      overridden_config.compile_info  = deep_clone(@compile_info)
-      overridden_config.unittest_info = deep_clone(@unittest_info)
+      overridden_config = clone
       overridden_config.load_yaml(path)
       overridden_config
     end
 
+    # Override these settings with settings from a hash
+    # @param config_hash [Hash] A configuration hash
+    # @return [ArduinoCI::CIConfig] the new settings object
+    def with_override_config(config_hash)
+      overridden_config = clone
+      overridden_config.apply_configuration(config_hash)
+      overridden_config
+    end
+
     # Get the config file at a given path, if it exists, and pass that to a block.
     # Many config files may exist, but only the first match is used
     # @param base_dir [String] The directory in which to search for a config file

spec/ci_config_spec.rb

@@ -39,6 +39,46 @@
     end
   end
 
+  context "clone" do
+    it "creates a copy" do
+      base = ArduinoCI::CIConfig.new
+      base.load_yaml(File.join(File.dirname(__FILE__), "yaml", "o2.yaml"))
+
+      expect(base.to_h).to eq(
+        packages: {},
+        platforms: {
+          "bogo"=> {
+            board: "fakeduino:beep:bogo"
+          },
+        },
+        compile: {
+          libraries: ["zip"],
+          platforms: ["bogo"]
+        },
+        unittest: {
+          testfiles: {
+            select: ["*-*.*"],
+            reject: ["sam-squamsh.*"]
+          },
+          libraries: ["def456"],
+          platforms: ["bogo"]
+        }
+      )
+    end
+  end
+
+  context "clone" do
+    it "creates a copy" do
+      base = ArduinoCI::CIConfig.default
+      orig = base.to_h
+      clone1 = orig.clone.to_h
+      clone2 = orig.clone.to_h
+
+      expect(orig).to eq(clone1)
+      expect(clone1).to eq(clone2)
+    end
+  end
+
   context "with_override" do
     it "loads from yaml" do
       override_file = File.join(File.dirname(__FILE__), "yaml", "o1.yaml")

spec/yaml/o2.yaml

@@ -0,0 +1,20 @@
+platforms:
+  bogo:
+    board: fakeduino:beep:bogo
+
+compile:
+  libraries:
+    - "zip"
+  platforms:
+    - bogo
+
+unittest:
+  testfiles:
+    select:
+      - "*-*.*"
+    reject:
+      - "sam-squamsh.*"
+  libraries:
+    - "def456"
+  platforms:
+    - bogo