tools/check: Add suite (message content) parser.

Some main benefits of adding both tools/content scripts to tools/check
are that you don't need to manage your directories for storing the
zuliprc files and corpuses, or specify any command line options for
fetching message contents and running the unimplemented features test.

The script is intended to be run manually, not as a part of the CI,
because it is very slow, and it relies on some out of tree files like
API configs (zuliprc files) and big dumps of chat history.

Fixes: zulip#190

Signed-off-by: Zixuan James Li <[email protected]>

Author: PIG208

.gitignore

@@ -53,3 +53,7 @@ app.*.map.json
 
 # Old scaffolding hack
 lib/credential_fixture.dart
+
+# Directories used by `tools/check parser`
+.corpuses
+.api_configs

tools/check

@@ -82,7 +82,7 @@ while (( $# )); do
         --all) opt_files=all; opt_all=1; shift;;
         --fix) opt_fix=1; shift;;
         --verbose) opt_verbose=1; shift;;
-        analyze|test|build_runner|drift|pigeon|icons|android|shellcheck)
+        analyze|test|build_runner|drift|pigeon|icons|android|shellcheck|parser)
             opt_suites+=("$1"); shift;;
         *) usage;;
     esac
@@ -416,6 +416,48 @@ EOF
     shellcheck -x --shell=bash -- "${targets[@]}"
 }
 
+run_parser() {
+    # This is needed to check if there is no match for the
+    # `config_files` and `corpus_files` globs.
+    shopt -s nullglob
+
+    local config_dir=.api_configs
+    local config_files=("$config_dir"/*)
+
+    # `tools/content/fetch_messages.dart` will create corpus files in
+    # `corpus_dir` for each host from the zuliprc files, which is needed for
+    # running `tools/content/unimplemented_features_test.dart`.
+    local corpus_dir=.corpuses
+    for config in "${config_files[@]}"; do
+        echo "Fetching all public messages using API config \"$config\"."\
+             "This can take a long time."
+        tools/content/fetch_messages.dart --config-file "$config" \
+            --corpus-dir $corpus_dir \
+        || return 1
+    done
+    local corpus_files=("$corpus_dir"/*)
+
+    if (( ! "${#config_files[@]}" )) && (( ! ${#corpus_files[@]} )); then
+        cat >&2 <<EOF
+tools/check parser: bad configuration
+
+Running checks for unimplemented features requires either zuliprc files in
+$config_dir to fetch corpuses or existing corpus files in $corpus_dir,
+but both directories are found non-existent or empty.
+
+Consider downloading zuliprc to $config_dir:
+  https://zulip.com/api/configuring-python-bindings#download-a-zuliprc-file
+or copying pre-fetched corpuses to $corpus_dir.
+EOF
+        return 1
+    fi
+
+    flutter test tools/content/unimplemented_features_test.dart \
+        --dart-define=corpusDir=$corpus_dir \
+        --dart-define=verbose=$opt_verbose \
+    || return 1
+}
+
 describe_git_head() {
     local name="$1" repo_path="$2"
     local commit_data
@@ -461,6 +503,7 @@ for suite in "${opt_suites[@]}"; do
         icons)        run_icons ;;
         android)      run_android ;;
         shellcheck)   run_shellcheck ;;
+        parser)       run_parser ;;
         *)            echo >&2 "Internal error: unknown suite $suite" ;;
     esac || failed+=( "$suite" )
 done

tools/content/fetch_messages.dart

@@ -21,6 +21,10 @@ import 'model.dart';
 /// Because message IDs are only unique within a single server, the script
 /// names corpuses from each server differently (if --corpus-dir is specified).
 ///
+/// This script is meant to be run via `tools/check parser`.
+///
+/// For more help, run `tools/content/fetch_message.dart --help`.
+///
 /// See tools/content/unimplemented_features_test.dart for more details.
 void main(List<String> args) async {
   final argParser = ArgParser();

tools/content/unimplemented_features_test.dart

@@ -16,9 +16,9 @@ import 'model.dart';
 /// Check if there are unimplemented features from the given corpuses of HTML
 /// contents from Zulip messages.
 ///
-/// This test is meant to be manually run.
+/// This test is meant to be run via `tools/check parser`.
 ///
-/// To run it, use:
+/// To run it directly, use:
 ///
 ///   flutter test tools/content --dart-define=corpusDir=path/to/corpusDir
 ///