The test set was mistakenly used as both the validation set and testing set !?

[Minqi824]

According to the scripts (e.g., scripts/exp2-odds/run_anollm.sh), AnoLLM is first trained by the train_anollm.py and then evaluated by the evaluate_anollm.py. However:

• In train_anollm.py, the training & testing set are generated via the load_data function, while the X_test & y_test are used as validation data (data_val=X_test, label_val=y_test) to perform early stopping.

• However, X_test is used again in evaluate_anollm.py to generate testing anomaly score, in order to acquire model's performance in corresponding dataset. In other word, the testing data is regraded as both validation set and testing set? Did I misunderstand something there? Thanks

[chepingt]

Thank you for your question, and apologies for the delayed response.

To clarify, we do not use the validation set for hyperparameter tuning or early stopping, given the unsupervised nature of the task. Instead, we adopt the fixed hyperparameters reported in the original AnoLLM paper, which are kept consistent across all datasets.

The only adjustments we make are to parameters such as batch size or whether to use LoRA, which are determined based on hardware constraints (e.g., GPU memory availability). Additionally, for some text-based datasets where the training loss converges more slowly, we set a higher value for max_steps to ensure sufficient training.

Importantly, we have observed that once the training loss converges, the test performance tends to remain stable. Therefore, we suggest that users determine the number of training epochs or steps based on the convergence of the training loss, rather than relying on validation performance.

We appreciate your thoughtful question and hope this helps clarify the design choices.