[FEATURE REQUEST]: Support for categoricals in `optimize_acqf_mixed_alternating`

[TobyBoyne]

Motivation

Currently, I don't believe BoTorch supports mixed optimization over mixed spaces with categorical features. I would like to extend optimize_acqf_mixed_alternating to also optimize over such spaces. This would essentially be closing the following TODO:

botorch/botorch/optim/optimize_mixed.py

Line 576 in d247a33

# TODO: Support categorical variables.

Describe the solution you'd like to see implemented in BoTorch.

During the discrete step, this function currently proposes small perturbations in the 1-Manhattan distance ball around the incumbent. We can similarly propose perturbations in the 1-Hamming distance ball around the incumbent for categorical features. That is, we consider two points to be neighbours if all-but-one of their features are the same, and the remaining (categorical) feature takes any two distinct values.

Describe any alternatives you've considered to the above solution.

1. You could enumerate over all possible combinations of categorical variables, then solve the continuous-only problem for each combination. This guarantees the best performance, but is costly, and infeasible in high-dim settings. For example, the popular Pest Control benchmark has 5^25 possible combinations.
2. You could treat the categorical features as ordinal discrete features. However, this imposes order to the categories that could harm the optimization.

Is this related to an existing issue in BoTorch or another repository? If so please include links to those Issues here.

I have a working implementation in my fork of BoTorch, at https://github.com/TobyBoyne/botorch/tree/optimize-cat-alternating. I would be happy to develop the tests, and create a pull request.

Pull Request

Yes

Code of Conduct

• I agree to follow BoTorch's Code of Conduct

[Balandat]

@TobyBoyne this sounds great, we'd love to accept a PR with this functionality. Just submit and we'll review - I peeked at your code and the high level design seems sensible, we can discuss implementation details in the review.

[TobyBoyne]

Added in #2866