Refactor tokenizer test and add to cmake #8450

lucylq · 2025-02-13T04:18:50Z

Summary

Refactor tokenizer test to use env instead of resource path.

Add to cmake tests.

Test plan

build et

./install_executorch.sh

build test

CMAKE_PREFIX_PATH="$(python3 -c 'import torch as _; print(_.__path__[0])')"
  cmake . \
    -DCMAKE_INSTALL_PREFIX=cmake-out \
    -DCMAKE_PREFIX_PATH="${CMAKE_PREFIX_PATH}" \
    -DEXECUTORCH_USE_CPP_CODE_COVERAGE=ON \
    -DEXECUTORCH_BUILD_KERNELS_OPTIMIZED=ON \
    -DEXECUTORCH_BUILD_KERNELS_QUANTIZED=ON \
    -DEXECUTORCH_BUILD_EXTENSION_DATA_LOADER=ON \
    -DEXECUTORCH_BUILD_EXTENSION_FLAT_TENSOR=ON \
    -DEXECUTORCH_BUILD_EXTENSION_LLM=ON \
    -DEXECUTORCH_BUILD_EXTENSION_MODULE=ON \
    -DEXECUTORCH_BUILD_EXTENSION_RUNNER_UTIL=ON \
    -DEXECUTORCH_BUILD_EXTENSION_TENSOR=ON \
    -DEXECUTORCH_BUILD_DEVTOOLS=ON \
    -DEXECUTORCH_BUILD_XNNPACK=ON \
    -DEXECUTORCH_BUILD_TESTS=ON \
    -Bcmake-out
  cmake --build cmake-out -j9 --target install

test

cd cmake-out
ctest -R tokenizer

Test project /data/users/lfq/executorch/cmake-out
    Start 54: extension_llm_tokenizer_test
1/1 Test #54: extension_llm_tokenizer_test .....   Passed    3.66 sec

100% tests passed, 0 tests failed out of 1

Total Test time (real) =   3.67 sec

pytorch-bot · 2025-02-13T04:18:53Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/8450

📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

✅ You can merge normally! (1 Unrelated Failure)

As of commit b1a9127 with merge base 8148603 ():

BROKEN TRUNK - The following job failed but were present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

pull / android / run-emulator (gh) (trunk failure)
The process '/usr/bin/sh' failed with exit code 1

This comment was automatically generated by Dr. CI and updates every 15 minutes.

facebook-github-bot · 2025-02-14T05:18:25Z

@lucylq has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

Summary: Refactor tokenizer test to use env instead of resource path. Add to cmake tests. Test Plan: ## Internal ``` buck2 test fbsource//xplat/executorch/extension/llm/tokenizer/test:test_tiktoken buck2 test fbsource//xplat/executorch/extension/llm/tokenizer/test:test_bpe_tokenizer buck2 test fbcode//executorch/extension/llm/tokenizer/test:test_tiktoken buck2 test fbcode//executorch/extension/llm/tokenizer/test:test_bpe_tokenizer ``` ## OSS build et ``` ./install_executorch.sh ``` build test ``` CMAKE_PREFIX_PATH="$(python3 -c 'import torch as _; print(_.__path__[0])')" cmake . \ -DCMAKE_INSTALL_PREFIX=cmake-out \ -DCMAKE_PREFIX_PATH="${CMAKE_PREFIX_PATH}" \ -DEXECUTORCH_USE_CPP_CODE_COVERAGE=ON \ -DEXECUTORCH_BUILD_KERNELS_OPTIMIZED=ON \ -DEXECUTORCH_BUILD_KERNELS_QUANTIZED=ON \ -DEXECUTORCH_BUILD_EXTENSION_DATA_LOADER=ON \ -DEXECUTORCH_BUILD_EXTENSION_FLAT_TENSOR=ON \ -DEXECUTORCH_BUILD_EXTENSION_LLM=ON \ -DEXECUTORCH_BUILD_EXTENSION_MODULE=ON \ -DEXECUTORCH_BUILD_EXTENSION_RUNNER_UTIL=ON \ -DEXECUTORCH_BUILD_EXTENSION_TENSOR=ON \ -DEXECUTORCH_BUILD_DEVTOOLS=ON \ -DEXECUTORCH_BUILD_XNNPACK=ON \ -DEXECUTORCH_BUILD_TESTS=ON \ -Bcmake-out cmake --build cmake-out -j9 --target install ``` test ``` cd cmake-out ctest -R tokenizer Test project /data/users/lfq/executorch/cmake-out Start 54: extension_llm_tokenizer_test 1/1 Test #54: extension_llm_tokenizer_test ..... Passed 3.66 sec 100% tests passed, 0 tests failed out of 1 Total Test time (real) =--sanitized-- Differential Revision: D69642007 Pulled By: lucylq

facebook-github-bot · 2025-02-14T21:12:46Z

This pull request was exported from Phabricator. Differential Revision: D69642007

Summary: Refactor tokenizer test to use env instead of resource path. Add to cmake tests. Test Plan: ## Internal ``` buck2 test fbsource//xplat/executorch/extension/llm/tokenizer/test:test_tiktoken buck2 test fbsource//xplat/executorch/extension/llm/tokenizer/test:test_bpe_tokenizer buck2 test fbcode//executorch/extension/llm/tokenizer/test:test_tiktoken buck2 test fbcode//executorch/extension/llm/tokenizer/test:test_bpe_tokenizer ``` ## OSS build et ``` ./install_executorch.sh ``` build test ``` CMAKE_PREFIX_PATH="$(python3 -c 'import torch as _; print(_.__path__[0])')" cmake . \ -DCMAKE_INSTALL_PREFIX=cmake-out \ -DCMAKE_PREFIX_PATH="${CMAKE_PREFIX_PATH}" \ -DEXECUTORCH_USE_CPP_CODE_COVERAGE=ON \ -DEXECUTORCH_BUILD_KERNELS_OPTIMIZED=ON \ -DEXECUTORCH_BUILD_KERNELS_QUANTIZED=ON \ -DEXECUTORCH_BUILD_EXTENSION_DATA_LOADER=ON \ -DEXECUTORCH_BUILD_EXTENSION_FLAT_TENSOR=ON \ -DEXECUTORCH_BUILD_EXTENSION_LLM=ON \ -DEXECUTORCH_BUILD_EXTENSION_MODULE=ON \ -DEXECUTORCH_BUILD_EXTENSION_RUNNER_UTIL=ON \ -DEXECUTORCH_BUILD_EXTENSION_TENSOR=ON \ -DEXECUTORCH_BUILD_DEVTOOLS=ON \ -DEXECUTORCH_BUILD_XNNPACK=OFF \ -DEXECUTORCH_BUILD_TESTS=ON \ -Bcmake-out cmake --build cmake-out -j9 --target install ``` test ``` cd cmake-out ctest -R tokenizer Test project /data/users/lfq/executorch/cmake-out Start 54: extension_llm_tokenizer_test 1/1 Test #54: extension_llm_tokenizer_test ..... Passed 3.66 sec 100% tests passed, 0 tests failed out of 1 Total Test time (real) =--sanitized-- Differential Revision: D69642007 Pulled By: lucylq

facebook-github-bot · 2025-02-14T23:38:06Z

This pull request was exported from Phabricator. Differential Revision: D69642007

Summary: Refactor tokenizer test to use env instead of resource path. Add to cmake tests. Test Plan: ## Internal ``` buck2 test fbsource//xplat/executorch/extension/llm/tokenizer/test:test_tiktoken buck2 test fbsource//xplat/executorch/extension/llm/tokenizer/test:test_bpe_tokenizer buck2 test fbcode//executorch/extension/llm/tokenizer/test:test_tiktoken buck2 test fbcode//executorch/extension/llm/tokenizer/test:test_bpe_tokenizer ``` ## OSS build et ``` ./install_executorch.sh ``` build test ``` CMAKE_PREFIX_PATH="$(python3 -c 'import torch as _; print(_.__path__[0])')" cmake . \ -DCMAKE_INSTALL_PREFIX=cmake-out \ -DCMAKE_PREFIX_PATH="${CMAKE_PREFIX_PATH}" \ -DEXECUTORCH_USE_CPP_CODE_COVERAGE=ON \ -DEXECUTORCH_BUILD_KERNELS_OPTIMIZED=ON \ -DEXECUTORCH_BUILD_KERNELS_QUANTIZED=ON \ -DEXECUTORCH_BUILD_EXTENSION_DATA_LOADER=ON \ -DEXECUTORCH_BUILD_EXTENSION_FLAT_TENSOR=ON \ -DEXECUTORCH_BUILD_EXTENSION_LLM=ON \ -DEXECUTORCH_BUILD_EXTENSION_MODULE=ON \ -DEXECUTORCH_BUILD_EXTENSION_RUNNER_UTIL=ON \ -DEXECUTORCH_BUILD_EXTENSION_TENSOR=ON \ -DEXECUTORCH_BUILD_DEVTOOLS=ON \ -DEXECUTORCH_BUILD_XNNPACK=OFF \ -DEXECUTORCH_BUILD_TESTS=ON \ -Bcmake-out cmake --build cmake-out -j9 --target install ``` test ``` cd cmake-out ctest -R tokenizer Test project /data/users/lfq/executorch/cmake-out Start 54: extension_llm_tokenizer_test 1/1 Test #54: extension_llm_tokenizer_test ..... Passed 3.66 sec 100% tests passed, 0 tests failed out of 1 Total Test time (real) =--sanitized-- Differential Revision: D69642007 Pulled By: lucylq

facebook-github-bot · 2025-02-14T23:39:53Z

This pull request was exported from Phabricator. Differential Revision: D69642007

facebook-github-bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Feb 13, 2025

lucylq force-pushed the lfq.cmake-tokenizer-test branch 6 times, most recently from 41b88a2 to 15d4cd0 Compare February 14, 2025 05:14

lucylq changed the title ~~tokenizer~~ Refactor tokenizer test and add to cmake Feb 14, 2025

lucylq force-pushed the lfq.cmake-tokenizer-test branch from 15d4cd0 to 71976c7 Compare February 14, 2025 05:17

lucylq marked this pull request as ready for review February 14, 2025 05:18

lucylq added the release notes: build Changes related to build, including dependency upgrades, build flags, optimizations, etc. label Feb 14, 2025

facebook-github-bot force-pushed the lfq.cmake-tokenizer-test branch from 71976c7 to ddccce2 Compare February 14, 2025 21:12

facebook-github-bot added the fb-exported label Feb 14, 2025

facebook-github-bot force-pushed the lfq.cmake-tokenizer-test branch from ddccce2 to a72e252 Compare February 14, 2025 23:37

facebook-github-bot force-pushed the lfq.cmake-tokenizer-test branch from a72e252 to b1a9127 Compare February 14, 2025 23:39

larryliu0820 approved these changes Feb 15, 2025

View reviewed changes

facebook-github-bot merged commit adf3956 into main Feb 15, 2025
45 of 48 checks passed

facebook-github-bot deleted the lfq.cmake-tokenizer-test branch February 15, 2025 01:26

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Refactor tokenizer test and add to cmake #8450

Refactor tokenizer test and add to cmake #8450

lucylq commented Feb 13, 2025 •

edited

Loading

pytorch-bot bot commented Feb 13, 2025 •

edited

Loading

facebook-github-bot commented Feb 14, 2025

facebook-github-bot commented Feb 14, 2025

facebook-github-bot commented Feb 14, 2025

facebook-github-bot commented Feb 14, 2025

Refactor tokenizer test and add to cmake #8450

Refactor tokenizer test and add to cmake #8450

Conversation

lucylq commented Feb 13, 2025 • edited Loading

Summary

Test plan

pytorch-bot bot commented Feb 13, 2025 • edited Loading

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/8450

✅ You can merge normally! (1 Unrelated Failure)

facebook-github-bot commented Feb 14, 2025

facebook-github-bot commented Feb 14, 2025

facebook-github-bot commented Feb 14, 2025

facebook-github-bot commented Feb 14, 2025

lucylq commented Feb 13, 2025 •

edited

Loading

pytorch-bot bot commented Feb 13, 2025 •

edited

Loading