-
Notifications
You must be signed in to change notification settings - Fork 522
[TensorFlow][Training][Sagemaker] TensorFlow 2.19.0 Currency Release #4789
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
sirutBuasai
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Overall great job on your first PRs. A few nit organizations here and there. I would retest the images especially EC2 image since the tag would've been wrong in your testing so we should figure out why the EC2 image passed your test even with the wrong image tag.
test/sagemaker_tests/tensorflow/tensorflow2_training/integration/local/test_smppy_local.py
Outdated
Show resolved
Hide resolved
test/sagemaker_tests/tensorflow/tensorflow2_training/integration/local/test_training.py
Outdated
Show resolved
Hide resolved
test/sagemaker_tests/tensorflow/tensorflow2_training/integration/sagemaker/test_smppy.py
Outdated
Show resolved
Hide resolved
test/sagemaker_tests/tensorflow/tensorflow2_training/integration/sagemaker/test_smppy.py
Outdated
Show resolved
Hide resolved
test/sagemaker_tests/tensorflow/tensorflow2_training/integration/sagemaker/test_smppy.py
Show resolved
Hide resolved
test/sagemaker_tests/tensorflow/tensorflow2_training/integration/local/test_smppy_local.py
Show resolved
Hide resolved
2. added --no-cache-dir -U
3. used {TF_Version} for better maintainability.
…tests_on_heavy_instances
|
Failed to start build_tensorflow_training_<X.Y>_sm. |
GitHub Issue #, if available:
Note:
If merging this PR should also close the associated Issue, please also add that Issue # to the Linked Issues section on the right.
All PR's are checked weekly for staleness. This PR will be closed if not updated in 30 days.
Description
Tests run
Tensorflow 2.19 Training -
ca63142 - passed ec2, ecs, eks, sanity, security, sagemaker_local, sagemaker tests
4186d27 - passed ec2 benchmark, sagemaker_efa, sagemaker_rc, sagemaker_benchmark tests
2db2329 - passed ec2_heavy_instance tests
NOTE: By default, docker builds are disabled. In order to build your container, please update dlc_developer_config.toml and specify the framework to build in "build_frameworks"
Confused on how to run tests? Try using the helper utility...
Assuming your remote is called
origin(you can find out more withgit remote -v)...python src/prepare_dlc_dev_environment.py -b </path/to/buildspec.yml> -cp originpython src/prepare_dlc_dev_environment.py -b </path/to/buildspec.yml> -t sanity_tests -cp originpython src/prepare_dlc_dev_environment.py -rcp originNOTE: If you are creating a PR for a new framework version, please ensure success of the standard, rc, and efa sagemaker remote tests by updating the dlc_developer_config.toml file:
Expand
sagemaker_remote_tests = truesagemaker_efa_tests = truesagemaker_rc_tests = trueAdditionally, please run the sagemaker local tests in at least one revision:
sagemaker_local_tests = trueFormatting
black -l 100on my code (formatting tool: https://black.readthedocs.io/en/stable/getting_started.html)DLC image/dockerfile
Builds to Execute
Expand
Fill out the template and click the checkbox of the builds you'd like to execute
Note: Replace with <X.Y> with the major.minor framework version (i.e. 2.2) you would like to start.
build_pytorch_training_<X.Y>_sm
build_pytorch_training_<X.Y>_ec2
build_pytorch_inference_<X.Y>_sm
build_pytorch_inference_<X.Y>_ec2
build_pytorch_inference_<X.Y>_graviton
build_tensorflow_training_<X.Y>_sm
build_tensorflow_training_<X.Y>_ec2
build_tensorflow_inference_<X.Y>_sm
build_tensorflow_inference_<X.Y>_ec2
build_tensorflow_inference_<X.Y>_graviton
Additional context
PR Checklist
Expand
NEURON/GRAVITON Testing Checklist
dlc_developer_config.tomlin my PR branch by settingneuron_mode = trueorgraviton_mode = trueBenchmark Testing Checklist
dlc_developer_config.tomlin my PR branch by settingec2_benchmark_tests = trueorsagemaker_benchmark_tests = truePytest Marker Checklist
Expand
@pytest.mark.model("<model-type>")to the new tests which I have added, to specify the Deep Learning model that is used in the test (use"N/A"if the test doesn't use a model)@pytest.mark.integration("<feature-being-tested>")to the new tests which I have added, to specify the feature that will be tested@pytest.mark.multinode(<integer-num-nodes>)to the new tests which I have added, to specify the number of nodes used on a multi-node test@pytest.mark.processor(<"cpu"/"gpu"/"eia"/"neuron">)to the new tests which I have added, if a test is specifically applicable to only one processor typeBy submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.