[TensorFlow][Training][Sagemaker] TensorFlow 2.19.0 Currency Release #4789

bhanutejagk · 2025-05-08T00:57:25Z

GitHub Issue #, if available:

Note:

If merging this PR should also close the associated Issue, please also add that Issue # to the Linked Issues section on the right.
All PR's are checked weekly for staleness. This PR will be closed if not updated in 30 days.

Description

Tests run

NOTE: By default, docker builds are disabled. In order to build your container, please update dlc_developer_config.toml and specify the framework to build in "build_frameworks"

I have run builds/tests on commit for my changes.

Confused on how to run tests? Try using the helper utility...

Assuming your remote is called origin (you can find out more with git remote -v)...

Run default builds and tests for a particular buildspec - also commits and pushes changes to remote; Example:

python src/prepare_dlc_dev_environment.py -b </path/to/buildspec.yml> -cp origin

Enable specific tests for a buildspec or set of buildspecs - also commits and pushes changes to remote; Example:

python src/prepare_dlc_dev_environment.py -b </path/to/buildspec.yml> -t sanity_tests -cp origin

Restore TOML file when ready to merge

python src/prepare_dlc_dev_environment.py -rcp origin

NOTE: If you are creating a PR for a new framework version, please ensure success of the standard, rc, and efa sagemaker remote tests by updating the dlc_developer_config.toml file:

Expand

sagemaker_remote_tests = true
sagemaker_efa_tests = true
sagemaker_rc_tests = true

Additionally, please run the sagemaker local tests in at least one revision:

sagemaker_local_tests = true

Formatting

I have run black -l 100 on my code (formatting tool: https://black.readthedocs.io/en/stable/getting_started.html)

DLC image/dockerfile

Builds to Execute

Expand

Fill out the template and click the checkbox of the builds you'd like to execute

Note: Replace with <X.Y> with the major.minor framework version (i.e. 2.2) you would like to start.

Additional context

PR Checklist

Expand

I've prepended PR tag with frameworks/job this applies to : [mxnet, tensorflow, pytorch] | [ei/neuron/graviton] | [build] | [test] | [benchmark] | [ec2, ecs, eks, sagemaker]
If the PR changes affects SM test, I've modified dlc_developer_config.toml in my PR branch by setting sagemaker_tests = true and efa_tests = true
If this PR changes existing code, the change fully backward compatible with pre-existing code. (Non backward-compatible changes need special approval.)
(If applicable) I've documented below the DLC image/dockerfile this relates to
(If applicable) I've documented below the tests I've run on the DLC image
(If applicable) I've reviewed the licenses of updated and new binaries and their dependencies to make sure all licenses are on the Apache Software Foundation Third Party License Policy Category A or Category B license list. See https://www.apache.org/legal/resolved.html.
(If applicable) I've scanned the updated and new binaries to make sure they do not have vulnerabilities associated with them.

NEURON/GRAVITON Testing Checklist

When creating a PR:

I've modified dlc_developer_config.toml in my PR branch by setting neuron_mode = true or graviton_mode = true

Benchmark Testing Checklist

When creating a PR:

I've modified dlc_developer_config.toml in my PR branch by setting ec2_benchmark_tests = true or sagemaker_benchmark_tests = true

Pytest Marker Checklist

Expand

(If applicable) I have added the marker @pytest.mark.model("<model-type>") to the new tests which I have added, to specify the Deep Learning model that is used in the test (use "N/A" if the test doesn't use a model)
(If applicable) I have added the marker @pytest.mark.integration("<feature-being-tested>") to the new tests which I have added, to specify the feature that will be tested
(If applicable) I have added the marker @pytest.mark.multinode(<integer-num-nodes>) to the new tests which I have added, to specify the number of nodes used on a multi-node test
(If applicable) I have added the marker @pytest.mark.processor(<"cpu"/"gpu"/"eia"/"neuron">) to the new tests which I have added, if a test is specifically applicable to only one processor type

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Yadan-Wei · 2025-05-08T01:57:10Z

tensorflow/training/docker/2.18/py3/cu125/Dockerfile.gpu

@@ -346,7 +346,7 @@ RUN pip install wheel \
 # tf-models does not respect existing installations of TF and always installs open source TF
 RUN ${PIP} install --no-cache-dir -U \
    tf-models-official==2.18.0 \
-    tensorflow-text==2.18.1 \
+    tensorflow-text==2.19.1 \


This is TF2.18, it shouldn't be modified

Yadan-Wei · 2025-05-08T01:58:07Z

tensorflow/training/docker/2.19/Dockerfile.cpu

@@ -0,0 +1,300 @@
+ARG TF_URL=https://storage.googleapis.com/tensorflow/versions/2.19.0/tensorflow_cpu-2.19.0-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl


The path of this file is incorrect, refer to TF2.18 to see the file structure. It should be located inside py3 folder.

Yadan-Wei · 2025-05-08T01:59:30Z

dlc_developer_config.toml



 # By default we build both training and inference containers. Set true/false values to determine which to build.
 build_training = true
-build_inference = true
+build_inference = false


I couldn't add comments on line 108, but it needs a value like dlc-pr-tensorflow-2-training = "tensorflow/training/buildspec-2-19-ec2.yml" to let the system know which image it gonna build.

Yadan-Wei · 2025-05-08T20:10:07Z

tensorflow/training/docker/2.19/py3/cu125/Dockerfile.gpu

+
+ARG OPEN_MPI_PATH=/opt/amazon/openmpi
+ARG EFA_PATH=/opt/amazon/efa
+ARG EFA_VERSION=1.37.0


latest EFA version is 1.40.0

But i see 1.37 in TF 2.18

That's because at that time latest is 1.37, now EFA installer launched new version.
Refer to bottom of this page https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/efa-verify.html

Yadan-Wei · 2025-05-08T20:12:48Z

tensorflow/training/docker/2.19/py3/cu125/Dockerfile.gpu

+ARG EFA_PATH=/opt/amazon/efa
+ARG EFA_VERSION=1.37.0
+ARG OMPI_VERSION=4.1.7
+ARG BRANCH_OFI=1.13.2-aws


OFI now is embedded with EFA installer, so no need to install this.

Yadan-Wei · 2025-05-08T20:14:01Z

tensorflow/training/docker/2.19/py3/cu125/Dockerfile.gpu

+   urllib3 \
+   "protobuf<4"
+
+# Install AWS OFI NCCL plug-in


No need to install ofi-nccl since it's embedded in EFA installer

Yadan-Wei · 2025-05-08T20:17:07Z

tensorflow/training/docker/2.19/py3/cu125/Dockerfile.gpu

+# https://github.com/tensorflow/models/issues/9267
+# tf-models does not respect existing installations of TF and always installs open source TF
+RUN ${PIP} install --no-cache-dir -U \
+    tf-models-official==2.19.0 \


the latest version of this is 2.19.1 https://pypi.org/project/tf-models-official/#history

Yadan-Wei · 2025-05-08T20:18:40Z

tensorflow/training/docker/2.19/py3/cu125/Dockerfile.gpu

+# tf-models does not respect existing installations of TF and always installs open source TF
+RUN ${PIP} install --no-cache-dir -U \
+    tf-models-official==2.19.0 \
+    tensorflow-text==2.19.1 \


The latest version of this is 2.19.0 https://pypi.org/project/tensorflow-text/#history

Yadan-Wei · 2025-05-08T20:20:35Z

tensorflow/training/docker/2.19/py3/Dockerfile.cpu

+# https://github.com/tensorflow/models/issues/9267
+# tf-models does not respect existing installations of TF and always installs open source TF
+RUN ${PIP} install --no-cache-dir -U \
+    tf-models-official==2.19.0 \


Update these versions too.

Yadan-Wei · 2025-05-08T20:24:50Z

tensorflow/training/docker/2.19/py3/cu125/Dockerfile.gpu

+FROM common AS sagemaker
+
+# Install SageMaker Profiler binary
+ARG SMP_URL=https://smppy.s3.amazonaws.com/tensorflow/cu118/smprof-0.3.332-cp310-cp310-linux_x86_64.whl


I am asking SMPPY team about this file, you can leave it here or comment out for now.

…mage

…not installed

…tecustomize.py

…a to handle pip check test failure

…ant resolve

…10 and debugging py version

Yadan-Wei · 2025-05-16T17:29:14Z

tensorflow/training/docker/2.19/py3/cu125/Dockerfile.gpu

+FROM common AS sagemaker
+
+# Install SageMaker Profiler binary
+#ARG SMP_URL=https://smppy.s3.amazonaws.com/tensorflow/cu118/smprof-0.3.332-cp310-cp310-linux_x86_64.whl


You can remove SMP as they won't give us a new version

Building for training TF 2.19 EC2

3dc6380

bhanutejagk requested a review from a team as a code owner May 8, 2025 00:57

aws-deep-learning-containers-ci bot added authorized build Reflects file change in build folder Size:S Determines the size of the PR tensorflow Reflects file change in tensorflow folder labels May 8, 2025

bhanutejagk changed the title ~~Building for training TF 2.19 EC2~~ [TensorFlow][Training][EC2] TensorFlow 2.19.0 Currency Release May 8, 2025

bhanutejagk closed this May 8, 2025

bhanutejagk reopened this May 8, 2025

[TensorFlow][Training][EC2] TensorFlow 2.19.0 Currency Release

bd6674d

Yadan-Wei reviewed May 8, 2025

View reviewed changes

Bhanu Teja Goshikonda added 4 commits May 8, 2025 09:44

[TensorFlow][Training][EC2] TensorFlow 2.19.0 Currency Release

c3259be

[TensorFlow][Training][EC2] TensorFlow 2.19.0 Currency Release

cd16864

[TensorFlow][Training][EC2] TensorFlow 2.19.0 Currency Release

80faa58

[TensorFlow][Training][EC2] TensorFlow 2.19.0 Currency Release

edfaf85

Yadan-Wei reviewed May 8, 2025

View reviewed changes

Bhanu Teja Goshikonda and others added 6 commits May 8, 2025 14:22

[TensorFlow][Training][EC2] TensorFlow 2.19.0 Currency Release

9eb5188

Added buildspec-2-19-sm.yml file and made other changes to build sm i…

3db0623

…mage

Merge branch 'master' into TF2.19

45502f6

commented autopatch_build line in buildspec

70408d0

Commented out SMP_URL and its installation code logic

f8bd89a

Added logic for timout and retry for failed downloads

7087d67

Bhanu Teja Goshikonda and others added 7 commits May 8, 2025 19:24

Resolved error due to '=' instead of '=='

9d58e54

Modified 'default-timeout' to '--default-timeout'

2a7ef55

Merge branch 'master' into TF2.19

ef5743b

Merge branch 'master' into TF2.19

b38268a

Merge branch 'master' into TF2.19

6b02887

changed build to false

dea10d5

Skipped tests which lead to failures and updated python version to 312

11d7cb7

bhanutejagk requested a review from a team as a code owner May 13, 2025 18:28

aws-deep-learning-containers-ci bot added ec2 Reflects file change in dlc_tests/ec2 folder sagemaker_tests sanity Reflects file change in dlc_tests/sanity folder test Reflects file change in test folder labels May 13, 2025

bhanutejagk and others added 10 commits May 13, 2025 12:46

Merge branch 'master' into TF2.19

845638d

debugging with print statements in test_pip_check

7fcd1e9

Added decorater to skip the sagemaker tests as sm profiler binary is …

8aa3c2a

…not installed

modified dlc_template.py and added copy command to dockerfiles for si…

f7acca8

…tecustomize.py

Building the image since copy command in docker files is added

34f04c4

Modified main to remove telemetry test failures

169a633

Pinned previous versions for tensorflow-datasets & tensorflow-metadat…

fa1b332

…a to handle pip check test failure

added 3.12 py_version as not included previously

3bd7184

added 2 allowlist files to handle unremovable errors

66e090e

reverted back to py3.10 from py 3.12 as protobuf confict occurs and c…

84ff7e3

…ant resolve

bhanutejagk changed the title ~~[TensorFlow][Training][EC2] TensorFlow 2.19.0 Currency Release~~ [TensorFlow][Training][Sagemaker] TensorFlow 2.19.0 Currency Release May 15, 2025

Bhanu Teja Goshikonda and others added 3 commits May 15, 2025 17:22

removed prev tags to tensorflow metadata since we reverted back to 3.…

10a1b4f

…10 and debugging py version

Merge branch 'master' into TF2.19

b0cf14c

changed tensorflow-metadata version as it is incompatible with protobuf

ae2f718

Yadan-Wei reviewed May 16, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[TensorFlow][Training][Sagemaker] TensorFlow 2.19.0 Currency Release #4789

[TensorFlow][Training][Sagemaker] TensorFlow 2.19.0 Currency Release #4789

bhanutejagk commented May 8, 2025 •

edited

Loading

Yadan-Wei May 8, 2025

Yadan-Wei May 8, 2025 •

edited

Loading

Yadan-Wei May 8, 2025

Yadan-Wei May 8, 2025 •

edited

Loading

bhanutejagk May 8, 2025

Yadan-Wei May 8, 2025 •

edited

Loading

Yadan-Wei May 8, 2025

Yadan-Wei May 8, 2025

Yadan-Wei May 8, 2025

Yadan-Wei May 8, 2025

Yadan-Wei May 8, 2025

Yadan-Wei May 8, 2025 •

edited

Loading

Yadan-Wei May 16, 2025

		@@ -0,0 +1,300 @@
		ARG TF_URL=https://storage.googleapis.com/tensorflow/versions/2.19.0/tensorflow_cpu-2.19.0-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl

[TensorFlow][Training][Sagemaker] TensorFlow 2.19.0 Currency Release #4789

Are you sure you want to change the base?

[TensorFlow][Training][Sagemaker] TensorFlow 2.19.0 Currency Release #4789

Conversation

bhanutejagk commented May 8, 2025 • edited Loading

Description

Tests run

Formatting

DLC image/dockerfile

Builds to Execute

Additional context

PR Checklist

NEURON/GRAVITON Testing Checklist

Benchmark Testing Checklist

Pytest Marker Checklist

Choose a reason for hiding this comment

Yadan-Wei May 8, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Yadan-Wei May 8, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Yadan-Wei May 8, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Yadan-Wei May 8, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bhanutejagk commented May 8, 2025 •

edited

Loading

Yadan-Wei May 8, 2025 •

edited

Loading

Yadan-Wei May 8, 2025 •

edited

Loading

Yadan-Wei May 8, 2025 •

edited

Loading

Yadan-Wei May 8, 2025 •

edited

Loading