HDFS-16644.java.io.IOException Invalid token in javax.security.sasl.qop #5962

zhuzilong2013 · 2023-08-18T14:59:33Z

Description of PR

This change prevents qop values from being overwritten with illegal values.
JIRA: HDFS-16644

How was this patch tested?

Test in a production environment

For code changes:

Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
If applicable, have you updated the LICENSE, LICENSE-binary, NOTICE-binary files?

hadoop-yetus · 2023-08-18T20:11:04Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 30s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
-1 ❌	test4tests	0m 0s		The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
			_ trunk Compile Tests _
+1 💚	mvninstall	40m 53s		trunk passed
+1 💚	compile	0m 59s		trunk passed with JDK Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04
+1 💚	compile	0m 51s		trunk passed with JDK Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05
+1 💚	checkstyle	0m 50s		trunk passed
+1 💚	mvnsite	1m 3s		trunk passed
+1 💚	javadoc	0m 55s		trunk passed with JDK Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04
+1 💚	javadoc	1m 28s		trunk passed with JDK Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05
+1 💚	spotbugs	2m 17s		trunk passed
+1 💚	shadedclient	28m 22s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 51s		the patch passed
+1 💚	compile	0m 55s		the patch passed with JDK Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04
+1 💚	javac	0m 55s		the patch passed
+1 💚	compile	0m 47s		the patch passed with JDK Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05
+1 💚	javac	0m 47s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	0m 41s		the patch passed
+1 💚	mvnsite	0m 51s		the patch passed
+1 💚	javadoc	0m 40s		the patch passed with JDK Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04
+1 💚	javadoc	1m 14s		the patch passed with JDK Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05
+1 💚	spotbugs	2m 20s		the patch passed
+1 💚	shadedclient	28m 39s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
-1 ❌	unit	193m 17s	/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt	hadoop-hdfs in the patch passed.
+1 💚	asflicense	0m 39s		The patch does not generate ASF License warnings.
		310m 1s

Reason	Tests
Failed junit tests	hadoop.hdfs.server.namenode.TestReconstructStripedBlocks
	hadoop.hdfs.server.blockmanagement.TestBlockManager

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5962/1/artifact/out/Dockerfile
GITHUB PR	#5962
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname	Linux 594a45e7b0c0 4.15.0-213-generic #224-Ubuntu SMP Mon Jun 19 13:30:12 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `7d3cd37`
Default Java	Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5962/1/testReport/
Max. process+thread count	3352 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5962/1/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

zhuzilong2013 · 2023-08-19T02:45:18Z

Hi @Hexiaoqiao @haiyang1987 Could you help review this when you have time? Thanks a lot～

zhuzilong2013 · 2023-08-22T08:11:24Z

Those failed unit tests were unrelated to the change. And they work fine locally.

zhuzilong2013 · 2023-09-03T06:28:22Z

This commit can help us avoid issues where the datanode fails to work properly after throwing exceptions with the 2.10.1 and 2.10.2 clients. I believe it's meaningful. @ayushtkn @linyiqun @aajisaka could you please help me review it？

Copilot

Pull Request Overview

This pull request addresses a potential issue where illegal qop values could overwrite existing configuration by creating a modified copy of the SASL properties.

Introduces a new TreeMap (dynamicSaslProps) to hold a copy of the original SASL properties.
Updates the SASL handshake process to use the modified properties map when setting the qop value and during validation.

Comments suppressed due to low confidence (1)

hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/sasl/SaslDataTransferServer.java:382

[nitpick] Consider renaming 'dynamicSaslProps' to 'updatedSaslProps' to more clearly reflect that this map is a modified copy of the original SASL properties.

Map<String, String> dynamicSaslProps = new TreeMap<>(saslProps);

jojochuang

I see. This is because the saslProps object is initialized once and used again and again.

jojochuang

LGTM.

It's a partial fix but at least it limits the damage to 2.10 clients only.

hadoop-yetus · 2025-03-27T21:45:57Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 19s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
-1 ❌	test4tests	0m 0s		The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
			_ trunk Compile Tests _
+1 💚	mvninstall	23m 37s		trunk passed
+1 💚	compile	0m 46s		trunk passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04
+1 💚	compile	0m 37s		trunk passed with JDK Private Build-1.8.0_442-8u442-b06~~us1-0ubuntu1~~20.04-b06
+1 💚	checkstyle	0m 38s		trunk passed
+1 💚	mvnsite	0m 43s		trunk passed
+1 💚	javadoc	0m 44s		trunk passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04
+1 💚	javadoc	1m 2s		trunk passed with JDK Private Build-1.8.0_442-8u442-b06~~us1-0ubuntu1~~20.04-b06
+1 💚	spotbugs	1m 41s		trunk passed
+1 💚	shadedclient	20m 44s		branch has no errors when building and testing our client artifacts.
-0 ⚠️	patch	20m 56s		Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 39s		the patch passed
+1 💚	compile	0m 40s		the patch passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04
+1 💚	javac	0m 40s		the patch passed
+1 💚	compile	0m 33s		the patch passed with JDK Private Build-1.8.0_442-8u442-b06~~us1-0ubuntu1~~20.04-b06
+1 💚	javac	0m 33s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	0m 29s		the patch passed
+1 💚	mvnsite	0m 37s		the patch passed
+1 💚	javadoc	0m 34s		the patch passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04
+1 💚	javadoc	1m 0s		the patch passed with JDK Private Build-1.8.0_442-8u442-b06~~us1-0ubuntu1~~20.04-b06
+1 💚	spotbugs	1m 37s		the patch passed
+1 💚	shadedclient	20m 26s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	1m 0s		hadoop-hdfs in the patch passed.
+1 💚	asflicense	0m 24s		The patch does not generate ASF License warnings.
		78m 33s

Subsystem	Report/Notes
Docker	ClientAPI=1.48 ServerAPI=1.48 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5962/2/artifact/out/Dockerfile
GITHUB PR	#5962
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname	Linux e9cba58a24bb 5.15.0-130-generic #140-Ubuntu SMP Wed Dec 18 17:59:53 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `cf88ca8`
Default Java	Private Build-1.8.0_442-8u442-b06~~us1-0ubuntu1~~20.04-b06
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_442-8u442-b06~~us1-0ubuntu1~~20.04-b06
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5962/2/testReport/
Max. process+thread count	552 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5962/2/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

…qop (#5962) (cherry picked from commit 9d93d08)

…qop (#5962) (cherry picked from commit 9d93d08) (cherry picked from commit cc0580c)

…qop (#5962) (cherry picked from commit 9d93d08) (cherry picked from commit cc0580c) (cherry picked from commit 2736485)

HDFS-16644.java.io.IOException Invalid token in javax.security.sasl.qop

7d3cd37

github-actions bot added HDFS trunk labels Aug 18, 2023

jojochuang requested a review from Copilot March 27, 2025 16:28

Copilot AI reviewed Mar 27, 2025

View reviewed changes

jojochuang reviewed Mar 27, 2025

View reviewed changes

Merge branch 'trunk' into HDFS-16644

cf88ca8

jojochuang approved these changes Mar 27, 2025

View reviewed changes

jojochuang merged commit 9d93d08 into apache:trunk Mar 28, 2025
1 of 3 checks passed

jojochuang pushed a commit that referenced this pull request Mar 28, 2025

HDFS-16644. java.io.IOException Invalid token in javax.security.sasl.…

cc0580c

…qop (#5962) (cherry picked from commit 9d93d08)

jojochuang pushed a commit that referenced this pull request Mar 28, 2025

HDFS-16644. java.io.IOException Invalid token in javax.security.sasl.…

2736485

…qop (#5962) (cherry picked from commit 9d93d08) (cherry picked from commit cc0580c)

jojochuang pushed a commit that referenced this pull request Mar 28, 2025

HDFS-16644. java.io.IOException Invalid token in javax.security.sasl.…

2a8a893

…qop (#5962) (cherry picked from commit 9d93d08) (cherry picked from commit cc0580c) (cherry picked from commit 2736485)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

HDFS-16644.java.io.IOException Invalid token in javax.security.sasl.qop #5962

HDFS-16644.java.io.IOException Invalid token in javax.security.sasl.qop #5962

Uh oh!

zhuzilong2013 commented Aug 18, 2023

Uh oh!

hadoop-yetus commented Aug 18, 2023

Uh oh!

zhuzilong2013 commented Aug 19, 2023

Uh oh!

zhuzilong2013 commented Aug 22, 2023

Uh oh!

zhuzilong2013 commented Sep 3, 2023

Uh oh!

Copilot AI left a comment

Uh oh!

jojochuang left a comment •

edited

Loading

Uh oh!

jojochuang left a comment

Uh oh!

hadoop-yetus commented Mar 27, 2025

Uh oh!

Uh oh!

Uh oh!

HDFS-16644.java.io.IOException Invalid token in javax.security.sasl.qop #5962

HDFS-16644.java.io.IOException Invalid token in javax.security.sasl.qop #5962

Uh oh!

Conversation

zhuzilong2013 commented Aug 18, 2023

Description of PR

How was this patch tested?

For code changes:

Uh oh!

hadoop-yetus commented Aug 18, 2023

Uh oh!

zhuzilong2013 commented Aug 19, 2023

Uh oh!

zhuzilong2013 commented Aug 22, 2023

Uh oh!

zhuzilong2013 commented Sep 3, 2023

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Uh oh!

jojochuang left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jojochuang left a comment

Choose a reason for hiding this comment

Uh oh!

hadoop-yetus commented Mar 27, 2025

Uh oh!

Uh oh!

Uh oh!

jojochuang left a comment •

edited

Loading