Fix Bundler removing executables after creating them #7580

Job	Run time
Rubygems on Ubuntu (3.4)	3m 54s
Rubygems on Windows (3.3)	3m 39s
Rubygems on Windows (3.2)	3m 44s
Rubygems on macOS (3.2)	2m 14s
Rubygems on Ubuntu (3.3)	1m 48s
Rubygems on Ubuntu (3.2)	2m 1s
Rubygems on macOS (3.3)	2m 42s
Rubygems on Ubuntu (jruby)	7m 38s
Rubygems on Windows (3.4)	3m 53s
Rubygems on macOS (3.4)	3m 35s
Rubygems on Ubuntu (truffleruby)	3m 33s
All rubygems jobs pass	5s
	38m 46s

Provide feedback