Skip to content

Commit 55efb2e

Browse files
eerhardtdseefeld
eerhardt
authored and
dseefeld
committed
Merge dev/release/2.1 into master (#577)
* Stabilize CLI. (#565) * Stabilize CLI. * Use stable MSBuild. * Fix MSBuild version to match prodcon build. * Modify CLI build file to always ignore NU5104 * Add web to smoke-test, with and without dev certs (#539) * Add web to smoke-test, with and without dev certs * Only reset caches when a test will run Avoid conflicting with a hanging process if there's no need to delete the files anyway. * Fix some minor issues with the bootstrap documentation. * Update CoreClr, ProdCon to rtm-26515-03, 20180516-07-1693122, respectively (#553) * Only set DropSuffix to true when UseStableVersions is true * Update dotnet-dev-certs version This version needs to be compatible with the current runtime being produced.
1 parent 4d468b8 commit 55efb2e

File tree

10 files changed

+168
-56
lines changed

10 files changed

+168
-56
lines changed

‎Documentation/boostrap-new-os.md

Lines changed: 3 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -44,15 +44,14 @@ For versions earlier than .NET Core 2.1, following dependencies are also require
4444
### Building the bootstrap CLI
4545
There is a bash script file that automatizes most of the process of building the bootstrap CLI end to end. The bash script is located at `dotnet/source-build/scripts/bootstrap/buildbootstrapcli.sh`. It first creates a folder named by the new target RID, clones the coreclr, corefx and core-setup repos into it and checks out the same commit of each of the repos as the one that was used to build the seed CLI. This first step is skipped if the coreclr, corefx and core-setup folders already exist. This is important so that the sources can be modified to fix possible build issues and to target the new RID.
4646

47-
The script needs to be passed several arguments. The target architecture, the build configuration, the target OS, the new RID and path to the folder with the untared seed CLI. There is also an optional option to specify the version of the Clang compiler to use to compile the native code. There is also an option to pass in a System.Private.CoreLib.dll built elsewhere and an option to crossgen it. This is useful if you are building debug configuration, since the seed CLI that comes from Azure is built for release configuration and the release version of System.Private.CoreLib.dll is not compatible with debug version of libcoreclr.so.
47+
The script needs to be passed several arguments. The target architecture, the build configuration, the target OS, the new RID and path to the folder with the untared seed CLI. There is also an optional option to specify the version of the Clang compiler to use to compile the native code. There is also an option to pass in a System.Private.CoreLib.dll built elsewhere. This is useful if you are building debug configuration, since the seed CLI that comes from Azure is built for release configuration and the release version of System.Private.CoreLib.dll is not compatible with debug version of libcoreclr.so.
4848
Here is the summary of the options that you get from running the script with `--help` option:
4949
```
50-
Usage: buildbootstrapcli.sh [BuildType] --rid <Rid> --seedcli <SeedCli> [--os <OS>] [--clang <Major.Minor>] [--corelib <CoreLib>]
50+
Usage: buildbootstrapcli.sh [BuildType] -rid <Rid> -seedcli <SeedCli> [-os <OS>] [-clang <Major.Minor>] [-corelib <CoreLib>]
5151
5252
Options:
5353
BuildType Type of build (-debug, -release), default: -debug
5454
-clang <Major.Minor> Override of the version of clang compiler to use
55-
-config <Configuration> Build configuration (debug, release), default: debug
5655
-corelib <CoreLib> Path to System.Private.CoreLib.dll, default: use the System.Private.CoreLib.dll from the seed CLI
5756
-os <OS> Operating system (used for corefx build), default: Linux
5857
-rid <Rid> Runtime identifier including the architecture part (e.g. rhel.6-x64)
@@ -61,7 +60,7 @@ Options:
6160
```
6261
So, for example, when we were creating bootstrap CLI for RHEL / CentOS 6, the command was:
6362
```bash
64-
./buildbootstrapcli.sh -rid rhel.6 -release -os Linux -seedcli ~/seed-cli
63+
./buildbootstrapcli.sh -release -rid rhel.6-x64 -os Linux -seedcli ~/seed-cli
6564
```
6665
After running the script, check the console output. If the last line printed is `**** Bootstrap CLI was successfully built ****`, then everything went fine and the bootstrap CLI is ready. You can find it in the `<Rid>-<Architecture>/dotnetcli` subfolder. So for the example command above, it would be `rhel.6-x64/dotnetcli`.
6766
If there were build errors, they need to be looked into and fixed. After that run the `buildbootstrapcli.sh` with the same arguments again. Repeat until everything builds.

‎dependencies.props

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -5,8 +5,8 @@
55
for which dotnet/versions commit was last used to update the dependency.
66
-->
77
<PropertyGroup>
8-
<ProdConCurrentRef>b0f08270da85e75c1fc9ac0b8b56937a4708958e</ProdConCurrentRef>
9-
<CoreClrCurrentRef>b0f08270da85e75c1fc9ac0b8b56937a4708958e</CoreClrCurrentRef>
8+
<ProdConCurrentRef>79731c509a9d31b921f531ee63c48daa8e0c30d5</ProdConCurrentRef>
9+
<CoreClrCurrentRef>79731c509a9d31b921f531ee63c48daa8e0c30d5</CoreClrCurrentRef>
1010
<BuildToolsCurrentRef>54227d85bec7093210eef26b6013d1435c6fc21e</BuildToolsCurrentRef>
1111
</PropertyGroup>
1212

‎netci.groovy

Lines changed: 7 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -13,8 +13,13 @@ def addBuildStepsAndSetMachineAffinity(def job, String os, String configuration)
1313
}
1414
else {
1515
shell("git submodule update --init --recursive");
16-
shell("./build.sh /p:Configuration=${configuration} ${loggingOptions}")
17-
shell("./smoke-test.sh --minimal --projectOutput --configuration ${configuration}")
16+
shell("./build.sh /p:Configuration=${configuration} ${loggingOptions}");
17+
smokeTestExcludes = "";
18+
if (os == "Fedora24" || os == "OSX10.12") {
19+
// Dev certs doesn't seem to work in these platforms. https://github.com/dotnet/source-build/issues/560
20+
smokeTestExcludes += " --excludeWebHttpsTests";
21+
}
22+
shell("./smoke-test.sh --minimal --projectOutput --configuration ${configuration} ${smokeTestExcludes}");
1823
}
1924
};
2025
};

‎repos/cli.proj

Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -94,6 +94,11 @@
9494
Unable to find package dotnet-watch.
9595
-->
9696
<EnvironmentVariables Include="CLIBUILD_SKIP_BUNDLEDDOTNETTOOLS=true" />
97+
98+
<!-- Produce stable versions for RTM only. When https://github.com/dotnet/source-build/issues/550
99+
is fixed we will be able to determine this automatically. -->
100+
<EnvironmentVariables Include="DropSuffix=true"
101+
Condition="'$(UseStableVersions)' == 'true'" />
97102
</ItemGroup>
98103

99104
<Target Name="AddProdConFeed" BeforeTargets="Build">
@@ -113,6 +118,28 @@
113118
Overwrite="true" />
114119
</Target>
115120

121+
<!--
122+
Ignore this error in the CLI build, which doesn't seem to happen in ProdCon:
123+
124+
error NU5104: A stable release of a package should not have a prerelease dependency. Either
125+
modify the version spec of dependency "NuGet.Frameworks [4.7.0-rtm.5156, )" or update the
126+
version field in the nuspec.
127+
-->
128+
<Target Name="AddNoWarnNU5104ByDefault" BeforeTargets="Build">
129+
<PropertyGroup>
130+
<BuildDefaultsFile>$(ProjectDirectory)build\BuildDefaults.props</BuildDefaultsFile>
131+
<ReplacementEndMark>&lt;/NoWarn&gt;</ReplacementEndMark>
132+
<InsertItemLine>;NU5104$(ReplacementEndMark)</InsertItemLine>
133+
134+
<BuildDefaultsPropsContents>$([System.IO.File]::ReadAllText('$(BuildDefaultsFile)'))</BuildDefaultsPropsContents>
135+
</PropertyGroup>
136+
137+
<WriteLinesToFile File="$(BuildDefaultsFile)"
138+
Lines="$(BuildDefaultsPropsContents.Replace('$(ReplacementEndMark)', '$(InsertItemLine)'))"
139+
Condition="! $(BuildDefaultsPropsContents.Contains('$(InsertItemLine)'))"
140+
Overwrite="true" />
141+
</Target>
142+
116143
<Target Name="CopyTarBall" AfterTargets="CopyPackage">
117144
<ItemGroup>
118145
<CliTarBalls Include="$(TarBallOutput)*$(TarBallExtension)" />

‎repos/msbuild.proj

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -6,7 +6,7 @@
66
<!-- Override value from commit to match expected build. -->
77
<OfficialBuildId>20180426.1</OfficialBuildId>
88
<PackagesOutput>$(ProjectDirectory)/artifacts/$(Configuration)/packages</PackagesOutput>
9-
<OutputVersionArgs>/p:VersionPrefix=15.6.82 /p:DisableNerdbankVersioning=true /p:PreReleaseVersionLabel="rc" /p:BUILD_BUILDNUMBER=$(OfficialBuildId)</OutputVersionArgs>
9+
<OutputVersionArgs>/p:VersionPrefix=15.7.179 /p:DisableNerdbankVersioning=true /p:PreReleaseVersionLabel="" /p:BUILD_BUILDNUMBER=$(OfficialBuildId)</OutputVersionArgs>
1010
<BuildCommand>$(ProjectDirectory)build/build$(ShellExtension) build -DotNetBuildFromSource -DotNetCoreSdkDir $(DotNetCliToolDir) -bootstraponly -skiptests -pack -configuration $(Configuration) /p:GitHeadSha=$(GitCommitHash) $(OutputVersionArgs)</BuildCommand>
1111
<SourceOverrideRepoApiImplemented>true</SourceOverrideRepoApiImplemented>
1212
<RepoApiImplemented>false</RepoApiImplemented>

‎scripts/bootstrap/buildbootstrapcli.sh

Lines changed: 1 addition & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -5,12 +5,11 @@ set -o pipefail
55
usage()
66
{
77
echo "Builds a bootstrap CLI from sources"
8-
echo "Usage: $0 [BuildType] --rid <Rid> --seedcli <SeedCli> [--os <OS>] [--clang <Major.Minor>] [--corelib <CoreLib>]"
8+
echo "Usage: $0 [BuildType] -rid <Rid> -seedcli <SeedCli> [-os <OS>] [-clang <Major.Minor>] [-corelib <CoreLib>]"
99
echo ""
1010
echo "Options:"
1111
echo " BuildType Type of build (-debug, -release), default: -debug"
1212
echo " -clang <Major.Minor> Override of the version of clang compiler to use"
13-
echo " -config <Configuration> Build configuration (debug, release), default: debug"
1413
echo " -corelib <CoreLib> Path to System.Private.CoreLib.dll, default: use the System.Private.CoreLib.dll from the seed CLI"
1514
echo " -os <OS> Operating system (used for corefx build), default: Linux"
1615
echo " -rid <Rid> Runtime identifier including the architecture part (e.g. rhel.6-x64)"

‎smoke-test.sh

Lines changed: 124 additions & 42 deletions
Original file line numberDiff line numberDiff line change
@@ -4,30 +4,42 @@ set -euo pipefail
44
SCRIPT_ROOT="$(cd -P "$( dirname "$0" )" && pwd)"
55
TARBALL_PREFIX=dotnet-sdk-
66
VERSION_PREFIX=2.1
7+
# See https://github.com/dotnet/source-build/issues/579, this version
8+
# needs to be compatible with the runtime produced from source-build
9+
DEV_CERTS_VERSION_DEFAULT=2.1.0-rtm-30762
710

811
projectOutput=false
912
keepProjects=false
1013
dotnetDir=""
1114
configuration="Release"
15+
excludeNonWebTests=false
1216
excludeWebTests=false
17+
excludeWebNoHttpsTests=false
18+
excludeWebHttpsTests=false
1319
excludeLocalTests=false
1420
excludeOnlineTests=false
21+
devCertsVersion="$DEV_CERTS_VERSION_DEFAULT"
1522
testingDir="$SCRIPT_ROOT/testing-smoke"
1623
cliDir="$testingDir/builtCli"
1724
logFile="$testingDir/smoke-test.log"
1825
restoredPackagesDir="$testingDir/packages"
26+
testingHome="$testingDir/home"
1927

2028
function usage() {
2129
echo ""
2230
echo "usage:"
23-
echo " --dotnetDir the directory from which to run dotnet"
24-
echo " --configuration the configuration being tested (default=Release)"
25-
echo " --projectOutput echo dotnet's output to console"
26-
echo " --keepProjects keep projects after tests are complete"
27-
echo " --minimal run minimal set of tests - local sources only, no web"
28-
echo " --excludeWebTests don't run tests for web projects"
29-
echo " --excludeLocalTests exclude tests that use local sources for nuget packages"
30-
echo " --excludeOnlineTests exclude test that use online sources for nuget packages"
31+
echo " --dotnetDir the directory from which to run dotnet"
32+
echo " --configuration the configuration being tested (default=Release)"
33+
echo " --projectOutput echo dotnet's output to console"
34+
echo " --keepProjects keep projects after tests are complete"
35+
echo " --minimal run minimal set of tests - local sources only, no web"
36+
echo " --excludeNonWebTests don't run tests for non-web projects"
37+
echo " --excludeWebTests don't run tests for web projects"
38+
echo " --excludeWebNoHttpsTests don't run web project tests with --no-https"
39+
echo " --excludeWebHttpsTests don't run web project tests with https using dotnet-dev-certs"
40+
echo " --excludeLocalTests exclude tests that use local sources for nuget packages"
41+
echo " --excludeOnlineTests exclude test that use online sources for nuget packages"
42+
echo " --devCertsVersion <version> use dotnet-dev-certs <version> instead of default $DEV_CERTS_VERSION_DEFAULT"
3143
echo ""
3244
}
3345

@@ -57,18 +69,30 @@ while :; do
5769
keepProjects=true
5870
;;
5971
--minimal)
60-
excludeWebTests=true
6172
excludeOnlineTests=true
6273
;;
74+
--excludenonwebtests)
75+
excludeNonWebTests=true
76+
;;
6377
--excludewebtests)
6478
excludeWebTests=true
6579
;;
80+
--excludewebnohttpstests)
81+
excludeWebNoHttpsTests=true
82+
;;
83+
--excludewebhttpstests)
84+
excludeWebHttpsTests=true
85+
;;
6686
--excludelocaltests)
6787
excludeLocalTests=true
6888
;;
6989
--excludeonlinetests)
7090
excludeOnlineTests=true
7191
;;
92+
--devcertsversion)
93+
shift
94+
devCertsVersion="$1"
95+
;;
7296
*)
7397
usage
7498
exit 1
@@ -90,6 +114,24 @@ function doCommand() {
90114
mkdir "${lang}_${proj}"
91115
cd "${lang}_${proj}"
92116

117+
newArgs="new $proj -lang $lang"
118+
119+
while :; do
120+
if [ $# -le 0 ]; then
121+
break
122+
fi
123+
case "$1" in
124+
--new-arg)
125+
shift
126+
newArgs="$newArgs $1"
127+
;;
128+
*)
129+
break
130+
;;
131+
esac
132+
shift
133+
done
134+
93135
while :; do
94136
if [ $# -le 0 ]; then
95137
break
@@ -99,9 +141,9 @@ function doCommand() {
99141

100142
if [ "$1" == "new" ]; then
101143
if [ "$projectOutput" == "true" ]; then
102-
"${dotnetCmd}" new $proj -lang $lang | tee -a "$logFile"
144+
"${dotnetCmd}" $newArgs | tee -a "$logFile"
103145
else
104-
"${dotnetCmd}" new $proj -lang $lang >> "$logFile" 2>&1
146+
"${dotnetCmd}" $newArgs >> "$logFile" 2>&1
105147
fi
106148
elif [[ "$1" == "run" && "$proj" =~ ^(web|mvc|webapi|razor)$ ]]; then
107149
if [ "$projectOutput" == "true" ]; then
@@ -110,12 +152,20 @@ function doCommand() {
110152
"${dotnetCmd}" $1 >> "$logFile" 2>&1 &
111153
fi
112154
webPid=$!
113-
echo " waiting 20 seconds for web project with pid $webPid"
114-
sleep 20
115-
echo " sending SIGINT to $webPid"
116-
pkill -SIGINT -P $webPid
155+
killCommand="pkill -SIGTERM -P $webPid"
156+
echo " waiting up to 20 seconds for web project with pid $webPid..."
157+
echo " to clean up manually after an interactive cancellation, run: $killCommand"
158+
for seconds in $(seq 20); do
159+
if [ "$(tail -n 1 "$logFile")" = 'Application started. Press Ctrl+C to shut down.' ]; then
160+
echo " app ready for shutdown after $seconds seconds"
161+
break
162+
fi
163+
sleep 1
164+
done
165+
echo " stopping $webPid" | tee -a "$logFile"
166+
$killCommand
117167
wait $!
118-
echo " terminated with exit code $?"
168+
echo " terminated with exit code $?" | tee -a "$logFile"
119169
else
120170
if [ "$projectOutput" == "true" ]; then
121171
"${dotnetCmd}" $1 | tee -a "$logFile"
@@ -141,35 +191,68 @@ function doCommand() {
141191
echo "finished language $lang, type $proj" | tee -a smoke-test.log
142192
}
143193

194+
function setupDevCerts() {
195+
echo "Setting up dotnet-dev-certs $devCertsVersion to generate dev certificate" | tee -a "$logFile"
196+
(
197+
set -x
198+
"$dotnetDir/dotnet" tool install -g dotnet-dev-certs --version "$devCertsVersion" --add-source https://dotnet.myget.org/F/dotnet-core/api/v3/index.json
199+
export DOTNET_ROOT="$dotnetDir"
200+
"$testingHome/.dotnet/tools/dotnet-dev-certs" https
201+
) >> "$logFile" 2>&1
202+
}
203+
144204
function runAllTests() {
145205
# Run tests for each language and template
146-
doCommand C# console new restore run
147-
doCommand C# classlib new restore build
148-
doCommand C# xunit new restore test
149-
doCommand C# mstest new restore test
150-
151-
doCommand VB console new restore run
152-
doCommand VB classlib new restore build
153-
doCommand VB xunit new restore test
154-
doCommand VB mstest new restore test
155-
156-
doCommand F# console new restore run
157-
doCommand F# classlib new restore build
158-
doCommand F# xunit new restore test
159-
doCommand F# mstest new restore test
160-
206+
if [ "$excludeNonWebTests" == "false" ]; then
207+
doCommand C# console new restore run
208+
doCommand C# classlib new restore build
209+
doCommand C# xunit new restore test
210+
doCommand C# mstest new restore test
211+
212+
doCommand VB console new restore run
213+
doCommand VB classlib new restore build
214+
doCommand VB xunit new restore test
215+
doCommand VB mstest new restore test
216+
217+
doCommand F# console new restore run
218+
doCommand F# classlib new restore build
219+
doCommand F# xunit new restore test
220+
doCommand F# mstest new restore test
221+
fi
222+
161223
if [ "$excludeWebTests" == "false" ]; then
162-
doCommand C# web new restore run
163-
doCommand C# mvc new restore run
164-
doCommand C# webapi new restore run
165-
doCommand C# razor new restore run
166-
167-
doCommand F# web new restore run
168-
doCommand F# mvc new restore run
169-
doCommand F# webapi new restore run
224+
if [ "$excludeWebNoHttpsTests" == "false" ]; then
225+
runWebTests --new-arg --no-https
226+
fi
227+
228+
if [ "$excludeWebHttpsTests" == "false" ]; then
229+
setupDevCerts
230+
runWebTests
231+
fi
170232
fi
171233
}
172234

235+
function runWebTests() {
236+
doCommand C# web "$@" new restore run
237+
doCommand C# mvc "$@" new restore run
238+
doCommand C# webapi "$@" new restore run
239+
doCommand C# razor "$@" new restore run
240+
241+
doCommand F# web "$@" new restore run
242+
doCommand F# mvc "$@" new restore run
243+
doCommand F# webapi "$@" new restore run
244+
}
245+
246+
function resetCaches() {
247+
rm -rf "$testingHome"
248+
mkdir "$testingHome"
249+
250+
HOME="$testingHome"
251+
252+
# clean restore path
253+
rm -rf "$restoredPackagesDir"
254+
}
255+
173256
function setupProdConFeed() {
174257
sed -i.bakProdCon "s|PRODUCT_CONTRUCTION_PACKAGES|$prodConFeedUrl|g" "$testingDir/NuGet.Config"
175258
}
@@ -207,6 +290,7 @@ prodConFeedUrl="$(cat "$SCRIPT_ROOT/ProdConFeed.txt")"
207290

208291
# Run all tests, local restore sources first, online restore sources second
209292
if [ "$excludeLocalTests" == "false" ]; then
293+
resetCaches
210294
# Setup NuGet.Config with local restore source
211295
if [ -e "$SCRIPT_ROOT/smoke-testNuGet.Config" ]; then
212296
cp "$SCRIPT_ROOT/smoke-testNuGet.Config" "$testingDir/NuGet.Config"
@@ -220,10 +304,8 @@ if [ "$excludeLocalTests" == "false" ]; then
220304
echo "LOCAL RESTORE SOURCE - ALL TESTS PASSED!"
221305
fi
222306

223-
# clean restore path
224-
rm -rf "$restoredPackagesDir"
225-
226307
if [ "$excludeOnlineTests" == "false" ]; then
308+
resetCaches
227309
# Setup NuGet.Config to use online restore sources
228310
if [ -e "$SCRIPT_ROOT/smoke-testNuGet.Config" ]; then
229311
cp "$SCRIPT_ROOT/smoke-testNuGet.Config" "$testingDir/NuGet.Config"

‎src/cli

‎src/cli-migrate

‎tools-local/ILAsmVersion.txt

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1 +1 @@
1-
2.1.0-rtm-26514-02
1+
2.1.0-rtm-26515-03

0 commit comments

Comments
 (0)